W3C home > Mailing lists > Public > public-ldp-wg@w3.org > October 2012

Re: ldp-ISSUE-18 (container membership): container membership and robust pagination [Use Cases and Requirements]

From: John Arwe <johnarwe@us.ibm.com>
Date: Mon, 8 Oct 2012 08:53:00 -0400
To: Linked Data Platform (LDP) Working Group <public-ldp-wg@w3.org>
Message-ID: <OF3D51A447.5517D38F-ON85257A91.0045A0E9-85257A91.0046C5C8@us.ibm.com>
> * Should there be a guaranteed way to get the full container 
> membership without pagination. ie. Should GET on <containerURL> 
> ''ever'' be redirected to the paginated <containerURL>?firstpage ?

This is a scaling issue, and as often is the case it has knock-on quality 
of service effects.
If a container has billions of members, you're unlikely to get answer at 
all (out of memory building the response, or consuming it) if you attempt 
to force a server to respond in this way.  Even at tens of thousands of 
members things can get dicey.
We chose to allow the server to initiate paging defensively.

> * The current approach to paging is not robust to deletions and 
> insertions of container members. Insertions/deletions will have the 
> side effect of moving page boundaries. This could be detected by 
> qualifying the request with 'If-match'. The ?firstPage still lists 
> the full membership so this can be correlated with the results, 
> however, a page boundary change makes it hard to recover resources 
> that have 'fallen through the cracks.

Again, this came from implementation experience on large databases.  If 
you want stable results, you have to hold locks and then the requests are 
no longer independent (you end up maintaining session state on the 
There is nothing wrong with implementations offering stable paging, but it 
comes with a cost high enough that making it anything more than optional 
creates scaling issues for large datasets.
If you want to make the problem really hard, assume the request is for a 
sorted result set.

> * Should the number of resources per page be an explicit property of
> the container?

A server might choose to make that a constant, and to expose it as such or 
as a URI template parameter.  I would be careful about inviting clients to 
depend on a constant page size however.

Best Regards, John

Voice US 845-435-9470  BluePages
Tivoli OSLC Lead - Show me the Scenario
Received on Monday, 8 October 2012 12:53:39 UTC

This archive was generated by hypermail 2.4.0 : Friday, 17 January 2020 16:17:32 UTC