Re: ldp-ISSUE-18 (container membership): container membership and robust pagination [Use Cases and Requirements]

> * Should there be a guaranteed way to get the full container 
> membership without pagination. ie. Should GET on <containerURL> 
> ''ever'' be redirected to the paginated <containerURL>?firstpage ?

This is a scaling issue, and as often is the case it has knock-on quality 
of service effects.
If a container has billions of members, you're unlikely to get answer at 
all (out of memory building the response, or consuming it) if you attempt 
to force a server to respond in this way.  Even at tens of thousands of 
members things can get dicey.
We chose to allow the server to initiate paging defensively.

> * The current approach to paging is not robust to deletions and 
> insertions of container members. Insertions/deletions will have the 
> side effect of moving page boundaries. This could be detected by 
> qualifying the request with 'If-match'. The ?firstPage still lists 
> the full membership so this can be correlated with the results, 
> however, a page boundary change makes it hard to recover resources 
> that have 'fallen through the cracks.

Again, this came from implementation experience on large databases.  If 
you want stable results, you have to hold locks and then the requests are 
no longer independent (you end up maintaining session state on the 
server). 
There is nothing wrong with implementations offering stable paging, but it 
comes with a cost high enough that making it anything more than optional 
creates scaling issues for large datasets.
If you want to make the problem really hard, assume the request is for a 
sorted result set.

> * Should the number of resources per page be an explicit property of
> the container?

A server might choose to make that a constant, and to expose it as such or 
as a URI template parameter.  I would be careful about inviting clients to 
depend on a constant page size however.

Best Regards, John

Voice US 845-435-9470  BluePages
Tivoli OSLC Lead - Show me the Scenario