Re: To sort or not to sort

Hi,

In cases where a server builds a fragments feed from several queries to an
underlying system or to several underlying systems it is not always
possible to execute a query that does the ordering for you. So, in the case
of the 60,000 fragments the server will need to get these all in memory
then sort them, or put them into temporary storage and then use that
storage to pull things back in order.  What is very nice about not
requiring ordering is that the amount of in-memory or duplicated state can
be kept at a minimum.

The large fragments feed usually only occurs when a client starts up and
doesn't initialise from a snapshot. This has actually tended to happen
quite a bit as fetching large snapshots has not been made easy by the
protocol until recently.

I feel that with improvements in how snapshots work and the potential for
'fat' server requests that on balance I like things as they are.

Gra

On 12 October 2012 12:24, Tom Bech <tom.bech@bouvet.no> wrote:

> Hi!
>
> (Sorry for the quotestyle - courtesy of Outlook.)
>
> Based on practical experience, i.e in concrete projects for customers
> involving writing both SDshare clients and servers, I'd say the benefits of
> having a sorted fragments list outweighs any
> inconvenience on the server side; Lars Marius pretty much summed up the
> benefits.
>
> Tom
>
> -----Original Message-----
> From: Lars Marius Garshol [mailto:lars.garshol@bouvet.no]
> Sent: 12. oktober 2012 12:13
> To: public-sdshare@w3.org
> Subject: To sort or not to sort
>
>
> The current draft[1] says that the order of fragments in the fragment feed
> is undefined.
>
> This is good for server implementors, as it means they can pump the
> fragments out in any order they choose. Less work for them.
>
> However, it has a cost for clients. It means that if the fragment feed is
> large, and something goes wrong during processing of fragment number 60,000
> the client cannot update its local "last fragment seen" time, because the
> feed may contain other fragments dated before that time which it hasn't
> read yet. So if it sends a new request with ?since updated it may lose
> fragments. That means it has to go through all the 60,000 fragments again,
> which takes time.
>
> It's also clumsy when debugging. In practical use you often make a change
> in the data source and want to see if it shows up in the feed as it should.
> Now you have to search through the entire fragment feed to find your
> change.. If the feed were ordered by time you'd know that your last change
> should be at the end of the feed.
>
> So what is the right choice here? Should we require fragments to be
> ordered by time ascending (oldest first), or leave it undefined?
>
>
>
> (To those of you who are new to SDshare: if you feel you need some sort of
> introduction to the spec before you can discuss this sort of issue, please
> let us know, and we'll see what we can do.)
>
>
> [1] http://www.sdshare.org/spec/sdshare-current.html
>
> --
> Lars Marius Garshol  |  Consultant
> Bouvet ASA Sandakerveien 24C D11 Postboks 4430 Nydalen NO-0403 Oslo
> Phone: +47 23 40 60 00 | Fax: +47 23 40 60 01 | Mobile: +47 98 21 55 50
> http://www.bouvet.no
>
>
>
>
>
>


-- 
Graham Moore
gra@brightstardb.com
BrightstarDB