Re: To sort or not to sort

On 15 October 2012 14:53, Lars Marius Garshol <lars.garshol@bouvet.no>wrote:

>
> * Graham Moore
> >
> > In cases where a server builds a fragments feed from several queries to
> an underlying system or to several underlying systems it is not always
> possible to execute a query that does the ordering for you. So, in the case
> of the 60,000 fragments the server will need to get these all in memory
> then sort them, or put them into temporary storage and then use that
> storage to pull things back in order.
>
> Well, not necessarily. It's possible to execute each query in a separate
> connection and then do a merge sort, always stepping along the cursor that
> currently holds the most recent change.
>
> It's not really a very pretty approach, and I'm guessing it will be prone
> to connections failing in some cases. But it would work, and it would not
> require prohibitive amounts of memory or CPU.
>

Yep this will work. But does requrie that the individual connection or data
source can be sorted, and maybe this is the majority of cases.

But there are systems where you can ask for the changes but not get them
sorted by date. One example, a web cms I've been using has a web service
API where I can execute a query, but only one that matches a bunch of
criteria, I cant get it to return me the sorted list. In this case the
server will need to bunch up all the data before it can start streaming it.
I see one key group of adopters of sdshare (the customers) being adaptor
writers, developers on projects who use a pluggable sdshare server to
expose their or their customers' data source. Unless there are compelling
benefits client side then I'd like their job to be as easy as possible
given its they who will need to work with a large number of weird and
wonderful systems.

What I like about sdshare is that is is a streaming protocol, things just
get passed through all over. With SdShare it feels like you turn on taps on
information and it flows. I'm not convinced that the tradeoff of client
benefits outweighs what it would mean to the overall protocol footprint if
we introduce requirements that *could* in some cases lead to bottlenecks,
large resource usage and the inability to 'stream through'.

Cheers,

Graham

-- 
Graham Moore
gra@brightstardb.com
BrightstarDB