Re: Streams in an unreliable world.

Dear Andy,

thank you for this post. I answer in line below.

On Oct 5, 2013, at 6:48 PM, Andy Seaborne <andy@apache.org>
 wrote:

> On the web, and indeed in any system of more than modest size and in one management domain, issues of
> 
> * out of order delivery, including arbitrarily late arrival

I believe that out of order is in scope, but not arbitrarily late arrival. We should allow for a maximum delay that guarantees for a minimum quality of answers. In many relevant use cases I have in mind reactiveness is more important than correctness, because either an answer arrive within a given time limit or it has no value. If we allow for arbitrarily late arrival, we cannot meet the reactivity requirement even if enough information arrived for a good enough answer. 

> * new stream producers coming online, old stream producers ending
>     (discovery, joining, leaving)

this is in scope and I believe it should be discuss under the topic RSP services.

> * consumers joining and leaving

same as above

> * streams becoming unavailable

this appears to be a difficult to handle point. What's the difference between a silent stream and an unavailable stream? However, I'd like to discuss this requirement.

> * ... then restarting (with or without loss of potential events)

in scope, even if, as for out of order delivery, I would suggest to remember how central to RDF stream processing is the reactivity requirement. Making sure to deliver an answer, which has no longer any value, may be irrelevant. Systems may even decide to drop events without processing them.

> leading to design points on
> 
> + choice of timestamps

we are discussing it

> + delivery ordering semantics

this also relates to query models

> + delivery guarantees (at least once, exactly once, at most once)

interesting...

> + persistence, and for how long
>   (forwards, for guaranteed delivery and backwards for consumers to
>    catch up).

Yes, I believe this is in scope as part of the RSP service. 

Bests,

Emanuele

Received on Monday, 7 October 2013 18:09:36 UTC