Re: Issue with "bytes" Range Unit and live streaming

------ Original Message ------
From: "Craig Pratt" <craig@ecaspia.com>
To: "Adrien de Croy" <adrien@qbik.com>; "ietf-http-wg@w3.org" 
<ietf-http-wg@w3.org>
Sent: 12/05/2016 3:37:14 p.m.
Subject: Re: Issue with "bytes" Range Unit and live streaming

>On 5/11/16 7:55 PM, Adrien de Croy wrote:
>>
>>
>>------ Original Message ------
>>From: "Craig Pratt" mailto:craig@ecaspia.com
>>To: mailto:ietf-http-wg@w3.orgmailto:ietf-http-wg@w3.org
>>Sent: 12/05/2016 2:01:39 p.m.
>>Subject: Re: Issue with "bytes" Range Unit and live streaming
>>
>>>Reply in-line.
>>>
>>>cp
>>>
>>>On 5/11/16 6:16 PM, Adrien de Croy wrote:
>>>>
>>>>unless we allow Content-Length to also use the new unit (which I'd 
>>>>be dead against), I'd suggest we steer clear of minting any new 
>>>>range units.
>>>>
>>>[cp] I don't see how a Range Unit would apply to Content-Length.
>>yes. That's the problem.
>>
>>And that's why suggestions for alternative range units keep cropping 
>>up.
>>
>>If you're an intermediate that has no knowledge of pages or frames, 
>>it's impossible to comply with any other range than bytes.
>>
>>It's not hard to ask for
>>
>>GET something?page=1
>>
>>Which is cachable, vs making life impossible for intermediaries.
>>
>>Adrien
>>
>[cp] This is not a good solution for large representations.
>
>There's no way for a cache to know that "GET something?pages=1-10" has 
>some of the same content that "GET something?pages=2-11" since it 
>doesn't (can't) have any understanding of the URI params. So you'll end 
>up with two copies of pages 2-10 in the cache.

the problem is that fundamentally without some form of mapping between 
whatever other range units would be used, and bytes (e.g. a table of 
contents), then the intermediary can't resolve a range request and would 
have to reject it (return the entire resource).

Maybe the table of contents should be requested separately (or it should 
form the start of the data).  then the client can just request ranges in 
bytes.

Or maybe define a new multipart response type that includes a 
proxy-readable table of contents to map the parts of the payload.

Otherwise you'll find caching just ends up getting suppressed.

Adrien



>
>
>And at the same time, the cache can't service a request of the form 
>"GET something?page=3" for the same reason - even though you would have 
>2 copies of page 3 in the cache. The request will go to the origin and 
>you'll now have 3 copies of page 3 in the cache.
>
>Obviously, an application can define it's own request header and mark 
>the response data non-cacheable. And this would still be better than 
>the above. But what I would rather do is ensure that any Range Units 
>that are defined include a bytes range that would at least *allow* for 
>caching. A Vary can probably take care of preventing the kind of cache 
>problems like those described above - given a little thought... I'll 
>try to write up something concrete.
>>
>>>
>>>Regardless of how a range is requested, the server is always 
>>>transferring bytes in the response body - whether indicated by 
>>>Content-Length, chunks, or whatever. The Content-Range response 
>>>header just lets the server indicate the returned range associated 
>>>with the response body *in terms of the indicated range unit* and *in 
>>>the domain of the representation*.
>>>
>>>e.g. If a Range Unit called "blobs" is used, and a request is made 
>>>with "Range: blobs=5-11", the response could have header 
>>>"Content-Range: blobs=5-11/50" indicating that 7 blobs are being 
>>>returned (out of 50) in the response body and a "Content-Length: 
>>>123456" could also be supplied designating that the response body 
>>>(associated with blobs 5-11) is 123456 bytes.
>>>
>>>Am I missing something?
>>>
>>>>All the proposed use cases for new range units that I've seen are 
>>>>application specific, and could arguably better be dealt with using 
>>>>another header.
>>>>
>>>>Adrien
>>>[cp] Doesn't the fact that there's a Range Unit Registry mean that 
>>>Range Units can be added that are application-specific - assuming 
>>>there's sufficient demand? It's just a different mechanism than 
>>>adding custom headers, IMHO - with well-defined semantics.
>>>
>>>>
>>>>
>>>>------ Original Message ------
>>>>From: "Mark Nottingham" mailto:mnot@mnot.net
>>>>To: "Darshak Thakore" mailto:d.thakore@cablelabs.com
>>>>Cc: mailto:ietf-http-wg@w3.orgmailto:ietf-http-wg@w3.org
>>>>Sent: 12/05/2016 12:59:25 p.m.
>>>>Subject: Re: Issue with "bytes" Range Unit and live streaming
>>>>
>>>>>Hi Darshak,
>>>>>
>>>>>I don't think that's where we're at. Based on the discussion so 
>>>>>far, it seems like there are two possible paths forward:
>>>>>
>>>>>1. Changing the 'bytes' range-unit to allow this use case
>>>>>2. Minting a new range-unit
>>>>>
>>>>>As discussed, both have downsides. What we need is data about how 
>>>>>current implementations -- especially caching intermediaries -- 
>>>>>behave when faced with a) 'bytes' used in the desired way, and b) a 
>>>>>new range-unit.
>>>>>
>>>>>Cheers,
>>>>>
>>>>>
>>>>>>  On 11 May 2016, at 5:57 AM, Darshak Thakore 
>>>>>>mailto:d.thakore@cablelabs.com wrote:
>>>>>>
>>>>>>  Hi all,
>>>>>>
>>>>>>  Based on feedback on this thread, it seems like the need for 
>>>>>>being able to send an open ended (read as unknown-last-byte-pos) 
>>>>>>Range response has been discussed a couple of times (with 
>>>>>>different use cases). Also there seems to be somewhat general 
>>>>>>agreement that the Content-Range ABNF in RFC 7233 is deficient in 
>>>>>>providing this. The initial argument has been, “is there a 
>>>>>>compelling enough need” and i think with different use cases 
>>>>>>popping up (log files, media streaming, gzip… others ??) there 
>>>>>>seems to be some value in defining a non-application specific 
>>>>>>Range unit that plugs this gap. Clearly fixing RFC 7233 is 
>>>>>>invasive and will result in thing breaking in unknown ways so 
>>>>>>that’s a no-go. With that, we can:
>>>>>>   • Ensure that the scope of this work item is narrow and 
>>>>>>restricted only to fixing the gap in RFC 7233
>>>>>>   • Define a new range unit (anything with “bytes” in it seems 
>>>>>>like a bad idea, so maybe call it “live-octets”, “blive”, “b-add" 
>>>>>>- suggestions welcome….)
>>>>>>   • Decide if there is enough interest/reason to do this as a WG 
>>>>>>item
>>>>>>  Any objects/suggestions to any of the above ?
>>>>>>
>>>>>>  Regards,
>>>>>>  Darshak
>>>>>>
>>>>>>
>>>>>>  From: mailto:K.Morgan@iaea.orgmailto:K.Morgan@iaea.org
>>>>>>  Date: Thursday, April 21, 2016 at 4:25 AM
>>>>>>  To: 'Craig Pratt' mailto:craig@ecaspia.com, 
>>>>>>mailto:fielding@gbiv.commailto:fielding@gbiv.com
>>>>>>  Cc: mailto:ietf-http-wg@w3.orgmailto:ietf-http-wg@w3.org
>>>>>>  Subject: RE: Issue with "bytes" Range Unit and live streaming
>>>>>>  Resent-From: mailto:ietf-http-wg@w3.org
>>>>>>  Resent-Date: Thursday, April 21, 2016 at 4:25 AM
>>>>>>
>>>>>>  On Thursday,21 April 2016 05:18 craig@ecaspia.com wrote:
>>>>>>  >
>>>>>>  > Re: Representation caching
>>>>>>  >
>>>>>>  > Whether a representation is considered cacheable in this use 
>>>>>>case is at
>>>>>>  > the discretion of the origin server and specific to the use
>>>>>>  > case/application - as it should be (imho). There's no 
>>>>>>*necessity* in
>>>>>>  > having a periodically-appended resource marked non-cachable, 
>>>>>>correct? If
>>>>>>  > the resource mutates, it's not cacheable. If it's just being 
>>>>>>appended
>>>>>>  > to, it is cacheable. And if an appended resource stops being 
>>>>>>appended
>>>>>>  > to, it doesn't invalidate the cached representation.
>>>>>>  >
>>>>>>
>>>>>>  I couldn't agree more. However, it seemed the prevailing 
>>>>>>sentiment when we tried to resolve the related issue of ranges 
>>>>>>before content codings, with a new bbcc unit 
>>>>>>(bytes-before-content-coding), was that the use cases for 
>>>>>>append-only growth represent an insignificant portion of HTTP 
>>>>>>traffic. “We live by app-specific protocols to handle these cases. 
>>>>>>What is so special ... that it must be addressed by http (in a 
>>>>>>very ugly way)?” [1]
>>>>>>
>>>>>>  [1] 
>>>>>>https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/1383.html
>>>>>>
>>>>>>
>>>>>>
>>>>>>  This email message is intended only for the use of the named 
>>>>>>recipient. Information contained in this email message and its 
>>>>>>attachments may be privileged, confidential and protected from 
>>>>>>disclosure. If you are not the intended recipient, please do not 
>>>>>>read, copy, use or disclose this communication to others. Also 
>>>>>>please notify the sender by replying to this message and then 
>>>>>>delete it from your system.
>>>>>
>>>>>-- Mark Nottingham   https://www.mnot.net/
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>--
>>>craig pratt
>>>
>>>Caspia Consulting
>>>
>>>craig@ecaspia.com
>>>
>>>503.746.8008
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
>
>--
>craig pratt
>
>Caspia Consulting
>
>craig@ecaspia.com
>
>503.746.8008
>
>
>
>
>
>
>