RE: #445: Transfer-Codings

On Wednesday,23 April 2014 23:43, Martin Thomson wrote:
>>> In particular, frames that are not compressed cannot be compressed, and frames that are separately compressed cannot be merged into a single frame.

>>

>> Why is merging disallowed? Two (or more) gzip files can be concatenated to form a single gzip. So unless there is a security issue I'm missing, I don't see why this is a problem.

>

> As long as the compression context is shared for the merged fragments, then it is OK I suppose.  The concern here is that an origin server might split frames intentionally

> in order to make it clear to downstream consumers that the content of those frames cannot be mixed in the same compression context.  Having an intermediary undo that would be bad.



This is a really good point.  Although merging two compressed frames still keeps the compression contexts separate, merging can create a problem if later the merged frame is decompressed* and recompressed - because the data would then share a single compression context (i.e. possibly merging confidential and attacker-controlled data into a single context).  In other words, this becomes relevant if decompressing and recompressing is allowed (see below).  So I suggest keeping the requirement to not merge frames, but it would be good to add some sort of explanation as to why this is disallowed otherwise I suspect there will be others reading the spec that will have the same thought I did and disregard this as a nonsense requirement.



(*For example, as far as I know the zlib library will merrily decompress two contexts which are merged without so much as telling you that it found separate contexts because concatenated contexts is essentially a built-in feature.  From the zlib manual: "If flush is set to Z_FULL_FLUSH, all output is flushed as with Z_SYNC_FLUSH, and the compression state is reset so that decompression can restart from this point if previous compressed data has been damaged or if random access is desired. " [1]).





On Wednesday,23 April 2014 23:47, Martin Thomson wrote:

> Missed this one.

> On 23 April 2014 14:21,  <K.Morgan@iaea.org> wrote:

>>  Finally, what about removing and then re-applying gzip (e.g. to extract a range).  That should be ok too.

>

> Yes.  But do you really think that more text is really necessary for that?



In light of the really good point you made about the problem with merging frames, now I do think this has to be explicit.



Here is my suggested change. It’s tough to explain/justify this succinctly. I’m sure you will be able to write it more eloquently :) …



Intermediaries MUST NOT alter the compression of DATA frames unless additional information is available that allows the intermediary to identify the source of data. In particular, frames that are not compressed cannot be compressed, and frames that are separately compressed cannot be merged into a single frame. However, compressed frames MAY be decompressed (note that it may be necessary to split the uncompressed data into multiple DATA frames). Additionally, compressed frames MAY be decompressed and then recompressed (e.g. to extract a range), however the uncompressed data from separate frames MUST NOT be merged.  Merging uncompressed data from separate frames creates a shared compression context that could allow an attacker to recover secret data if merging combines confidential and attacker-controlled data.  Likewise, merging compressed frames could allow an attacker to recover secret data if a downstream intermediary elects to decompress then recompress the merged frame, potentially combining confidential and attacker-controlled data into a shared compression context.





[1] http://www.zlib.net/manual.html#Basic




This email message is intended only for the use of the named recipient. Information contained in this email message and its attachments may be privileged, confidential and protected from disclosure. If you are not the intended recipient, please do not read, copy, use or disclose this communication to others. Also please notify the sender by replying to this message and then delete it from your system.