Re: Using HTTP Trailers [was: Content-Integrity header]

On Tue, Jul 10, 2012 at 7:50 PM, James M Snell <jasnell@gmail.com> wrote:

> Yeah, this has been my experience as well. Unfortunately, I'm not sure
> if this is something that can be effectively fixed. Off the top of my
> head, I don't know of a single server or client side http stack that
> is capable of easily setting Trailers in the stream. I just checked
> and support for trailers is still missing in the Servlet API, for
> instance.
>
There is no new API required, necessarily, to support trailers:
1. app code needs set the header Trailer: <header_name> before the response
is committed;
2. container will ignore the header  <header_name> when the response is
committed;
3. container will output the trailer when the response is finished.

The request header "TE: trailers" may or may not be respected.



> I know this kind of thing is generally frowned upon, but one possible
> approach is to make use of a special content-type... for instance...
>
>   HTTP/1.1 200 OK
>   Trailer: Content-Integrity
>   Content-Type: application/http-trailers; type="text/plain"
>
>   foobar
>
>   Content-Integrity:
> sha-256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f
>
> This is just a strawman, but essentially, it would append a checksum
> to the end of the content stream. It's not a trailer in the
> traditional sense, it's actually encoded as part of the payload. In
> lieu of proper Trailer support this is the only other way I can see it
> working effectively in HTTP/1.1.
>
> For HTTP/2.0, I suppose it would be possible for us to take a more
> appropriate approach either with trailers or by building the integrity
> mechanism into the framing somehow.
>
> (Warning: thinking out loud *before* having consumed any beer... the
> ideas expressed above may not make any sense. May need to apply
> alcohol and try again)
>
> - James
>
> On Tue, Jul 10, 2012 at 4:59 PM, Mark Nottingham <mnot@mnot.net> wrote:
> > chunk-extensions aren't widely supported in APIs, and usually dropped
> hop-by-hop (then again, trailers can be too).
> >
> > Some discussion of trailer support in Apache:
> >
> http://apache-http-server.18135.n6.nabble.com/HTTP-trailers-td4796242.html
> >
> > … and using Trailers to surface debug information in Firebug:
> >   https://groups.google.com/forum/?fromgroups#!topic/firebug/v5ldjoVThH8
> >
> > (yes, this is a bit of a project for me)
> >
> > Finally, I know some server-side implementers would would LOVE to be
> able to set things like ETag and Last-Modified in trailers, and have them
> used by caches...
> >
> > Cheers,
> >
> >
> > On 11/07/2012, at 4:33 AM, Ludin, Stephen wrote:
> >
> >> I really like the idea of placing the Digest in the chunk trailers.
>  Being
> >> able to calculate these digests on the fly and not buffer the entire
> >> message is critical in my opinion.
> >>
> >> Another concept that I have been playing with is providing digests on
> >> individual chunks using chunk-extension.  The rational for this is for
> >> very large objects.  With per-chunk digests the client would have the
> >> ability to re-request a specific corrupted section of an object using a
> >> range request rather than the entire object.  This can have enormous
> >> perceived performance and reliability benefits for consumers of things
> >> such as software download and large media files.
> >>
> >> I was working on a draft to propose this, but I didn't feel it was well
> >> baked enough to share.  If there is interest in this type of
> functionality
> >> I will polish it up and post it.
> >>
> >> One issue to point is is that for these types of "frame" based integrity
> >> checks I generally feel like we are reinventing the content integrity
> >> portion of SSL/TLS.  Though I see the value in begin able to do this
> apart
> >> from SSL it forces the question at what point do you just switch over to
> >> SSL to get the desired functionality?
> >>
> >> -stephen
> >>
> >>
> >>
> >> On 7/9/12 4:00 PM, "Amos Jeffries" <squid3@treenet.co.nz> wrote:
> >>
> >>> On 10.07.2012 07:08, HAYASHI, Tatsuya wrote:
> >>>> +1
> >>>>
> >>>> I know that this is demanded.
> >>>> When I discussion about http-authentication and phishing,
> >>>> it is requested by many people.
> >>>> It is a difficult problem.
> >>>> ex) proxy...
> >>>>
> >>>> I think that it is good to do this discussion now.
> >>>>
> >>>> ---
> >>>> Tatsuya
> >>>>
> >>>> On Sat, Jul 7, 2012 at 8:23 AM, James M Snell wrote:
> >>>>> In general, I'm +1 on the general idea albeit with a few caveats...
> >>>>>
> >>>>> 1. To minimize complexity, only a single Content-Integrity header
> >>>>> should be used. I don't want, as Roy points out, to have to iterated
> >>>>> through a bunch of unsupported header values looking for the one I
> >>>>> want. Just as it makes very little sense for an implementor to
> >>>>> provide
> >>>>> multiple Last-Modified, Etag and Content-Type headers in a single
> >>>>> message; there should be only a single Content-Integrity statement
> >>>>> and
> >>>>> I either understand it or I don't.
> >>>
> >>> Either the client advertises what it supports (opening itself to
> >>> middleware erasing options they can't modify). Or the server uses
> >>> multiple algorithms in hopes that the middleware cannot violate them
> >>> all.
> >>> It makes perfect sense to have several levels of integrity check. MD5,
> >>> SHA1, AES in one response and allow the client to validate the
> strongest
> >>> it can handle.
> >>>
> >>> There is also an arguable case for middleware wanting to add its own
> >>> hash to inform the client essentially "this is what I got given". So
> the
> >>> point of manipulation can be back-traced when the more secure
> end-to-end
> >>> checks fail.
> >>>
> >>> If you want end-to-end integrity, don't stop at half measures.
> >>> Particularly at half measures which can be corrupted.
> >>>
> >>>
> >>>>>
> >>>>> 2. The performance impact of calculating the digest needs to be
> >>>>> carefully considered. I'd rather not be required to buffer a full
> >>>>> representation in memory all the time just to calculate a header
> >>>>> value. I know it's largely unavoidable, but perhaps there's some
> >>>>> currently elusive solution that can be considered. For instance..
> >>>>> allowing Content-Integrity to appear as a trailer at the end of a
> >>>>> chunked response.
> >>>
> >>> As has been said Trailers happen here.
> >>>
> >>>>>
> >>>>> 3. Something needs to be said about what happens if the
> >>>>> Content-Integrity check fails. For instance, if a request containing
> >>>>> Content-Integrity is sent to the server and the server detects that
> >>>>> the signature is invalid, what should happen? what must happen?
> >>>>> Likewise, how are intermediaries expected to treat the
> >>>>> Content-Integrity header given that any intermediary is able to
> >>>>> modify
> >>>>> the payload at any time?
> >>>
> >>> This is going to be most useful on request/responses sent with
> >>> "no-transform" of course.
> >>> If the integrity was only a MD5 or SHA1 hash which middleware can edit
> >>> easily there is no end-to-end integrity, just hop-by-hop integrity.
> >>>
> >>>
> >>> Also, there has to be a mutual secret between origin server and client.
> >>> Without that, when integrity is compromised the transforming hop will
> >>> simply erase or replace the Content-Integrity header value. A secret
> key
> >>> unknown to that middleware is required to make the integrity hash break
> >>> when it tries this.
> >>>
> >>> AYJ
> >>>
> >>
> >>
> >
> > --
> > Mark Nottingham
> > http://www.mnot.net/
> >
> >
> >
> >
> >
>
>