Re: The use of trailers

Thanks for your points on this.  more comments below

Henrik Nordstrom wrote:
> mån 2007-05-28 klockan 14:30 +1200 skrev Adrien de Croy:
>
>> It seems there are a few features in HTTP that primarily consider agents 
>> and servers, but not proxies, or specifically administrative 
>> requirements for proxies.
>
> Well, there is considerably more focus on the endpoint, but I am not
> sure I agree in this case.
>
I meant many do not primarily consider admin requirements for proxies.  
Most primarily consider servers and clients.

Anyway, not much value in my original comment - apologies.  I do get 
frustrated trying to design an administrative interface for a proxy 
product where the protocol makes things difficult, and the value to the 
end-user isn't clearly understood or obviously justified.

I see HTTP getting more and more complex, and hope that it doesn't go 
the way of IMAP.  History has shown complexity and achieved 
interoperability to be conflicting / divergent paths.
>> It makes it very difficult to make decisions based on header values, 
>> when the header itself may arrive a long time after the main headers of 
>> a response in some chunk down the line.
>
> Sure, but in most cases you don't need to, and it's easy to indicate you
> don't support trailers by simply not sending a TE: trailers header.
>

A proxy needs to understand the TE header to strip it.

There are plenty of good use-cases for switching policy on return 
headers.  Content-Type is a classic one, where certain types of content 
scanning for instance break streaming media that uses HTTP transport.  
Or blocking things like CGI proxies based on a signature header etc.

There's no end to some of the policies admins wish to implement, who am 
I to deny them the capability :)

>> And it added a lot of complexity to header processing.
>
> Not really, just that you need to be able to add headers. Quite the same
> requirement as the need to be able to rewrite headers on a 304..

I meant processing of all headers (including trailers) particularly for 
policy just got split over potentially a zillion places instead of one.


>
>> the options available to an administrator don't lend themselves to good 
>> user experiences.
>
> Not sure I get the point here. The alternatives is

Users don't even want to know about let alone think about trailers.  Any 
incursion of trailers onto their reality is seen by most to be an 
annoyance.  So, it's really hard to justify providing GUI in a product 
to deal with how to cope with trailers.


>
> a) Not emit the header at all if it's value can't be realistically
> determined before the body has been sent.
>
> b) Emit the header in a trailer for those who can handle it.
>
> and I prefer having the 'b' option available than only having 'a'.

I'd be interested if you can point me to any docs on real-life uses of 
option b.  I still haven't seen a server return any trailers, few 
clients advertise support for them.  Is there a restriction on what 
headers may appear in trailers?  A site to test against would also be 
useful.

>
>> For instance a proxy administrator wishing to strictly enforce actions 
>> based on response header values has a big problem with trailers.  
>> Stripping any TE header in requests, or blocking responses from servers 
>> that indicate there will be trailers are about the only decent options.
>
> Sure, and where is the problem? 
the problem is that when trying to figure out what to do about trailers, 
and policy enforcement, when putting together a request for upstream, 
it's all to easy to decide to simply strip out any TE header from any 
request received before passing it on.  This has the same effect as if 
trailers didn't exist.  It's a simple way to deal with the problem.  But 
what does it break?

the arguably more correct way to deal with it is to allow the user to 
configure what to do if and when headers arrive attached to chunks.  
This has certain downsides:

1. user education. If I don't even understand the true user benefits of 
trailers, how can I educate my users.  Even if I did, it would be 
non-trivial.  Providing an understandable user interface that doesn't 
just put people off the product... etc etc.
2. performance.  Need to evaluate policy on headers possibly many more 
times - whenever a trailer arrives. 
3. A subsequent trailer can potentially invalidate previous policy 
considerations.  Policy logic may not lend itself to correct evaluation 
when  matching is piecemeal.  This is the clincher for me.

So for the moment, I have chosen to deprecate trailers, which leaves me 
uncomfortable that some day, someone will require them to work and I'll 
be faced with this problem again.


> Such setups gets restricted to 'a' or
> requires the server to handle the request in a more complex manner, just
> as they would be if trailers did not exists.
>

It seems to me that implementation of trailers in a server would 
generally be thought by developers to be more trouble than they are 
worth.  If they have to support the non-trailer case anyway.  Is support 
for trailers optional?  I didn't think it was.


>> The overall structure of a message seems to have been muddied with the 
>> addition of trailers - from a clear delineation between headers and 
>> resource, separated by 2 CRLF, we now no longer enjoy this clarity.
>
> Yes.
>
>> Processing of "content" now has to be able to encounter and deal with 
>> new "headers".  This affects all resource processing, e.g caching, 
>> scanning etc.
>
> Well, it's still clearly separated at the transport level (not message
> level).
>
>> It would be nice if the concept could be deprecated or re-thought before 
>> someone goes ahead and implements it, but I'm dreaming there I guess.
>
> Don't think that's a good alternative. trailers is a good thing when
> they can be used.
>
> But there should be a note that servers should if possible try to send
> the headers in the message-header when possible, and only use the
> trailer if it can't compute the header value before the response.
>
> It's also worth noting that the server MUST know it's going to emit the
> header. It's not allowed to add random headers to the trailer without
> advertising them in the message-header. So for a server to emit Vary in
> the trailer the following conditions must all be true
>
> a) The request had a "TE: trailer"
>
> b) The server realized the content varies, and emits a "Trailer: Vary"
> in the message-header telling the recipient that the response varies but
> that it's not yet known on which details..
>
>
hmmm, maybe I need to re-read the section on trailers.  I remember the 
bit about advertising which headers will be in trailers, but I don't 
recall it covering things like how many times a trailer may appear, 
whether there are any headers that cannot appear in a trailer (and must 
be in the response) etc.  Obviously the Trailer header must not appear 
in the trailer I guess.  But what about headers like Content-Type

there may be an opportunity for optimisation of policy where advertised 
trailer headers aren't used in the policy evaluation.

Still, I'm having fun trying to figure out how for instance a server can 
provide an API to a CGI/ISAPI/NSAPI/etc so that the module can emit 
trailers.  unless the module does the chunking itself as well.  Maybe 
that's why I've never seen one.

Speaking of which, is there a decent specification for the allowable 
output of a CGI?  Is it valid for a CGI process to emit chunked data?

Thanks

Adrien

> Regards
> Henrik

Received on Tuesday, 29 May 2007 10:30:42 UTC