Re: "actual content length", was: Handling multiple headers when only one is allowed

On Fri, 11 Jun 2010 15:24:23 +0200, Adrien de Croy <adrien@qbik.com> wrote:

>
> I raised this problem a while back.
>
> All the browsers except Opera (limited case) make no complaint when a  
> download is truncated.  This is whether it's chunked and doesn't receive

When accessing the server directly, that is correct.

When accessing through a proxy Opera will assume a broken connection means  
all was loaded anyway. The proxy special case is due to how a number of  
adbuster proxies edited content without removing the content-length  
header, causing the resend request heuristic to spin out of control.

> a final 0 chunk, or whether there's a content length and the connection  
> is closed (whether or not the server indicated it would close) prior to  
> that many bytes being transferred.
>
> I personally view this as highly problematic, and it's tied in with the  
> work I've been doing recently with scanning at a proxy.
>
> The reason it's problematic, is because every single proxy I've tested  
> (TMG/ISA, WinRoute, WinGate, Webmarshall - admittedly there are many  
> more) does something called either "drip-feeding" or "trickling".  If  
> you're downloading a file through one of these proxies, they will send  
> you a portion of the resource as it's coming down to the proxy.  When  
> the proxy has received the whole file, it scans it and sends the rest if  
> it's ok, but if it's not ok, it has 1 option only - abort the connection.
>
> Since the browsers ignore the connection having been aborted, and  
> present the downloaded file as if nothing was wrong, then any malware  
> purveyor need only pad their malware out, so that the executable part  
> will fall within the drip-feeding window.  It basically renders AV at  
> gateway potentially useless.

There is only one realistic way to avoid that: Not forward a byte of  
content until it has been checked. The recent discussion of keep alive 1xx  
responses might be useful here, but OTOH Opera (at least) will not shut  
down a normal connection while waiting for content; the only case we do  
that is for special requests with a timeout, like requests for CRLs  
(considering that some requests can take minutes or  more we can't tell  
what is the expected time to complete the operation).

> If OTOH the browsers were to act on the fact that the download was  
> aborted, this wouldn't be nearly as big a security risk.
>
> Regards
>
> Adrien
>
>
> On 12/06/2010 12:51 a.m., Dan Winship wrote:
>> On 06/09/2010 08:24 AM, Julian Reschke wrote:
>>
>>>>      http://code.google.com/p/browsersec/wiki/Part1#Hypertext_Transfer_Protocol
>>>>
>>>> ...
>>>>
>>> Interesting.
>>>
>>> That text mentions the test
>>>
>>>    "Content-Length header value overrides actual content length?"
>>>
>>> I have trouble understanding what this means... Unless the connection  
>>> is
>>> closed, or chunked encoding is in place, or the message is by  
>>> definition
>>> not having a body (HEAD response), there *is* no other signal than
>>> Content-Length to find out the actual content length.
>>>
>> It's about the case when you know the connection will be closed at the
>> end of the response, but the actual number of bytes sent by the server
>> before closing the connection doesn't match the Content-Length header.
>>
>> Eg, if you GET http://sourceforge.net/apps/wordpress/sourceforge, you'll
>> get back a 302 response with "Content-Length: 338" and "Connection:
>> close", followed by 8 bytes of body data before the server closes the
>> connection. A client should consider that to be a truncated response,
>> but as noted by the page above, most web browsers instead just assume
>> that the Content-Length header was wrong. (I assume this was originally
>> another case of bug-compatibility with IE to support broken web sites.)
>>
>> -- Dan
>>
>>
>


-- 
Sincerely,
Yngve N. Pettersen
********************************************************************
Senior Developer		     Email: yngve@opera.com
Opera Software ASA                   http://www.opera.com/
Phone:  +47 23 69 32 60              Fax:    +47 23 69 24 01
********************************************************************

Received on Friday, 11 June 2010 13:39:59 UTC