Post Bug ID'd Here for those who want it

Libwww'ers:

SYMPTOM:
Occassionally a 100 Continue coming from a server causes libwww to terminate
the request prematurely.  This appears to happen on a post when pipelining
servers that are actually pipelining send a 100 continue as a consequence of
a post.  The return data from the post (a 200 OK, and then the page) is
usually right behind the 100 so you lose the data.  In another variation
libwww gets confused and you lose data by prematurely terminating the next
block of data coming from the server.

NOTE:  This fix has been extensively tested on at least 30 large web sites
and is known to work.  If anyone finds a situation wherein this fix appears
to break a site, let me know and I'll run it down.  Even though we are using
it production, ones more familiar with the libwww architecture may be able
to point to a more appropriate fix based on my comments.

ANALYSIS:
I traced this down to HTTP.c  HTTPStatus_put_block()
What is happening is that if we get a HT_CONTINUE situation, the code:

{the code uses a lowercase 'L' variable, so the code below is for an alpha
'l', not a numeric '1'}

b += (length-l);

is incorrect, since the 100 processing in the routine ultimately called by

status = (*me->info_target->isa->put_block)(me->info_target, b, l+1);

does not (in some cases) 'consume' 'l' worth of bytes.  In addition, if the
remaining length is 0, the response length should be reset to -1 for reasons
I (unfortunately) cannot remember at this time (ran that down about 3 months
ago).

SOLUTION:
To determine the 'real' bytes consumed, I look at HTHost_Remaining bytes
before and after the call to put_block.  I use this to increment the b ptr.
In addition if there are no bytes remaining to consume, I reset the response
length.

CODE:
	if (me->info_target) {
		h = HTNet_host(HTRequest_net(me->request));
		firstremain = HTHost_remainingRead(h);
    	/* Put data down the 1xx return code parser until we are done. */
	    status = (*me->info_target->isa->put_block)(me->info_target, b, l+1);
		if (status != HT_CONTINUE) return status;
		/* Now free the info stream */
		(*me->info_target->isa->_free)(me->info_target);
		me->info_target = NULL;

	    /* Update where we are in the stream */
		l = HTHost_remainingRead(HTNet_host(HTRequest_net(me->request)));
		b += firstremain - l;
		if (l == 0)
			HTResponse_setLength(me->request->response,-1);

		length = l;
	    if (l <= 0) break;

Regards,
Fred Covely

Received on Thursday, 8 May 2003 11:22:29 UTC