Re: Making a converter(proxy) HTTP server with wwwlib?

Markku Savela writes:
> Remember me? I wanted to do a proxy/converter using WWW library

Yep - good to hear from you!

> 
> Client                        Proxy/Converter              The WEB
> (requestor)                                            (real WWW server)
> 
>           GET URL             (pass through)        GET URL
> ----------------------------> .............. ------------> ........
>                               ..............               ........
>                               ..............   HTTP Reply  ........
>                               .............. <------------ ........
>                                 HTServHTTP
>                                HTMIMEConvert
>                           + the whole converter
>                         stacking and type matching
>                             stuff sitting here
>    application/x-mheg-5       ..............
> <---------------------------- ..............
> 
> With the library version 4.0D and following setup
> 
> HTProtocol_add("http", NO, HTLoadHTTP, HTServHTTP);
> HTProtocol_add("file", NO, HTLoadFile, NULL);
> HTProtocol_add("ftp", NO, HTLoadFTP, NULL);
> HTProtocol_add("nntp", NO, HTLoadNews, NULL);
> HTProtocol_add("news", NO, HTLoadNews, NULL);
> HTProtocol_add("gopher", NO, HTLoadGopher, NULL);
> conv = HTList_new();
> HTConversion_add(conv,"text/x-http","*/*",HTTPStatus_new,1.0, 0.0, 0.0);
> HTConversion_add(conv,"text/*","*/*", TextObject, 1.0,0.0,0.0);
> HTConversion_add(conv,"image/*","*/*", ImageObject, 1.0,0.0,0.0);
> HTConversion_add(conv,"message/x-rfc822-head","*/*",HTMIMEHeader,1.0,0.0,0.0);
> HTConversion_add(conv,"message/rfc822","*/*",HTMIMEConvert,1.0,0.0,0.0);
> HTConversion_add(conv,"text/html","*/*", HtmlObject, 1.0,0.0,0.0);
> 
> ..and with a HTTP/1.0 request  "GET <some image URL>" I manage to get
> a call to my "ImageObject" stream, which
> 
> 	- successfully receives and buffers the image content
> 
> 	- in the free "method" of the HTStream I actually convert the
> 	  image to the output format (x-mheg-5) and write the byte
> 	  stream to target stream.
> 
> The problem:
> 
> 	- what do I have to do to get the output target (which now is
> 	  actually HTTPServ.c/HTTPReply) actually generate my new
> 	  reply headers, most importantly
> 
> 		content-type: application/x-mheg-5

The response is generated by the HTTPReply stream in the HTTPServ module. This 
stream is added as the output stream of the client request object so that when 
data is starting to flow from the client request, you can paste in the HTTP 
response header. When the HTTPReply is called with data, it sets up the 
remaining part of the response which is the MIME header lines.

One of these streams is the HTMIMERequest() stream which uses the anchor 
object to generate the entity headers like Content-Type, Content-Length, etc. 
So when you have converted the data object you should at the same time change 
the content type in the anchor object so that it matches the new one. Then the 
rest should work as expected.

> What happens now is that
> 
> 	- HTTPServ/HTTPReply calls MakeReplyPipe, which finds out that
> 	  the client->net pointer exists, and goes into "Direct
> 	  output" mode without generating any headers! (Thus, my
> 	  binary MHEG-5 stream is returned to original client without
> 	  any header information).

This is the default way that the mini server does it. In most cases you don't 
need to touch the response coming from the origin server. You can just pass it 
directly to the proxy client.
 
> Solution? Of course, knowing that this "Direct output" happens, I
> could just write the headers myself to the target, before the actual
> content, but
> 
> 	- is this correct solution in respect to WWWLIB architecture?
> 
> 	- will it always work this way (I would be hosed, if the
> 	  library in some cases would not go into "direct output")?
> 
> 	- if not, how do I instruct the HTTPReply to generate the
> 	  headers I want? (If this is documented, just mail me the
> 	  appropriate WWW pointer :-)

When you are converting format on the fly then you can not just pass the 
response directly to the proxy client as above. You have to parse the whole 
header as there is at least the 

	Content-Length
	Content-Type

that you'll need to change. As you don't know the position of these in the 
response you have to use the full HTMIME stream. Hence, I suggest that you 
enable the HTTP message parser using something like

	HTConversion_add(c,"text/x-http", "*/*", HTTPStatus_new, 1.0, 0.0, 0.0);

And then generate a new HTTP response header using the HTTPReply header but 
not in "direct output" mode. You can handle unknown MIME headers by using the 
HTHeader interface so this is not a problem.

> Actually there is an other issue in the general picture: in my
> converter proxy, I may get client request using HTTP/0.9 protocol, but
> might always wish to forward the requests as HTTP/1.0. But, if such
> "upgrade" happens, I still have to remember the original client
> protocol and reply to client according to that. Any comments on this?
> It already is taken care by the library? Or, I have to to something
> special to support it?

In the current version of the MakeReplyPipe you'll find the comment:

	/* We are not using the version info for the moment */

So it's not implemented right now. What it should do was to check that the 
version response header matches the request version and if not then in the 
case you describe above take out the headers all together.

-- 

Henrik Frystyk Nielsen, <frystyk@w3.org>
World-Wide Web Consortium, MIT/LCS NE43-356
545 Technology Square, Cambridge MA 02139, USA

Received on Monday, 26 February 1996 13:28:22 UTC