Re: Making a converter(proxy) HTTP server with wwwlib?
To: firstname.lastname@example.org (Markku Savela)
Subject: Re: Making a converter(proxy) HTTP server with wwwlib?
From: Henrik Frystyk Nielsen <email@example.com>
Date: Mon, 26 Feb 1996 13:28:08 -0500
From firstname.lastname@example.org Mon Feb 26 13: 28:22 1996
Reply-To: Henrik Frystyk Nielsen <email@example.com>
X-Mailer: exmh version 1.6.2 7/18/95
Markku Savela writes:
> Remember me? I wanted to do a proxy/converter using WWW library
Yep - good to hear from you!
> Client Proxy/Converter The WEB
> (requestor) (real WWW server)
> GET URL (pass through) GET URL
> ----------------------------> .............. ------------> ........
> .............. ........
> .............. HTTP Reply ........
> .............. <------------ ........
> + the whole converter
> stacking and type matching
> stuff sitting here
> application/x-mheg-5 ..............
> <---------------------------- ..............
> With the library version 4.0D and following setup
> HTProtocol_add("http", NO, HTLoadHTTP, HTServHTTP);
> HTProtocol_add("file", NO, HTLoadFile, NULL);
> HTProtocol_add("ftp", NO, HTLoadFTP, NULL);
> HTProtocol_add("nntp", NO, HTLoadNews, NULL);
> HTProtocol_add("news", NO, HTLoadNews, NULL);
> HTProtocol_add("gopher", NO, HTLoadGopher, NULL);
> conv = HTList_new();
> HTConversion_add(conv,"text/x-http","*/*",HTTPStatus_new,1.0, 0.0, 0.0);
> HTConversion_add(conv,"text/*","*/*", TextObject, 1.0,0.0,0.0);
> HTConversion_add(conv,"image/*","*/*", ImageObject, 1.0,0.0,0.0);
> HTConversion_add(conv,"text/html","*/*", HtmlObject, 1.0,0.0,0.0);
> ..and with a HTTP/1.0 request "GET <some image URL>" I manage to get
> a call to my "ImageObject" stream, which
> - successfully receives and buffers the image content
> - in the free "method" of the HTStream I actually convert the
> image to the output format (x-mheg-5) and write the byte
> stream to target stream.
> The problem:
> - what do I have to do to get the output target (which now is
> actually HTTPServ.c/HTTPReply) actually generate my new
> reply headers, most importantly
> content-type: application/x-mheg-5
The response is generated by the HTTPReply stream in the HTTPServ module. This
stream is added as the output stream of the client request object so that when
data is starting to flow from the client request, you can paste in the HTTP
response header. When the HTTPReply is called with data, it sets up the
remaining part of the response which is the MIME header lines.
One of these streams is the HTMIMERequest() stream which uses the anchor
object to generate the entity headers like Content-Type, Content-Length, etc.
So when you have converted the data object you should at the same time change
the content type in the anchor object so that it matches the new one. Then the
rest should work as expected.
> What happens now is that
> - HTTPServ/HTTPReply calls MakeReplyPipe, which finds out that
> the client->net pointer exists, and goes into "Direct
> output" mode without generating any headers! (Thus, my
> binary MHEG-5 stream is returned to original client without
> any header information).
This is the default way that the mini server does it. In most cases you don't
need to touch the response coming from the origin server. You can just pass it
directly to the proxy client.
> Solution? Of course, knowing that this "Direct output" happens, I
> could just write the headers myself to the target, before the actual
> content, but
> - is this correct solution in respect to WWWLIB architecture?
> - will it always work this way (I would be hosed, if the
> library in some cases would not go into "direct output")?
> - if not, how do I instruct the HTTPReply to generate the
> headers I want? (If this is documented, just mail me the
> appropriate WWW pointer :-)
When you are converting format on the fly then you can not just pass the
response directly to the proxy client as above. You have to parse the whole
header as there is at least the
that you'll need to change. As you don't know the position of these in the
response you have to use the full HTMIME stream. Hence, I suggest that you
enable the HTTP message parser using something like
HTConversion_add(c,"text/x-http", "*/*", HTTPStatus_new, 1.0, 0.0, 0.0);
And then generate a new HTTP response header using the HTTPReply header but
not in "direct output" mode. You can handle unknown MIME headers by using the
HTHeader interface so this is not a problem.
> Actually there is an other issue in the general picture: in my
> converter proxy, I may get client request using HTTP/0.9 protocol, but
> might always wish to forward the requests as HTTP/1.0. But, if such
> "upgrade" happens, I still have to remember the original client
> protocol and reply to client according to that. Any comments on this?
> It already is taken care by the library? Or, I have to to something
> special to support it?
In the current version of the MakeReplyPipe you'll find the comment:
/* We are not using the version info for the moment */
So it's not implemented right now. What it should do was to check that the
version response header matches the request version and if not then in the
case you describe above take out the headers all together.
Henrik Frystyk Nielsen, <firstname.lastname@example.org>
World-Wide Web Consortium, MIT/LCS NE43-356
545 Technology Square, Cambridge MA 02139, USA