Re: What's next after the common log format?

On Fri, 23 Jun 1995, Andrew Payne wrote:
> Has anyone thought about "CLF-ng"?  What's next for the Common Log Format 
> in a world of additional logging demands (security info, additional header 
> info, etc.)?

We've tossed a few ideas around on the apache developers list about this, and
I think the rough consensus was that the best approach was take a step back
and define an API to a "logging process", which would get fed data down a
pipe by the actual document-serving daemons.  Using pipes this logging
process could even be written in perl or tcl if the load isn't too high. 
Data sent down the pipe would consist of some randomly-generated transaction
ID, an attribute name, and a value.  The daemon(s) would send this info to
the logging process as soon as it encountered it, sending a "logit" command
at the end of the transaction (or perhaps starting a new one indicates the
end of the old one, in the case where the daemon doesn't reach a recognizable
end state).  The daemons could send down most of the HTTP request and many of
the response headers pretty much untouched, adding the unique transaction ID. 
It could also spit out information such as the actual object returned (if it
was different than the object requested), the start time and end time, and
even progress reports for large files.  This logging process would be
responsible for threading the requests together (as usually servers are
responding to more than one request at a time) and storing those requests in
some format, however the administrator would like.  This is where one would
plug in SQL calls, for example.  The logging process could also have a GUI
displaying charts of response times, traffic flow, how many processes are
busy (in the fixed-children model of NetSite, HTTP 1.4, and Apache 0.7) So,
the communication might look like this: 

AN342sQ StartRequestTime: 804531212.825
AN342sQ Method: GET
AN342sQ URI: /foo.html
AN342sQ HTTP-level: HTTP/1.0
AN342sQ Accept: text/html, image/gif, */*
AN342sQ Referrer: http://host.com/bar.html
AN342sQ Response: 200
AN342sQ DataStartTime: 804531213.027
AN342sQ Progress: .2, 804531213.127
AN342sQ Progress: .4, 804531213.227
AN342sQ Progress: .6, 804531213.327
etc...
AN342sQ Done

This is all very preliminary and such, but it's conceptually pretty 
clean.  In my opinion "CLF2" should perhaps be a definition for the API 
between servers and logging processes, with some sort of extension 
mechanisms.  This is a fair amount more verbose than other proposals, but 
this data isn't meant to be logged as is, but rather massaged somewhat by 
the logging daemon.

Thoughts?  We're busy plugging holes in Apache 0.7 right now, so we 
haven't started building TOO much yet.

	Brian

--=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=--
brian@organic.com  brian@hyperreal.com  http://www.[hyperreal,organic].com/

Received on Saturday, 24 June 1995 01:06:00 UTC