What's next after the common log format?

Has anyone thought about "CLF-ng"?  What's next for the Common Log Format 
in a world of additional logging demands (security info, additional header 
info, etc.)?

The current format is positional, so there's no clean way to add new items 
to the format without breaking existing tools.  Also, it doesn't work well 
when you have a variety of extensions in use (your server wants to log 
session IDs, but my server wants to log security stuff).

Another approach is to create new log files for new items (access_log, 
error_log, security_log, etc.).  This makes it easy to accomodate new item 
without breaking existing tools, but creates a problem when you have to 
correlate across files to find all of the info associated with a particular 
client request.  It also can create performance problems under heavy load, 
because you are now writing many small log entries, instead of a single 
larger entry.  There's also a bit of expansion, because you usually 
replicate the timestamp or some other request info.

To solve these problems, we designed a new format based on name-value 
pairs.  Fields are named so that you can accomodate new stuff without 
breaking tools (which ignore fields they don't know about).  It's also an 
integrated format:  all of the info associated with the request (access, 
error, security info, etc.) is written in a single log entry.

We ended up using a Tcl list-based format for the entries, that look like 
this (wrapped for the reader's benefit):

log {start 803173054.917815} {method GET} {url /~payne/link.html} \
    {bytes 0} {error {file not found}} {status 404} {end 803173054.930446} \
    {host localhost}

We pay a little bit (30% or so) over the Common log format.  But most 
people end up compressing logs anyway, and the name tags end up getting 
compressed away.

Has anyone else wrestled with designing better log formats?  Are there any 
other formats in use?

-andy

Received on Friday, 23 June 1995 20:43:05 UTC