dqd_log Module for AOLserver

by Rob Mayoff 2000-09-16 04:25:17
This is a logging module for AOLserver 3. It is loosely based on the source code of the standard nslog module, but virtually all of the code has been removed or replaced.

This module is designed to let AOLserver use multilog from the daemontools package for its access log.

Compiling and Installing dqd_log

Run "make" in the dqd_log directory, using a command like this: 
make INST=/path/to/aolserver install
INST should be the location where you installed AOLserver.

Telling AOLserver to Load dqd_log

Tell AOLserver to load it by adding lines like this to your config file: 
ns_section ns/server/server1/modules
    ns_param dqd_log dqd_log.so
(Substitute the correct server name in place of "server1".)

Configuring the File Descriptor for the Log

You must specify the file descriptor to which the module will send the log, and arrange for that file descriptor to be open. Typically the file descriptor is 3: 

ns_section ns/server/server1/module/dqd_log
    ns_param fd 3
To have file descriptor 3 open to a file, you would add a redirection on the AOLserver command line. For example: 
/usr/local/aolserver/bin/nsd8x -zft nsd.tcl 2>&1 3>access-log

Example Using Multilog

To use multilog to manage your access log:

  1. Create a service directory and a FIFO for the access log multilog: 
    mkdir -p /usr/local/web-access-log/main
cd /usr/local/web-access-log
mkfifo pipe
cat > run <<EOF
#!/bin/sh
( echo -n '' > pipe & )
exec /usr/local/bin/multilog t ./main 0<pipe 3>pipe
EOF
chmod +x run
    This run script is designed to get multilog running immediately and keep it running even if AOLserver exits.

  2. Link the new service into /service: 
    ln -s /usr/local/web-access-log /service

  3. Modify your AOLserver startup script to set up file descriptor 3: 
    #!/bin/sh
exec /usr/local/aolserver/bin/nsd8x -zft /usr/local/web-service/nsd.tcl 2>&1 3>/usr/local/web-access-log/pipe

  4. Add the dqd_log configuration to your nsd.tcl: 
    ns_section ns/server/server1/modules
    ns_param dqd_log dqd_log.so

ns_section ns/server/server1/module/dqd_log
    ns_param fd 3

  5. Restart AOLserver.

Module Operation

Typically the file descriptor refers to a pipe or FIFO. The program on the other end of the pipe should simply read lines from the pipe and process them according to your wishes. It should close the pipe or exit only if it encounters end-of-file on the pipe. If the module receives an EPIPE (pipe closed) error, it will make AOLserver exit, dropping any pending log entries.

If the module blocks while writing to the file descriptor, AOLserver will block. The module will not drop any log entries in this case.

Log Format

Each log entry is a Tcl list followed by a newline. The list elements are, in order:

  1. Peer address
  2. Username, according to HTTP authentication
  3. Request line (e.g. "GET /foo/bar HTTP/1.0")
  4. HTTP status of the response
  5. Content bytes transmitted in the response
  6. Arbitrary request headers (default: Referer, User-Agent)
  7. Arbitrary response headers (default: none)
The list does not include a timestamp because multilog will prefix each entry with a timestamp (when given the "t" option).

Customizing the Logged Peer Address

If you run AOLserver behind an HTTP accelerator such as squid, it will only see the IP address where the accelerator is running. It will not see the real client's IP address. However, squid can append the client's IP address to the X-Forwarded-For header. You can have this module log the last word of the X-Forwarded-For headers in place of the peer address. Set the value configuration parameter "AccelAddresses" to the list of IP addresses where squid is running. For example: 

ns_section ns/server/server1/module/dqd_log
    # Squid running on localhost and another machine on my private network
    ns_param AccelAddresses {127.0.0.1 10.0.0.2}
The module will only honor the X-Forwarded-For header if the request came from an address listed in AccelAddresses.

Customizing Request Header Logging

If you wish to log request headers other than Referer and User-Agent, you may specify the "Headers" configuration parameter. It must be a Tcl list off all the request headers you wish to log. If you wish to log the Referer, User-Agent, and Cookie headers, you must list all three, in the order you want them to appear in your log. For example: 

ns_section ns/server/server1/module/dqd_log
    ns_param Headers {Referer User-Agent Cookie}
By default, dqd_log logs the Referer and User-Agent headers. If you do not wish to log any request headers, specify the empty list: 
ns_section ns/server/server1/module/dqd_log
    ns_param Headers {}

Customizing Reply Header Logging

If you wish to log any reply headers, you may specify the "OutputHeaders" configuration parameter. It must be a Tcl list of all the reply headers you wish to log. For example, if you wish to log the Set-Cookie and Content-Type headers you send in each reply: 

ns_section ns/server/server1/module/dqd_log
    ns_param OutputHeaders {Set-Cookie Content-Type}
By default, dqd_log does not log any reply headers.

Order of Header Logging

The request headers are logged in the order listed, followed by the reply headers in the order listed.

Bugs

  1. Only one instance of each header is logged per request/reply. If a reply contains ten Set-Cookie headers, only the first will be logged. On the other hand, if the request and the reply both contain Foo headers, and you list Foo in both headers and outputheaders, then the first Foo header in the request and the first Foo header in the reply will both be logged.

History

mayoff-webpage@dqd.com

$Header: /u/cvsroot/nsd-modules/dqd_log/index.html,v 1.6 2004/10/16 02:43:11 mayoff Exp $