As the Log Scrolls By...

As a web hosting company, there are times when it's vitally important to see what our Apache web server is dishing up to the outside world at any one time—and to see this as quickly as possible.

Just as a system administrator needs to be able to monitor system log files, a web administrator should be able to do the same with web logs. Noting the number of utilities that display system log information in real time, I was sure there'd be similar programs for monitoring web logs. After a search through Freshmeat.net and other on-line resources, I didn't find anything that met all of my needs. Several came close, but most would only monitor one file, and the few that monitored several files would leave me lost trying to wade through tabs for the 30 or so logs that I watch.

The solution I found isn't in one program, but in making a few changes in Apache's httpd.conf file I found I could have a specialized disposable log file containing only the information I require, for all of our web hosts. The “disposable” monitoring log is then displayed using colortail (with additional configuration settings) on an external monitor in our workshop. I can see at a glance which of our hosts have current web activity, where the traffic is coming from and the pages that are being accessed. This has also allowed us to deal quickly with problems such as script kiddies and rogue search engine robots. The system has worked so well for us that we added system logging to it as well.

In addition to the LogFormat for general logging, I added a new format labeled “webmonitor”:

LogFormat "[%v] %h %u \"%r\"%>s%b\n\"%{Referrer}i\" \"%{User-Agent}i\"%t" webmonitor

This displays log information with the Referrer and User-Agent on a second line, making it clearer to read. The log file could be in any format—even the “common” one we use for standard logging. I decided to change it for purposes of clarity and because Apache is flexible enough to allow this.

Since graphic files such as GIFs, JPEGs or PNGs files can clutter up the display, I exclude them by adding the following three lines to the general log section in httpd.conf:

SetEnvIf Request_URI \.gif$ unwanted
SetEnvIf Request_URI \.jpg$ unwanted
SetEnvIf Request_URI \.png$ unwanted

We use name-based virtual hosts, and each host has their own <VirtualHost></VirtualHost> container. In addition to their permanent log file, we add an additional CustomLog command for our webmonitor file for each of our hosts. For example:

<VirtualHost someisp.com>
...
CustomLog /var/log/httpd/someisp.com-access_log combined
CustomLog /var/log/httpd/webmonitor_log webmonitor env=!unwanted
...
</VirtualHost>

CustomLog /var/log/httpd/webmonitor_log webmonitor env=!unwanted

The ability to see what's happening on the server with an external monitor proved so useful that we also included system logging information in the same file. To do this, we edited /etc/tem syslog.conf to include the following command:

kern.*;authpriv.*;*.crit;*.error;*.warning;*.emerg /var/log/httpd/webmonitor_log

Colortail was written by Joakim Andersson (pt98jan@student.hk-r.se) and is available from www.student.hk-r.se/~pt98jan/colortail.html under the GNU Public License.

While we could simply tail the webmonitor log file, adding color to the display is a nice touch and gives us an indication of which web host is seeing activity even if we happen to be on the other side of the workshop.

Colortail comes with several sample configuration files; none really suited web logs, although conf.xferlog comes close. After a bit of tweaking, this is the format we've been using. It's a hybrid as it includes both web and system-log-related items.

Listing 1. colortail.conf

To use colortail locally, you could use a command such as:

colortail -f -k /etc/colortail /var/log/httpd/
 <@cont_arrow><\#229><@$p>webmonitor_log &

This is fine except that it doesn't allow us to have it on screen all the time, and I'd often need to switch to the particular console or X window displaying the log.

To be able to monitor activity better, we display the colortail output on a Commodore 128D computer connected to the system. Our particular set up has our C128 connected to a private server using a null modem and PPP connection. From here, we log in to the server with the log files. You can use any inexpensive spare computer you may have lying around for this purpose, as long as it's capable of handling ANSI or VT100 emulation and has an 80-column display. PPP isn't a requirement.

Rather than type the command to start the colortail on the Commodore machine, we use a nightly cron program that rotates the log file and then sends the colortail output to the PTY device. See Listing 2 for the file used for this purpose.

Listing 2. Cron Program

There are probably as many ways to monitor log files as there are Linux users, but that's part of the fun. While there really isn't anything “new” about using colortail to display log files, this is a different combination of resources from those I've read about, and it works for my requirements. Hopefully, this article will help others looking for a way to view real-time web activity.