Take command of your log files by learning to handle those pesky logging daemons.
Most UNIX-like systems since the early days of BSD (and Linux certainly falls in this category) have provided an API for application programs to send log messages to the system, where they can be centrally handled at the discretion of the system operator. Prior to the creation of this facility, each application program would handle log messages in its own way. Some would write to STDERR, some would write to a file, some would write to a pipe, and some would offer all these options or more.
As the number and complexity of applications on a system grows, so, too, does the complexity of the system administrator's job. Applications and their messages vary widely in their significance to certain audiences. If a number of applications are considered “critical” and their status is the system administrator's responsibility, he does not want to search to find out where and how every critical application logs its status. That's where syslogd comes in.
BSD added an API for logging to the standard library. Linux also offers it. This API consists of three function calls:
#include <syslog.h> void openlog( char *ident, int option, int facility) void syslog ( int priority, char *format, ...) void closelog( void )
An application that wishes to use syslogd for logging uses these calls. A brief introduction to the use of this API can be found in the sidebar “Using the syslog API in Applications”. The one critical thing to know is that all messages from this API have, at a minimum, a “facility” and a “priority”.
Using the syslog API in Applications
Facilities include LOG_AUTHPRIV, LOG_CRON, LOG_KERN, LOG_DAEMON and so forth. These serve to identify the “system” of origin. Note that it is not the “program” of origin. For example, many different programs make up UUCP, but they all log as LOG_UUCP. The program name can be a component of a log message, but this has nothing to do with the facility. Some programs will log as more than one facility. For example, telnetd might log failed logins as LOG_AUTHPRIV, but it might log other messages as LOG_DAEMON. “Priorities” specify the “severity level” or “level of attention” the message merits. We'll discuss these concepts at greater length throughout this piece.
Our primary focus here is the tool that handles messages sent via this API. While syslogd was originally developed for 4.2BSD, we are going to cover the version that ships with most Linux distributions today, specifically syslogd version 1.3-3. The syslogd utility normally reads a configuration file at startup to determine how messages are to be handled. This file, normally /etc/syslog.conf, tells syslogd what to do with messages. Much of the rest of this document will describe how to use syslog.conf to customize logging on your system.
The syslog.conf file follows the more or less ubiquitous UNIX convention of using the pound sign (#) as a comment character. We'll use the sample syslog.conf file in Listing 1 for the rest of our discussion. This is the “out-of-the-box” syslog.conf from my Red Hat 6.1 laptop. I use other distributions personally, primarily Debian and SuSE, but Red Hat seems to be the most popular. Let's understand what this file is doing.
“Rules” in syslog.conf are a single line which consists of two parts. The first is a “selector”, which specifies the set of messages on which the rule is to act. The second is an action, which specifies what is to be done with messages that match the selector.
The selector is further divided into a “facility” and a “priority”. Yes, these match the terms mentioned above in the brief description of the syslog API. The facilities and levels have numeric values and you can use them in the syslog.conf file, but it is strongly advised that you do not. Symbolic values are supported, and if the syslog API is ever changed, the numeric values might change, whereas one would expect that the symbolic names would be kept in alignment with any such change. So, be safe and use the symbolic names.
The symbolic facility names are auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and local0 through local7. The security keyword is deprecated in favor of the auth keyword. The mark keyword is internal and should not be used by applications; the syslogd program can be set to produce a mark periodically, which provides a means to tell if you aren't getting messages, because none are being generated or because syslogd has died. The rest of them correspond to the major subsystems on your Linux box.
The priority keywords are debug, info, notice, warning, warn (same as warning), err, error (same as err), crit, alert, emerg and panic (same as emerg). The keywords error, warn and panic are deprecated and should no longer be used.
A selector consists of a facility and a priority separated by a period (.) character. Thus, mail.crit would select all critical messages from the mail facility.
The default behavior of the BSD syslog system is for all messages of the specified or higher priority to be handled by the action. The Linux syslogd does the same by default. It does have a number of extensions, however.
You may use the asterisk (*) character to indicate all facilities or all priorities (depending on whether it appears before or after the period). Thus, the authpriv.* line in the example sends all messages from the authentication facility, no matter what priority, to /var/log/secure. You may use the special priority none to indicate that no messages from a given facility rule are to be acted upon by the action.
You may specify multiple facilities with the same priority in a single rule by listing the facilities separated by commas (,) before the period. Thus, the uucp,news.crit line sends all critical and above priority messages from the mail and news facilities to /var/log/spooler.
You may specify multiple selectors for a single action by listing them separated by the semicolon character. Each subsequent selector may override the previous. Thus, the *.info;mail.none;news.none;authpriv.none rule would send all messages above info priority from all facilities (because of the *) to /var/log/messages, except messages from mail, news or authentication facilities (because of the none keyword and because rules are applied in order, left to right).
More than one rule may apply to a message! It is important to understand that a message will be sent to all actions with matching selectors. It is not as if a message, once matched, is gobbled up. That means you can store a single message to multiple actions if it is matched by multiple selectors.
There are more priority selection extensions. First, remember that the default is to select messages of the stated or higher priority. You may also reverse the sense with the exclamation mark. So, for example, a rule such as
*.!err /var/log/routine
would send all messages not at err or above to /var/log/routine (a file meant for “routine” messages, apparently).
You can also restrict the selection to an exact priority instead of to a given priority and higher with the equals sign (=). Thus, the news.=crit rule out of our example would send only critical messages from the news facility to /var/log/news.crit.
At this point, you may be a little foggy on what, precisely, these various priorities are meant to denote. Let's shed a little light on that issue by looking at the “Priorities” table.
A classic problem in designing software is trying to figure out an empirical way to tell the difference between a condition you would report as crit vs. alert. In fact, it is sometimes even more difficult to decide when you should use notice vs. warn. You won't find total agreement between packages on what level of message falls where. One difficulty lies in trying to decide who will read the log. An emerg to a business unit might be only a warn to a network administrator.
There's no one good answer to this problem. At least, by creating a uniform method for handling program messages, we avoid a proliferation of different reporting systems, and some conventions have emerged with time. Because we are able to match a message to more than one action, we can output messages to targeted audiences. For example, we could report all auth messages to the security department's home directory, but the system administrator might choose to receive only those of “crit” or above. The syslogd lets us do this.
There are some drawbacks to using the “out-of-the-box” Red Hat syslog.conf file. Notably, all “actions” are basically to write to local files. The syslogd dæmon can do much more than that. Let's take a look at actions next.
Actions may send messages to any of these destinations:
A regular file: this is what you see in our example. This is simply the name of a file to which the message is appended.
A named pipe: named pipes, or FIFOs (First-In, First-Out), are a simple form of inter-process communication supported by Linux and many other operating systems. You create a named pipe with the mkfifo command; a FIFO appears in the file system. You tell syslogd it is writing to a FIFO instead of a file by putting a pipe character (|) in front of the FIFO name. Take a look at the man pages for mkfifo, both the command and the system call, and the man page for “fifo”, which is a description of the special file. You read and write FIFOs with the normal file system calls. A description of FIFO programming is beyond our scope, although I can highly recommend the excellent book UNIX Network Programming by W. Richard Stevens, from Prentice-Hall.
A terminal or console: if you specify a tty device (such as /dev/console), syslogd is smart enough to figure out that it is a device, not a file, and treat it accordingly. This can be fun if you have a dumb terminal—you can send all your messages to /dev/ttyS1 (for example) and get all your messages on the terminal screen while you work on your console. This is state-of-the-art 1970 technology—I love glass teletypes!
A remote machine: now this is the true power. Let's assume you have many Linux boxes on a network. Do you want to log in to each to check their logs for certain conditions? Of course you don't, and you don't have to. Optionally, the syslogd listens on the network for messages as well. Just put an at sign (@) followed by the host name:
*.crit @sol.n0zes.ampr.org
This last will send all critical and above messages from all facilities to sol.n0zes.ampr.org, which will then apply its own syslog.conf file to save them. Syslogd will not forward a message received from the network to another host: in other words, you get one and only one hop. This may be overridden with switches when syslogd is invoked. It seems like a reasonable thing to do, since even the possibility of circular message routing would be enough to scare the dickens out of any network administrator.
This capability has obvious advantages for centralized logging and log scanning for security violations and so forth. It also has obvious deficiencies. It is hard to maintain a complete log when your network is down. Take advantage of the fact that you can route messages to more than one action by making sure every message finds its way to a file before you send it to remote logger.
One feature that newer users of Linux may not be aware of is console messaging. This isn't used very much any more, thanks to talk and irc and other much more interactive “chat” mechanisms with much cleaner user interfaces. You can, however, send a text message to any user logged in to your system with the “write” command. This is an unpopular facility for several reasons. First, in today's windowed environments, a user probably has many “terminals” active and it is hard to know which one to write to. Second, if they are in the middle of some intense full-screen activity (such as editing a large file with vi) and you blast a bunch of text at them that confuses their editor and screws up their screen, they will not like you very much. Most Linuxes I have seen default their users to messaging being off. This facility uses the same ability to write to a user's console to send messages directly to their screen. Just put a comma-separated list of user names as the action. Save this for truly critical stuff. You might turn this on to try it, but I bet you will turn it off again before too long.
There is a similar method, called wall or write-to-all. This lets you send a text message to every user logged in to the system. The superuser can do this whether you choose to accept messages or not. This is how shutdown sends its warning messages. You can have syslogd send a message to everyone by this mechanism by specifying an asterisk (*) as the action. Save this one for the most dire of dire messages, if you must use it. This should be used for warning of an impending crash—anything less is probably overkill.
At this point, you may be asking what klogd has to do with any of this. The answer to that is simple. The kernel can't call the syslog API. There are many reasons for this. The core reason, and the simplest to understand, is that Linux actually provides two completely separate APIs. The more familiar one is the “standard library” used by user-space applications; this is the one that uses syslog. The other API is normally not used by applications: this is the kernel API code that runs as part of the kernel. This code needs services similar to those offered by the applications programming interface, but for numerous technical (and a few aesthetic) reasons, it is not possible for kernel code to use the application's API. For this reason, the kernel has its own entirely separate mechanism for generating messages. The klog dæmon, klogd, is an application process that ties the kernel messaging system to syslogd. Actually, it can also dispatch kernel messages to files, but most configurations use the klogd default, which is to prepare kernel messages, and in essence, resubmit them through syslog.
There is quite a bit more to klogd if you wish to delve into the depths, but for the purposes of this article, it is sufficient to know that klogd feeds kernel messages to syslogd, where they appear to be coming from the kern facility.
syslogd provides a powerful and simple mechanism for managing messages from multiple applications in a highly configurable manner. Its ability to “demultiplex” the message stream makes using the syslog API an appealing option for applications developers, and I would encourage you to consider using that API in your own programs.
