How to Log Friends and Influence People

The syslog daemon (known as syslogd) is used to receive messages from various parts of the Linux (or other Unix) system and distribute these messages to other locations. This allows you, the administrator, to review system activities by keeping a log of what goes on behind the scenes. It's a great way of tracking down problems and also of reviewing your system security.

For example, you can keep track of serious errors, such as TCP/IP problems or lack of memory (for when ghostscript and X get a bit out of hand). At the same time, you can debug that pesky mail problem that a user is having, or you can keep track of who has used the su program to assume root status, and you can also find out who failed trying to do so.

To get a copy of the syslog program, see the sidebar on obtaining syslog from an ftp site near you.

Syslogd has a setup file, called /etc/syslog.conf, which tells syslogd what messages to log, and what to do with the messages it receives. By default (i.e., the /etc/syslog.conf file is empty), messages are not stored anywhere. Most distributions of Linux have some basic setup of the syslog.conf file included, and the syslogd program is often started in the /etc/rc.d/rc.inet2 file.

The setup of your syslog.conf file is in three parts. Let's look at a sample syslog.conf file:

# /etc/syslog.conf
 *.info;*.notice    /usr/adm/messages
 *.debug           /usr/adm/debug
 *.warn            /usr/adm/syslog

The first line is a comment. Any lines starting with # or completely blank lines are considered comments and are ignored by syslogd. You can see two columns here, separated by at least one tab. The left side (the one with *.info;*.notice in it) defines the facility and the logging level. The right side defines the destination of the messages.

The facility can be defined as the “kind” of program that is running. For example, “mail” is the facility used whether you are running sendmail or smail. There are nine pre-defined facilities, plus eight facilities that are locally defined, a “wildcard” facility that means “all facilities” and one facility for issuing timestamps.

One special note about the mark facility. This facility is created by syslogd every twenty minutes at the info level. This will allow you to quickly see time change in a log file, along with making sure that syslogd is running (and logging). And as noted above, the special * facility (which means all the available facilities) does not include mark.

Each facility can have a logging (or severity) level to it. This is used to indicate which types of messages you want to record. The high severity levels (such as emerg) require the most attention, as an emerg usually indicates that the program will fail soon. These severity levels decrease in importance all the way down to the debug level, which is used for properly setting up your software. Once your software is configured correctly, you can change the severity level to a higher one. Check the documentation on your software for its interpretation of the various levels. From highest severity to lowest:emergalertcriterrwarningnoticeinfodebugnone

The severity of none means that no messages from that facility are to be recorded. These levels mean that messages of that level and above are recorded. For example, say you have the following two lines:

mail.debug      /var/adm/syslog.mail
mail.emerg      /var/adm/syslog.mail.emerg

When sendmail (or smail), or whatever program is logging as the mail facility) sends syslog a message with a level of debug, it gets placed in the syslog.mail file. Any other messages, of any level from info up to emerg, also get placed in syslog.mail. Emergency messages from mail get sent to both syslog.mail and to syslog.mail.emerg. This setup makes it extremely easy to check for emergency conditions for your software, since a glance at the directory listing for syslog.mail.emerg will tell you if the file has changed recently, and you can easily type tail/var/adm/syslog.mail.emerg to glance at the ten most recent entries. In addition, you can find the emergency message in syslog.mail and see the other messages surrounding it to determine the events leading up to the emergency message.

In another example, we can see the use of the “none” severity level:

*.alert;user.none       /var/adm/syslog.alert
user.alert              /var/adm/syslog.user.alert

Here, all messages at alert severity and above are sent to syslog.alert, except for the ones in the user facility. Those get sent to the syslog.user.alert file instead, as specified by the second line.

Now you are probably wondering about the right hand side of the lines. By now, you have figured out that this can be the name of a file. But it can also be the name of a user or of a remote host. If the intended recipient of the message is a file, the name of the file must start with a /, indicating that you have to give a full path for the filename. Note that this can be just about any file, including /dev/console, which will print the message to the console of the machine, or /dev/lp2 to print your message on paper.

If the recipient is a remote host, it will start with an @ followed by the name of the host. The intended host will receive the message via TCP/IP port 514, and from this point, it is just like a message to syslog locally. A message to the remote host must go through syslog and get logged to the appropriate location. This will allow you to monitor a network of machines from one location. You can also include a comma-separated list of users and the messages will be printed on those users' screens if they are logged in. You can also include a *, which means that every user that is logged in gets the message. This is very useful for extreme emergency conditions where you want the users to realize that something is very wrong and they should log out.

#Log mail errors to the mail host for
#the postmaster to deal with
mail.*         @mailhost
#Send kernel emergency warnings to all
#users so they know what's up
kern.emerg     *

Once you have your /etc/syslog.conf file set up, you have to first set up the files that will be storing the messages, then restart syslogd.

Setting your files up for syslog merely means reviewing the permissions of the log files. By default, files are set up owned by root, in the root group, readable by everyone, and writable only by the root user. For a one-user system, this should be fine. But for a multi-user system, with (possibly) logs of who sent and received e-mail, many of these files should only be read by the root user. A good suggestion for file setup is to make it read-write by root and read-only by the wheel group. This allows you to set up some security (by controlling who is in the wheel group) without having to `su' to root every time you want to check some of the files. Re-starting syslogd does not require killing the syslogd program. All that is needed is to send a SIGHUP (signal number 1) to the process, and it will re-read its configuration files. To help you out even more, the PID of syslogd is stored in the /etc/syslog.pid file. As root:

kill -HUP `cat /etc/syslog.pid`

will re-start the syslog daemon and put your changes into effect. Some Linux packages also include a script called /etc/syslogd.restart, which is merely a script that runs the above kill program. If for some reason you want to actually kill the syslogd program, a TERM signal (or 15) will kill the program (kill -TERM `cat /etc/syslog.pid`)

If you have trouble with either of these, consider using the killall program. This is a linux-specific program, but is quite useful:

killall -HUP syslogd

will work on almost any Linux box. The only possible problem is if you are trying to learn how to administer Unix systems in general, you may be spoiled by using the convenient, but Linux-specific killall command.

For more information, look at the man pages for syslogd and syslog.conf. The Sendmail book published by O'Reilly & Associates (often referred to as the “Bat book” because it features a picture of a bat on the cover) also devotes a few pages to syslog (aside from being a great book on configuring sendmail).

In my January 1995 article rm your way to fun and adventure dealing with the problems of un-deleting, I made a few critical errors which made their way into print. Fortunately, Matt Welsh caught some of these and pointed them out to me.

First, my alias for rm was a bit off. It only copied the first file you gave it (which made it not-usable if you gave it a * or list of files as an argument). The best way to replace it is to create a shell function in the bash shell:

rm()
{
mv $@ ~/.rm
}

Note that you can use /tmp/.rm/$LOGNAME if you preferred storing files in /tmp as originally suggested:

mv $@ /tmp/.rm/$LOGNAME

/bin/tcsh uses $LOGIN for storing the username. /bin/bash uses $LOGNAME. This often creates fun for writing shell scripts. If you are using a shell other than /bin/bash, test it first!!

So, the /etc/profile should have these lines:

alias waste=/bin/rm
rm()
{
mv $@ ~/.rm
}
if [ ! -e ~/.rm ];
then
  mkdir ~/.rm
  chmod og-rwx ~/.rm
fi

if you wish to have the deleted files stored in the user's home directory or:

alias waste=/bin/rm
rm()
{
mv $@ /tmp/.rm/$LOGNAME
}
if [ ! -e /tmp/.rm/$LOGNAME ];
then
  mkdir /tmp/.rm/$LOGNAME
  chmod og-rwx /tmp/.rm/$LOGNAME
fi

if you wish to have deleted files stored under /tmp/.rm.

Using this method will create some problems for users:

1) This method will not preserve the directory structure. If a user does:

rm foo
cd ..
rm foo

only the second foo file will be available for deletion, as the original foo was overwritten by the second one.

2) Certain switches to rm that users will be accustomed to, such as -r, will not be available to them or may do something different.

3) Make sure that /tmp/.rm has full write permissions to it, so that users can create /tmp/.rm/$LOGNAME if it does not already exist.

4) With the second method, if you delete a directory and /tmp and the directory you are deleting are on two different filesystems, mv will complain about moving directories across filesystems. This can happen, but is less likely to occur, with the first method as well.

And as a side note, if you just type crontab, many versions of crontab will drop you into the editor as opposed to crontab -l that I had listed. Mine does not, but yours might. The dcron22 used by slackware will automatically drop you into the vi editor. The Vixie Cron used by some other installations will check the value of the EDITOR or VISUAL environmental variables if you wish to use another editor.

So what (you might ask) is being done to prevent this from happening again? Good question. I'm setting up a small mailing list which will have as members some good Linux experts, along with some not-so-experienced users so that any potential confusion or problems get ironed out before we go to print. If you would like to join this mailing list, please drop me an e-mail note at komarimf@craft.camp.clarkson.edu and ask.