LJ Archive

Security Monitoring and Enforcement with Cfengine 3

Aleksey Tsalolikhin

Issue #205, May 2011

Cfengine is well known as a powerful system configuration management tool, but did you know you also can use it to secure your systems?

Cfengine, from the start, has had security as a key part of its design and use scenarios. Here, I demonstrate how Cfengine 3 can be used to increase the security of a Linux system by monitoring file checksums, monitoring filesystems for suspicious filenames, monitoring running processes, monitoring open ports and managing sshd.conf.

Because Cfengine 3 is under active development, I suggest you install the latest version from the Cfengine Source Archive (see Resources).

The purpose of this article is to give practical examples of how you can use Cfengine to increase security on a Linux system. See the Quick Start Guide in the Resources section of this article for help in learning the Cfengine language. (I don't provide a tutorial on the Cfengine language here.) This article is based on Cfengine version 3.1.5a1.

Monitoring File Checksums

Cfengine 3.1.4 shipped with 214 unit tests that can double as examples of Cfengine's functionality. They are installed to /usr/local/share/doc/cfengine/. I've adopted unit_change_detect.cf into detect_changes_in_etc.cf (Listing 1).

Run this with:

cf-agent -KIf detect_changes_in_etc.cf

cf-agent is the component of Cfengine that actually makes changes to the system. (There are other components to serve files, monitor system activity and so on. cf-agent is the piece that makes changes to the system, and the one you'd use to start learning Cfengine.) In the command above:

  • -K — tells cf-agent to ignore time-based locks and allows you to run cf-agent repeatedly (no “cool-off” period, which might otherwise kick in to prevent system overload).

  • -I — tells cf-agent to inform you of its actions and any changes made to the system.

  • -f — specifies the policy filename.

On the first pass, cf-agent builds a file information database containing file timestamps and inode numbers and builds an MD5 hash for each file. You should see something like this:

# cf-agent -KIf detect_changes_in_etc.cf
  !! File /etc/hosts.allow was not in MD5
     database - new file found 
  I: Made in version 'not specified' of
     'detect_changes_in_etc.cf' near line 22

There are two messages here, alert and info.

Cfengine prefixes its output to help you understand what kind of output it is (in other words, metadata):

  • Informational messages start with “I”.

  • Reports start with “R”.

  • Alerts start with !! or ALERT.

  • Notice of changes to the system starts with ->.

In the above case, the alert message is accompanied with an info message about the policy that was in effect when the alert was produced, its version number (if supplied) and the line number.

I didn't specify the version number, but the line number is useful. Line 22 is:

changes      => detect_all_change,

This is the line responsible for Cfengine adding /etc/passwd to the MD5 database. It tells Cfengine what to do about changes—to detect them.

Now, I run cf-agent again, and it runs quietly. The contents of /etc match the MD5 sum database:

# cf-agent -KIf detect_changes_in_etc.cf

Next, I edit /etc/hosts.allow to add “sshd: ALL” to simulate an unauthorized change. Watch cf-agent scream:

# cf-agent -KIf detect_changes_in_etc.cf
ALERT: Hash (MD5) for /etc/hosts.allow changed!
 -> Updating hash for /etc/hosts.allow to
I: Made in version 'not specified' of
'./detect_changes_in_etc.cf' near line 22
ALERT: inode for /etc/hosts.allow changed
38901878 -> 38901854
ALERT: Last modified time for /etc/hosts.allow
changed Sat Jan 29 17:09:26
2011 -> Mon Jan 31 08:00:02 2011

There are three alerts:

  1. MD5 hash changed (because the contents changed).

  2. The inode number changed (when vi saved the file).

  3. The modification time changed (when vi saved the file).

Reminder: messages about actions that Cfengine takes are prefixed with “->”:

 -> Updating hash for /etc/hosts.allow to

You can set up Cfengine to complain via e-mail or syslog, so even if the intruder tampers with the MD5 database, the alarm will sound. In commercial versions of Cfengine (Nova), you can set up multiple Cfengine nodes to share their MD5 databases and monitor and cross-check each other.

You can run this check fairly often—every five minutes, if you like and if your hardware will sustain it. (Computing lots of MD5 sums can be expensive on CPU and disk I/O.) Is the added security worth it to you?

Monitoring for Suspicious Filenames

Cfengine has a special cf-agent control variable called suspiciousnames. You can put a list of names into it to warn about during any file search (such as was done during the MD5 hash check). If Cfengine sees these names during recursive (depth) file searches, it will warn about them. If suspiciousnames is not set, cf-agent won't check for them. It's not set by default.

Let me demonstrate how this works by adding the following control block to detect_changes_in_etc.cf:

body agent control
suspiciousnames => { ".mo", "lrk3", "rootkit" };

A cf-agent control block controls the behavior of cf-agent. This is where you can set things like dry-run mode (don't change anything but report only on what changes would have been made—useful for learning Cfengine), the largest file size Cfengine will edit and so on. So the suspiciousnames variable is set in the agent control block. It's an array of strings.

Let's create a suspiciously named file to see cf-agent get excited:

# date > /etc/rootkit
# cf-agent -IKf detect_changes_in_etc.cf
Suspicious file rootkit found in /etc

So, if you're scanning your system directories for an MD5 hash check, you can add the suspicious name check too.

Monitoring Running Processes

I follow the best practice of securing servers by disabling unnecessary services. I often want to make sure my Web servers are not running CUPS—usually, a Web server does not need to print!

The example shown in Listing 2 is based on the Cfengine unit test unit_process_kill.cf.

The line of interest in Listing 2 is:

processes: "cupsd"  signals => { "term", "kill" };

This means if there is an entry in the process table matching “cupsd”, that process will be sent TERM and then KILL signals:

# cf-agent -IKf cups_not_running.cf
 -> Signalled 'term' (15) to observed process match '28140'
 -> Signalled 'kill' (9) to observed process match '28140'

But, let's not be so brutal. Cfengine can report suspicious process names. You can keep an eye out for password sniffers, crackers, IRC bots and so on with the policy shown in Listing 3.

The key line here is:

vars: "suspicious_process_names" slist => { "sniff",
    "eggdrop", "r00t", "^\./", "john", "crack" };

A variable called “suspicious_process_names” is a list of strings; what we deem as suspicious process names includes, let's say, any processes starting with ./. As you can see, this list can include regular expressions. Cfengine uses Perl-compatible regular expressions.

You can set the contents of this array to reflect what you consider suspicious process names. Then, Cfengine scans the entire process table (that's the processes: .*) and loops over the contents of the “suspicious_process_names” array. Cfengine has implicit looping over arrays, so if you have an array @{suspicious_process_names} and you reference ${suspicious_process_names}, you're actually saying:

for ${suspicious_process_names} in (@{suspicious_process_names}

That's what happens when you say process_select => proc_finder("$(suspicious_process_names)"); You're actually saying, for each element in @(suspicious_process_names), find processes that match that regex.

Anyway, I want this to be a security demonstration rather than a language primer, so let's continue:

# cf-agent -IKf report_suspicious_process_names.cf
 !! Matched: root     20044 20002 20044  0.0  0.0  
    4956  19   664    1 22:05 00:00:00 ./eggdrop 

The first numeric field (20044) is the PID. The last field is the process name. (Why is there an IRC bot on my Web server?)

Monitoring Open Ports

You can increase your security situational awareness by knowing on what ports your server is listening. Intruders may install an FTP server to host warez or install an IRC server for bot command and control. Either way, your server's TCP profile has changed (increased) in terms of on what TCP ports it listens.

By constantly comparing desired and actual open TCP ports, Cfengine quickly can detect an intrusion. Cfengine 3 runs every five minutes by default, so it can detect a compromise pretty fast.

The code example shown in Listing 4 starts with hard-coded lists of what TCP ports and corresponding process names are expected on the system: 22 sshd 80 httpd 443 httpd 5308 cf-server. It then uses lsof to get the actual list of TCP ports and process names, compare them and report DANGER if the comparison fails.

Here's an example run:

# cf-agent -IKf ./check_listening_ports.cf
DANGER! Expected open ports and processes:
DANGER! 22 sshd 80 httpd 443 httpd 5308 cf-server
DANGER! Actual open ports and processes:
DANGER! 22 sshd 80 httpd 443 httpd 3306 mysqld 5308 cf-server

Again, this is a security demonstration, not a language primer, but if you want to understand the policy, follow the Quick Start Guide for Cfengine. If you need any help understanding this policy, come to the help-cfengine mailing list or ask me directly at aleksey@verticalsysadmin.com.

Managing sshd.conf

The next example is Diego Zamboni's Cfengine bundle for editing the sshd configuration file and restarting sshd if any changes were made. It has two parts (to abstract the under-the-hood details). In the first part, the sysadmin edits the sshd array to set variables corresponding to the sshd configuration parameters. For example, to mandate Protocol 2 of SSH, set:

"sshd[Protocol]" string => "2";

If the parameter is commented out, Cfengine uncomments it and sets it to the desired value. If the parameter is absent, Cfengine adds it and sets it to the desired value. Additionally, if any changes were made to sshd_config, sshd restarts to activate the change.

For an example of changes made, run diff of sshd_config before and after Cfengine edited it to set Protocol, X11Forwarding and UseDNS:

# diff /etc/ssh/sshd_config /etc/ssh/sshd_config.cf-before-edit
< #Protocol 2,1
> Protocol 2
< #X11Forwarding no
< X11Forwarding no
> X11Forwarding yes
> X11Forwarding yes
< #UseDNS yes
> UseDNS no

You may notice X11Forwarding is there twice after the edit, because it was in the file twice before the edit, once commented and once uncommented. But, this does not break things. Having X11Forwarding yes is valid syntax, and the /usr/sbin/sshd -t syntax checker does not complain.

You also may notice that cf-agent saved a copy of the original file, just in case.

Learning More

Download the source and follow the Recommended Reading on the Quick Start Guide site. Also, please visit us on the help-cfengine mailing list to share your ideas on automating security with Cfengine.

Aleksey Tsalolikhin has been a UNIX systems administrator for 13 years, including seven at EarthLink. Wrangling EarthLink's server farms by hand, he developed an abiding interest in automating server configuration management. Aleksey taught “Introduction to Automating System Administration with Cfengine 3” at Ohio Linux Fest 2010 and Southern California Linux Expo 2011 as an instructor from the League of Professional System Administrators.

LJ Archive