Transferring Conserver Logs to Elasticsearch

If your organization manages Linux, AIX, HP-UX or Solaris servers in-house, chances are your system administrators at least occasionally need low-level access to those devices. Typically, administrators use some kind of serial console—for example, traditional serial port, Serial-over-LAN or Intelligent Platform Management Interface (IPMI). Managing and auditing console access is not trivial, so many organizations rely on the Conserver application to create session logs when accessing these servers via the serial console. These logs can be useful for various reasons—for example, maintenance or troubleshooting (to review why something crashed), security (to find out who did what—connecting user names to actual users) or compliance (to provide detailed session logs).

This article covers the following:

Conserver is a wonderful piece of software that lets you manage your infrastructure's serial consoles, whether they be old-style hardware serial ports or state-of-the-art Serial-over-LAN (SOL) baseboards. Its distributed design permits a decentralized user experience using a secure, TLS-encrypted protocol. The straightforward workflow consists of the user connecting to any conserver master node, which then forwards the traffic to the node that manages the console you want to access. As all masters share the same configuration file, it is very straightforward to redistribute consoles among servers automatically (provided they are virtual SOL devices, like IPMI) using configuration management (for example, using the Puppet module we developed at CC-IN2P3).

So where is the catch? As far as we are concerned at the Computing Centre of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3), the logging mechanism could be greatly improved, because conserver's design is quite ancient now. For example, it does not support logging to syslog. From a user perspective, logging is awesome, as you can use a keystroke to access the logs of the console, and the console logs contain the complete session. From an architecture perspective though, things are not so great, as every conserver master stores the logs of the consoles it manages locally in a file.

This is where syslog-ng Open Source Edition comes into play. The idea is to transport the logfiles of the conserver masters to our favorite event store back ends, which are Riemann and Elasticsearch. They provide powerful real-time stream processing and long-term indexed storage capabilities, respectively. In addition, with syslog-ng, you simply can send the logs to Riemann and Elasticsearch directly; there is no need for any additional agents (like Logstash). To see how this system works before going into configuration details, watch this video: https://webcast.in2p3.fr/videos-syslog_ng_conserver.

The video shows what the user does in the console (in the top-right section of the screen), its effect on the real-time Riemann-dash dashboard (bottom-right) and the near-real-time Elasticsearch front end (Kibana|Elastic, on the left).

As you can see, the user activities and events of the session are transported to the back ends, including useful metadata like conserver.is_attached: true. This tells you whether or not someone was attached to the console (which is obviously the case in this example).

To create the system shown in the demo video, you need the following software:

Note that this article does not cover how to install, configure (in general) and get the above software working. You can find plenty of related tutorials on-line. If you need help with these tasks, check the documentation, mailing lists or on-line forums for the software you need help with. The following sections of this article explain how to configure the components of this infrastructure for the specific needs of our scenario as an example.

The conserver configuration in our setup is nothing special. It creates a unified logfile, which will serve as the glue between conserver and syslog-ng. You can activate the unified logfile using either of the following methods:

1) Run the conserver executable with the -U /var/log/console.log flag.

2) Use the following configuration block in conserver.cf:

<config * { unifiedlog * /var/log/console.log; }

You also can set the server's general logfile (where conserver stores the global messages that are unrelated to individual consoles)—for example, to /var/log/conserver.log.

Both /var/log/conserver.log and /var/log/console.log will be inputs for syslog-ng. You might want to take special care of the log rotation of these files. As you are sending them to Elasticsearch, there is no need to keep them for too long.

You need to install syslog-ng locally on each conserver master and on a central host (that is, your logserver) that will gather all the events from the conserver hosts. The local instances will parse, process and enrich the console output, while the central host will collect them and send them over to the two back-end systems, Riemann and Elasticsearch. Note that you could get the same results using only local instances, but most people prefer to centralize first, for various reasons.

Configuring syslog-ng on the Conserver Hosts:

The following is an example configuration file for running syslog-ng on the conserver hosts. As you can see, it has three sources:

The s_console and s_conserver sources process conserver's unified logfile. Since the format of the console and conserver messages is different, we have to configure syslog-ng to parse them differently, then forward them to the central syslog-ng server (you can add any other sources as needed for your environment):

@version: 3.7

@include scl.conf

options {
    threaded(yes);
};

source s_conserver {
  channel {
    source {
      file(
        '/var/log/conserver.log',
        flags(no-parse)
      );
    };
    parser {
      csv-parser(
        columns(tmp.date,PROGRAM,PID,MESSAGE)
        delimiters(' :')
        quote-pairs('[]()')
        flags(greedy)
      );
    };
    rewrite {
      set('$(strip $MESSAGE)', value(MESSAGE));
    };
  };
};

source s_console {
  channel {
    source {
      file('/var/log/console.log');
    };
    junction {
      channel {
        filter{
          program('\*/div>);
        };
        rewrite {
          subst('\*/div>, '', value(PROGRAM));
          set(
            'true',
            value('.SDATA.console.is_attached')
          );
        };
        flags(final);
      };
      channel {
        rewrite {
          set(
            'false',
            value('.SDATA.console.is_attached')
          );
        };
        flags(fallback);
      };
    };
    rewrite {
      set('$PROGRAM', value(HOST));
      set('console', value(PROGRAM));
    };
  };
};

source s_internal {
  internal();
};

destination d_remote {
  network(
      "",
      transport(tcp),
      port(514),
      flags(syslog-protocol)
  );
};

destination d_internal {
  file("/var/log/syslog-ng.log");
};

log {
  source(s_console);
  source(s_conserver);
  destination(d_remote);
};

log {
  source(s_internal);
  destination(d_internal);
};

Global Conserver Logs—the s_conserver Source:

If you are not familiar with syslog-ng, the s_conserver and s_console sections can be a bit intimidating. To better understand how they work, take a look at a sample message conserver produces:

[Thu Sep  3 22:29:52 2015] conserver (13550): 
 ↪[node42] automatic reinitialization

The related source definition contains three blocks:

This configuration parses the above example message into the following structured data:

tmp.date: Thu Sep  3 22:29:52 2015
PROGRAM: conserver
PID: 13550
MESSAGE: [node42] automatic reinitialization

Console Logs—the s_console Source:

Here are two example messages from two different consoles:

node03: ACPI: No handler for Region [POWR] 
 ↪(ffff8808248bb150) [IPMI]

node66*: node66 login: root

The first one is an unattended message probably produced by an ACPI signal. The second one, as hinted by the * (asterisk) character appended to the name of the console, is a message produced while someone was attached to the console (using console node66). We will use this hint to produce additional metadata. The source consists again of three parts:

So in the above examples, the messages are parsed into the following structured data:

PROGRAM: console
HOST: node03
MESSAGE: ACPI: No handler for Region [POWR] 
 ↪(ffff8808248bb150) [IPMI]
.SDATA.console.is_attached: false

PROGRAM: console
HOST: node66
MESSAGE: node66 login: root
.SDATA.console.is_attached: true

Forwarding the Logs to the Central syslog-ng server, d_remote:

The rest of the syslog-ng configuration is simple: we just send the structured payload using the syslog IETF RFC5424 protocol (hence the syslog-protocol flag) to the central syslog-ng server. All RFC5424 keys, including .SDATA.*, are sent over to the central syslog-ng server automatically. The only part that we parsed from the conserver logs that is not transferred to the central server is the tmp.date field. Instead, we will use the time when syslog-ng processes the message (which is a good approximation for current logs). If you absolutely want to use the value from tmp.date (because, for example, you want to send old conserver logs to the remote server), you can use the date parser from the syslog-ng-incubator project.

Configuring syslog-ng on the Central Logserver:

On the central syslog-ng server, we have to route the console and conserver messages received from the conserver hosts to the Riemann and Elasticsearch back ends. The following syslog-ng configuration does exactly that; the only adjustment is that it removes the .SDATA. prefix from the fields, so they are more readable:

@version: 3.7

@include scl.conf

options {
  threaded(yes);
};

block destination realtime (
  host()
  port(5555)
  type("udp")
  throttle(0)
  flush-lines(1)
)
{
  riemann(
    flush-lines(`flush-lines`)
    throttle(`throttle`)
    server(`host`)
    port(`port`)
    type(`type`)
    ttl("${ttl:-300}")
    host("$HOST")
    description("$MESSAGE")
    attributes(
      scope(all-nv-pairs)
      key(".SDATA.*"
        rekey( shift(7) )
      )
    )
  );
};

source s_remote_tcp {
  channel {
    source {
      network(
        transport(tcp)
        port(514)
        flags(syslog-protocol)
        tags("syslog")
        so-rcvbuf(8388608)
      );
    };
  };
};

source s_internal {
  internal();
};

destination d_elasticsearch {
  elasticsearch(
    index("syslog-${YEAR}.${MONTH}.${DAY}"),
    type("syslog"),
    flush-limit(1),
    template("$(format-json -s all-nv-pairs --rekey 
     ↪.SDATA.* --shift 7 --key ISODATE)")
    cluster("elasticsearch")
    port(9300)
    server("localhost")
    client_mode("transport")
    time-zone("UTC")
  );
};

destination d_internal {
  file("/var/log/syslog-ng.log");
};

destination d_riemann {
  realtime(
    host("riemann"),
  );
};

log {
  source(s_remote_tcp);
  destination(d_riemann);
  destination(d_elasticsearch);
};

log {
  source(s_internal);
  destination(d_internal);
};

From this article, you have learned how to create a system that allows you to review serial console logs in real time and make them accessible for free-text searching on a modern user interface. This is helpful for maintenance and troubleshooting purposes, and also for meeting auditing and compliance requirements. To achieve these goals, conserver can be integrated with Riemann and Elasticsearch. To integrate these services, you can use syslog-ng Open Source Edition, a flexible log collecting and processing application that can collect and parse the log messages and forward them to the Riemann Elasticsearch back end.

The syslog-ng application is very flexible and has powerful message-processing capabilities. If you learn a bit about its possibilities, you can find several ways to improve the configuration described in the article. Here are some ideas that you can do with syslog-ng: