Security Distribution for Linux Clusters

This article is a follow-up to previous articles in LJ that discuss the Distributed Security Infrastructure (DSI) and the Linux Distributed Security Module (DSM) [see “Linux Distributed Security Module”, LJ, October 2002, available at /article/6215, and “DSI: a New Architecture for Secure Carrier-Class Linux Clusters”, available at www.linuxjournal.com/article/6053]. In this article, we focus on how we used IP options in DSM to send security information in a distributed environment for the process level of security. We discuss network buffer handling, adding hooks into the kernel, IP options and modifying IP headers. We then cover the network hooks in DSM and present some early performance results.

The Open System Lab at Ericsson Research started the Open Source DSI Project to design and develop a cluster security infrastructure targeted at soft, real-time telecom applications running on Linux carrier-grade clusters. These clusters are expected to operate nonstop, regardless of any hardware or software errors. They must allow operators to upgrade hardware and software, kernel and applications, during normal operations, without any scheduled downtime and without affecting the offered services.

DSI originally was designed to offer carrier-grade characteristics, such as reliability, scalability, high availability and efficient performance. Furthermore, it supports several other important features, including a coherent framework, a process-level approach and support for both preemptive security and dynamic security policies.

One important feature of DSI is its process-level access control. Currently implemented security mechanisms are based on user privileges and do not support authentication checks for interactions between two processes belonging to the same user, even if the processes are created on remote processors. For telecom applications, only a few users run the same application for a long period of time without any interruption. Applying the above concept grants the same security privileges to all processes created on different nodes, which leads to no security checks for many actions through the distributed system. The granularity of the basic entity for the above security control is the user. For carrier-class applications, this granularity is not sufficient, and the need for a more fine-grained basic entity, the individual process, is required and thus supported in DSI.

The DSM is a core component of DSI that provides the implementation of mandatory access control within a Linux cluster. The DSM is responsible for enforcing access control and providing labeling for the IP messages with the security attributes of the sending process and node across the nodes of the cluster.

The DSM is implemented as a Linux module using the Linux Security Module (LSM) hooks. The development started using Linux kernel 2.4.17 along with the appropriate LSM kernel patch. The implementation was based on CIPSO and FIPS 188 standards, which specify the IP header modification.

One important aspect of the DSM implementation is its distributed nature. Access control in a cluster can be performed from a subject located on one node to a resource located on another node. Therefore, a need exists to transfer the security information between the nodes in the same cluster. The distributed nature of DSM provides location transparency of the security resources in the cluster from the security point of view.

Here, we briefly discuss the topic of the network buffer handling to provide a better understanding of how security information is embedded into the network packet. We describe how the kernel handles network buffers starting from the application layer down to the hardware layer and vice versa.

Figure 1. Network Packet Flow in the Kernel

Figure 1 shows the flow of the network packet in the kernel. Packet handling occurs in two cases, the incoming packet and outgoing packet. The outgoing network packet is handled as follows, starting from the application layer: the application prepares the data to be sent on the network; the application issues a system call to the kernel to send a packet; the packet, in the form of an sk_buff structure, goes through filters and routing functions inside the kernel; and the packet then is passed to the network driver that sends it to the network card (DMA).

The incoming network packet, starting from the network card, begins with the network card capturing the network packets either with its own address or the broadcast address; it then reads them to the network memory and generates an interrupt. The interrupt service routine, which is triggered by the hardware interrupt and is a part of the network card driver that runs inside the kernel, allocates an sk_buff and moves the data from the card memory into this buffer (DMA). Next, the packet is put on the CPU queue for upper-layer processing, and the processing is deferred to a later time when interrupts are enabled. Finally, the packets go through the filters and the routing functions and are passed to the application layer.

Based on the generic information on how the network buffers are handled in the Linux kernel, we now demonstrate how this information can be used to extend the kernel security. We look into the hooks added to the IP routing functions that allow us to manipulate the IP packets and add extra security to the IP messages.

The security module can influence the routing decision based on the security hook implementation. A few things need to be remembered when programming the routing hooks, because those hooks are executed as normal kernel functions for every packet coming in and out the kernel.

A module that registers a function must specify the priority of the function within the hook. The net filter hooks are called from the kernel code in the order of priorities. The user functions are free to manipulate the IP packet. The user function must return one of the following values in order for the networking code to decide what to do with the packet:

This function shows us how the packet is manipulated before it enters the system and before it is sent out. What we are still missing is: what kind of information or options can we add to the packet? How? And, will the changes coexist with the current implementation? We answer these questions in the following sections.

A little-known fact about Internet Protocol is that an IP packet can contain a variable amount of extra information (maximum of 40 bytes) following the standard 20-byte header. These extension bytes are called IP options, and some of the options are defined to carry security information.

Currently, the Internet Protocol includes two security options. One of them is the DoD Basic Security Option (BSO—Option Type 130), which allows IP datagrams to be labeled with security classifications. This option provides 16 security classifications and a variable number of handling restrictions. To handle additional security information, such as security categories or compartments, a second security option (ESO—Option Type 133) exists and is referred to as the DoD Extended Security Option (ESO). The Defense Information Systems Agency (DISA) is responsible for administrating the values for the fixed fields within these two options.

Computer vendors now are building commercial operating systems with mandatory access controls and multilevel security. These systems are no longer built specifically for a particular group in the defense or intelligence communities. They are generally available commercial systems for use in a variety of government and civil sector environments.

The small number of ESO format codes cannot support all the possible applications of a commercial security option. The BSO and ESO were designed to support only the United States DoD. Commercial IP Security Option (CIPSO) has been designed to support multiple security policies. The Internet draft provides the format and procedures required to support a mandatory access control (MAC) security policy.

The IP options used to label packets in our implementation are based on the FIPS 188 standard and the Commercial IP Security Option (CIPSO) draft. In our implementation, the IP header is changed using these standards, so we can add the security information to the IP header and send it over the network.

The security information we want to transfer using IP options are Security ID (SID) and Security Node ID (NID). The DSM modifies every IP packet by supplying our security information as its IP options. Figure 2 shows the format of the modified IP header.

Figure 2. Security Options in the IP Header

Here is a list of header options:

We used the LSM security hooks in the DSM to add our security labels to the IP messages. We now demonstrate how we achieved this by presenting an example of an application that sends a packet over the network by writing to a socket. The application uses some of the library calls. At one point, a system call is generated that passes the message to the Linux kernel. The entry point to the kernel socket implementation is the function sys_socketcall(), located in net/socket.c. In the chain of calls, the sock_sendmsg() function (Listing 1) in net/socket.c is executed.

sock_sendmsg
(struck socket *sock, struct msghdr *msg, int size)
{
  int err;
  struct scm_cookie scm;

  err =
    security_ops->socket_ops->sendmsg(sock,
                                      msg, size);
  if(err)
    return(err);
  ...

}

One of the first actions in the function is to execute the security hook (security_ops->socket_ops->sendmsg(...)). This hook ends up in the DSM socket hook that modifies the IP packet, as shown in Listing 2.

int dsi_socket_sendmsg(struct socket *sock,
                       struct msghdr *msg, int size)
{
...

inode_security_t *isec;
struck sock sk;
struct ip_options *opt =  NULL;
int optlen = NSID_BASE_LEN + NSID_SSID_LEN +
             NSID_NODEID_LEN; //8 +_6 + 6
unsigned char optptr[optlen];

  ...

  sk = sock->sk;
  opt = sk->protinfo.af_inet.opt;
  dsi_options_fill (isec, optptr, optlen);
  dsi_ip_options_get(&opt, optptr, optlen);
  opt = xchg(&sk->protinfo.af_inet.opt, opt);

...
}

The function dsi_options_fill sets up the security information to the buffer as specified in the previous paragraph. Later, in subsequent functions, this security information is attached to the IP message as options. The SID is derived from the socket security ID, and the NID is global for the whole node—there is no need to pass it as a parameter to the function.

After this action, the modified packet with the security information added is forwarded for normal processing in the kernel and finally is sent over the network. At the receiving side, the incoming messages are stored in the sk_buff structures and preprocessed in a series of functions and hooks. One of these functions is ip_options_compile (Listing 3) in /net/ipv4/ip_options.c, where the options are processed.

int
ip_options_compile (struct ip_options *opt,
                    struct sk_buff *skb)
{
unsigned char *pp_ptr;
unsigned char *optptr;

  ...

  case IPOPT_CIPSO:

    if(security_ops->ip_ops->decode_options(skb,
                                    optptr, &pp_ptr)
      goto error;
      break;
  ...
}

For the CIPSO case, the security hook decode_options is called. This hook is replaced by the DSM dsi_decode_options hook, where the security parameters (SID, NID) from the incoming packet are read and stored in the security structure attached to this sk_buff. The sk_buff buffers, populated with the security information, are attached to the receiving socket queue, where they are waiting to be read by the receiving application. In order to read them, the application issues the system call sys_socketcall (), as it did for the sending packet. The call once again goes through the DSM security hook, where the receiving socket security ID is validated against the sk_buff security of the incoming packet. If the socket is not allowed to receive the packets with a given security ID, then those packets are dropped. Listing 4 shows the kernel function in include/net/sock.h.

int
sock_queue_rcv_skb (struct sock *sk,
                    struct sk_buff *skb)
{
int err=0;

  ...

  err=security_ops->socket_ops->sock_rcv_skb (sk,
                                              skb);

  if(err)
    return (err);
  ...
}

As we can see, the security hook sock_rcv_skb is called. This hook then is replaced by the DSM function dsi_sock_rcv_skb when the DSM is loaded. In this function, the security validation is performed. From the example code we can see work needs to be done to manipulate the security labels.

We performed several benchmarking tests in order to verify whether adding options to the IP header affects the overall performance and by how much. One test was to send a UDP packet between nodes of the cluster and measure the performance degradation that consists of the packet security modification on the sending side, including the packet security extracting on the receiving side. The average overhead of adding extra security based on our implementation is 30%. Most of the overhead (around 25%) is related to the IP packet modification based on the IP security option. The remaining overhead (around 5%) is contributed by the security hooks infrastructure in the Linux kernel, such as the socket hooks. As we can see, most of the overhead is related to the IP packet modification based on the IP options, with only a small fraction of the overhead caused by the security hooks infrastructure.

Our future efforts will be directed at improving the IP modification algorithms as we continue to use IP options as the security transport mechanism.

By changing the IP options, we were able to distribute security information to nodes of the cluster with the DSM. We have optimized the IP packet modification and our primary results show significant improvements—the 30% overhead has dropped to 14%. These performance results are promising, and we see more opportunities for further optimizations to attain a lower overhead. Nevertheless, the results demonstrate the challenges facing the development of efficient distributed security. We hope you try out DSI and DSM and send us your feedback.

David Gordon, co-op intern from Sherbrooke University, for his contributions to DSM.