Supporting Multiple Kernel Versions

I work in the Atlas experiment at CERN. Many of the groups in Atlas are beginning to turn to Linux as the operating system for the next generation of particle-physics experiments. Among the teams working on the data-acquisition, there is often a need for specific versions of the Linux kernel. Typically, the teams are using special PC cards, such as ATM cards, where the drivers may not yet be available for all kernel versions, or for which patches must first be applied to the kernel. Both SMP and uniprocessor machines are used, and team members want the same kernel with the same patches for both flavors. They also wish to share software and meaningfully compare results.

This article describes how my team supports this need. I will assume you are familiar with the basic process of configuring and compiling a kernel.

We needed an environment in which a range of kernels could be configured, built, packaged for distribution and later installed in a coherent and consistent manner. The result is our “kernel repository”, containing tar files of the source code for several kernel versions with different patches applied, tar files of the compiled kernels, and a set of scripts used to compile and install the kernels. We wanted to ensure that the configure and build steps were fully recorded and any kernel configuration could be reproduced at a later date, even if the details of the build environment had changed. Also, we wanted to keep up with newer kernel versions (at the time, 2.2.0 was about to be released), so we tried to make it easy to add new versions as they came out. We also wanted the distributions to be easy to install, so that people could use them without knowing the details.

Our kernel repository and all its associated tools are accessible on the WWW at www.cern.ch/Atlas/project/kernels/www/kernels.html. The kernels are available as separate distributions of precompiled binaries and source code. A technical note is included, which goes into greater detail on some points and helped form the basis of this article.

The original kernel sources were downloaded from the Web. Each kernel unpacks into a single subdirectory—/linux. Since users may want several kernel source trees available at the same time, we rename this directory to /linux-kernel_version_number.orig and repack the tree using tar and bzip2. To give a clear example, if I had downloaded the tar file for version 2.2.0 of the kernel, I would repack it with these commands:

cat linux-2.2.0.tar.bz2 | bzip2 -d | tar xf -
mv linux linux-2.2.0.orig
tar cf - linux-2.2.0.orig | bzip2 >linux-2.2.0.orig.tar.bz2

The kernel repository includes scripts which will fetch and repack kernels for you.

Whenever any patches were applied, the corresponding source tree was renamed to reflect the patch and sometimes the version of the patch. For example, 2.0.36 with the “bigphysarea” patch is packed as linux-2.0.36.bphys.tar.bz2.

To configure the kernels, I wrote an Expect script called KernelConfig.exp. Expect is a tool for automating interactive processes (see “Automating Tasks with Expect” by Vinnie Saladino, Linux Journal, October 1998), and it is ideally suited to this task. KernelConfig.exp runs make config at the top of the kernel source tree and answers the questions for you. The beauty of controlling the configuration by an Expect script is that it is insensitive to the kernel version used with it. This script should be able to configure any Linux kernel version. I have run it on all stable kernels from 2.0.33 to 2.2.7 and on a number of the 2.1-series kernels. While it may not give an optimal configuration for any kernel (whatever that might mean), it does provide consistent and reproducible configurations.

Some configuration options are hardwired in the script, such as:

Some hardware support is plainly excluded, even though it is available as modules, because at least one of the kernels in the range we are using does not compile. Such problems are normally caught and fixed rapidly, but if the software is not important to us, I leave it disabled for consistency. This happens for a few sound cards, the infrared devices (broken in 2.2.6) and at least one ATM card we don't use. This list may grow as more kernels are produced, so you should look at the configuration log or the script to see what it does. For all other options, the script will select to compile the code as a module if possible; otherwise, it will use the default offered by the configuration. The output of the configuration is stored in the KernelConfig.log file in the current directory. If this file already exists, the output will be appended to it, not written over existing information.

Two words of caution are necessary. The script reacts to the defaults and will therefore react differently if the default changes. For some code, the default changes over time. Something available only as built into or excluded from the kernel in the 2.0.x series might be available as a module in the 2.2.x series. In this case, it will be built in or excluded from the 2.0.x series, according to the default, but will certainly be built as a module for the 2.2.x series. If you need that feature at boot time, be warned.

Secondly, the defaults are taken from the .config file if it exists, or from the file arch/i386/defconfig if the sources have never been configured. If you configure the kernel by hand and set some options before running KernelConfig.exp, it will accept those settings as default. For truly consistent results, run make mrproper before running KernelConfig.exp.

SMP support is tricky. In the 2.0.x series, SMP support had to be enabled by editing the Makefile or by building the kernel with the command make SMP=1 bzImage or a similar one. In the 2.2.x series, SMP support is a configuration-time option, and the Makefile no longer needs to be changed. KernelConfig.exp enables or disables SMP support, depending on the value of a user-defined environment variable SMP_SUPPORT. If this variable is not defined or is empty, the script will not enable SMP support. If it is non-empty, the script will enable SMP support in the 2.2.x series.

This is not enough for the 2.0.x series, where the value of the make-macro SMP must be true when the kernel is compiled, not when it is configured. I get around this by defining SMP_SUPPORT to have the value SMP=1. I can then run KernelConfig to configure the kernel, and make $SMP_SUPPORT bzImage afterwards. For the 2.0.x series kernels, the value of SMP_SUPPORT ensures that the kernel is built with SMP enabled at compilation time. For the 2.2.x series, the very fact that the variable is defined causes KernelConfig.exp to enable SMP support at configuration time. This gives a consistent approach to SMP for both the 2.0.x and 2.2.x series kernels.

I use a Bash script called KernelBuild.sh to compile the kernels and produce the binary distributions. It takes one argument, the name of a kernel source file (without the “.tar.bz2” extension—e.g., ./KernelBuild.sh linux-2.0.36.bphys). It starts by defining a few environment variables:

The MY_* environment variables may be defined externally if you wish, and will not be overridden by the script. KERNEL_VERSION will always be set from the input argument.

KernelBuild.sh does not actually do the work of compiling the kernels. For this, it uses two other scripts, Meanwhile.pl and KernelBuild.cmds. Meanwhile.pl is a Perl script which will execute a Bash script in the background, log all the output and send an e-mail message when it is done.

The real workhorse is KernelBuild.cmds, which can be executed as a stand-alone script, although normally you would use KernelBuild.sh. It unpacks the source code tree, uses KernelConfig.exp to configure it, compiles the uniprocessor version of the kernel, packs it into a binary distribution file, packs the header files into a header-file-distribution, then repeats the process for the SMP version.

KernelConfig.exp determines how a kernel is to be configured, but KernelBuild.cmds determines how it is to be built and installed. The boundaries between the two are a bit blurred because of the way in which SMP support has changed from the 2.0.x series to the 2.2.x series, as mentioned earlier. If you wish to customise the build, it is these two scripts that you will want to change.

KernelBuild.cmds makes use of the fact that anything stored in a file called .name at the top level of the kernel source will be incorporated into the kernel name and can be retrieved later using the commands uname -v or cat /proc/version. I use this to record the kernel version string, including the distinction between uniprocessor and SMP versions. For the 2.2.x series, the Makefile includes this distinction, but for the 2.0.x series it does not. KernelBuild.cmds uses a bit of sed, smoke and mirrors to smooth out the differences.

Finally, the binary and header file distributions are stored in the /dist directory, packed using tar, and compressed with bzip2. The binary distribution contains the kernel image, the System.map and all modules. It also contains a copy of KernelConfig.exp, so in the likely event that this script is updated, you will still have access to the exact version used to compile any particular distribution of the kernel. For the same reason, the log file of the configuration is also packed in the distribution. When the distribution is installed, these will find their way into the directory /log/kernel-version.

The kernels can be installed using the InstallKernel.pl script. InstallKernel.pl takes the full name of the kernel distribution file as input, with the “.tar.bz2” extensions. First, it checks that the distribution will not overwrite any existing file—if so, it aborts execution unless you specifically tell it to go ahead. It installs the kernel and its modules, and adds an entry to /etc/lilo.conf for this kernel. It is quite careful about how it does this. It creates a backup copy of /etc/lilo.conf, then scans it line by line until it finds a root= entry. It uses this to set the root for the new kernel. If it finds a later root= entry that specifies a different root partition, it will warn you, but will continue, using the first one it found. It will not add an entry if it finds an existing entry for this kernel image. The last thing it does is show you the differences between the saved lilo.conf and the one it has just created. InstallKernel.pl will not run LILO for you—you must do that yourself.

Another script, InstallHeaders.pl, will take care of installing the header files for you. The headers are installed as subdirectories of /usr/src/linux-headers. If you set the link /usr/src/linux to point to one of these installed sets of header files, you can compile your driver or program for a version of the kernel different from the one you are actually running. I make use of this to compile the ARLA AFS clone for all the kernels I support, without rebooting my machine.

Whichever distribution of Linux you are using, you will probably have to modify the way it decides which set of kernel modules to use. The details vary from distribution to distribution, so it is not possible to describe all the necessary changes here.

Since these kernels rely heavily on the use of modules, you may also need to create an initial RAM disk for your specific machine. This is certainly true if you have a SCSI-based system. See the man page for the mkinitrd command for details.

In order to clone the repository to build your own kernels, copy the contents of the /bin and /source directories, and modify them as you wish. KernelBuild.sh will need modifying in order to set the MY_* variables correctly. KernelConfig.exp may also need modifying to enable or disable any specific options—this may not be a trivial task. KernelBuild.cmds will need to be modified if you wish to actually change the way the kernels are built. The other scripts should never need to be altered.

At present, about 30 kernel source distributions are included in the repository, representing kernels from 2.0.34 to 2.0.36 and 2.2.0 to 2.2.7 with various patches. As the person who manages the machines running these different kernels, I find that this standardization has simplified my tasks considerably.