BYOC: Build Your Own Cluster, Part II—Installation

In Part I of this three-part series, we left off with bare-metal hardware assembled, a disk partitioning scheme for the head node and compute nodes, and a design for the network.

In this article, we bootstrap ourselves up to performing fully automated operating system installations on both the head node and compute nodes, a vitally important step for the cluster to be scalable to large numbers of compute nodes. To create the kickstart scripts used in automated installations, we'll perform the installation many times, each time with a larger amount of the process being automated. By the end of this article, we'll have an operating system on all nodes and network connectivity.

We like to start with doing things manually for several reasons. First, it's a great opportunity to make sure that the hardware is set up correctly. Second, a manual installation generates the kickstart file template that will be used in subsequent installations. Although you can find examples of kickstart files online, it's more useful to generate it yourself, because then you can be sure it contains the specifics for your hardware setup.

To perform the installation, you'll need to download the distribution as an ISO file, burn it to a DVD and boot from it. This guide uses CentOS 7, the redistributable version of Red Hat Enterprise Linux. Alternatively, you could burn it to a USB drive; however, Linux currently names hard disks and flash drives in an arbitrary order, changing the sda and sdb labels on different boots. This will become a problem when you start kickstarting installations. There are workarounds, like using the more verbose UUID naming scheme, but to avoid confusion, we're going to assume you use the DVD.

When installing, we assume the disk partitions as described in Part I of this series:

For networking, configure the interface on the external network as your network administrator dictates. As you've probably discovered by now, it's important always to be on your network administrator's good side, since he or she has nearly unlimited power over your internet connectivity. But, you have full control over the internal network of the cluster.

When configuring the network interface on the internal network, set it to use a statically assigned IP address. The address should be selected from one of the private IP address ranges, which are the following:

In this article, we use a subset of the 192.168 range, specifically 192.168.1.100 – 192.168.1.199, which provides 100 IP addresses. The head node should use the first IP address in the range. Therefore, the configuration for the head node on the internal network is:

When you select packages to install, choose Server With GUI, and choose E-mail Server and Development Tools as add-ons. We'll fine-tune this selection later, but this is a good starting point.

To achieve reliability, one needs to have a reproducible installation method that provides consistent results. Luckily, Red Hat's installer, anaconda, has a reliable and scalable method called kickstart. Kickstarting means that the installer uses a configuration file to install Linux automatically. Anaconda generates a kickstart file after every installation, which you can find at /root/anaconda-ks.cfg.

After manually installing the head node, locate this file and copy it as ks.cfg to a separate computer as a backup. You'll be editing ks.cfg over the course of this article if you're following along, and if the only copy resides on the same machine being re-installed, a typo could result in its destruction.

You can edit the kickstart file to include all desired installation options and post-installation configurations. The kickstart file must use UNIX end-of-line characters, so if you're going to edit the file on a Windows machine, use an editor like notepad++ that respects this difference. Otherwise, it's easiest just to edit it on the head node and back it up to some other machine.

Kickstart files have a specific formatting so that they can be parsed by the system installer. A few important features include:

Creating an automated installation is an iterative process in which one modifies the kickstart file, re-installs the system and verifies that the changes took hold as intended. For example, to check that /admin and /home (and any others that you specified) don't get formatted during the install, create a file on the partition and see if it exists after anaconda finishes re-installing. Also check that SELinux is indeed disabled. You will perform a multitude of installations during the creation of the cluster, each time automating and then testing a new capability or feature added to the cluster. Hence, it is vital to automate the process.

You can access the kickstart file for an automated installation in a variety of ways. The kickstart file can be located on an ext2- or fat32-formatted USB drive, or on a partition on the hard drive from where it can be read by the installer.

Table 1 summarizes the different installation methods we use to bootstrap ourselves up to a fully automated installation without removable media. In this table, the boot/installer is the collection of files from which the CentOS installer boots, which currently resides on the DVD. Likewise, the distro is the repository of software that the installer uses to set up the CentOS operating system, also currently located on the DVD.

The end goal is to eliminate the need for removable media and instead to use the hard disk for the entire installation process.

Before starting, this method has one caveat: depending on the hardware configuration of your system, when the installer boots, the USB drive may show up as a different drive from normal. For example, in a configuration with two hard drives, sda and sdb, when you insert a USB drive, it comes up as sdc. However, upon booting into the installer, the USB drive might be sda, while the hard drives are sdb and sdc. This scenario would require the kickstart file to be edited so that all references to any sdX hard drives are shifted one letter.

To install CentOS using a kickstart file on a USB drive, perform the following steps.

1) Copy the kickstart file to a blank ext2- or fat32-formatted USB drive. Make sure the repository line is set to:

cdrom

2) With the USB inserted into the head node, you now can use the kickstart file while booting from the DVD by pressing Tab at the initial welcome screen and appending the following to the boot options:

inst.ks=hd:sdX1:/ks.cfg

Note that sdX corresponds to the USB device when booting with it in place, and that the drive letter X might not be the same as when you inserted it into a running operating system. The installer will allow you to edit this line if it fails to find the kickstart file on the specified device.

By performing an installation this way, you have verified that the kickstart file is formatted properly. The next step is to eliminate the USB drive and reduce your reliance on the DVD.

The first step in migrating away from external media is to move the kickstart and distro to the hard drive, thus eliminating the USB drive and reducing the responsibility of the DVD down to just booting.

To use the kickstart and distro from the hard drive, complete the following steps.

1) Make sure you know the device name of the partition mounted as /admin. You can discover this information with the lsblk command.

2) Create the following folder hierarchy in /admin:

# mkdir -p /admin/iso/centos7/
# mkdir -p /admin/ks/headnode/

3) Copy the installation media to iso/centos7. If you have the original ISO file floating around, you simply can copy it across the network. If not, you can use the dd command to create it from the DVD:

# dd if=/dev/cdrom of=/admin/iso/centos7/CentOS-7-x86_64-DVD-1511.iso

4) Copy the kickstart file to the newly created ks/headnode folder. In the ks.cfg file, comment out the repository line and replace it with the following:

harddrive --partition=sdXY --dir=/iso/centos7/

Note that sdXY is the partition in which /admin is located. You can discover this using the lsblk command. Also, while you have ks.cfg open, verify that it isn't set to format /admin. You've just made some fairly significant additions to that partition, and it would be a bummer to wipe them all out.

5) Insert the DVD and reboot. Press Tab at the initial welcome screen and append the following options:

inst.ks=hd:sdXY:/ks/headnode/ks.cfg

The installation now should proceed using both ks.cfg and the ISO from the hard drive. This makes the installation go much faster, since it doesn't have to read everything from the DVD.

The next step in eliminating external media is to set up the /admin partition to be the installer.

Currently, the only role the DVD serves is to boot the installer. In this final step, you'll transfer that role to the hard drive, eliminating the need for all external media. To do so, you'll configure grub to boot directly into the installer, pass it the boot options for locating the kickstart file, and install CentOS completely automatically, without any external media or typing inst.ks=<location> options into the terminal.

To run the installer from the hard drive, do the following steps.

1) Make a directory to house boot files:

# mkdir /admin/boot/

2) Mount the iso and copy the internal files to the boot directory:

# mount -o loop /admin/iso/centos7/CentOS-7-x86_64-DVD-1511.iso /mnt
# cp -a /mnt/* /admin/boot/

3) At the bottom of /etc/grub.d/40_custom, insert the following:

menuentry "Install" {
    set root=(hdW,msdosZ)
    linux /boot/images/pxeboot/vmlinuz ks=hd:sdXY:/ks/headnode/ks.cfg
    initrd /boot/images/pxeboot/initrd.img
}

Note the (hdW,msdosZ) and ks=hd:sdXY lines. These correspond to the drive and partition for /admin. Use the lsblk command to find the partition in sdXY format. W then corresponds to the drive number; sda is 0, sdb is 1 and so forth. Z is the partition number /admin is on. So, if /admin is located on sda2, use (hd0,msdos2).

4) Regenerate grub.cfg:

# grub2-mkconfig -o /boot/grub2/grub.cfg

5) At the end of the ks.cfg file, between %post and %end, insert the following:

#grub configuration
cp -p /boot/grub/grub.conf /boot/grub/grub.conf.000
cat >> /etc/grub.d/40_custom << EOF
menuentry "Install" {
    set root=(hdW,msdosZ)
    linux /boot/images/pxeboot/vmlinuz ks=hd:sdXY:/ks/headnode/ks.cfg
    initrd /boot/images/pxeboot/initrd.img
}
EOF
grub2-mkconfig -o /boot/grub2/grub.cfg

This script now will modify your grub.cfg file automatically, as done in steps 3 and 4. Make sure to modify the (hdW,msdosZ) and sdXY lines as shown in step 3. The method used here for writing this text to /etc/grub.d/40_custom is called a here file. As opposed to using the echo command, here files have far fewer characters that must be escaped, although some characters still need to be—for example, every $ or ` symbol must be escaped with \$ or \`. Otherwise, bash will attempt to resolve it as a variable or executable script.

6) Reboot. At grub's splash screen, arrow down to the entry titled “Install” and press Enter. The system should install. Now the system installs automatically without external media. This is a very useful ability to have when rebuilding the head node, and it will be essential for building compute nodes!

Before installing the compute nodes, Dynamic Host Configuration Protocol (DHCP) must be configured on the head node. DHCP allows the compute nodes to receive their network configuration from a central server. This step is vital for a scalable system, because it allows the nodes to be configured identically but have unique IP numbers.

To configure DHCP on the head node, complete these steps:

1) Install DHCP:

# yum install dhcp

2) Add the following to /etc/dhcp/dhcpd.conf:

#dhcpd config options
authoritative;
default-lease-time -1;
option broadcast-address 192.168.1.255;
ddns-update-style none;
next-server 192.168.1.100;
filename "pxelinux.0";

subnet 192.168.1.0 netmask 255.255.255.0 {
    range 192.168.1.101 192.168.1.199;
    option subnet-mask 255.255.255.0;
    option domain-name-servers 8.8.8.8;
    option routers 192.168.1.100;
}

Change the range option to include enough addresses for all of your nodes. Note that some options may need to be adjusted if the head node's internal IP is not 192.168.1.100. The domain-name-servers option may point to whatever DNS you desire; in this example we use Google's at 8.8.8.8, although some network administrators may require you to use their own.

3) To start the DHCP service on the head node at boot time, execute the command:

# systemctl enable dhcpd

And to start the service immediately, execute the command:

# systemctl start dhcpd

4) To automate this process, in ks.cfg, append dhcp to the %packages list, and between the %post and %end tags, add the following:

#dhcp
cp -p /etc/dhcp/dhcpd.conf /etc/dhcp/dhcpd.conf.000
cat > /etc/dhcp/dhcpd.conf << EOF
	[Contents of dhcpd.conf as determined in step 2 go here]
EOF
systemctl enable dhcpd

5) To use DHCP, the firewall must allow it. Since you want all communications on the internal network to go unobstructed, you simply can add the interface on that network to firewalld's trusted zone:

# firewall-cmd --permanent --zone=trusted 
 ↪--change-interface=[INTERNAL INTERFACE]
# echo "ZONE=trusted" >> 
 ↪/etc/sysconfig/network-scripts/ifcfg-[INTERNAL INTERFACE]
# nmcli con reload
# firewall-cmd --reload

Use the ip addr command to discover [INTERNAL INTERFACE]. Add these changes to the kickstart using the following lines:

firewall-offline-cmd --zone=trusted 
 ↪--change-interface=[INTERNAL INTERFACE]
echo "ZONE=trusted" >> 
 ↪/etc/sysconfig/network-scripts/ifcfg-[INTERNAL INTERFACE]

Note: we'll include a more thorough explanation on firewalld in Part III of this series.

Now that the head node is installed fully automatically, and support for a basic network is in place, you can proceed to the compute nodes.

On a single compute node, boot from the installation DVD and perform a manual installation.

Compute nodes have a different partition table, network setup and package selection from the head node. The example partitioning scheme we used in Part 1 of this series is:

When configuring networking for the compute node, connect using a dynamic IP address. If DHCP is set up on the head node correctly, you should receive an address.

For now, make the package selection a Minimal installation.

The generated kickstart file on the newly installed compute node is located at /root/anaconda-ks.cfg and should be transferred to the head node. In /admin, create a directory to house the compute node kickstart file:

# mkdir /admin/ks/computenode/

Copy it over the network from the compute node to the head node:

# scp 192.168.1.101:/root/anaconda-ks.cfg /admin/ks/computenode/ks.cfg

substituting the node's actual IP address as determined by running the ip addr command on it.

Automated installations are incredibly important for compute nodes in a cluster. While manual installations may be practical (though still undesirable because of reliability) for small test clusters, they scale poorly and are impractical for medium to large clusters.

For the compute nodes, the process of automating the installation will be similar to that of the head node. In the kickstart file, be sure to disable SELinux as you did for the head node, and also disable the firewall because the compute nodes won't be connected to the outside world anyway:

selinux --disabled
firewall --disabled

Like with the head node, you'll transfer responsibilities away from the DVD. This time, rather than the final destination for installation files being the compute node itself, it will be the head node accessed over the network. In the final configuration shown in Table 2, the compute nodes will boot over Pre-boot eXecution Environment (PXE), and they'll retrieve their kickstart files and distributions via a Network File System (NFS).

The procedure here is the same as under the head node. If you are still learning about kickstart files, feel free to follow the DVD-Based Booting with Kickstart on USB instructions under the head node section. But otherwise, save yourself time and use NFS as described in the next section.

NFS allows compute nodes to access files stored on the head node over the internal network. This is a necessary step for the scalability of the cluster, since it would be impractical to have a copy of the kickstart and distro locally on each node at installation time.

In this section, we assume that you've configured the directory structure on the head node as follows:

If your setup differs, modify the following commands accordingly.

To install over NFS, do the following steps.

1) On the head node, open the /etc/exports file in a text editor and add the following:

/admin 192.168.1.100/255.255.255.0(ro,sync,no_root_squash)

This gives all computers on the 192.168.1.0/24 subnet read-only (ro) access to the /admin directory.

2) Restart the NFS service so that this change takes effect. If NFS wasn't already running, this will start it:

# systemctl restart nfs

3) In the compute node ks.cfg file (located on the head node), comment out the repository line and replace it with:

nfs --server=192.168.1.100 --dir=/admin/boot

This tells the installer to look for the installation files on the network rather than on a DVD.

4) On a compute node, determine the name of the network interface that connects to the internal network using the ip addr command. The interface name could be formatted in one of several different ways depending on your motherboard, such as ethX, enoX, enpXsY, or if you're unlucky, en<mac address>.

5) Now you can use the kickstart file while booting from the DVD by pressing Tab at the initial welcome screen and appending the following to the boot options:

ks=nfs:192.168.1.100:/admin/ks/computenode/ks.cfg ksdevice=ethX

substituting ethX for the interface name found in step 4.

6) To make the changes on the head node persistent between installs, add the following to the end of the ks.cfg file for the head node between the %post and %end tags:

#nfs
echo "/admin 192.168.1.100/255.255.255.0(rw,sync,no_root_squash)" >> 
 ↪/etc/exports
systemctl enable nfs

The compute nodes now are capable of retrieving kickstarts and installation files over the network, but that is only half the story. To make the installation proceed without any media, the nodes must boot over the network as well.

Pre-boot eXecution Environment (PXE), called MBA on some BIOSes, allows nodes to retrieve their boot media via the network. Here are some basic prerequisites before using PXE:

PXE allows you to perform kickstart installations on the nodes without having to load any disks physically. It's then possible to start an automated installation merely by powering on the cluster. Think about it: throw a switch, take a lunch break, and when you get back, the entire cluster is installed. That's scalability!

To make this claim a reality and install the compute nodes from the head node, do the following.

1) On the head node, install syslinux, tftp-server and tftp using yum. Add these to the Packages section of the kickstart file for documentation and re-installation purposes.

2) On the head node, make and populate the directory /admin/tftpboot:

# mkdir /admin/tftpboot
# cp /usr/share/syslinux/pxelinux.0 /admin/tftpboot/
# cp /usr/share/syslinux/menu.c32 /admin/tftpboot/
# mkdir -p /admin/tftpboot/images/centos7/

3) Copy in a compressed kernel and initial ramdisk from which the compute nodes can boot:

# cp /admin/boot/images/pxeboot/vmlinuz 
 ↪/admin/tftpboot/images/centos7/
# cp /admin/boot/images/pxeboot/initrd.img 
 ↪/admin/tftpboot/images/centos7/

Those two files are necessary for booting a bare-bones Linux system. vmlinuz is a compressed Linux kernel, and initrd.img is a temporary root filesystem. When you boot a compute node over PXE, those two files will be transferred to the compute node, giving it the software required to access the kickstart file and the rest of the installation files over NFS.

4) Create a directory to hold the PXE configuration files:

# mkdir /admin/tftpboot/pxelinux.cfg

5) Create the new /admin/tftpboot/pxelinux.cfg/default file containing the following:

DEFAULT menu.c32
PROMPT 0
TIMEOUT 100
ONTIMEOUT kickstart
MENU TITLE PXE Menu
MENU separator
LABEL local
LOCALBOOT 0
MENU separator
LABEL   kickstart
        kernel images/centos7/vmlinuz
        append initrd=images/centos7/initrd.img
         ↪ks=nfs:192.168.1.100:/admin/ks/computenode/ks.cfg 
         ↪ksdevice=ethX

Modify the ksdevice as needed. Notice that when the menu times out (ONTIMEOUT), it defaults to the kickstart option. This is useful for installing the cluster without manual intervention. But this must be changed later to local for the cluster to reboot without re-installing the OS.

6) Edit the /usr/lib/systemd/system/tftp.service file, changing the line:

ExecStart/usr/sbin/in.tftpd -s /var/lib/tftpboot

to:

ExecStart/usr/sbin/in.tftpd -s /admin/tftpboot

and restart the service so that this change takes effect. This change could be added to the head node's kickstart file using a here file, but it's more concise to use sed:

# sed -i.000 's|/var/lib/tftpboot|/admin/tftpboot|'
# /usr/lib/systemd/system/tftp.service

In this command, the -i.000 flag specifies that sed is to do the modification in place—that is, it will perform the change and write it back to the original file. The .000 part makes it so that sed will save the original file with a .000 extension as a backup. The next component, in single quotes, specifies the operation that sed is to perform. The s tells it to do a substitution (as opposed to a deletion or insertion), the pipe (|) serves as a delimiter between parts of the command, and the two strings are the parts to exchange. Finally, the file path at the end of the command specifies the file that sed operates on.

7) Reboot a compute node—or all of them. If everything is set up correctly, they will install automatically.

8) Change ONTIMEOUT kickstart to ONTIMEOUT local. Otherwise, every reboot will result in a re-install.

An explanation on the inner workings of PXE booting is in order. When pxelinux.0 boots on a machine, it tftps back to the boot server and tries to find a configuration file in the pxelinux.cfg directory. The filename is determined by converting the IP address given to it by the DHCP server to hexadecimal. For example:

192	168	1	101
C0	A8	01	65	-> C0A80165

pxelinux.0 attempts to find files in pxelinux.cfg in the following order:

This PXE feature is useful for supplying specific kickstart files for different sets of compute nodes because of hardware differences; for example, it can supply different partitioning schemes for different hard disk sizes. Right now DHCP on the head node is configured to dole out IP addresses in an arbitrary order, making it hard to take advantage of this feature. (In the next article, we'll fix that.)

When booting a compute node over the network, vmlinuz (the compressed kernel) and initrd.img (the compressed initial filesystem) are transferred back to the compute node, along with the boot options.

In the case with the kickstart option, the boot options tell the installer (included in initrd.img) the location from which to retrieve ks.cfg, which in turn includes the location of the distro.

PXE also is useful for booting other images, such as Memtest and other diagnostic tools.

In this article, we covered the basics of kickstart files on CentOS, and we set up a scalable method for installing the entire cluster. The resulting system is capable of intercommunication over ssh as root, but it doesn't have any useful cluster-wide application software or users on it yet.

In the final article, we'll address the Linux services that are vital for cluster operation, culminating on a resource manager called SLURM. With this software in place, the cluster will be fully fledged and ready for its end users.