Vagrant

How many times you have been hit by unit tests failing because of environment differences between you and other team members? How easy is it to build your project and have it ready for development? Vagrant provides a method for creating repeatable development environments across a range of operating systems for solving these problems. It is a thin layer that sits on top of existing technologies that allows people working on a project to reproduce development environments with a single command: vagrant up.

Vagrant is open-source software, and it sees lots of contributions from its community in every release. It was first released in 2010. Its founder Mitchell Hashimoto founded the company HashiCorp to work on Vagrant and a number of other DevOps tools in November 2012. HashiCorp has since released a number of other tools that are worth checking out, such as Packer, Consul and Serf. These are for different areas of the development lifecycle.

Vagrant uses virtual machines from various providers. It allows for easy configuration using a Vagrantfile. This allows development teams to set up the virtual machine the same way every time in a programmable fashion. This repeatability encourages developers to tear down and bring up environments more often. The development environment that all team members are developing against now can be “stored” in the version control system in such a way that anyone (new or otherwise) can have an environment set up after issuing a single command.

Continuous integration systems can use the same method. Building an environment and setting up a project in this way allows the developer to be sure that once other environments are provisioned in the same way, the project will run as it did in development. Environment-specific unit test failures is a class of problems completely removed by Vagrant.

Vagrant takes the building and provisioning of your software from tribal knowledge within your team to a reproducible provisioning script that is stored alongside your code. Others can read your Vagrantfile and provisioning scripts and have your project working on their system in minutes. This helps integrate new developers into your team quickly. It is easier to enable cross-team development within a larger company as it's easier to get up and running with a project. It also allows designers or technical writers to get quick access to your project without having to know any technical details about how to set up your project.

Originally, Vagrant supported only VirtualBox as a provider. A provider is the underlying virtualization technology that Vagrant will interact with. At the time of this writing, Vagrant supports a wide variety of providers, VMware, hyper-v and, most recently, Docker. Vagrant only ships with VirtualBox support by default. Other providers can be added as plugins using Vagrant's plugin system. For the purpose of this article, I use VirtualBox as the provider, as it is the default and is likely the most accessible to everyone. It is not the most performant provider; VMware in most cases out-performs it, and Docker (although it is not a drop-in replacement) is a very exciting technology in terms of speed and definitely is worth checking out.

The next configurable part of Vagrant is provisioner support. After a virtual machine has been started, Vagrant uses “provisioners” to provision the box in a repeatable way. This is done as part of bringing the virtual machine up in most cases. The simplest type of provisioner is “shell”, which runs a defined shell script at boot of the VM as the root user. More advanced provisioners, such as Salt, Chef and Puppet, also are available. In my workplace, we use Puppet to provision our virtual machines so as to better match our development systems with machines running in the production environments.

Depending on the provider, Vagrant also allows for “Synced Folders”; this allows very easy sharing of folders between the host and guest machine. This means every developer on your team can use a different IDE and tooling that runs on the host, and yet develop on the same guest machine. Vagrant makes these things incredibly easy to set up as you will see later in the article.

Vagrant has installers for all of the main operating systems: Windows, Linux (deb and rpm) and Mac OS X. This portability across host operating systems allows developers in a team to get the same local development experience on any environment.

For this article, I used Ubuntu 12.04 LTS, and I used the deb file from the downloads section of the Vagrant home page. For Linux distributions that don't accept deb or rpm package types, there is still the ability to run older versions of Vagrant, provided that the distribution has support for Ruby and Ruby gems, but this has been discontinued for newer releases of Vagrant.

If you have not already installed VirtualBox, you will need to install it. This can be done through the package manager, although I used the packages provided on the Oracle Web site.

This article uses Vagrant 1.6 and VirtualBox 4.3.

In this article, I run through setting up a project using Vagrant, including how to write the Vagrantfile, how to boot the virtual machine using Vagrant and interacting with that VM. The first thing you need is the base box on which you are going to build your project. A base box is a provider-specific bare-bones operating system disk image with some basic things installed. Vagrant uses SSH to connect to all its boxes, so an SSH server must be enabled, and a specific user must have SSH enabled. There are a number of other requirements to making a Vagrant-compatible box. The process is too involved for this article, but more specific instructions can be found on the Vagrant or specific providers' Web sites. Vagrant, by default, expects a vagrant user on the box and the password (by convention) should be “vagrant”.

For the purpose of this article, you will use a publicly available base box. HashiCorp provide a base box for Ubuntu 12.04 LTS. To add a box to Vagrant:

$ vagrant box add "hashicorp/precise32"

This command downloads the box from Vagrant cloud. Vagrant cloud is a service HashiCorp provides, which among other things offers box discovery. Community base boxes are uploaded and can be used. Vagrant knows that you are accessing a base box from the Vagrant cloud and will download it directly from there. Once downloaded, you will not need to download the base box again to use in other Vagrant projects on the same host. You can see available boxes by using the list command:

$ vagrant box list
hashicorp/precise32 (virtualbox, 1.0.0)

Now that the box has been added, you will craft a basic Vagrantfile. A Vagrantfile is a Ruby file that sets up certain variables for your box. Here's a basic Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/precise32"
end

This shows the power of Vagrant—with this simple file, an Ubuntu 12.04 LTS virtual guest machine can be created and started on any system Vagrant supports. To start the Vagrant VM, simply do the following:

$ vagrant up

It may take some time for the virtual machine to boot, but when the command returns to the prompt, you successfully have started your virtual machine. At this time, it might be worth opening the VirtualBox application so you can see the new virtual machine that has been created and ensure that it is marked as running.

Figure 1. VirtualBox Screenshot

As I mentioned previously, base boxes for use with Vagrant must be accessible via SSH. This allows you to attach to your new virtual machine simply by doing the following in the same directory as the Vagrantfile:

$ vagrant ssh

You will be left in a bash prompt in the virtual machine. In two commands, you have booted the VM and ssh'd in to the VM. One more thing to note before continuing on to using Vagrant more powerfully is that Vagrant, by default, shares the directory that the Vagrantfile is located in under the /vagrant folder. To demo this capability, while still in the SSH session, do the following:

Guest machine:

$  cd /vagrant
$ echo "test" > test

Host machine:

$ ls
test  Vagrantfile

As should be clear, you created a new file “test” on the guest, and the file appears in the same directory on the host. Vagrant uses VirtualBox shared folders to allow you to keep files in sync between the host and the guest very easily. This is especially useful when developing software projects using Vagrant when you want to be able to edit files locally and have those changes automatically appear in your development VM for testing purposes. It allows developers to run their choice of editor while developing on the same platform as anybody else in the team.

You've seen how to boot a simple virtual machine using Vagrant, so next, let's look at how to provision that VM in a repeatable way. As already mentioned, Vagrant supports a wide range of provisioners, but for the sake of this article, bash scripts will be used due to their familiarity. Adding a provisioning step is a matter of adding a single line inside the configure block in the Vagrantfile. The bootstrap also can be an inline script, but for most cases, it makes sense to supply a script file that contains the provisioning steps.

Add the following bootstrap.sh script to the same directory as the Vagrantfile:

#!/usr/bin/env bash
# provisioning is run as root by default

# make sure system sources are up to date.
apt-get update

# install a web server
echo "==== Installing NGINX ====" 
apt-get install -y nginx

echo "==== Start the NGINX Server ===="
service nginx start

This bootstrap script makes sure the system sources are up to date first. It then installs the nginx Web server. The provisioning is added to the Vagrantfile by adding the following line to the configure block so that your Vagrantfile now looks like this:

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/precise32"
  config.vm.provision :shell, :path => "bootstrap.sh"
end

To use the new provisioning, destroy the running VM. This will stop and remove the virtual machine. To destroy the VM, simply use the command vagrant destroy. The base box still will be available for use after running the destroy command. Now to test the new configuration, simply run vagrant up. When a provisioning step is present in the Vagrantfile, Vagrant automatically will provision the box once it has booted. You can turn off the provisioning by using the --no-provision flag.

When vagrant up is complete, you will be able to ssh in to the machine as before. As shown below, let's make sure that the provisioning has been successful. Note that nginx is now installed on the system and running on the system:

$ vagrant ssh
$ which nginx
/usr/sbin/nginx
$ netstat -at | grep http
tcp        0      0 *:http      *:*       LISTEN

The virtual machine has been provisioned, and nginx is set up and started. The Web server can be tested to be working, but it's more difficult if you have to ssh in to the box to test this. Next, let's look at how to configure networking.

The first option to access ports on your virtual machine is port forwarding. For this example, nginx serves http traffic on port 80. To forward the guest machine port 80 to port 8080 on the host, add the following line to the Vagrantfile:

config.vm.network "forwarded_port", guest: 80, host: 8080

As before, destroy the VM and do another vagrant up. Once the virtual machine has been brought up and provisioned, the Web server can be accessed on the host port 8080. On your favorite browser, visit http://localhost:8080/ to be greeted with the nginx home page. This is one of the easier ways to be able to connect to the virtual machine.

For example, let's modify the bootstrap.sh to link the “html” directory in the Vagrant directory to the nginx root directory. This would be an example of doing simple static HTML Web site development. This could be distributed to anybody to modify the HTML on the host and immediately see the results on the guest.

The revised bootstrap looks like this:

#!/usr/bin/env bash
# provisioning is run as root by default

# make sure system sources are up to date.
apt-get update
# install a web server
echo "==== Installing NGINX ====" 
apt-get install -y nginx

echo "==== Start the NGINX Server ===="
service nginx start

echo "==== Symlink Test Directory ===="
ln -s /vagrant/html/ /usr/share/nginx/www/html

You can access the html directory through the nginx Web server using the forwarded port once the VM has been booted and provisioned by Vagrant. For example, using curl:

$ curl "http://localhost:8080/html/tester.html"

This example assumes an html directory is in your Vagrant directory that has a tester.html file in it.

What if you want to provide a way to access the virtual machine as if it were another machine on the network? Vagrant provides a very easy way to use VirtualBox networking to set up the virtual machine with a static IP on the same LAN. Remove the port forwarding line from the Vagrantfile and replace it with the following:

config.vm.network "private_network", ip: "192.168.56.20"

Once you have re-ran vagrant up, your virtual machine will be accessible from your host using the IP address specified above. In my own projects, I use this private networking to assign my static IP a domain by modifying the /etc/hosts file. This allows testing of complicated http redirect flows that rely on certain domains.

It also allows multiple Vagrant virtual machines to contact one another, as it is a shared network between the host and all other virtual machines that are on the network. For distributed projects, being able to control the networking access of multiple standalone components all of which have their own Vagrantfile (or shared) is very powerful.

Again, in my work environment, it's not uncommon to spin up multiple virtual machines, one for a database, one for an app server and one for a squid proxy. Running functional tests against these tests a production-like setup. Vagrant allows you to do this in a simple, easy-to-repeat way, and the private network means there is no hassle in setting up intercommunication between the various components.

The last part of Vagrant that I want to cover is having multiple machines in the one Vagrantfile. I already have given an example of why you might want to have multiple Vagrantfiles for different components, so that they can be tested/developed on their own or as a collective. Having multiple machines in the one Vagrantfile is powerful when a single project can have multiple parts that don't belong on the one machine. Often in a Web development project, you will run the app server and database on the same machine for testing but will have multiple app servers and databases when that system gets into the production environment. This is the cause of a lot of production bugs due to untested component configuration.

Vagrant allows you to manage this all through a single Vagrantfile. If the idea of multiple machines is one that interests you, I also recommend looking closely at the new Docker container option for Vagrant. Docker is built on top of Linux containers and allows you to spin up machines much faster than VirtualBox. Multiple machine support in Vagrant is all the more powerful if the provider level is much faster to spin up machines.

For multiple machine support, the Vagrantfile allows you to configure multiple machines, each having their own configuration. The Vagrantfile below shows an example of what a MySQL master and slave Vagrantfile might look like:

Vagrant.configure("2") do |config|
  config.vm.box = "hashicorp/precise32"

  config.vm.define "master" do |master|
      master.vm.provision :shell, :path => "master_bootstrap.sh"
      master.vm.network "private_network", ip: "192.168.56.20"
  end
  config.vm.define "slave" do |slave|
      slave.vm.provision :shell, :path => "slave_bootstrap.sh"
      slave.vm.network "private_network", ip: "192.168.56.21"
  end
end

Each machine is defined with the first config block as separate blocks. They support all of the same options as the normal one-machine Vagrant config blocks. For example, for the master here, you define the shell provisioning script to be run as well as the static IP address to give this virtual machine. This allows you to test distributed systems in the same easy repeatable way as you could with a single machine.

When using a multiple machine configuration like the above, there are a few changes to the way you have to run Vagrant commands. Running vagrant up will bring up both boxes automatically. The master will be provisioned first and then the slave. You can choose to bring up only one of the machines by supplying the machine name:

$ vagrant up master

When the boxes have booted and been provisioned, you also have to specify the machine name when using vagrant ssh.

Let's reflect on what you have achieved here: multiple virtual machines booting, being provisioned automatically and consistent IP addresses using one command—vagrant up. This is repeatable across the three major operating systems and a number of different providers including the most popular virtualization technologies and cloud offerings.

This removes so many of the pain points from working on systems in local development. There are a lot of advantages to making the switch to using Vagrant for both local development and your continuous integration systems. All developers' test machines are running the same code and are provisioned in the same fashion. Test failures are not tied to a specific developer's setup and should be reproducable on anyone's machine by running vagrant up. This article covered only some of the amazing capabilities of Vagrant, so I encourage you also to read over the brilliant documentation at docs.vagrantup.com.