Drupal Distributions: Working the Linux Model Up the Stack

In the eight years since Linux Journal switched over to Drupal (that fateful Halloween morning in 2004), the Drupal project has undergone explosive growth, maturing from a small but powerful content management system into an enterprise-class development framework powering some of the largest sites on the Web. This process has been driven by a highly collaborative ecology later augmented by the ability to bundle up completed sites and pass them around for use and further development.

In this article, I look at the big picture of this Drupal evolution and where it's going (so that LJ readers can better anticipate what's coming) with an emphasis on how the underlying architecture is giving rise to a distribution-based development ecology (in many ways reminiscent of the heyday of Linux distros) and explain how to leverage existing Drupal distributions as well as build your own.

Drupal's evolutionary arc, in many ways, recapitulates the history of Linux development, with the adoption of more automated build tools and package management systems giving rise to a victory garden of different distributions. Drupal distributions are based on installation profiles, which date back to around the time LJ switched over its site, and preceded the existence of Drupal shell tools, such as drush and drush make.

Around the time of Drupal 4.5 (the version Linux Journal used when it launched its Drupal Web site), the need was recognized for installation profiles as a way to focus “the highly configurable, but largely baffling initial experience” with Drupal into something that could be used and adopted easily by both individuals and groups. Back in the day when the Drupal learning curve was more like a learning brick wall, the idea was to make Drupal make sense to first-time users and to allow it to be used to meet their needs without all the unnecessary complication.

CivicSpace was one of the very first “distributions”, although it wasn't necessarily a distribution in the specific sense in which the term is used in Drupal today. Aimed primarily at political campaigns, it was the one that started it all, demonstrating how the Drupal CMS could be used to build an easily expanded, complex distributed network of sites that all communicated with each other. It even included integration with CiviCRM, which was considered pretty advanced for the time.

It wasn't a distribution in the contemporary sense but more correctly an installation profile. Back then, both terms had different meanings from what they have today (as the process has been greatly improved), but all along there has been the goal of facilitating a way to share Drupal site recipes and complete installations easily.

In those days, an installation profile was more like a formula than a completed thing. It included a list of sub-components that would be needed in addition to Drupal core (which wasn't itself even included), such as modules, themes and a profile file that contained scripts required to execute predefined functions and configure the completed site.

Building an installation profile required considerable knowledge of Drupal APIs and a fair amount of time and dedication. In addition to merely having to assemble manually the list of the modules that were needed for the particular installation, you had to write your own custom installation scripts. These scripts invoked PHP functions that could be fired in place of Drupal's default installer and were needed to manage the additional configuration at install time. Since this was before CTools and the Features module allowed you to define exportables, there was no way to prepackage the configuration; you had to do it by writing custom scripts that built the configuration in real time, essentially “compiling” a working site, but from executable code, not configuration flags.

The final step of using this install profile to generate a finished site—although not involving code writing—was also a bit tedious. The installation profile wasn't a complete distribution, but included a list of everything you would need to get to put it together. You then had to download Drupal core separately, as well as the .profile itself, and then individually download each and every one of the modules. Imagine if you had to manage dependent libraries on a Linux server this way; it's not surprising that it never took off as quickly as some had hoped. Then, you still had to run the installation profile scripts to stitch it all together, and, if all went well, you'd have an up-and-running custom-configured Drupal site.

The introduction of drush or drupal shell (drupal.org/project/drush), a command-line shell and scripting tool, completely changed all that and more. At the very least, it made possible far-reaching changes in Drupal development practices. By providing a set of tools that allowed nearly every aspect of Drupal development to be done completely from within a shell environment, it created a foundation for further advancements and techniques—one that borrowed heavily from approaches to Linux development.

Drush is a tool that lives outside the Drupal installation. Installing it on a Linux system is as easy as curling it to the right directory. Drush also can be installed via Debian package management (packages.debian.org/sid/drush) or by using its custom PEAR channel:

$ pear channel-discover pear.drush.org
$ pear install drush/drush

Drush can interact with nearly every aspect of the Drupal API and has rapidly grown in popularity, expanding to be able to do more and more using succinct and direct invocations and arguments. Using drush to download every module needed for a given profile meant speeding things up a little, but the giant leap forward was made possible by the release of drush make—essentially a build script that accepts a structured manifest with instructions for “compiling” a specific working Drupal installation. (Of course, it isn't strictly compiled in the technical sense, but the metaphor holds.) This makefile (or “dot makefile”) is a text-based configuration file using Drupal's .info format (derived from PHP's .ini format) that's essentially a list of all the required/needed subcomponents. This file describes all the components needed to assemble a full Drupal installation profile, where to get them, how to get them and even what patches need to be applied.

Just a few short months after the first version of drush make was released in mid-2009, it was used to create a Drupal distribution called Managing News, which prior to that had been a downloadable package, but was not based on any universal package management framework. With the adoption of drush, it became the first install profile to include a makefile. This was somewhat of a watershed moment for Drupal, as it was the first Drupal installation profile that automated the build process, instead of requiring you to go fetch all the components yourself. After this, the genie was out of the bottle. Drush make allowed installation profiles to be assembled and served up as a single downloadable tarball, instead of requiring the user to go fetch all the pieces individually from different locations and put them all together.

The next important milestone was Pressflow, which wasn't technically a distribution, or even an profile, but a retooling of Drupal 6 to make its HTTP headers compatible with the Varnish reverse proxy server. Pressflow was important in that it wasn't just a collection of Drupal features aimed at a specific interest group (in fact, it included no bundled modules at all) but was purely a technical distribution that allowed you to use Drupal to do something that wasn't strictly possible with vanilla core: running it on a reverse proxy server to be able to scale site performance. This advancement ushered in a whole new era of highly performant Drupal sites able to serve up content at Web scale, and was later included by default in Drupal 7.

Pressflow also diverged in the sense of being a completely packaged Drupal install containing all components, as opposed to a manifest that you had to build with drush, anticipating the Drupal distribution era. A full distribution doesn't need anything to be fetched over the network but includes all components in a single bundle, vastly simplifying the process of circulating finished Drupal sites with complex sophisticated functionality.

In December 2009, drupal.org updated its infrastructure to integrate the drush make tool as a packaging system (in large part due to the work of Derek Wright). Up until that point, there were no distributions on the main site, but only installation profiles. The integration of drush make turned drupal.org into an on-line package management system that automated the process of turning a Drupal site manifest into a finished Drupal tarball ready for download.

Internationalism blazed the trail, providing one of the first true distributions to be made available there, with the stated goal of making it easy just to install a language-localized version of Drupal with no manual intervention or additional configuration whatsoever. Since then, the Internationalism distro (aka “i10n”) has continued to surf the edge; the latest version can be found at drupal.org/project/l10n_install.

Note: one important distinction between distributions and installation profiles (in the older sense of the term) is that not all distributions can be made directly available on drupal.org, because there is no third-party code hosted there. Those distributions bundling third-party code will have to be content with having only their manifests there and/or making a full tarball (or zip file) available elsewhere.

It's also important to note that the terms “installation profile” and “distribution” have morphed in meaning as the development process has become more defined. Originally, an installation profile was a list of dependencies the user had to retrieve manually with a script or set of scripts that ran ancillary to the default installer script (install.php), which provided a number of places for them to hook into or override the standard behavior. Nowadays, an install profile is thought of as a specifically formatted set of manifests that are executed by the Drupal 7 installer that can optionally invoke ancillary scripts, which may be less necessary for distros that are mainly bundled features.

Thus, an install profile technically has essentially the same meaning it did originally; however, the way it's used has changed as the packaging system and drupal.org infrastructure has evolved. Install profiles come packaged with distributions in the sense that every distribution has at least one install profile, but an install profile by itself is not a distribution. Drupal 7 is the first version to ship with more than one installation profile from which to choose.

Properly speaking, a distribution is a full copy of Drupal packaged together with the additional modules, themes and scripts that give you everything you need to produce a complete and complex Drupal Web site in a single downloadable file. It's a ready-made tarball (or zip) that's completely self-contained.

The now outmoded method of assembling all these components manually was a tedious process and one that required you to know a good bit about Drupal's terminology and file structure, not to mention having to implement all the required function calls yourself. The process was repetitive and predictable, which made it a great target for automation. In addition to the main benefit of automation and ease, distributions can provide:

Drupal distributions improve the factory model when you need to able to spin up or deliver Drupal solutions quickly and efficiently, and like the original installation profiles that gave rise to them, they are especially effective at meeting a widespread use case (just as they are much more efficient at meeting it). Essentially, distributions finally realize the vision of install profiles both in terms of vastly improving the Drupal experience and facilitating an ecosystem now heavily populated with hundreds of distros of every stripe and color, such as the Spark distro, which brings in-place editing to WYSIWYG content management; Open Atrium, an extremely popular groupware/intranet (the Whitehouse uses it as an internal collaboration tool); DataPublic, for creating open data portals; a paste-bin distro; a port of Joomla called GLORilla; and the official TedX Web site, just to mention a few of the hundreds of freely available distros, from educational to “corporative”, from scientific to religious, all of which can be downloaded directly from drupal.org/project/distributions.

Note, however, that Drupal doesn't have a defined way to separate upstream and local configuration and settings. There's no “/usr/local”, and apart from workflow-based solutions, it's a missing piece that arguably would be of great benefit to making Drupal distributions even better.

Distributions are essentially finished installation profiles. The basic components of a distribution are Drupal core plus any of the following additional components:

Just to be clear, a distribution is an install profile, packaged with Drupal core and any required contrib modules. The install profile that comes with a distribution may in fact have additional scripts that run at installation time in its .profile file. Although distributions and installation profiles seem very straightforward once you understand how they work, you may find the terminology seemingly inconsistent, both in the official documentation and elsewhere. Building your own Drupal distribution, however, and showcasing it on Drupal, isn't too complicated. The following is a short guide on how to roll your own drupal distro and submit it.

The first step is, of course, to make an awesome Drupal site that other people would be interested in downloading and using. You can use an existing distribution as your starting point, building on the work of others. For example, Panopoly (drupal.org/project/panopoly) is an Apps-enabled distribution of Drupal designed to be both a general foundation for site building and a base framework upon which to build other Drupal distributions. After you have a completed site, you're ready to package it up for sharing. The site from which you're making a distribution could be a clean install of the architecture you've built (meaning it could have no created content yet), or it could be a full site with content that's been created by multiple users. The distribution build process will pack up only the modules, not any of the configuration, content or users. To include configuration, you'll need to write a .profile with bundled scripts (or capture it all in Features). To provide default content or even users “out of the box”, you'll have to use the Profiler module (drupal.org/project/profiler).

The second step is to generate the makefile. This file is a manifest specifying the version of Drupal core that is needed to build the site (if none is given, the latest stable release will be used) along with a list of all contributed and custom modules. Contrib modules optionally may be given a version number, and custom modules generally will need to have a path to a repository to download them from, such as GitHub. (You also can use a repository other than drupal.org for contrib modules, such as running your own local repository to speed up package retrieval.) The easiest way to generate the basic site manifest is to use:

$ drush make-generate my-site-name.make

which will auto-generate the drush makefile based on your current site configuration. For any information it can't find (the path to a custom repository, for example, which isn't stored anywhere else in configuration), it will insert placeholders reminding you to fill this in manually. And, of course, if you're using alternate or dev versions of any contrib modules, they will be included too. Inspect the file it created for you and make sure any placeholders are replaced with corrected paths. (Covering the exact syntax for each of these entries is outside the scope of this article.)

Note that drush make handles only modules and libraries (downloadable code) and not the configuration stored in the database (including content types and so on) for which you'll need a manually built install .profile or bundled “features” (using the Features module).

The next step is to execute your makefile to prepare your distribution using the prepare-install command:

$ drush make --prepare-install --tar my-site-name.make my-site-name

(Note: this step can take several minutes depending on the size of your site.)

At the completion of this step, you will have a finished Drupal distribution that anyone can use to spin up an instance of the kind of Drupal site you've built, without downloading anything additional over the Internet. You could offer it for download at the site of your choosing, or you could put it on a site specifically dedicated to showing off your distribution and making it available for download.

You even could add your custom distro to the growing number available for download directly from the drupal.org Web site. However, to do this, you won't submit a full package made with the prepare-install command, but instead generate the makefile only:

$ drush make --generate-makefile drupal-org.make

(Note: at the time of this writing, you do have to make one manual edit to this file. See drupal.org/node/642116 for instructions.)

Then, verify that your makefile is ready using:

$ drush verify-makefile

Finally, you'll want to execute the makefile locally (just to confirm it's working) and then commit the manifest only (that is, the drush makefile, not the fully packaged distribution) to the distribution's repository using Git, the version control system used on drupal.org (and created to manage the Linux kernel). To release a distro on drupal.org, you will need the ability to create full projects there, which involves a one-time approval process. This approval is not instantaneous, so if you don't have it, don't plan for an “immediate” release of your distro.

After you've successfully created and tested your distribution, and uploaded the make manifest, the drupal.org packaging scripts will handle all the rest for you.

For full details on submitting your distro to drupal.org, see drupal.org/node/642116.