Project Utopia

For the better part of a decade, Linux enthusiasts have waxed poetic on the inherent greatness and looming success of Desktop Linux. Our kernel is so robust! Our applications are infinitely configurable! This is the year of Desktop Linux! Who would ever use Microsoft Windows? These claims and similar—particularly when made back in the 20th century—seem, in retrospect, so trite. Not that I stand righteous—oh no, I laid the praise down as thick as anyone else did. I too was a convert.

At least, I did until I realized that hardware support in Linux was awful. Like a deck of cards, my rosy view of Desktop Linux came crashing down, making an 180 degree turn from glowing to ghastly. Heartbroken, I cried myself to sleep every night and went on an inexplicable diet consisting only of cheese and pudding.

But this did not last long. One day, the Linux community decided to do something about it. We outlined a plan not only to reach feature parity with the other desktop operating systems, but also to surpass them. Hardware support has come a long way in the last year and a half. This is the story of just how far.

The steps for installing a new hardware peripheral on a Mac might go a bit like this:

Most of us would not even include these two items as steps. The first is a physical necessity; the second is the original and ultimate goal. Lost, somewhere between steps one and two, are 39 other steps, right? Kernel modules? Configuration files? Rebooting? Extensive mastery of sed and awk?

At some point in Linux's history, support for new hardware could easily require compiling a new kernel module, becoming root, editing configuration files, loading said module, checking dmesg, cursing, removing the module, unplugging the hardware, plugging the hardware back in, reloading the module and so on.

Forgotten, perhaps clouded by a love for free software and the invigoration of do it yourself, is the notion that stuff should just work. As fun as writing my own kernel module might be—and I use the term fun here loosely—sometimes I just want to plug in my camera, get my photos and be done with it.

In late 2003, the Linux system was well primed for the emergence of a new architecture for managing hardware on the desktop. The 2.6 Linux kernel was out and rapidly gaining adoption. It brought, among numerous other new features and improvements, a new mechanism for handling device drivers, called the device model. The device model allowed, for the first time, the kernel to build an in-memory tree of the devices it supported. For example, both my mouse and my keyboard are connected to my USB hub, which is connected to my third USB port, which is on my first PCI bus. Such a rich hierarchy provides all sorts of opportunities to the kernel. One of the most promising, however, was sysfs.

sysfs exports this device hierarchy as a filesystem. One directory lists all the buses on a system. For each bus, another directory lists all of the devices on a given bus. Files for a given device could link to the associated module. Walking the sysfs tree, therefore, would allow user space to build a comprehensive picture of the system's physical device hierarchy, exactly as the kernel sees it.

That alone is incredibly useful. But another kernel feature, hotplug, broadened the horizon even more. The kernel's hotplug infrastructure notifies user space whenever a device is added to or removed from the system. This allows applications to become aware of changes to sysfs in real time. It also allowed for the creation of udev.

udev is a user-space implementation of devfs—an automated and dynamic manager of device nodes. Instead of a /dev created once, statically, udev updates /dev on the fly, in response to the exact hardware available to the system. More important, however, is that udev places intimate knowledge of devices and their device nodes in user space. Hotplug, sysfs and udev together allow user space a complete view of the system's hardware.

Now user space needed to capitalize on the opportunity.

This was 2003. That summer, I attended a BOF at the Ottawa Linux Symposium on improving the Linux desktop by Robert Sanford Havoc Pennington. In the BOF, Havoc referenced a whitepaper of his entitled “Making Hardware Just Work”, in which he unveiled a utopian view of hardware management on the Linux desktop. Intrigued, I took notes—see Figure 1.

Figure 1. Early Project Utopia Notes

We ended up speaking to the group on this utopia and discussing possible implementations. The BOF ended without much traction from the audience, but Havoc and I had a firm understanding of the situation and potential solutions. Other responsibilities kept me from immediately acting on my crude sketches, and so they sat idle on the pages of my notebook.

Two things happened that lifted the pages to life without my immediate realization.

First, David Zeuthen, then living in Copenhagen, decided to bring Havoc's documents to life by beginning the HAL Project. HAL, originally hardware abstraction layer but now not an abstraction of anything whatsoever, is a system-level dæmon that ties together hotplug, sysfs and udev in order to provide a Linux system with a single, comprehensive view of hardware, accessible via a standardized interface. HAL makes it possible for an application to say, “give me the device nodes of all input devices” or to ask, “is there a camera connected to this computer?” With HAL, what was once a hundred lines of hacks, operating on hard-coded device nodes with intimate knowledge of Linux internals, is now a single, elegant HAL request. David's HAL, in effect, brought a 21st century hardware infrastructure to Linux.

The second disruptive event came in December of the same year, when I accepted a job with Ximian, recently acquired by Novell, as a kernel hacker dedicated to the desktop. My first mission was to figure out the hardware situation. I teamed up with a colleague, the inimitable Joey Shaw, an Ohio native, and we sat down and hashed out our utopian view of hardware management.

Both Joey and I recognized the strong foundation that the 2.6 Linux kernel, sysfs, hotplug, udev and now HAL supplied. We concluded that the missing pieces were the layers on top of HAL. We had a rich infrastructure in place; we just had to do something with it.

HAL uses a then-nascent but always-promising project called D-BUS as its communications mechanism. On one side, D-BUS is a run-of-the-mill interprocess communication (IPC) system—like CORBA, but a lot easier to use. On the other side, however, D-BUS introduces the concept of the system-wide message bus. In addition to per-user process-to-process communication, D-BUS allows components in a Linux system to send out signals, announcing events or providing information to all who care to listen. Signals can announce when a network connection is obtained or when the laptop battery is running low. Interested applications higher up the stack can listen for these signals and, upon receipt, react.

Our plan was literally to flood the system with D-BUS signals. Have HAL and other lower-level components of the Linux system generate numerous useful signals and have higher-level components respond, evolve and react. In effect, make the Linux system much more dynamic and, ultimately, make hardware just work.

Joey and I decided to create an umbrella project—a meta-project. The plan was to spur development of HAL-aware applications that can provide hardware policy on the desktop. Never should a user need to configure hardware. It should happen automatically in response to the user plugging the hardware in. Never should the user (or even the programmer) have to mess with device nodes and esoteric settings. HAL should provide all of that, on the fly, to the applications. Never should the user have to guess how to use new hardware. If I plug in a camera, my photo application should run. If I insert a DVD, it should start playing. All of this should happen magically, automatically and cleanly.

I coined the name Project Utopia. It was, after all, a bit utopian.

We did not have a central Web site or source repository or cute logo. Project Utopia was a cause and a way of thinking. We had a goal and a set of use cases and a growing disgust toward things not working. We blogged and spoke at conferences and wrote code. One by one, piece by piece, we started to build a set of policy pieces on top of HAL, guided by the following rules:

I began writing GNOME Volume Manager in late December 2003. It was originally a proof of concept—a test bed for my ideas. I wanted to see how feasible hardware management on top of HAL could be. The plan was to respond to events such as “new hardware” or “audio CD inserted” with specific actions. GNOME Volume Manager is nothing but a simple finite state machine, receiving hardware-related events on one end and replying with hardware-induced actions on the other. The tricky part was to do it all with HAL: no polling, no hacks.

GNOME Volume Manager implemented the Project Utopia policy related to block devices. When the user inserted an audio CD, GNOME Volume Manager would play it. When the user inserted a USB keychain device, GNOME Volume Manager would mount it and open a Nautilus window. When the user plugged in a camera, GNOME Volume Manager would ask if it should automatically import the photos into the user's photo management application (Figures 2 and 3). A recently added feature even found GNOME Volume Manager managing iPods!

Figure 2. GNOME Volume Manager Prompting on Discovery of New Photos

Figure 3. F-Spot, a Photo Management Application

The next step was bringing HAL support to more applications, a process Joey and I call halification. The following months witnessed additional policy pieces, such as automatic printer configuration and seamless network management (Figure 4).

Figure 4. NetworkManager's Network-Switching Applet

For printers, Joey wrote a HAL back end for CUPS, the Common UNIX Printing System, allowing CUPS to query HAL on the availability of printers. The result: plug in a printer and configure it automatically, on the spot.

The ambitious NetworkManager Project, started by hackers at Red Hat, aimed to solve networking woes. Seth Nickell, an early designer on the project, described the intended use case as an electrical outlet: “you plug it in and [it's] on.” For example, plug a laptop in to a docking station, and it instantly switches to the station's Ethernet. Walk into your favorite coffee shop and instantly begin using the wireless networking. NetworkManager made networking simple, automatically choosing the optimal solution for networking connectivity.

NetworkManager's architecture is two-part. First, a root-level dæmon sits alongside HAL, responding to HAL events and communicating with the system's networking hardware. Second, one or more user-level components implement policy and provide a user interface. Together, the components provide a complete solution for networking. Figure 5 is a diagram of the architecture.

Figure 5. NetworkManager Architecture

Today, the Project Utopia mindset continues to foster new applications, interesting hacks and fresh projects aimed at making hardware just work. Linux distributions from Novell, Red Hat and others sport powerful HAL-based infrastructures. The GNOME Project is integrating HAL and D-BUS across the board. The Project Utopia cause is spreading beyond GNOME too, as other platforms implement HAL-based solutions in a similar vein.

Linux development has never stood still, however. Like a rabid cheetah, development sprints forward toward better, faster, simpler solutions. Support for new hardware continues to roll in, and solutions in the spirit of Project Utopia are continually implemented to provide a seamless user experience.

Cute hacks such as having your music player mute when your Bluetooth-enabled cell phone receives a call are not a dream but the reality in which we live. What cute hacks will tomorrow bring? What new hardware will we support next? What application will be halified next? Join in and answer those questions yourself!

Resources for this article: /article/8459.