Open Source Software for Real-Time Solutions

Much hype is circulating in the industry today about open-source software and Linux. These discussions have focused on how Linux represents the first major threat to Microsoft's domination of the desktop. However, open-source software is also moving into the embedded real-time marketplace. Last year, Cygnus Solutions unveiled a new initiative called eCos (embedded Cygnus operating system). eCos is available as both open source and royalty free. Today, eCos has been downloaded by well over 10,000 developers. Why didn't Cygnus choose Linux as its open-source real-time alternative? This article will compare and contrast these two alternatives and explain why each can have a place in your real-time solution.

To see why Cygnus developed its own real-time operating system (RTOS), you must look at what is on the market and the type of job Cygnus wanted to take on with its RTOS.

Cygnus was founded in 1989, effectively pioneering the open source business model. While Linux is the poster child for open source today, Linus Torvalds was in fact inspired by earlier open-source projects, especially the GNU C compiler (gcc). The Free Software Foundation's (FSF) Richard Stallman first released the GNU C compiler in mid-1987, and Michael Tiemann, then a research scientist, quickly adopted it. Tiemann's contributions to the GNU C compiler, which included ports to several RISC and CISC microprocessors, a complete native-code C++ front-end, and optimization passes for instruction scheduling and branch scheduling, created more demand for code development and support than he (or any of the other gcc hackers) could provide in their spare time. Tiemann decided that this demand was not a fluke, but a result of deeply seated economic principles, and founded Cygnus to prove it. (See Michael Tiemann, “Future of Cygnus Solutions” in Open Sources: Voices from the Open Source Revolution, O'Reilly, 1999.)

Cygnus was in the real-time business almost from the beginning. When the GNU C compiler acquired a reputation for runtime speed and compact code, it came to the attention of real-time developers. Exercising the freedoms uniquely available from the open-source bazaar, these savvy developers asked for, and contributed back, further enhancements to the compiler, making it an even better choice. Cygnus, their customers and the Net community were seeing the power of the bazaar in action.

Early on, Cygnus identified configuration tools as one of the key technologies for making it possible to support many microprocessors from a single source base. They created the “configure” script (one half of the famous “configure; make” process for building FSF and Linux applications and utilities), opening the doors to a true multiplatform solution. Many companies that depend on embedded processor projects move rapidly from processor to processor to gain new capabilities or to get the same capabilities in a smaller package. As a result, the GNU C compiler supports some 50 processors today, and 125 host-target combinations are available for cross compilation. In two years, Cygnus may see a 90% rollover in the processors it will support with gcc. They may write 24 new processor ports in a year, a tremendous output for a company that doesn't even own the product it is supporting.

Meanwhile, Cygnus assiduously sought out its customers' wants and wish lists. They had to understand why they were getting—or losing—sales. By 1995, they knew their customers needed standardized runtime support that integrated completely with the GNU gcc compiler, gdb debugger and tools on the host. The runtime support needed to include a C library, hardware abstraction layer (HAL), debugger support and a wide range of RTOS functionality. As with gcc, portability was a key ingredient, as it had to be available for a wide range of architectures and evaluation boards.

In moving into the RTOS business, Cygnus had to consider its own strengths and weaknesses. To judge by the plethora of proprietary closed-source RTOSs (over 120 in 1995), anyone can make a buck with a proprietary RTOS. At least, that's what many people would like to think. Actually, an RTOS, like a restaurant, is a truly good way to go broke fast, unless you know exactly what you are doing.

What Cygnus needed was an open-source RTOS. Many products actually come quite close. Typically, the user has access to the source but the vendor owns it, and you don't see the source until after you buy the product. Part of the purchase is a non-disclosure agreement, which prevents you from trading improvements with other users of the same product. These constraints are anathema to the open-source model.

A number of Linux variants offer real-time capabilities, all of which are open source. Probably the best-known is Victor Yodaiken's RTLinux. More are listed at the Linux Embedded web site. Any RTOS must be highly configurable, because embedded projects run on very different hardware, almost always custom hardware. If you think PCs vary greatly, you haven't looked at embedded projects.

To give an example, Linux might support an infinite number of some resource—say, mutexes or timers—or it might supply a fixed but very large number of them. It must provide that flexibility, because the designers have no idea which applications the user will run. That flexibility has a cost: the resources, such as memory, to support it, or the code to spawn a new one. An embedded project designer knows exactly how many of something the project will need, so he can select that number before compiling. The embedded project trades runtime flexibility for compile-time flexibility and gains simplicity and reliability. Thus, in addition to open source, Cygnus needed an easily configured RTOS.

Offering real-time capabilities isn't the same as offering a full-blown RTOS. To see why, let's look at real-time programming and projects.

Dean Koester, Cygnus' Director of Product Marketing, sees a whole spectrum of real-time applications. They vary throughout that spectrum according to their requirements. We will look at Koester's spectrum by examining the two ends, and see why RTLinux and eCos fall where they do on the spectrum. While we are at it, we will look at the economic implications of some of those requirements.

The term “real time” has become a buzz word, as in “real time stock quotes”. If the NYSE ticker is running an hour late, your quotes aren't in real time and that's that. Real time, according to Koester, means getting the job done on time, every time. Not some of the time, or most of the time, but every time. This means your primary concern is not average response time, but worst-case response time.

Take a simple example: a processor that controls a servo based on one analog-to-digital converter's input. The processor reads the A/D converter and adjusts the servo accordingly. Painting pretty pictures based on the data or shuffling it off to a log file are secondary.

Linux, as Linus Torvalds and his team of kernel hackers provide it, simply is not a real-time operating system, nor was it ever intended to be one. It is designed to provide the best average response time. As Linux is loaded down, it gracefully degrades the performance of all tasks. This is not acceptable for real-time computing. The data acquisition and control functions must respond on time, every time. They cannot degrade, gracefully or otherwise.

The RTLinux response to this is to write a minimal RTOS, then run Linux as a background task under the minimal RTOS. As long as the real-time functions are getting done, and there are resources to spare, Linux is permitted to run whatever tasks are assigned to it. The typical RTLinux application runs one or two real-time tasks. They collect data and stuff it into FIFOs. All processing is done in the background, using Linux and all the neat programming tools that come with it.

This makes RTLinux suitable for what we might call “desktop real time”: data collection where overhead such as several hundred megabytes of Linux are acceptable, even when you don't use them in the application. It means applications where a typical desktop computer, with its keyboard, video display (even running X) and other extraneous processes are acceptable overhead. The extreme here is a student project which collects a few streams of analog data and uses them to control some device.

While RTLinux is excellent for desktop real time, it is by no means restricted to the desktop. Applications using RTLinux running on stripped-down PC hardware can be embedded into products.

RTLinux also offers very fast development for real-time programming. The data collection driver is a Linux module, which means you can remove it, recompile it and re-insert it without rebooting. All of the post-collection data messaging is done with the tried-and-true tools we all know and love, such as Perl and Ghostscript.

Also, RTLinux offers very inexpensive R&D economics. If you have a desktop computer with Linux on it, you already have most of the capital cost of implementing a minimal RTLinux project. You can spend money for custom PC-compatible hardware if volume and real estate are constraints. Many RTLinux projects will do this.

However, Cygnus wasn't after “desktop real time”. The term “embedded” originally meant putting a microprocessor into some product in such a way that it wasn't obvious a computer was even there. A classic case is a photocopier. Even a simple home photocopier has some microcontroller running it. But it doesn't look like a computer: there is no QWERTY keyboard, no monitor, no GUI. The key to an embedded application is that absolutely no inessential overhead is acceptable.

There are two classic embedded project paradigms. The first is a mass production product, where tens of thousands or millions of pieces will be manufactured. Spending hundreds of thousands of dollars up front to shave ten cents off a single unit is economical because of the production run involved. An automobile brake controller is an archetypal application.

The other classic embedded project is where the customer is the European Space Agency (ESA) or the Department of Defense, and spending vast piles of money to shave a gram of weight is considered economical. For example, the Jet Propulsion Lab's Mars Pathfinder or Voyager spacecraft fit this paradigm.

Cygnus decided it was simpler to start from scratch and build an RTOS for embedded applications than it would be to strip unnecessary components from Linux to produce an RTOS suitable for embedded projects. eCos was designed to scale down to just a few kilobytes of code if all you require is a HAL and C library. An attempt to strip Linux that far down would produce something you would not recognize as Linux.

An RTOS suitable for embedding in applications where lives are at stake, such as automobile brakes or pacemakers, must be provably reliable. An application where high production runs are planned must also be highly reliable, because the costs of replacement are much higher than the costs of making sure the product is reliable.

A key component of reliability is simplicity: the simpler an RTOS is, the faster you can prove its reliability because there are fewer interactions to test. So the time-to-market you may gain in fast prototyping with RTLinux, you may lose proving that your product is reliable.

If an application is to run on batteries or any other limited-power environment (solar panels, radio-isotope powered generators, etc.), a host of considerations come into play. The obvious one is that the hardware must be able to run with limited power. In general, the more powerful a processor is, the higher the power consumption. The more overhead imposed by an RTOS, the more powerful the processor must be in order to do its job. That translates into shorter battery life, which translates into annoyed customers and lost sales.

Most battery-powered devices have extreme size and weight constraints. The cell phone makes a good example. If your “full-featured” RTOS has a large ROM or RAM footprint, you will pay for that footprint not only in memory cost, but in battery life as well. Similarly, if your RTOS is not efficient and requires more cycles than another RTOS to do the same job, then even if it meets your real-time constraints, it may still blow your power budget.

Not so obvious are the implications of power failures. For example, if a user has just punched a new phone number into her cell phone and the battery dies, she will be annoyed if the new number wasn't saved. You cannot afford file corruption caused by power failure. This means the designer has a tight window between the time the last digit is pressed and when the data is saved to permanent storage.

Linux, with its lazy writing to non-volatile storage and the multi-user overhead in its file systems, was not deemed acceptable by Cygnus. A battery-powered application requires an RTOS designed from the ground up for speed, efficiency and utter reliability.

One thing that drives almost all real-time projects is time to market (even student projects are driven by time to grades). Cygnus offers two features that will appeal to the embedded processor engineer.

First is ease of configuration. Having learned from the GNU compiler, Cygnus provides compile-time configuration tools that allow the user to select the components to include or exclude. In addition, the components themselves are configurable, and you have full access to the source for each component and can customize it to suit your application.

eCos achieves globally the sort of modularity that Linux achieves with its drivers. Linux was driven to modules by management issues and to minimize interfaces. eCos was driven to greater modularity by the customers' desire for configurability. As it happens, both get portability as a byproduct, and eCos will gain from the decentralized management made possible by its modularity. That decentralization is essential to the success of an open-source product; something RTOS vendors who make it difficult to access their source have missed. (See Linus Torvalds, “The Linux Edge”, also in Open Sources: Voices from the Open Source Revolution, O'Reilly, 1999.)

eCos is currently configured with scripts on Linux and by a GUI program for Microsoft Windows.

Figure 1.

Some RTOSes, designed to target the consumer market, do offer the capability of adding an application framework like a Java virtual machine. Again, open source and the bazaar come to the rescue: third-party virtual machines have been ported to eCos, which allow dynamic addition and removal of applications.

The other feature Cygnus offers is its customer support. When time to market is a big factor, fast, expert support is essential and worth the cost. Cygnus' experience supporting gcc will come in handy here.

This is not to say that you must buy Cygnus' support to use eCos. Cygnus offers an e-mail list for eCos users, for which it charges nothing. It also publishes the source on the World Wide Web. If you want to support yourself by looking through the code, you can do that, too.

Koester contends that eCos and RTLinux don't really compete. The market for RTLinux is desktop real-time and other high-end applications where overhead is less of a factor or not a factor at all. The market for Cygnus' eCos is the embedded processor market, where overhead is a major factor.

Both eCos and RTLinux have their places, and, says Koester, can coexist. For many real-time applications, RTLinux is overkill at the expense of power, price, and product performance. eCos can be driven to the higher end of the real-time spectrum, at the cost of losing the standard desktop applications that make RTLinux so attractive.

While very different in the markets they are intended to serve, Linux and eCos are in fact similar in fundamental concept. Each is a monolithic kernel. Both use modularity, and both are portable because both embody some abstraction of the hardware layer.

Ultimately, says Koester, it is consumer requirements which drive RTOS requirements. Consumer requirements are greatly varied, and so there is room for any number of open source RTOSes.

Resources