An Interview with Red Hat's Michael Tiemann

Red Hat Chief Technical Officer Michael Tiemann is well known for his extensive and important work on g++ and gdb. A founding member and former CTO of Cygnus Support, Michael joined Red Hat when they purchased Cygnus. He now serves as Red Hat's CTO.

LJ: Tell us about Red Hat and Embedded Systems.

Michael: Let me start first by talking about one of the keynotes I've been giving this year. It is based on a book called Guns, Germs, and Steel by Jared Diamond, and I don't know if you've heard of this book. It won the Pulitzer Prize in 1998. He is a sort of an anthropologist and linguist and biologist all wrapped into one. He uses his extensive array of scientific techniques to answer interesting questions like, “Why did people from Spain set sail to the New World and conquer large civilizations instead of the other way around?”

With respect to embedded systems the question that I'm thinking the most about today is, “Will open source change embedded, or the other way around?” The reason that this is an interesting question is because the dynamics that open source enjoys-readily downloadable software, a large developer community sharing software which can easily be downloaded, installed and modified-is quite different than what we think of in the traditional embedded systems world where things like ROM are where programs are stored. The interesting question will be, “what extent are the constraints of the embedded systems world going to change the open-source approach to programming?” There's a lot of interesting stuff going on in that direction. Or, the other way around: is open source actually going to change the way people design embedded systems. I could give you a couple of examples of both.

One is eCos, the Embedded Configurable Operating System. eCos was designed to fit the constraints, whatever they were, of embedded systems quite efficiently. What we provide with eCos is a configuration technology which enables source-level configuration. What this means is that instead of developers needing to touch actual source code to make it conform to their world, they can use a tool to control over 200 different configuration points. The consequence is that you can deliver a real-time operating system whose memory footprint ranges in size depending on the needs and the assumptions of the application.

There are some applications which have already implemented their own cooperative threading model and have already implemented other resource allocation routines similar to those provided by today's conventional real-time operating systems. In the case where the application does all the work and they simply need a hardware abstraction layer, eCos can deliver that with less than 1,000 bytes of memory footprint. But if you want to turn on scheduling as a function of the kernel, or you want to turn on management of memory, or you want to turn on the existence of a file system, et cetera, et cetera, et cetera, you can do all of this incrementally and largely independently, and scale that up to typically 100 or 200 kilobytes at the high end.

We have customers who have not been able to configure conventional proprietary real-time OSes smaller than about 55 kilobytes. In eCos in this exact customer situation, where the customer's design requirements were 24K of memory footprint to fit in the on-chip ROM, eCos came in at 11K by utilizing its source-level configurability.

So, in the case where the requirements of embedded systems are changing the shape of open source, we have built shape-changing technology for eCos.

LJ: Is that shape-changing technology actually part of the base open-source release of eCos?

Michael: Yes it is. We announced the open-source availability of CDL, the Configuration Description Language, I think last month.

Now, putting the shoe on the other foot, and looking at how open source is changing embedded, I have seen a tremendous shift in interest from traditional proprietary real-time operating systems, to using embedded Linux. In fact, I can tell you that up until September of last year, the customers that I talked to tended to profile the exact same way that IDC profiled the entire market. In other words, X percent were interested in proprietary RT/OS “A”, and Y percent were interested in proprietary RT/OS “B”e and my own statistical sampling of the market when I talked to customers corresponded very accurately to what IDC was reporting in their vendor numbers.

Since the announcement of commercially available embedded Linux was made by a number of companies in September of last year, I have spoken with exactly zero people who are designing new embedded devices using proprietary operating systems.

LJ: That's quite a sea change.

Michael: That is. That's not to say that people are not buying these operating systems for legacy applications. I continue to meet with a lot of people who complain about how difficult it is to change. But in terms of new designs, it has been like night and day.

LJ: Does that include WinCE and PalmOS?

Michael: I have not talked to anybody, ever, who has been interested in WinCE. I don't know about Palm.

I would say that if you're simply looking to replace a proprietary RT/OS with embedded Linux, just to change the RT/OS in the socket, that's not very interesting. What's really interesting here is if you recognize not only the development community that comes with Linux, but also the extensibility and the openness of the platform. With the ability to create new kinds of computing devices that very much and very effectively blur the lines between the traditional desktop model, which is sort of going away, and a pervasive post-PC model, which is coming to the fore, it looks like embedded Linux may very well be the ingredient that has been missing in this new post-PC world.

LJ: Some of the developers I know are feeling that they're on a treadmill, keeping up with the accelerating change in hardware, and one of their problems is constantly having to rewrite device drivers to adapt to the new chip that replaces the one that's no longer available. I understand that one of the attractions of Linux is that other people are writing a lot of the device drivers.

Michael: Yes. Other people are writing the device drivers. But also, it is a common practice for people who write device drivers for Linux to make those drivers open source.

LJ: Exactly. Is the same dynamic happening with eCos?

Michael: Certainly. That is a dynamic which we're increasingly seeing. eCos went through what many real-time operating systems go through, which is that we announced it two years ago at Embedded Systems West in San Jose. It went through a gestation period. Of course there are always the innovators and early adopters who pick up stuff aggressively, but by and large people waited to see whether or not eCos would mark its first, and now its second, anniversary. One thing that we've noticed is that as we are marking its second anniversary, the number of designs has increased exponentially. With those designs comes an increasing number of participants in the development community. That means new board-support packages, ports to new architectures, device drivers, et cetera. In other words, it is really beginning to reach its critical mass.

LJ: So what kind of quantities of “design wins” are you looking at these days?

Michael: It's a little tricky. You could be on the list of people who will get the pre-briefings about announcements we're going to make at ESC West. But I can't spill those beans too much.

LJ: I'm just looking for a ballpark. Ten solid wins, a hundred?

Michael: I think we've got in the neighborhood of tens of announceable wins.

LJ: Could you compare and contrast the areas of application of embedded Linux with that of eCos?

Michael: eCos is really for devices that want to have a very simple footprint, the sort of classic embedded system design where you have a microcontroller and a device to be controlled. The benefit that eCos gives you is the ability to use open-source availability, the ability to squeeze a particular design into much smaller spaces than would be practical with Linux.

One that we did announce previously, for example, is an operating system for Brother's 2400cen printer family. The jobs of these printers are to be peripherals that apply ink to paper. It is not likely that users are going to want to be running high volume web servers on their printers. It is also not likely that people are going to want to use their printers as additional computing devices for other purposes. Of course, I'm sure the SETI@Home guys would love to double their listening capability by running on unused cycles of color and black-and-white printers. But seriously, eCos is much more designed for purpose-specific devices.

Linux is appearing to be a real runaway success story in the space of Internet appliances, both in the server and client space. Last month we announced a design success with Ericsson, who has actually demonstrated a screen-phone platform running on top of Red Hat Embedded Linux. Our hope is that we will have at least a picture, if not a model, of this phone on display at the conference. This is Ericsson's bid for a new, richer, completely connected device on the client side. There's also a lot of activity on the server side.

There are server appliances, residential gateway servers, small office/home office servers, where you don't want to be running a full desktop distribution of Linux, and you don't necessarily want to be running it as a complete Linux server, but there are capabilities you want, like web serving or firewall or e-mail or file serving et cetera. The reason you want to have these devices configured more narrowly is because it dramatically reduces the complexity of administration.

LJ: Bring it home and plug it in.

Michael: If you want to do something different, you can just download a different profile and it will have those properties. When you have a normal Linux server doing lots of different jobs, it increases your need to have people who understand network security, and it also means you have to make more complex resource allocation and monitoring decisions.

LJ: With freedom comes responsibility.

Michael: Exactly.

LJ: You mentioned Red Hat Embedded Linux. Is that a distinct distribution at this point?

Michael: It's a toolkit. It uses our graphical user-interface technology, called Source Navigator, as a front end, and what we have right now is a standard Red Hat distribution configured by this graphical user interface. So it's not source-level configurability, like we have with eCos, and in fact it is an open question whether the embedded systems market will change Linux to be more supportive of source-level configuration. The current thinking is “no”, but thinking about the future is uncertain. If we don't have source-level configurability, then we still have module-level configurability, and that's what the graphical user interface from Red Hat offers.

LJ: Is that a distinct Red Hat product?

Michael: Yes. We call it the Embedded DevKit. You can actually get it off our web site for $199 (US).

LJ: What's happening with your embedded API proposal, EL/IX?

Michael: Everywhere I talk about EL/IX, I talk about POSIX 1003.13. I gave a couple of talks about embedded Linux at LinuxWorld, and I talked about the fact that we announced EL/IX two weeks before the POSIX committee went public with their 1003.13 recommendations. We had known that they were brewing this, but we also had no certainty as to what the timeline would be. Because we knew what they were doing, and because we liked their approach, we took what we figured to be their ideas. We said, okay, there are going to be four profiles, and they are going to have these bundles of functionality.

Then what happened is 1003.13 came out, and PE51, PE52, PE53, and PE54 corresponded to the four levels that we had picked. Had POSIX come out with thirteen profiles, or twelve, or seven, or two, then our work wouldn't have looked at all like theirs. But the fact that they chose four profiles, as did we, and that those four profiles do have a pretty straightforward alignment, means we're getting behind POSIX 1003.12 as a recognized international standard, and we're lobbying to incorporate more of what we consider to be the true embedded developer's viewpoint. For example, POSIX is very proud of their signal model. But, not a lot of embedded developers that I have talked to like that signal model very much. POSIX signals are not an option in these profiles. However, they are an option, in or out, in ours. The way to look at EL/IX is four profiles: from minimal controller to minimal real time to single process to full-blown multiprocess system.

Number one, we think the theory is correct, but we want to offer people additional configuration options. Whether or not you have signals, whether or not you have networking support, whether or not you have file system support, is (or can be) orthogonal to the four levels.

LJ: So your energy at this point is going behind the POSIX effort, and bringing that forward and making it useful to embedded developers?

Michael: Yes.

LJ: Then the process of EL/IX implementation, either as a layer on top of eCos or as additional elements in the Linux kernel, needs to wait until some finalization on the POSIX 1003.13?

Michael: No. The thing that needs to wait is whether or not Linux is going to support the concept of source-level configurability. As far as what's in POSIX 1003.13, and as far as what options and configuration bundles we want to offer, we believe that our approach is consistent with what POSIX is doing. We anticipate that we will offer something which will be POSIX-conforming. It will also have other options. The fact that you can always get to POSIX from Linux is going to be something which will make people very comfortable.

LJ: So you're moving your efforts forward anticipating converging with POSIX as it moves along?

Michael: Yes. I think that last year Linus Torvalds said he's happy to see POSIX become more Linux-compliant.

LJ: Your EL/IX process is moving forward at this point?

Michael: Yes.

LJ: The EL/IX draft standard 1.1 on your web site is dated in January. Is there going to be a new draft?

Michael: I am hoping so. We've got people under the gun for the Embedded Systems Conference. I can't make any promises. But we'd certainly like to have something fresh for the September show.

LJ: How's the Red Hat/Cygnus integration going along?

Michael: It's going along very well. The magnitude of the merger was pretty substantial. We hit the ground running at the beginning of this year, and in another ten days we'll be announcing earnings. Last quarter when we did our earnings announcement, there were some very clear ways in which Cygnus enhanced the Red Hat business with key customer wins, and since then we've publically announced the relationship with Motorola for high-availability embedded Linux servers, also with Ericsson on the screen phone. We've probably got some other announcements coming out.

If we were a private company I could tell you a lot more exciting details.

LJ: You mentioned Hitchiker's Guide to the Galaxy in another recent interview. Who's your favorite character in that book?

Michael: I think it would have to be Ford Prefect.

LJ: Why would that be?

Michael: Because he is so oblivious of his own absurdity.

LJ: I guess that would be a good model for most of us.

Michael: I also enjoy his optimism.

LJ: Thanks so much, Michael, for taking the time to talk with us.

Michael: Thank you, Dan, and I hope I see you down at ESC West.

Resources