Linux on Alpha AXP—Milo, The Mini-loader

Late in 1994, I was in the US visiting my home group in Hudson, Massachusetts, where the Alpha AXP processors are built. On a free morning, I followed up a rumour that I had heard a few weeks before. The rumour was that Jim Paradis was porting Linux to Alpha. At that time I did not know anything about Linux other than that it was a freeware version of Unix developed by a student in Finland. When I caught up with Jim, I was in for a surprise. Well, two surprises actually: Linux—running on an Alpha laptop. We chatted for a while and I soon became infected (if that's the right phrase) with Jim's enthusiasm.

One subject that we discussed was Linux/Alpha's need for a small loader. On an Intel PC system, the firmware that initialises the system when it is powered on is known as BIOS. There are several very well known providers of BIOS code, and PCs are built to conform to a very rigid set of rules, which means that PCs are very similar to each other in hardware terms. The equivalent software in an Alpha-based system from Digital (and even in a VAX-based system) is the console, and within Digital it is known as the SRM console, since its interface is described in the System Reference Manual.

So, just why did Linux need the SRM console? Firstly, Linux needs to be loaded from some media, and the SRM contains device drivers to do just that. Secondly, Linux needed the Digital Unix PALcode. PALcode can be thought of as a tiny software layer that tailors the chip to a particular operating system. It runs in a special mode (PALmode) and has certain restrictions, but it uses the standard Alpha instruction set. In this way, the Alpha chip can run such diverse operating systems as Windows NT, Open VMS, Digital Unix and, of course, Linux. Finally, Jim was using the SRM call backs in his prototype device drivers. From Linux's point of view, though, the SRM Console does too much. It contains call back procedures that allow the running operating system to write messages to the console or to write environment variables and so on. Linux makes no use of these functions; in fact, the only part of the SRM console needed, once it loads Linux, is the PALcode.

I volunteered to write a loader that would be small and would do only those things that Linux needed. Like all good projects, my one-man effort had some straightforward project goals. First and foremost, the software would be under free licence, built and freely distributed as part of standard Linux distributions. Secondly, Linux drivers should be able to be used within Milo without modification or even re-compilation. Thirdly, it should maximize the amount of memory available to Linux. Little did I realise quite what I was letting myself in for.

Milo contains the following functional pieces:

Finding Digital Unix PALcode was no problem. Back in 1992 when Digital announced the Alpha processor, it also announced that it was moving into the merchant chip market. I joined a small group in the UK to provide engineering effort in Europe to further these aims. We are a small offshoot of the main group, which is based in the silicon factory in Hudson, Mass. We build evaluation boards for the Alpha processors and PCI peripheral chips. These systems included a very low level Evaluation Board Debug Monitor that uses Digital Unix PALcode. The sources for the Evaluation Board Debug Monitor and the PALcode are under free licence.

Although this PALcode is fully compliant with the interface described in the Alpha Architecture Manual, there are some differences between it and the SRM console's PALcode. One of the differences between Alpha based systems is the way interrupts are handled outside of the processor itself. There are a limited number of interrupt signals into the CPU itself, and the number varies from CPU to CPU, but there are typically three: timer, I/O and non-maskable interrupt.

The way in which real device interrupts are mapped onto CPU interrupts is system-specific. Most current Alpha systems include an ISA bus, whose interrupts are routed through a pair of 8259s in the same way as on x86 PCs. The SRM console PALcode handles these differences and interprets the interrupt, passing it to the OS's interrupt handling code as an “SCB offset” (described by Jim Paradis in last month's Kernel Korner). The PALcode used in Milo does not do this interpretation, so the OS's interrupt handling code must do it instead. Particularly with PCI devices, there must be code in the PCI BIOS code and within the interrupt handler that understands how interrupts are routed in the system. One side effect of this is that when Linux has been loaded by Milo, the interrupt handler can handle more than one device's interrupt each time it is called.

One interesting and useful feature of the example PALcode I adopted from our Alpha evaluation boards is that it allows the Evaluation Board Debug Monitor to run in 1-to-1 physical addressing mode. Bit 0 of the virtual page table base register turns this feature on and off. When translation buffer misses occur, the PALcode builds a new page table entry and inserts it into the cache. Pretty much the first thing that Milo does when it is loaded is to swap to this PALcode in physical address mode. The last thing that the Mini loader does is to swap to this PALcode again, this time passing final control to the Linux kernel.

Milo must turn virtual memory mapping on as it passes control to the kernel, because Linux expects that a control structure called the Hardware Restart Parameter Block (HWRPB) is at the right virtual address. Amongst other things, this describes the type of system and how much memory is free, together with where the memory is. As Linux was first loaded via the SRM console, it naturally used the interface provided by the SRM, which was the HWRPB, as described in the Alpha Architecture Manual. I could see no reason to change this interface: there are enough interfaces in the world, so why invent yet another one?

In order for Milo to set up the memory mapping correctly, it must itself have a good idea of what memory is available and what it is being used for. It finds the amount of memory available because after the PAL reset code has been executed, the size of memory is put into the impure area, a data structure shared between the PALcode and the console or Evaluation Board Debug Monitor. Milo keeps a memory map describing what each page in the system is being used for. While device drivers are running, they allocate temporary memory and use it. Just before control is passed to Linux, Milo must build a correct memory cluster description in the HWRPB, and the memory map is used to do this. Pages are marked as “free”, “allocated” or “temporarily allocated” in the memory map. When Milo builds the memory cluster descriptions in the HWRPB, it treats all temporarily allocated memory as free, since they will be free once Linux starts to run. In this way, the only memory that is marked as allocated is the memory containing the PALcode (8 pages), the memory for the HWRPB (1 page) and the memory for the level 2 and level 3 page tables (2 pages): 11 pages in total. I think I've succeeded in my goal of maximizing the amount of memory available to Linux.

One of the really excellent things about Linux is the number of device drivers that have been developed for it. It seems as if any commercially-available card or chip set has a driver already written for it. It therefore seemed vital that Milo should be able to make use of those drivers. This allows companies building Alpha-based systems to differentiate their products by having a vast choice of possible devices.

To run the device drivers unmodified, I had to duplicate some services of the Linux kernel. Originally, I planned not to have real interrupts, but instead to poll the drivers. This was the way that Milo worked in Linux 1.1.68. However, once I started to try and get the NCR 53C810 SCSI driver working in Milo, I ended up needing proper interrupt handling, and it seemed best to take the interrupt handling directly from Linux, which I did.

I have tried to keep the number of Linux services that I have had to duplicate in Milo to a minimum. After all, as Linux progresses, these routines tend to need re-writing. A good example is the change from 1.1.68 and 1.2.8; the floppy driver changed its way of determining that it was running during kernel initialisation. This caused me headaches as I figured this out.

Maybe over time I will incorporate more of the real Linux kernel into Milo, but it is supposed to be the Mini loader, so I do not want to add the whole of the kernel into it. Right now, Milo includes the PCI BIOS code, the block device code, the interrupt handling and DMA code directly from the kernel. The scheduling services are mine and I cannot see them changing unless I add multi-threading support.

The final functional piece of Milo is the part that most users see, the user interface. Milo can operate via the serial port, but mainly people use it via the system console. For this reason, it must have some keyboard and VGA initialization code.

The keyboard code is very very simple and does just enough to take in commands correctly. Linux itself assumes that some BIOS code has initialized the VGA device and its console device drivers just use it; that meant that Milo had to initialize the VGA device. There are two ways of doing this. The first is to have very simple ISA VGA initialisation code, and this is how Milo first operated. The second way is to include BIOS emulation code that can run the on-board initialisation (which is Intel x86 object code) from the different video cards. David Mosberger-Tang pulled this part of Milo together, with the result that it can successfully initialise a number of common ISA and PCI graphics cards.

The Milo interface is meant to be very simple and do just enough to get the right kernel loaded and to pass the right boot arguments to it. Typing anything other than a legal command displays all of the commands available. Right now, Milo assumes that all devices that it can see are available to boot from and will attempt to use the EXT2 file system with them.

Milo was developed on an Alpha evaluation board (an EB66, which is a 21066 based system similar to the AxpPCI33). This meant that loading and testing Milo was easy, since the Evaluation Board Debug Monitor was running. However, for real systems like the AxpPCI33 (Noname), Milo needs to be loaded some other way.

Alpha-based systems boot in several steps. The first step, immediately after power on, is to clock the SROM code directly into the I-Cache stream and then to start executing it. This code does really basic system set up such as figuring out how many DRAM slots are occupied, and with what size of memory. The next step varies from system to system, but essentially that SROM code loads the firmware code (whatever it is) into memory and passes control to it in PALmode. This is the PALcode reset entry point for the image. Some firmware, notably Milo, has the address of the entry point to the user-mode firmware defaulted in the PALcode, and that is where control is passed to when the PALcode reset code has finished initialising the system. Other systems have this information stored in NVRAM or infer it from jumper settings.

For a variety of reasons, Milo can be loaded from a failsafe boot block floppy, from flash, and via the Windows NT ARC firmware. What varies most from system to system is where the SROM code is able to load firmware from. On the AxpPCI33, the SROM code is capable of loading from flash, from a serial line, or from a failsafe boot block floppy. On the AlphaPC64, the failsafe boot block floppy is not supported. All of this is controlled by jumpers and/or boot options saved in NVRAM (in the TOY clock in the AlphaPC64's case). There are systems that do not support flash and instead have ROMs. These are not easy for users to change without access to a ROM blower. and so yet another way must be found for Milo to be loaded. Paradoxically, you could load Milo via the SRM console, but a more fruitful approach is to load it via the Windows NT ARC firmware, since that is the firmware these boards ship with.

There are a number of ways to put an image into flash, and for this reason Milo supports running any image, so long as it is linked to where the Linux kernel usually is. In this way, I can build images that update the flash when loaded and not burden Milo with knowing about the flash requirements of every different system that it runs on.

Loading via the Windows NT ARC console is interesting. On Alpha, Windows NT runs in “super paged” mode, which does not support KSEG addressing—which is unfortunately exactly what Linux needs for fully 64-bit operations. However, all PALcode implementations must support the “Swap Pal” call, and this allows you to change from one mode to another. The Windows NT ARC console has within it the notion of running images and providing services to them so long as they are built to run in the appropriate addressing mode and run at a safe place in memory.

Thus, the Windows NT OS loader is in fact an executable image which gets loaded in order to load Windows NT using the appropriate call backs. I have written a very simple OS loader whose only function is to load Milo, which in turn loads Linux. It is this simple loader which makes the “Swap Pal” call which causes control to be passed to Milo and KSEG addressing turned on. From then on, Milo operates exactly as before with the addition that it can execute commands passed via the [cw]OSLOADOPTIONS[ecw] environment variable for this boot option, and thus boot directly without pausing at the Milo prompt.

I have tried to eliminate this need for an image that is built under the Windows NT firmware development tree; unfortunately this is not possible, so I have kept the functionality of this part as minimal as possible.

Of course, loading Milo via the Windows NT ARC console is one way to get Milo running so that Milo can run the flash update utility to put itself into flash. Alternatively, it can be a way of running either operating system without one interfering with the other.

Milo is in its infancy and I hope to see the Linux community add into it what they need. After all, that is part of the attraction of Linux itself—a community of able, enthusiastic programmers adding to the effort.