AMD64 looks great on paper, but can you build a high-end system with 64-bit processors and have all your device drivers work? We mix the hottest new processor architecture with the latest video, audio and storage devices and watch for smoke.
For the Ultimate Linux Box this year, Editor in Chief Don Marti and I decided to build a multimedia workstation. I was thinking of doing a dual-Xeon unit, having helped build a few of them for a former employer, when Trey Harris of Monarch Computer Systems called to say, “The dual Opteron boards are out.” My mouth watered at the thought. The only formally released OS that runs on an Opteron (at press time) is Linux. We traded e-mails and machines, and ultimately we came up with a design that, although it has a rough edge or two as I write this, should be ready to go by the time this is printed.
The motherboard is an Arima HDAMB workstation board in an ATX form factor, with the AMD 8111/8151 Rhapsody chipset. It has slots for four DDR333 Registered ECC DIMMs (we used two 1GB Corsair CM74SD1024RLP-2700s), an 8X AGP card and five 32-bit PCI cards. The backplane has four USB jacks, a Broadcom gigabit Ethernet interface and a Realtek ALC650 sound system, as well as the usual parallel, serial and PS/2 keyboard and mouse ports (one of each). The test machine did not come with a FireWire interface or serial ATA ports, although the board does offer options for both. The twin Socket 940s were loaded with AMD Opteron 246s running at 2GHz and topped with Thermaltake AI724 coolers.
Video is provided by an NVIDIA Quadro FX 1000. This is a workstation-class, twin-head-capable card that I selected because NVIDIA released support for it on the AMD64 back in December 2002. (ATI has not released 64-bit support for its competing product, the Fire GL X1, as of press time.) Sound comes from the Creative Labs Sound Blaster Audigy 2, a 24-bit/192kHz, THX-certified digital card capable of 6.1 analog output or Dolby Digital EX on the digital side. It also has an IEEE 1394 (FireWire) interface.
The IDE interfaces are populated by a JLMS XJ-HD166S 16x DVD/ROM drive and a LITE-ON LTR-52327S 52/32/52x CD-RW drive. We chose to forego a DVD writer, because the whole issue of DVD+RW/DVD-RW/DVD-RAM—and Linux drivers for each—is a rather sticky issue. Part of the joy of Linux and of open standards such as IDE, however, is that experienced audiophiles are free to choose their own favorite and expect it to work when they plug it in. Indeed, at press time, all of the above combination drives are on the market, and both drives and drivers probably will make quantum leaps by the time this issue hits the newsstand.
If we're using the IDE interfaces for optical drives, what about hard disks? Indeed, aside from the Opterons themselves, hard disks are the most formidable part of this system. I was eager to try out 3ware's new Serial ATA (SATA) RAID card, the Escalade 8500-4. It is a four-channel, half-size 64-bit PCI card that also works well in 32-bit slots. At press time, the Escalade 8506s were becoming available, replacing the 8500s. According to 3ware's Product Transition Matrix, the second-generation controller is up to 25% faster than the 8500s. The 8500 is what we had to test with, though, and the statistics reflect that. The good news is the box you build or buy will be faster.
The 8500-4 has four SATA-150 ports on the back edge of the card (toward where the drives are located). This arrangement, combined with the much smaller form factor of SATA cables, allows for much better airflow behind the drives. And these drives need some pretty serious airflow; the four 36GB Western Digital Raptor WD360GD Serial ATA drives run at 10,000 RPM. We deliberately went with these, instead of larger but slower RPM drives, to optimize for speed over storage space. These drives also have jacks for standard Molex power connectors as well as the new SATA type adapters, which help the transition between platforms. One drive is mounted below the floppy; the other three are mounted vertically in the housing at the bottom of the case, in front of the fans.
Now, we arrive at the case itself. All of the previously named components, plus the ENERMAX 465P-VE-24P 460-watt power supply, are contained in a custom Lian-Li PC-6270 quiet case. We chose this case because it is capable of accommodating extended ATX motherboards. As usual, this case comes with all the Lian-Li bells and whistles, such as thumbscrew assembly almost everywhere. In addition, four 5.25" and three 3.5" bays are accessible from behind a removable door on the front; an additional horizontal bay and five vertical-mount drive bays are inside. The case also has a slide-out motherboard tray and a filtered air intake under the chin of the case for the twin hard drive fans. A second matched pair of fans pulls heat out of the back of the case. An adapter converts the lower five vertical-mount bays to a four-bay horizontal-mount arrangement, and a handy cable clamp mounted with adhesive foam on the bottom of the case keeps all the SATA cables out of the airflow. Two USB jacks are provided on the front of the case and are accessible when the door is closed. The case is kept quiet by the use of dense foam batting inside the top and side panels, and a rubber seal around the door also helps cut drive noise.
Now that we have all the hardware bases covered, we need an OS. I chose SuSE Linux Enterprise Server 8 (SLES 8) based on what I had seen in SuSE's 32-bit offerings. Although SLES 8 isn't a bells, whistles and eye-candy OS, I was impressed by its smoothness and ease of use. It comes with base GNOME and KDE environments and a full set of development tools, making it an excellent base on which to install or develop professional applications.
If that weren't enough, YaST2, SuSE's GUI setup tool, has completely sold me on SLES 8. YaST2 does everything from printer configuration and user additions to advanced firewall configuration, and it does so quickly, clearly and with few surprises. The only thing I found that it could not do was configure a non-ALSA sound card—more on that later.
Often, when installing many distributions on anything other than the x86 architecture, packages are out of date, things don't work quite right and other annoyances abound. SLES 8 comes with kernel 2.4.19, KDE 3.0, Samba 2.2 and XFree86 4.2. These packages aren't bleeding edge, but quite current. They work well together, too, and aside from that one sound annoyance, I didn't find anything wrong that I couldn't fix easily.
As I said, getting sound to work on this system was an adventure. The Audigy 2 is relatively new, and I had a feeling that getting it to work might be a chore. I was not disappointed. The ALSA configurator in YaST2 correctly identified the card, but no sound came out. I tried using the emu10k1 Project available on SourceForge. After hand-tweaking /etc/modules.conf for the latest version of the OSS driver, the module loaded, but the system still had no sound. Prior to giving up and submitting a bug, I searched the SourceForge bug database and found that the CVS version was reported to work. But, would it work on a 64-bit platform? After a few make scripts, I saw the CVS version conflict with the Realtek chip on the motherboard, so I took its listing out of modules.conf and rebooted. The subsequent strains of Theodorakis' “Ode to Zeus” pouring forth from the speakers heralded victory.
Lack of time and equipment precluded any sort of high-end technical test, but I consider myself a decent audiophile, so I set up a test to satisfy myself that the Audigy 2 is worth the investment. My wife and I have side-by-side machines with identical speakersets. I put the sound cable from her system in the appropriate jack on the Ultimate Linux Box and started Arnaud's “Bulger's Dream” from CD. I used XMMS with the EQ off for both tests, and I have to say, qualitatively, the Audigy 2 beat my Esoniq 5880 hands down for clarity and frequency response. It was quite a bit crisper and didn't develop any distortion or hum at high volume settings. Cards are available that probably can do quite a bit more than the Audigy 2, but we know this one is a good starting point. Plus, it works on the 64-bit platform, which is an achievement in itself.
Prior to press time, we were able to file down one rough edge—the video card. If you read my August 2003 Web article [www.linuxjournal.com/article/6922] on the subject, you may recall we rejected ATI's Fire GL X1 because it has drivers that work only on 32-bit platforms. My source at ATI was unwilling to comment on record about a 64-bit release. The ATI representative did at least ask me, unsolicited, what card I was running and why I had switched cards after reading my Web article.
When I initially tried to run the driver for the NVIDIA Quadro FX 1000 card, the X server hung in max CPU mode, even though the driver had been available more than six months. Monarch put me in touch with NVIDIA directly. The NVIDIA engineer, Mark Visconti, took a look at my current information and suggested I upgrade the BIOS, which appeared to be an engineering sample version. I dutifully downloaded the BIOS and flash utility, both of which refused to work. Fortunately, from previous issues with the motherboard, I had a contact at Arima, the board maker. As it turns out, some arcane arguments have to be given to PHLASH.EXE—even now, you still have to boot DOS to flash the BIOS—to get the new image to load. After this step, I went back and reset the BIOS settings Visconti recommended. When booting into Linux this time, startx finally rewarded me with the cautions-bomb-boxes background that is a root X login.
I was able to verify that the server was running in 3-D mode, but we did not have time before going to press to do video benchmarks. Given our tests of the NVIDIA card on a 32-bit machine, we should see frame rates in the mid- to high 50s, if not higher. If we do manage to get it tested at some point, you'll be able to find those results on our Web site.
The rough edges on this machine, however, are only in the eye candy. Aside from Chromium (the video test), we were able to run the rest of the Linux Journal Benchmark Bundle on the machine (bonnie++, postgres-test, tiobench and the kernel compile), and the results were impressive. The base kernel compile averaged one minute 35 seconds using both CPUs. As a point of comparison, my personal 1.1GHz Duron took six minutes 50 seconds to do the compile. Overall, the Opteron/3ware architecture seems to offer about a 15% advantage in speed, gigahertz for gigahertz. The Opteron has an integrated memory controller that bypasses the Northbridge and gives you a 64-bit duplex channel direct to the DIMMs. This particular machine also allows use of a singleton DIMM in 32-bit mode, which is handy for debugging hardware problems, at the expense of speed.
Tiobench produced the other interesting numbers, as shown in Listing 1. I compared the performance of the 3ware/Western Digital harness on an Athlon 2800 to its performance on the ULB. The numbers are not quite apples to apples, because I took advantage of the Opteron's 64-bit addressing to handle a larger file size—but that in itself produced interesting results. The 32-bit platform appeared to do better in single-threaded sequential reads but produced comparable numbers for the rest of the run. On random reads and writes, the absolute rate of the ULB lags behind, but the latency does not. The ULB soundly trounces the Athlon despite using a file more than double the size. I suspect that the drop in numbers at the end of the sequential read and write runs is an artifact of hitting the 3ware card's buffer limit, given the nice flat curve up to that point. On a side note, these numbers compare favorably in most departments in absolute terms—and very well in terms of bang for buck—with a Dell Precision 650n SCSI system I tested previously.
After receiving some feedback on our Web article series that discussed the ULB, we decided we needed to test noise levels. The ULB scored 50.5dBa at 10" in front of the case, 50.0dBa at the operator position (24" above the top of the case) and 60.0dBa at 10" from the back of the case. Obviously, these numbers aren't as low as we'd like them to be with the aforementioned Dell coming out at 47, 45 and 55dBa respectively. I think the culprit is the Thermaltake coolers; when we had the machine in-house previously with AMD coolers, it was quieter by far. However, the AMD version did get rather warm. The ULB version seems to be quite stable with respect to temperature, at the expense of extra noise. I suspect that between now and when this issue is on newsstands, companies such as Zalman and PC Power and Cooling will have offerings available for the Opteron like the ones they now have for noisy Athlons.
The Ultimate Linux Box, like Linux itself, is a work-in-progress. By the time you read this, new toys will be on the market, not to mention new software, maybe even a major kernel revision. The Escalade 8506 also will be available, as will newer, bigger Western Digital Raptors, DVD-plus-or-minus-RW combo drives—perhaps even an Athlon 64 CPU and motherboard. Use this design as a place to start and make your own improvements.
The author would like to thank 3ware, Western Digital, NVIDIA, Arima and SuSE for their contributions and Monarch Computer Systems for tying all the hardware together in one package and making it play.
