UpFront

Linus Torvalds recently said that the whole software suspend debate between Pavel Machek's swsusp and Nigel Cunningham's suspend2 was entirely misguided. Linus said he no longer has faith that either project can succeed. His hope now, he said, was that some new person would step in and take the whole concept in an entirely new direction. He also said that he believes software suspend is indeed a kernel issue and not something that should be entirely in user space. If he finds a solution he likes, he'll incorporate it into the kernel.

It also seems that Linus himself has solved some of the basic development issues, and various people are now telling him his ideas are crazy. Typically when this happens, folks will hurl insults at Linus for weeks or months, while he'll just continue to explain his ideas. Eventually, some kind of tipping point is reached, and a general perspective shift occurs. At this point, one or more folks will rush off to code up the new idea.

Alan Hourihane of Tungsten Graphics has been working on an FBDev driver for Intel's LE80578 chipset. Intel actually had been funding the work through Tungsten, and Alan posted his code recently to the linux-kernel mailing list. In response to various technical suggestions, Alan also posted updated patches addressing those issues—typically a sign that a piece of code is likely to be included in the official kernel sooner rather than later.

The question of what will become of the Reiser4 filesystem is still open. It could be adopted into the official kernel; it could transform from a primarily corporate project to a primarily volunteer one; it could become unmaintained and disappear completely. There are many possibilities. But, the fact of Hans Reiser's murder trial does not change most of the issues surrounding whether to include the code in the kernel.

One thing holding the project back is that kernel developers are still reluctant to give significant feedback on the code. Hans' habit of rejecting all commentary and engaging in ad hominem attacks on the commentators has left a mark that even his absence on murder charges has not eroded. A lot of folks who had put in a significant amount of time on giving analysis of Reiser4 code simply have moved on to other projects, and no one has arisen to take their place.

Without reviewers, the likelihood of Reiser4 going into the kernel diminishes greatly. A number of relatively large technical issues are standing in the way of inclusion, and reviews by kernel developers were one of the key ways the Reiser4 developers could understand those issues.

It seems possible that Andrew Morton might try to jump-start some kind of review process by accepting the patches into his -mm tree, with the implied threat/promise of submitting them to Linus Torvalds. This would certainly have a number of kernel developers up in arms and cause them to take a much closer look at the code. But, it also is a fairly drastic measure for Andrew to take.

It's also possible that any sort of review process might take too long. The Namesys engineers who maintain Reiser4 are having to confront a diminishing amount of funds available to support the project. At least one Namesys engineer, Edward Shishkin, has said he would still devote about 25% of his time to Reiser4 on a volunteer basis if his employment at Namesys went away.

As usually happens when issues like these are brought out in the open, recent discussions on linux-kernel have resulted in several folks showing interest in reviewing the Reiser4 code. None of the reviewers from the old days seem very interested, but maybe this new set of reviewers will be able to provide enough commentary to smooth the path a little bit.

Adrian Bunk continues his long-standing efforts to clean up the kernel. A lot of kernel code hangs around in the kernel for a long time, even after a better alternative comes along, and even when that better alternative makes its way into the official source tree.

In the current case, X86_ACPI_CPUFREQ is the new code, allowing users to change the clock speed of their CPUs on the fly, which can amount to a large power savings. The old code, X86_SPEEDSTEP_CENTRINO_ACPI, has been deprecated for a long time, and now its time has come.

Adrian posted a patch removing the older code, and there was a small bit of discussion—whether to include the patch in 2.6.21 or 2.6.22, and how to communicate the change to users who might rely on the older code. Bill Davidsen was particularly worried about this latter possibility, until Adrian reminded him that the older code had been marked as deprecated for a long time, and the feature-removal-schedule.txt file contained a clear explanation of the replacement.

Last month, we visited the overwhelming pro-Linux response to Dell's IdeaStorm site (“New Product Design: Now You Decide!”, www.ideastorm.com). As we go to press, there is much talk about Dell's plans to market PCs with Ubuntu pre-installed. That talk has been fueled by the biography page of company founder and namesake Michael Dell. At its top, above all the other computers he uses, is this:

1. Position of Linux-based Rackspace among Netcraft's most reliable hosting companies for April 2007: 1

2. Number of Linux-based hosting providers among Netcraft's top 10 for April 2007: 3

3. Number of open-source hosting providers among Netcraft's top 10 for April 2007: 6

4. Millions of dollars in 2006 revenues for Rackspace: 224

5. Rackspace revenue growth percentage over 2005: 61

6. Number of open-source venture capital deals in 2004: 36

7. Millions of dollars invested by VCs in open-source startups in 2004: 297

8. Number of open-source venture capital deals in 2005: 41

9. Millions of dollars invested by VCs in open-source startups in 2005: 306

10. Number of open-source venture capital deals in 2006: 48

11. Millions of dollars invested by VCs in open-source startups in 2006: 481

12. Billions of dollars invested by VCs in open-source startups since 2000: 1.9

13. Number of five-year road maps for the Linux kernel: 0

14. Number of operating systems that support more hardware than Linux: 0

15. Millions of digital cameras: 400

16. Millions of cell phones with cameras: 600

17. Millions of digital music players: 550

18. Millions of computers: 900

19. Millions of plasma TVs sold worldwide: 70

20. Year by which the amount of data to be stored exceeds the capacity of data storage devices: 2007

1–5: Netcraft.com

6–12, Matt Asay, Robin Vasan and Matthew Aslett

13, 14: Jonathan Corbett

15–20: IDC, via Freedom's Phoenix

First there was the Qtopia Greenphone (www.trolltech.com/products/qtopia/greenphone) from Trolltech: “a Linux mobile development device open for unlimited software innovation”. Then there was OpenMoko (wiki.openmoko.org/wiki/Introduction): “the world's first integrated open-source mobile communications platform”.

Now there's JavaFX Mobile: “a complete mobile operating and application environment built around Java and Linux open-source technologies”. Not coincidentally, the phone Sun used to demonstrate JavaFX Mobile at JavaOne in May 2007 was the OpenMoko design (known as Neo1973) that we covered in the February 2007 issue of Linux Journal.

Sun and Java have been around the cell-phone business for a long time. And, the platform has come a long way since the time I joked to Sun CEO Jonathan Schwartz that Java to me was “the logo I had to stare at for 16 seconds after I hit the wrong button on my cell phone”.

But, this is a huge new move, for five reasons. First, Java is being GPL'd (which was the cover story of our June 2007 issue). Second, Sun wants to fill the world with mobile phones that are open to development by players other than the phone builders and their carrier partners. Third, Sun actually has clout with phone makers. Fourth, it's not stopping at phones, but looking to put this in all kinds of embedded devices. Fifth, but hardly last, Schwartz & Co. have a bone to pick with Steve Jobs, who has dissed Java while getting huge PR for Apple's new (and closed) iPhone.

There are lots of dots to connect here.

There's SavaJe, the source of the code base for JavaFX Mobile, recently acquired by Sun. There are connections between SavaJe and longtime Linux hardware developer Tadpole Technology.

There's Ed Zander, the Motorola CEO who used to be Sun's President and CEO, and who can't be happy to have seen Motorola's phone “partnership” with Apple go south. Would Motorola be willing to go zig in the open direction while Apple zags in the closed? Good chance.

There's Eric Schmidt, the former Java development chief at Sun who is now CEO of Google. He's on Apple's board, but still, it's a connection.

Then, there's what Jonathan Schwartz says on his blog (blogs.sun.com/jonathan/entry/when_not_where): he wants the world to have an open platform that allows “any consumer electronics manufacturer to accelerate the delivery of Java/Linux-based devices, from phones to set tops and dashboards and everything else imaginable, without fear of format lock-in or disintermediation from a competitor”. All this from “a company with no agenda to disintermediate content owners and every interest in propelling the Open Source community (every portion of the content Sun contributes to the JavaFX product and community will be via the GPL license, at the core of Java and GNU/Linux)”.

He also wants this to make phones cheap as well as open, to “play a central role in bringing the Internet to the planet” and to “be the software to build the devices to bridge the digital divide”—like the One Laptop Per Child, only half the price or less.

Encouraging stuff. It'll be interesting to see what happens.

Support open spectrum. Demand real competition. Ask for investigations of past spectrum auctions and the return of frequencies obtained through fraud. Demand that systems be built-out. It will take political pressure to create the kind of competitive wireless market in which something like a $30 Internet phone can be made available. Time to bring it.

—Dana Blankenhorn, blogs.zdnet.com/open-source/?p=1042

I don't know who discovered water, but I'm pretty sure it wasn't a fish.

—Marshall McLuhan, multiple places on the Net

We're continuing to nurse along a few basically 15-year-old filesystems while we do have the brains, manpower and processes to implement a new, really great one.

—Andrew Morton, source: Jonathan Corbett, presentation at CELF, April 2007

It's so hard to write a graphics driver that open-sourcing it will not help.

—Andrew Fear, NVIDIA software product manager, source: Jonathan Corbett, presentation at CELF, April 2007

Linux cannot give in to binary-only drivers. That way leads to the end of our free system.

—Jonathan Corbett, presentation at CELF, April 2007

When you introduce Linux to a group of people unfamiliar with the idea, it's a lot like giving cough syrup to a five-year-old. Average users tend to lose sight of the things Linux can do for them and focus on the rough edges. As Linux “people”, our job is often just as political as it is technical, because let's face it, the software is generally the easiest part of the equation.

In 2003, Michigan's educational funding started a downward trend that continues into present day (2007). Every school in our state feels it, and every school is competing to keep students. In an unfortunate twist, 2003 also was the year we needed to change our student management system from our old, now defunct, Macintosh-based system to a Microsoft Windows-based solution. My suggested method to accomplish that was Linux—specifically, Linux thin clients.

I created a menu of proposals for the school board. One described the cost to switch entirely to a Microsoft-based infrastructure. This included workstations, servers, licensing, staff training and additional third-party needs, such as antivirus software and Microsoft Office. The other option required purchasing a handful of servers and using donated computers as thin clients. We would base the system on the Linux Terminal Server Project (LTSP). The first proposal came in at about $500,000 for the first year, and the latter came in at about $150,000. I didn't have to do much more selling—“Linux” was the school board's new favorite word.

It's important to note that I easily could have quoted the cost at $50,000 and still had a functional network. One of the biggest mistakes we as OSS advocates make is not spending enough money. Don't scrimp on hardware when you are saving so much on software. It's important to find a balance between savings and functionality. For example, the bare minimum I needed to convert our district to thin-client technology was:

However, my quote also included the following additional purchases:

The board approved the transition to Linux. We ordered the hardware and jumped feet first into an exciting summer.

LTSP is a system that works on top of basically any Linux distribution. I've always been rather fond of Debian, but for this huge install, I wanted to use something that had been tried and tested. The folks over at K12LTSP have taken Fedora and created an installer that integrates LTSP and scores of tweaks that really make the thin clients shine. The support from their mailing list is usually quick and always provides more help than any phone support I've ever used. In fact, the developers usually are available in real time on IRC if you have a mission-critical question. Unless you have a real aversion to Fedora-based distributions, there's really not any good reason to go elsewhere.

One of the more difficult parts of implementing a wide-scale thin-client rollout is to estimate server requirements. There is no simple way to determine how many thin clients a given server can handle. The requirements vary enormously depending on the applications that will run from the thin clients. A best guideline, based on discussion with the LTSP developers, is the following:

Following those guidelines, I figured in order to support the 75 thin clients I needed, I should get three LTSP servers, one file server and one Windows Terminal Server (for the Windows application we needed).

During the installation, one of my biggest worries was not having enough horsepower to run all the thin clients. The first concern was network bandwidth. We did purchase new switches throughout the district, but I wasn't familiar with Ethernet channel bonding, and I didn't want to experiment with the idea on the first day of school. I decided to split the network traffic into two segments. All traffic between the LTSP servers and the thin clients traveled over eth0 on our main network, and all file transfers between the LTSP servers and the NFS server flowed through a separate, gigabit switch over eth1. That effectively isolated the two main bandwidth usages and avoided the need for channel bonding. The solution was very effective.

I was still concerned about the performance issue and didn't want my ignorance to make LTSP or Linux look bad on day one. To make sure bandwidth and processor usage wouldn't be a problem, I decided to give the users a very spartan desktop when they logged in. The software I chose for a minimalistic Linux experience was:

The school year started, and from a technical standpoint, the system performed better than I anticipated. The servers were minimally loaded, and I never even had to bring the second and third LTSP servers on-line. That first day was great, but my excitement was short-lived.

Our teachers hated the thin clients. There were nasty complaints, grievances and even personal attacks pointed at me. In hindsight, I easily could have avoided the problems. Currently, four years later, almost the entire staff loves the thin clients. I have more requests for thin clients in the classroom than I have the hardware to handle. Now, four years later, I see the results I expected that first day of school in 2003.

The following are a few ways to introduce Linux successfully, specifically thin clients, to an organization, while avoiding some of the errors we made along the way.

I started testing LTSP with a willing teacher back in 2001 when it was very new. That 4th-grade classroom found a ton of bugs, but worked through them and really loved the technology. It was the right idea to start in such a way, but my focus was too narrow. Multiple pilot programs, with staff demonstrations, are a very effective way to get people talking positively about Linux. With LTSP, pilot programs are easy, because a standard workstation class computer easily can act as a server, supporting 4–6 thin clients. A few small-scale labs really showcase the power of thin clients, and they usually are almost free to implement. When staff members see firsthand the benefits OSS can offer, especially when money is tight, they are much more willing to try something new.

Linux is great. Thin clients are amazing. It's not necessary to claim thin clients are better in every way than traditional workstations. I made the mistake of talking about all the good things thin-client technology meant and failed to point out the shortcomings. As Linux users, we're very familiar with the reasons to use OSS, because we've already learned that the benefits outweigh the drawbacks. It's important to let new users judge those things for themselves and come to a decision on their own (see the sidebar outlining some pros and cons of thin clients). It's vital to be up front and informative on both sides. In fact, stressing the shortcomings sometimes pleasantly surprises people when they realize how unimportant those faults really are.

We hired a consultant. In our case, it was late in the game, and it was much more responsive than proactive. Hiring a professional consultant will be money well spent. I don't mean it's necessary to hire a consultant due to lack of personal ability, but rather as a political move with the advantage of another hired brain. The fact of the matter is, consultants actually do very little of the work. We hired a well-known educational technology consulting company in Michigan—Trimble Consulting. If I could do it over again, I happily would have traded one of my servers for a single day of their help and wisdom. That said, it is important to hire wisely. Consultants should be able to do the following:

Hiring consultants is different from hiring helpers. As I said, on the job work-wise, they do very little. The benefits they offer behind the scenes are incredible. The credibility they add to a project is priceless.

Here are some example explanations:

You get the idea. Communication is key, and being able to communicate very complex ideas simply is a huge asset. In fact, looking back over the past four years, our biggest difficulties were not technical, but social. In general, people are not keen to change, so helping them understand the process makes the change less foreign. It's impossible to over-communicate.

Our thin clients were donated Pentium 133 computers. Even in 2003, Pentium 133s were ancient. The big problem is that they looked as old as they were. In fact, most of them had been marked with permanent markers, and many of them had CD drawers broken off the front. They looked terrible.

So on the first day of school, teachers didn't notice the speed that a dual Xeon 3.2GHz server with 6GB of RAM gives a person. The brand-new gigabit backbone in the closet didn't impress them. The deafening sounds of cooling fans in the server room didn't make them feel special. They saw old, junky computers on the first day of school. In fact, the computers looked worse than the Macintosh 5400s they left behind the year before. I did get new 17" monitors, and brand-new optical mice (which were pretty new technology then), but in the end, they just saw cruddy old desktops. When staff members found out the “new” computers couldn't even play audio CDs, they were sure they'd been sold out.

In retrospect, for an additional 10% on the original purchase, I could have gotten everyone an actual thin-client device, such as the type sold by the LTSP developers at www.disklessworkstations.com. If that number would have been 15% more than the original, they could have had thin-client devices with LCD displays. Never underestimate the power of fancy new toys to win over hearts.

As I mentioned earlier, our four-year transition to thin clients has been a bumpy road. In the end, OSS was powerful enough to overcome even my shortsightedness. Our staff and our student body love Linux. We've expanded our number of thin clients from the original 75 to a couple hundred. This year, we introduced our first 24-station thin-client lab in the middle school. In fact, this year was the first year I needed all three of those original servers. I'm ashamed to admit that for three years, I had two brand-new servers unplugged. Thankfully, that's no longer the case.

The Linux Foundation now has a travel fund for developers who need to attend technical conferences but can't make it on their own. The fund is open to “elite community developers with a proven track record of open-source development achievements and who don't otherwise have access to funding for attending technical events”. In other words, it's “for the rock stars of the open-source world”, as Jim Zemlin, Executive Director of the Linux Foundation, puts it.

The fund covers travel to “LF Collaboration Summits, the Linux Foundation's Japan Symposiums, the Kernel Summit, Ottawa Linux Symposium, Linux.conf.au, desktop conferences, such as Guadec and Akademy, and other technical conferences where true collaboration takes place”. More pointedly, it “does not offer sponsorships to tradeshows or nontechnical conferences”.

If this is for you, or if you wish it were, visit www.linux-foundation.org/en/Travelfund.