The Politics of Porting

At each of the companies where I have worked in the past eight years, I have introduced Linux in some capacity. It often seemed the obvious choice, but what was obvious to me and other colleagues was not obvious to everyone. Some people needed persuading before they would accept what to them was an unconventional solution. In most cases, benchmarks and cost factors were enough to tip the balance. Novelty and openness to new ideas were sufficient for others. But, for a stubborn minority, nothing could persuade them. Although they participated in the debate, it appeared that for them discussion was simply a tactic to avoid implementation.

Faced with such opposition, arguments are pointless and demonstrations are a waste of time. In those situations, other tactics are called for. Here I describe how I persuaded my employer to port the company's flagship application to Linux literally before the opposition in management knew what was happening.

In 1998, I was UNIX and Oracle administrator for a company called Constellar Ltd. They specialised in data migration and Enterprise Application Integration (EAI) solutions for Global 2000 companies. What does that mean? At its simplest, it might mean transforming the data in a legacy mainframe database for insertion into a newly prepared database. Such tasks are by their nature repetitive and time consuming. They are ideal candidates for automation and that is where the Constellar Hub comes in.

Add a few extra offices in different parts of the world, more databases for customer data, warehouse inventories and order tracking and imagine how these assorted data stores and applications could be made to work together seamlessly. Throw in a live environment and a company that relies on accurate available data, and you begin to understand how complex the task could become.

The main application was a client server called the Constellar Hub, which was designed to extract data from disparate sources, processing, weeding and integrating it on the fly before writing or streaming the transformed data to its destination.

The server relied on an Oracle database to store data in transit, as well as the metadata in which the business rules were held, data types were defined and the relationship between the multifarious data sources and destinations were stored.

The combined transient data and metadata were held in upward of 200 database tables with many triggers and constraints to enforce the integrity of the data as it flowed through the bowels of the application. The engine of the application consisted primarily of C and SQL code, but an API provided the opportunity to extend functionality if required.

On a large project, two or many Hub servers might be required, sometimes located at remote sites around the world, feeding or being fed data through frame relay or less likely the Internet. Typically, an average project would process many gigabytes of data and usually hundreds, thousands or, more rarely, tens of thousands of business rules would be defined.

In many cases, the whole project of which the Hub was only a part—albeit a central part—would cost in the millions and involve dozens of technical and business staff.

Prospective customers would be given a relatively simple demonstration of the tool running on test data, typically on Sun hardware. The server had been ported to various other shades of UNIX, including Dynix, Data General UNIX, AIX and Digital UNIX.

It was also around that time that Windows NT was being promoted as an enterprise-ready OS destined to take over in many areas previously thought to be the exclusive domain of UNIX. Indeed, our vice president of engineering was convinced that the future was Microsoft and told us so. In the future, he said, most if not all our sales would be generated from the Windows NT version of the Hub. As a first move, they already were working on a Windows NT port and had announced the beta version in February 1998. Such a view was understandable, because the cost of a Windows NT box versus the various UNIX platforms tended to highlight only the expense of UNIX solutions. Linux was destined to change all that.

By that time, I already was using Linux on all my home PCs as well as my workplace workstation. I also followed Linux developments fairly closely, especially any that might affect the adoption of Linux in the enterprise workplace. Like many other Linux advocates, I saw the possibility of a computing future that might not be restricted to the products of a single company imposed on a captive customer base and wondered if the Constellar Hub could be ported to Linux.

There were two problems, however. First, the Hub made extensive use of Oracle databases and Oracle-specific features, and it would be a nightmare porting those dependencies to a Linux-friendly equivalent. Second, and this was a more difficult hurdle to clear, a few people in the company were not predisposed to see the opportunities presented by a Linux port.

I vaguely had suggested to one or two members of middle management that perhaps there ought to be a Linux port, but the response I received in return was along the lines that Linux is an amazing product for a bunch of amateurs to have produced, but it wasn't quite ready for the big leagues, and the company couldn't afford the time and expense of porting to it when there was serious business to deal with. I have heard that argument many times since then and have learned it is largely an excuse for not thinking.

Two things changed the nature of the ball game. First, in September 1998, Larry Ellison announced, with excellent timing, that Oracle would release a Linux version of its database server and made it free for download for development use. This had a dual effect. It demonstrated that some very large companies were taking Linux seriously, and at the same time, it made the possibility of a Linux port of the Constellar Hub feasible.

I decided that if I couldn't persuade them to port the Hub I would do it myself, so I copied the Hub source code and set about providing a proof of concept. I figured this either would get me the sack or provide a fait accompli with which those in opposition could not argue.

At this point I should say “Don't do what I did.” It is illegal to copy your employer's source code without permission, and you are likely to find yourself at the wrong end of a lawsuit if it all goes belly up, but sometimes dire circumstances require drastic solutions.

In late November 1998, I checked out a full set of source code and, along with a newly downloaded copy of Oracle 8.0.4, I set up the development environment on my PC at home; using the unedited Makefile from the Solaris development directory, I ran make. Not surprisingly, it all fell in a heap, throwing up pages of compile errors.

I devoted an hour or two most evenings to stepping through the errors and debugging them. Those I didn't understand I presented to Richard Glover, one of the senior UNIX developers at Constellar. He also was the release manager for the various UNIX versions of the Hub, so he was a good person to know.

Most of the problems revolved around the Oracle ProC precompiler, which didn't recognise some Linux-specific directives, such as the include_next statements in the Linux header files. This was resolved by copying the relevant files, stripping the #include_next statements out and placing the location of our customised version of these files first in the include path.

There also were assorted issues with library paths. These were trivial and related solely to the setup on my home PC. They were solved by adding entries to the LD_LIBRARY_PATH shell variable exactly as it would be with Solaris. Some macro definitions—ULONG_MAX, INT_MIN, INT_MAX, LONG_MAX—were missing from the copy of Red Hat that I was using at the time. I copied them from the relevant Solaris headers as a temporary fix.

I left some minor problems unresolved because my resources were somewhat limited. For instance, the Hub was required to connect to many and varied sources of data, and some of these required the use of IBM MQ series libraries, which I did not have available at the time. In any case, I did not have the facilities to test such functionality at home. I edited the main Makefile to disable the linking of these libraries and accepted that my version of the Hub would be a lightweight, more dynamic kind of Hub. If it compiled at all, I would be happy.

Some differences found in the Linux version of various utilities were sorted out by editing scripts to reflect the proper invocation of particular utilities. I worked my way around some of them, and I ignored others to save time. Richard would later track down the source of all these problems and make allowances for them in the platform-specific sections of various build scripts in the official build environment at work. Those utilities that required changes to the standard build scripts included various compiler and linker flags. We were obliged to use the Sun compiler for the official Solaris build, so moving to Linux and GCC required a bit of translation with regard to the flags and switches. The use of df, the shell built-in echo command, ftp, ldd, mknod, nm, ps and lex/yacc (used for the proprietary Transformation Definition Language) all required changes when invoked.

As Richard pointed out a few weeks later, the set -o posix option in bash resolved nearly all the shell script differences, and in any case, the command-line differences were quite trivial and the solution obvious. In some cases, it was simply a case of providing Berkeley versions of the utilities rather than the System V ones, or vice versa. The situation would be different today, because many of the GNU utilities standard on Linux are now included as standard with Solaris, so porting between the two has become even simpler.

In all, it probably took about two man-days to get a rough-and-ready port with a binary that at least looked like an executable.

I had a binary, now I needed to test it. I checked out a copy of the suite of test scripts and data used in the daily smoke test at work. After setting the environment variables for the smoke test, I set the test harness running.

The usual smoke test consisted of more than 200 transactions, and on the other platforms would take anything from six hours for UNIX versions to more than 24 hours for the Windows NT version—this is not good when testing a daily build. I ran the test on my home-built Linux port, and within a minute or two I had errors scrolling up the screen. Disaster. I checked the error logs, but they were no help. For some reason, the logs didn't provide any diagnostics for the first few dozen transactions, and this confused me.

It took a while to figure out that the reason there were no diagnostics in the error log for the initial set of transactions was simply because the Linux Hub already had run the first few dozen transactions successfully! This initially was hard to believe, because the same set of transactions took considerably longer to run on most of the other platforms to which the Hub was ported. This meant that my rough homebrew Linux version of the Hub was, without tuning, already faster on my home PC than some of the other UNIX ports, and several times faster than the Windows NT port. Even allowing for the differences between the hardware specifications on which these varied ports ran, it looked good.

I felt sure that if these results could be reproduced in the workplace, there would have to be an official Linux port.

My suggestion that the Hub be ported to Linux previously had been met with a lukewarm response from the middle management with whom I had broached the subject, so I decided to adopt a different tactic this time around.

Several others had expressed an interest in finding out more about Linux, so a couple of months previously we had formed a company Linux User Group with a mail list on the company mail server and a meeting to talk tech once a week. I let the other members—Ngozi Mayo, Anthony Durity and Olusola Ojeje—know what I had been up to and they provided useful feedback and advice.

After some thought, I realised that I had no idea who, if anyone, in senior management would lend support to the idea of a Linux port. I decided to e-mail all of them and the members of the board and hope that someone among them would take an interest.

I planned to announce the alpha port in the style of a press release and see if it would get me the sack or not. I wrote a draft release with a number of bullet points and submitted it for review to the members of the LUG. I made corrections, simplified the announcement and toned down the style as suggested, and then I sent off the revised release and waited for a response.

On February 26, 1999, I pointed out in my press release that the port had cost the company nothing, that the Linux port ran the smoke test several times faster than the Windows NT version, and at least as fast as the other UNIX ports, that there was an increasing trend toward Linux ports of enterprise software and that now, 1999, was the time to get on board. I requested that the company supply hardware to enable an official company Linux port and that it be defined as a tier-three platform to give it official status. Quotes from IDC, highlighting a 212% increase in Linux sales that year and 17% share of the enterprise market, helped shock the complacent into life. For many of the e-mail recipients these facts and figures were a revelation.

The e-mail generated an immediate response and much discussion. Some people sent congratulations and others speculated about possible marketing angles. One senior Sales Consulting Manager questioned the need for a Linux port at all, pointing out that in the previous two years not one customer had requested such a thing. We would all still be riding round in horse and carriage if that attitude prevailed. Someone suggested that “Enterprise deals are closed on enterprise platforms” without then explaining why Windows NT was an “Enterprise” platform or how Linux was not. Others didn't quite understand what Linux was and the LUG did their best to explain.

The final word went to our CEO Nic Birtles who proved to be both insightful and encouraging. In his pithy e-mail response to the ongoing debate, he simply asked what had to be done to be able to announce a Linux version of the HUB. Debate over.

We got our hardware for a dedicated Linux development machine and the Linux port became an official tier-three platform. Having produced a rough-and-ready port merely to prove a point, the task of producing a robust Linux binary was given to Richard Glover.

Everything went quiet for a while, and then one day in early May 1999, Richard quietly mentioned in passing that the Linux port had a 100% pass rate the previous night and that it had taken only five hours to complete. Good news. How long did the Windows NT version take? Fifteen hours on a good night, although some nights the process had to be killed because there weren't enough hours in a day.

Another in-house press release for management and senior staff was prepared and sent to announce the beta release and the amazing test results. It garnered a wide range of comments both positive and negative. In the latter category, our Director of European Marketing flirted with both sides in her e-mail: “Great job” but then spoiled it by suggesting that we didn't want to be a trailblazer in the Linux “space”, no matter that the Linux port ran faster than any of the other ports and three times faster than the Windows NT version. Fortunately, most reactions were much more positive and the most important one, the CEO's, was both congratulatory and helpful. He suggested that the Linux port could now become a tier-two platform alongside Windows NT and the other UNIXes. Only Solaris and AIX were tier-one.

By July 1999, the Linux port was added to the official application CD.

By the time we had an official Linux release of the Constellar Hub on the CD, 25% of the UK workforce had joined the Linux User Group, and the Linux objectors were a small and subdued minority keeping their thoughts to themselves.

The Constellar Hub subsequently was bought by Data Mirror who continues to offer and support the Linux version, which currently is at version 3.8. Thanks to a bit of guerilla porting and a CEO who knew a bargain when he saw one, this Linux application finally found its way out into the open.

So why is it so difficult to convince some people in the face of such revealing benchmarks? The cost of the port was close to zero, the performance was superior to almost all the other ports and the resulting Linux Hub was rock solid.

It think the answer to that question hinges around the personality of the individuals concerned. There seems to be two types of people involved in this debate, and although each may be amenable to argument, their instinct leaves them predisposed to favour particular forms of software solutions.

One group prefers a clear and rigid hierarchy. They want to know who they are answerable to and who is answerable to them. They want one organisation providing one solution and would like to dispense with the alternatives. They respect the dominant player, whoever it is. They feel at ease with conformity and would prefer it if you would conform too.

On the other hand, the second group values freedom of choice. They tend not to take anything for granted, want to see under the bonnet when considering a product and always are ready to look at alternatives. They certainly don't want to be told what to do but are amenable to argument and debate.

These two groups might be characterised as dogs and cats. The dogs listen to their masters' voice and defend their own territory and that of their master. They know their place and yours and would prefer to keep it that way.

The second group are cats. They don't recognise a master, never mind his voice. There are no limits to their territory, and they range widely. They are curious and seek out new pastures and are at ease with the unfamiliar.

Next time you are having problems introducing Linux into the workplace, the resistance you encounter might have nothing to do with the strengths and weaknesses of the software under discussion. It simply might be that you are a cool cat trying to persuade an old dog to learn new tricks.