Understanding and Replacing Microsoft Exchange

Linux by itself provides a formidable set of internet applications for mainframes, which have always needed them. IBM's eServer strategy seemed incomplete without a robust set of internet tools, which it promised to provide to all of its brands. Near the end of calendar year 2000, IBM demonstrated it could host a thousand instances of Linux on a single S/390 mainframe.

Figure 1. One mainframe supports thousands of Linux instances.

Even so, IBM realized that web servers and GNU applications didn't provide a complete value proposition. IBM needed an application that made Linux a host that reached further into mainstream computing. So they made a call on us.

In April 2001, when the sales manager for IBM's zSeries, the new name for the S/390, visited our company, Bynari, Inc., I did not understand his interest. After his visit, I understood it perfectly. Bynari, Inc. became IBM's first Linux Influencer Partner.

Initially, people knew us for making Linux and UNIX clients talk to Microsoft Exchange servers. Looking to broaden our market, I found the “Exchange Replacement HOWTO” by Johnson and Mead (www.bynari.net/whitepapers_howto.html). Using their work as a guide, we built a server for our Linux client and Microsoft Outlook. Our server and its Outlook Configuration Guide caught on with the reseller channel.

We didn't know anyone at IBM when we ported our server code to a Linux instance running on an S/390 Multiprise 3000. Jimmy Lee, then with Equant, provided the resources to see if we could do it. Gary Ernst of Equant configured the S/390 instance of Linux and provided assistance in getting our server to work.

As long as Microsoft Outlook had an Internet Mail Only mode and provided peer-to-peer folder sharing, we had a product that allowed UNIX and Outlook clients to schedule meetings and delegate calendar tasks. Our server scaled nicely, and we mimicked the Exchange global address list (GAL) while providing views of users' free/busy time and a decent administrative interface.

But then Microsoft released Office XP and made major changes in Outlook. Suddenly, our products needed server-side calendaring. We feared the growing appetite of IBM enterprise customers for a low-cost server solution for Outlook might wane. Therefore, we needed peer-to-peer calendar sharing in Outlook's Internet Mail Only mode, or we needed something on the server side.

With the January 2001 LinuxWorld Conference & Expo approaching, IBM continued marketing our server, somewhat blind to the needs of Outlook XP users. I knew we had to do something and do it fast or lose IBM's trust. At that time, the person who best understood the market problem was Roger Luca of Mainline Information Systems.

Fortunately, Roger and I developed a good working relationship. With Roger at the head of sales and marketing, Mainline became the largest reseller of IBM mainframes. They also were our biggest supporter outside IBM. Roger provided us with hardware resources to help us build server-side calendaring into our product, as well as with people to support us if we ran into hardware-related problems.

Imagine having completed an exhaustive year of development. You have trips scheduled for the holidays and other plans. Then you get a call on your cell phone; your boss asks you to attend an important meeting. That's what happened in our shop. I could hear the dread in the developers' voices when they answered my call.

We met on November 7, 2001, to see if we could deliver a server-side calendar solution by Christmas. Mainline had several sales pending, and they needed that functionality. Two of my people agreed to work with me to get the solution.

As the senior developer, I provided the project framework. In theory, a project has a variety of phases and processes. To shorten the project's life, I instituted a three-step approach that called for research, invention and execution. I gave each phase a milestone. So instead of starting with code, we started browsing the Internet and any books we could find.

Following a week of intensive research, we discovered our challenges. We had to figure out how Microsoft's DCE-RFC protocol stored and moved calendar events around. We had to interpret the stored data, provide a format that we could store on an IMAP server and then forward the data to an Outlook client in its familiar schema. We also had to provide access control over those information stores to allow a user to appoint and delegate control over his or her calendar to other users with varying permission levels.

We spent another intense week researching and discovered a consensus. Every expert, newsgroup, Outlook specialist and company that tried said one could not create a Microsoft Outlook calendar store on an IMAP server. Here's how we did it.

First, we observed that Outlook did around 95% of the work in the Microsoft Exchange model. Exchange, as its name implies, transfers data between or among Outlook users. Even so, Exchange controlled the transport protocol and that presented a problem. People use the term MAPI to describe the Exchange protocol for transferring data among Outlook users. I don't see MAPI as being the appropriate term.

When intercepting Outlook messages, we discovered large arrays of binary data instead of text. I recognized the data but didn't realize where I had seen it before.

Outlook runs in two different modes: Corporate Workgroup mode and Internet Mail Only mode. In the Corporate Workgroup mode, Microsoft turns on all of Outlook's highly regarded features. In the Internet Mail Only mode, Microsoft uses a completely different and undocumented application program interface (API) with a limited feature set. Without an Exchange server, Outlook doesn't function in the Corporate Workgroup mode at all and has a limited set of features.

In Outlook's Workgroup mode, or when connected to an Exchange server, its data moves around in binary form. That binary data becomes impossible to recognize by an uninformed observer.

While researching, I found a developers' site on SourceForge.net that was porting Open DCE to Linux. I e-mailed one of the developers who wrote back and mentioned that the Open Group contributed the code.

I went to the Open Group's web site, searched the archives and found an old article mentioning that Microsoft had licensed DCE. We downloaded the Open DCE code and, using the engine, shook hands with Outlook and then Exchange. We knew more about the transport protocol as a result. We also understood the presence of binary data streams.

What we discovered is Microsoft uses the distributed computing environment (DCE) as its transport when using Exchange and Outlook in the Corporate Workgroup mode. Microsoft provides a programming interface on top of DCE, which it calls MAPI. Still, underneath MAPI exists an open standards-based protocol (DCE), which Microsoft bought from the Open Group and modified.

One of the default functions in DCE automatically translates ASCII text into binary objects. Microsoft leaves the binary object undocumented. So most of the MAPI properties programmers tag wind up as binary code they would not recognize. To make matters a little more complex, Microsoft embeds the binary property code in a large array of null binary data, thus hiding it.

We began to understand the transport, but we realized that Outlook sent MIME attachments to other Outlook clients. Those attachments did not transform themselves into binary data. We concluded that Outlook also used encapsulation to pass attachments around, which led us to the TNEF object.

Microsoft Exchange uses a number of programs it calls “service providers” that Linux users might call dæmons. Exchange service providers handle objects, which have state and behavior.

Transport neutral encapsulation format (TNEF) is a method to pass ASCII text, other files and objects, along with binary message data. The binary message data makes up the bulk of each TNEF object. TNEF encapsulates MAPI properties into a binary stream that accompanies a message through transports and gateways. Outlook can decode the encapsulation to retrieve all the properties of the original message. The TNEF object hides in MIME as an attachment.

When we found the properties, which created calendar events, we built a TNEF encoder and soon began sending calendar events to and from Outlook clients with SMTP. We immediately recognized that we could use internet transport protocols and turn on Microsoft's Corporate Workgroup mode without MAPI. We knew we had arrived when we saw Microsoft Knowledge Base Article Q197204, which says that Microsoft does not support our transport protocol in the Workgroup mode.

With our primary goal being server-side calendaring, we needed to create a message store to hold our Outlook client objects. As we used an IMAP server, we needed IMAP support, which Microsoft did not provide in the Workgroup mode. So we had to find a way to add IMAP client support to Outlook. The facilitator for adding functionality to an existing Outlook client involved what most people think of as a plugin.

When Microsoft first released Exchange, Outlook didn't exist. Instead, Microsoft provided a set of Exchange messaging clients for its different Windows operating systems. Microsoft also provided an extensible architecture for those Exchange messaging clients. Client extensions provided developers with a way to change the default behavior of the Exchange client. When Microsoft released Outlook, it continued providing support for the Exchange client extension architecture for compatibility with existing client extension DLLs.

Client extensions allow one to alter the default behavior of the client. Microsoft saw the advantages of an extension as a convenient way to integrate customized features and new behavior directly into the client, instead of having to write a separate MAPI application. We, however, saw extensions as a way to add IMAP client services to Outlook in the Workgroup mode. Using this architecture, we added commands to Outlook menus, custom buttons to the toolbars and the ability to preprocess outgoing and incoming messages with IMAP client services.

Luckily, we had already written client libraries for IMAP when we built our Linux client the previous year. We simply needed to port them to Windows. Our familiarity with the function calls, headers and protocols reduced our overall effort.

Once we built a Microsoft DLL for the client functions, we added it as an Outlook extension. Luckily, it worked the first time we tried it. By choosing the rich text format (RTF) for mail and meeting invitations, our TNEF objects attached themselves to the messages. Because Outlook created the TNEF objects, it exchanged them without any problems.

At this point, we uploaded our messages to our IMAP folders using the Microsoft .pst file as store and swap space. Staying connected to Exchange and using our server for message stores, we noticed compatibility between the two systems. We dragged and dropped objects from the Exchange folders into our IMAP folders. By doing this we discovered that tasks, journal entries, calendar events and so on, all showed up in Outlook as if they arrived from Exchange. The calendar also worked perfectly.

When you look at Exchange and study its components, you find they number only four. The first is an information store or message store. The store holds individual user messages and has an access control list (ACL) engine associated with them. Similar to RFC-compliant IMAP servers, namespace differs according to whether the stores belong to individual users or whether the folders are public. Microsoft uses an Access database for storing message stores. The limitation of Microsoft's Jet Engine technology and the Access MDB file prevents vertical scalability.

Secondly, Exchange has a directory. Microsoft structured their Exchange directory with object classes and attributes. The Exchange directory structure resembles the RFC-compliant LDAP protocol. However, Microsoft added Object Classes and changed the attribute names within those and other classes.

Next, Exchange has a mail transfer agent or MTA. Microsoft's MTA appears similar to the MTA used in an earlier product called Microsoft Mail 3.5. The Microsoft Mail MTA requires connectors or gateways, which rewrite their proprietary mail headers to those that comply with foreign systems, such as Lotus Notes, X-400 and RFC 822 internet mail standards. Unlike sendmail and similar internet MTAs, Exchange's MTA lacks configuration options.

Finally, Exchange has a component called a system attendant. The attendant handles every action taken within Exchange, from sending and receiving e-mail to filling requests for addresses from the Exchange directory. In many ways the system attendant resembles an attempt to provide interprocess communication (IPC), which Microsoft's operating systems lack.

Our Linux server-side solution included similar components to those found in Exchange. The first is the Cyrus IMAP message store. Cyrus stores hold individual user messages and have an ACL engine associated with them. Namespace differs according to whether the stores belong to individual users or whether the folders are public. Cyrus uses the Berkeley Database from Sleepycat Software. Where Microsoft's Jet Engine and Access database technology prevents scaling, Berkeley DB's high performance and scalability support thousands of simultaneous users working on databases as large as 256 terabytes.

Secondly, Linux has a directory. While Microsoft structured their Exchange directory to resemble the Lightweight Directory Access Protocol (LDAP), the Linux solution uses OpenLDAP software, an open-source implementation of LDAP. To accommodate Outlook clients, we added the Exchange object classes and their noncompliant attribute names. We indexed the Microsoft-based distinguished names and created a high-performance global address list.

Like Exchange, the Linux solution has an MTA that can be managed and configured internally and doesn't need external connectors. The University of Cambridge developed the Linux MTA we use, called Exim. Exim has numerous configuration options, including file lookups, local delivery and regular expression support. In the context of the Linux MTA, users provide regular expressions to filter content coming in and going out.

In the “Exchange Replacement HOWTO”, Johnson and Mead leave the tasks of adding server-side messaging and the administrative console to the next generation of Linux developers. In this article, we explain how one could transform Exchange transports and message stores. We accomplish this in two steps. First, we capture Outlook messages and decode their TNEF objects. Secondly, we use the Exchange client extension architecture to add IMAP functionality to Outlook in its Corporate Workgroup mode.

These two steps can allow a programmer or a seasoned administrator to create an alternative service provider for Outlook and serve a number of conventional mail clients. Linux mail servers do not discriminate based on the platform one uses. One can use Netscape Mail, Outlook Express, Ximian Evolution, mutt or Pine, to mention a few of the available MUA.

Highly scalable Linux components, such as Cyrus IMAP, OpenLDAP and Exim, can replace dozens of Exchange servers on a single Intel platform. The layers of interfaces and outdated DCE components used by Exchange do not hinder Linux. With Linux on the zSeries mainframe, we can replace hundreds of Exchange servers.

If you're looking for a graphical administrative console, projects such as PHP Cyrus tools, cyrus_imap-sql, Webmin and Replex can make administration of the server a simple task.

In general, few people would consider replacing Exchange with Linux an easy task. In spite of that, our development team proved that it could be done. Hopefully, we have taken much of the mystery and intimidation out of the Exchange server.

Resources