Remote networking with high-frequency (HF) radio and Dan Bernstein's qmail.
Editors' Note: The following is the complete version of the article by the same title that appears in the November 2002 issue of Linux Journal.
Deep inside the warm green interior of Guinea, centered in the frontal lobe of West Africa, field personnel in the widely scattered village-towns of Dabola, Kissidougou and Nzerekore now enjoy access to regular internet e-mail, directly from their desktops. Here we have bridged the digital divide, and there isn't a telephone line or satellite dish in sight. Instead we are moving the mail over distances of hundreds of miles—over jungled mountains and high palmy savannahs—through wavelengths of high-frequency (HF) radio. Our project is called Radio E-mail, and here is its story.
The Republic of Guinea is a cashew-shaped nation with Atlantic view property, 10 degrees north of the equator in west West Africa. It is a beautiful and resource-rich nation, with an total land area about the size of Oregon. As far as African countries go, Guinea is a calm pocket of peace and stability, and it generally doesn't attract a lot of attention from beyond its own borders.
But Guinea has quietly played a heroic role in the theater of world events in recent years. It provides a safe and welcome refuge for as many as half a million people displaced by brutal wars and civil upheavals in the neighboring countries of Sierra Leone and Liberia.
The International Rescue Committee (IRC) has one of their largest operations in Guinea, providing services and support to a population of up to 200,000 refugees quartered in many camps established throughout the country. I became involved with IRC when my wife accepted the position of Country Director for the program in the summer of 2001. Soon we were traveling on an inspection tour of the camps, making the long road-trip to visit the program's three field offices up-country. Our first destination was a distant and dusty village, delightfully named Kissidougou—frequently called Kissi in the local vernacular.
Traveling outside the capital city of Conakry, one immediately finds that Guinea has little infrastructure, especially in the way of electrical grid and telecommunication systems—to say nothing of Starbucks and broadband access to the internet. So IRC field offices must provide their own infrastructure: diesel generators for electricity and high-frequency (HF), two-way radio sets to communicate with other offices and mobile units, up to hundreds of miles apart.
Expecting this isolation and general lack of connectivity, I was quite astonished when we arrived in Kissi. Here I found the radio operator using his equipment to make a binary file transfer from his desktop PC to another field office, wirelessly!
This capability surprised and intrigued me. On top of the operator's radio set, connected to the serial port of his PC, sat a dingy black box simply labeled 9002 HF Data Modem. I noticed the operator used a proprietary, MS-DOS program to make his file transfers, but I immediately began wondering: if this device is truly some kind of modem, moving binary data over the ether of radio, why couldn't we set it up with Linux and network with PPP connections as well?
After a little research and testing, I soon confirmed this equipment could indeed form the basis of a wide area network, providing full access to internet e-mail via the Conakry office for all personnel in each of the three field offices. Moreover, since IRC owned most of the equipment already—and since we would be using Linux and other freely available, open-source software—the system could be implemented at negligible cost, with no increase in operating expenses. For the price of some network cards and category 5 cable, we could connect our bush offices to the rest of the world. I developed a design and specification for the system, and the project we call Radio E-mail has been continuously operational since January 2002.
If you have been making the move to wireless lately, most likely you are working with the microwave, high bandwidth frequencies of 802.11b. If so, you know that on a clear day you maybe can get a line-of-sight connection out 10 miles or so. That surely won't do for the vast distances and wild terrain we need to cover in rural Africa.
HF radio is another animal. Its longer waves roll out across the landscape, reflecting off the ionosphere to follow the curvature of the earth. This gives HF signals a range in the hundreds of miles. From Conakry to Nzerekore—IRC Guinea's most distant field office—HF easily covers a straight-line distance of over 375 miles (600 kilometers.) The road that sometimes connects these two points is, of course, much longer—a gut-slamming, spine-jamming, two-day punishment for the damned.
So the great advantage of HF is it can go the distance, leaping the obstacles in its path with aplomb. Now for the bad news: where HF wins the wireless game in range, it loses its pants in data capacity. If 802.11b is considered broadband, think of HF as slim-to-none-band. The radio modems we are using here are speced at an anorexic 2400 baud!
And wait, it gets worse. Two-way radio is the classic half-duplex medium of communication; that is, you are either transmitting—push to talk—or receiving, not both at the same time. This, plus the robust error-checking protocols implemented by the modem hardware itself, means the actual link experience is more on the order of 300 baud. Does anyone remember 300 baud? Unless you measure your patience with radio-carbon, your dreams of remote login sessions will be dashed and splattered. As for on-line browsing, chat, video-conferencing and the like, well, best to not even think about it.
Yet for classic store-and-forward applications like text-based e-mail, the bandwidth limitation of HF radio is workable. We simply need to pay close attention to our configuration and try to optimize as much as possible. With HF radio, every packet is precious.
Given these capabilities and limitations of HF, our design strategy for the project uses radio modems in a topology among field offices, as shown in the reference network of Figure 1. In Conakry we have a full-time internet gateway on the host coyah. The radio modem on the host congo serves as the dial-in hub for each of the three field offices. We establish periodic PPP links between the field and Conakry and use our choice of carefully selected client-server protocols over TCP/IP. Although we might have implemented e-mail to the field using UUCP, our TCP/IP system is easier to configure and integrate with the existing network and e-mail system. This setup is also readily converted to use faster telecommunications linkups, such as through landlines or satellites. If and when these become available in the field (and as the budget may support them), the IRC network is ready to simply plug in.
Figure 1. Radio Network Topology
For now, though, we don't have the luxury of dedicating our radios to full-time connections for data communications. In fact, voice communication continues to be the central purpose of the radio equipment, providing an essential lifeline for the safety and security of field office and mobile unit personnel. Our implementation and procedures must supplement this lifeline, not impair it. Since data sessions over the radio block voice calls at each end of the link, we have policies and configurations to hold connections to less than 15 minutes per session. Otherwise, we keep the radios free for voice contact. In the field, radio operators have procedural protocols for making their periodic dial-up connections with Conakry for e-mail exchange at regular intervals throughout the day. A schedule is established for these transfers, but radio operators still maintain the latitude to adjust the schedule and break sessions as necessary to accommodate urgent voice communications. For these reasons, all dial-ups are necessarily under the control of the radio operators, rather than set up with cron jobs or diald, as we might do in other, better connected circumstances.
The equipment used in this project is made by Codan, an Australian manufacturer of two-way radio hardware. Although there are other manufacturers—including Motorola, Kenwood and Yaesu—Codan seems to be the radio of choice for international aid organizations in this part of the world. Wherever expats gather, you will see their big white Landcruisers with official markings and the substantial Codan whip antennas bolted conspicuously to every front bumper. While emphatically robust and utilitarian, the symbolic authoritative value of these thick black whips—such as when crossing the many and potentially intimidating military checkpoints—is certainly their most dominant feature.
The modem used in this project is the 9002 model. (Believe it or not, Codan also makes a 2400 baud fax modem, model 9001. We found a used one on the local market for $500.) These modems are equipped with a basic Hayes-like AT command set, so they are easy to configure, operate and trouble-shoot with any telecommunications application available for Linux, such as Minicom or Kermit.
There are some significant differences between this modem and that Sportster you may have jacked to your phone line, however. For one thing, the Codan unit is actually built as DTE (data terminal equipment), rather than DCE (data communication equipment). To connect it to your serial port, you need a special DB-9 null modem cable, wired exactly as diagrammed in David Lawler's “Text-Terminal-HOWTO”. Since not all cables described as null modem are wired the same, this detail is crucial. Also, the AT commands are not as extensive as, and vary slightly from, those you would expect to find on your standard USR model. And, of course, this modem moves data much more slowly than you can possibly imagine. The dripping of a famous-brand ketchup comes to mind, after refrigeration.
Returning to the network setup, the Conakry radio host congo—aliased as radiohub—is configured as a PPP server, ready to accept dial-in connections over the radio from offices in the field. As with a conventional telephone dial-in configuration, we use mgetty to watch the serial line, initialize the modem, wait for incoming calls, answer the RING and fire off the PPP daemon.
The 9002 works pretty much as mgetty expects, and the sample mgetty.config file (Listing 1) shows how to do it. The comments in the listing describe the key settings. The most important line in this file is the modem initialization string given by the init-chat parameter. Here we start by setting the modem to a known state with the factory defaults (AT&F) and then get defensively redundant: set all command, local and remote echo off (E0 L0 R0); ignore carrier (X0); use hardware flow control (&K3); auto-answer off (S0=0 &A=0); and reset the station address (&I=nnnn, where nnnn is like a phone number, the unique identifier that other radios call to reach this particular station.)
After mgetty answers the RING and gets a CONNECT back from the the other end, then what? This is controlled by mgetty's login.conf file (Listing 2). It is common in dial-in systems to let mgetty look for incoming PPP packets and then start the PPP daemon automagically, typically using CHAP in the link negotiation process for authentication. Such a configuration is handled by the line starting with the magic string /AutoPPP/.
In our experience, though, the high latency of the radio link makes an /AutoPPP/ configuration slow to kick in. So what we do instead will be shocking: we dispense with conventional authentication entirely! In our configuration, the login name provided by the chat script of the incoming connection is used to start the PPP daemon directly. When mgetty matches a login name with an entry in the first field of the login.conf file, such as Pklogin, it then runs the program specified in the fourth field, such as /usr/local/sbin/pppd.login.kenya. In essence, then, the login name used by the remote system serves to control access. Note that Pklogin is bogus as a system account, and you can be sure I haven't told you the string we are really using! (Note also that we have an implicit system of human authentication even before a connection is made, when operators agree by voice to lock on a specific channel before starting the link.)
When mgetty gets a login name listed in the login.conf file, it then passes control to the corresponding start-up script, such as pppd.login.kenya (Listing 3). This in turn starts the PPP daemon, using an options file for the particular remote host, such as options.kenya (Listing 4).
Here you may want to spare yourself bitter trails of sweat, tears and other emotional excretions, and pay attention to the lcp-restart and ipcp-restart options. These parameters give the time, in seconds, that pppd will wait to receive a reply to that particular phase of PPP negotiation before trying again. The default value of these parameters is three seconds, which is generally more than adequate when using regular telecommunications and is probably why you have never encountered these options before now.
Over the radio, though, if these restart defaults are not extended, you'll only snag yourself a useless snarling hairball. Here's what happens. As pppd starts up, each peer begins a negotiation process with the other to agree on all parameters for the connection. During these initial discussions, when one end of the link doesn't hear back from the other within its restart interval, pppd repeats the transmission. In the meantime, though, the remote end has received the original transmission and sends back its reply. The local end gets this response back to its first transmission, thinking it has a proper response to its second, and so proceeds to the next step of negotiation. But then the local server gets what is now the unexpected response back from its second transmission, and the negotiations break down in unresolved chaos.
By extending the lcp-restart and ipcp-restart parameters, you can delay sending out the repeat transmissions for a sufficient amount of time for the peers at each end to receive a response. Almost always the remote end will receive and reply to every original phase negotiation request, but the round-trip time can take several seconds. Here we have configured a generous 16 second delay and never have any more problems.
Turning our attention up-country, each remote host in the field is configured to dial-up the radio server in Conakry, using PPP configurations corresponding to those shown in Listings 5 - 8 for the kenya host in Kissidougou. These reflect the key configuration issues as already discussed for the server, such as the crucial modem initialization string (Listing 8) and the essential options for pppd (Listing 6).
After all our testing, failures and ordeals of travel, it was a happy and amazing day when we finally got our first link across radio waves spreading invisibly through the Guinean atmosphere. From kenya we actually made ssh connections with congo, just to express our exuberance over talk sessions with the radio operators watching their terminal screen in Conakry: “Greetings from Kissi!”
With the configuration details finally worked out, we found the PPP link, although slow, would come up reliably and remain stable at all times of the day, even over channels which otherwise sounded fuzzy. Of course, all radios and antennas should be tuned for their best performance. But once you establish a link, it is reassuring to find the radio modems are quite capable of maintaining it, even when conditions are less than optimal. Because somehow conditions always have a way of being less than optimal.
Here in West Africa, it seems nearly everything just barely hangs together with bent nails and spit. While this is a tribute to the resourcefulness and ingenuity of Guineans in the face of scarcity, darn if this doesn't make a rough neighborhood for hardware and networks. I have found power cords hacked apart and spliced together with scotch tape, motherboards rigged in place with twist ties and wedges of cardboard. It is forever hot, dusty, buggy and humid. Generators go out, plugs get kicked, lightning strikes, nothing is grounded. In short, we face the harshest of environments in trying to implement high-tech reliably.
For these reasons alone, we would probably use Dan Bernstein's outstandingly robust qmail tools to move the mail. qmail is designed with particular attention to reliability, making sure messages are safely written to disk at each stage of processing. If a message is handled by qmail—especially when delivered to a Maildir, the special mailbox directory format developed by Bernstein—you can be sure that message is well and truly delivered. Otherwise, the mail always remains safely queued on disk for delivery at a later time. This feature is essential for the frequent power and network disruptions we face in the African environment.
Going beyond qmail's core reliability, Bernstein also has developed a number of special purpose tools and applications perfectly suited to the Radio E-mail project. Central among these, qmail includes a QMTP server, implementing Bernstein's own Quick Mail Transport Protocol. QMTP is accessible from the serialmail package, a supplemental suite of programs designed specifically by Bernstein for moving mail over slow dial-up connections. And our connections would certainly qualify as slow! By using qmail, QMTP and serialmail—controlling the entire chain of e-mail delivery within our own network—we have an optimal solution for moving mail as efficiently and reliably as possible over HF radio.
As shown on the reference network in Figure 1, qmail runs on no less than five hosts: the central mail server coyah, the radio hub server congo and each of the three field office hosts. That's a lot of qmail! In fact, our production network adds a sixth qmail installation on a separate gateway/firewall system. This is configured with what is known as a mini-qmail system, using Bernstein's Quick Mail Queuing Protocol (QMQP) to queue incoming mail through the firewall directly to the mailhub host coyah. Table 1 shows the qmail components installed on each of these different systems.
gateway/firewall | mailhub | radiohub | fieldhost | |
qmail-smtpd | Y | Y | Y | Y |
qmail-qmqpc | Y | |||
qmail-qmqpd | Y | |||
qmail-qmtpd | Y | Y | ||
qmail-pop3d | Y | Y | ||
fastforward | Y | Y | ||
serialmail | Y | Y | ||
daemontools | Y | Y | Y | Y |
ucspi-tcpi | Y | Y | Y | Y |
djbdns | Y | Y | Y | |
qturn | Y |
With so many qmail servers to install and configure, you intimately become acquainted with all things qmail; it is a wonderfully capable and flexible system. See the Resources for some pointers, and prepare to spend some time with The Big Qmail Picture, Bernstein's FAQ documentation and the PIC files. All installations follow Dave Sill's “Life with qmail” instructions to the letter, the indispensable guide for setting up production qmail systems. The discussion that follows here assumes familiarity with qmail and its ways: its control files, the special user alias the qmail virtual domain mechanism, dot-qmail delivery instructions and Maildir mailbox delivery.
For the e-mail examples given in this article, we'll use the nonexistent refugee.ngo domain to represent IRC-Guinea's actual domain of irc.org.gn. Each user on the IRC-Guinea network, whether in Conakry or a field office, is assigned an e-mail address of the form firstname.lastname@refugee.ngo. All mail addressed to the domain refugee.ngo is processed by the mailhub host and routed as necessary from there. No sender needs to know anything about a user's particular location of assignment or otherwise use any special form of addressing.
To show how this can be done with qmail, let's first consider a message arriving (or originating) in Conakry for rhonda.rabbit@refugee.ngo, a user who happens to be posted to the field in Kissidougou. The key processing steps are sketched in Figure 2 and described in more detail below.
Figure 2. Radio E-mail to Kissidougou
On the mailhub server coyah, qmail finds the domain refugee.ngo in its locals control file and attempts to deliver the message locally, that is, to a user account on the host coyah. First it checks for a user named rhonda.rabbit in the system /etc/passwd database. When qmail fails to find the user there, it turns the message over to the special alias user, processing it in accordance with the instructions in /var/qmail/alias/.qmail-default. Here we have installed the fastforward command, directing a lookup in a central database built from the file /etc/aliases. Now an entry is found for rhonda.rabbit, and the e-mail message is returned to qmail with a new envelope address of rhonda.rabbit@kissidougou.refugee.ngo.
Still on coyah, qmail now does not find kissidougou.refugee.ngo in either its locals or virtualdomains control files, so it queues the message for remote delivery. But before qmail tries to find an MX record with a name server lookup for the kissidougou.refugee.ngo domain, which does not exist, it checks in the smtproutes control file and finds a match. In this way all mail destined for kissidougou.refugee.ngo, as well as for all other remote offices, is relayed directly to the radiohub host. (The smtproutes mechanism helps us avoid the complication of setting up and delegating a number of subdomains and MX records in our local nameserver.)
When the message arrives at the radio hub congo, qmail won't find the domain kissidougou.refugee.ngo in the locals control file. In fact the radiohub may be configured without a locals control file. Instead qmail finds a match in its virtualdomains control file. The matching entry in the file kissidougou.refugee.ngo:qturn-kissidougou instructs qmail to forward all messages addressed to the virtual domain kissidougou.refugee.ngo, prepending the string qturn-kissidougou- to the original envelope address. In the case of our example message then, the address is now rewritten as qturn-kissidougou-rhonda.rabbit@kissidougou.refugee.ngo and delivered locally to user qturn.
qturn is a special user account we have set up on the radio hub congo, with a home directory of /var/qmail/qturn, to receive all mail destined for delivery to remote offices via HF radio modem. Following qmail's dot-qmail conventions, qmail looks in qturn's home directory for local delivery instructions in /var/qmail/qturn/.qmail-kissidougou. Our instructions here tell qmail to deliver to the Maildir mailbox ./kissidougou/.QMAIL.PPP/. (Similarly for messages destined for Dabola, for example, the dot-qmail file ~/.qmail-dabola will instruct delivery to the Maildir mailbox ./dabola/.QMAIL.PPP/.)
qmail now considers the message delivered. It is out of the qmail queuing mechanism, and no further delivery or forwarding attempts are made autonomously.
Instead, we wait for the next PPP connection with the remote host kenya in Kissidougou. Then we can make use of the serialmail package to blast—relatively speaking—all the mail collected in the /var/qmail/qturn/kissidougou/.QMAIL.PPP/ mailbox across the link using QMTP. The command sequence that sends the collected mail over the link is:
# cd /var/qmail/qturn/kissidougou # maildirqmtp .QMAIL.PPP qturn-kissidougou- 10.0.10.243
The first argument to maildirqmtp is the path to the Maildir directory with the mail we want to transfer. The third argument is the IP address of the destination qmail-qmtpd server, in this case the static IP address we have assigned to the Kissi end of the PPP connection.
The purpose of the second argument may be less obvious. It provides the prepended string which should be removed from the envelope address on each message. Recall that the local part of the example message address has been rewritten as qturn-kissidougou-rhonda.rabbit. When maildirqmtp sends this message, the prepended qturn-kissidougou- is stripped off, and the envelope address is written once again as rhonda.rabbit@kissidougou.refugee.ngo.
The message then wings its way over Guinea, lofting through African skies on gossamer currents of HF ether, coming gently alight upon the Kissi radio aerial after a long and spectacular journey. Here the qmail system on kenya may now complete its delivery.
Again qmail first checks its locals control file and does find kissidougou.refugee.ngo among the entries, since here is where we want local delivery of the message. Although there is no user named rhonda.rabbit in kenya's /etc/passwd file, the fastforward lookup in /etc/aliases does find a match and sets up delivery of the message to the user account of rrabbit. The dot-qmail delivery instructions in /home/rrabbit/.qmail—as for all users set up to receive POP3 e-mail—is configured to deliver to the Maildir mailbox named ./.QMAIL.POP/ in the user's home directory.
By a happy coincidence, we have configured qmail's POP3 daemon to deliver mail from Maildirs named .QMAIL.POP in users' home directories. With her laptop connected to the local network in Kissidougou, eagerly awaiting word from the world beyond, Ms. Rabbit is now at last greeted with the joyous chimes of its arrival, “You have mail.” Radio E-mail works: they laughed, they cried, they danced.
Sometime sooner than later, users in Kissidougou will want to try sending their own messages to others. If you have followed along with qmail this far, you may already have some ideas for getting mail back out from the field to any address on the internet. But it turns out there are a couple of tricky parts in sending e-mail from Kissidougou to other refugee.ngo users outside of Kissidougou, whether in Conakry or another field office. Figure 3. considers the case of sending a message from Kissidougou to a user in Conakry, addressed to lois.lion@refugee.ngo.
Figure 3. Radio E-mail from Kissidougou
In Kissidougou as in Conakry, we want to keep the convention of addressing mail to users with firstname.lastname@refugee.ngo, no matter where the recipient is actually assigned. This means that refugee.ngo also must be listed in the locals control file on kenya, so qmail will accept and deliver mail originating on the local network in Kissi and addressed to recipients posted there. This avoids the delay and extra traffic of an unnecessary round trip to the mailhub in Conakry.
The question, then, is how to forward mail addressed to users at refugee.ngo, but who are not found locally. The first part of the solution is to set up fastforward in pass through mode by using the -p switch. With this option, if a user isn't found in the /etc/aliases database on kenya, fastforward exits 0, allowing the message to be processed by further instructions in the alias .qmail-default file.
Which brings us to the second part of the solution. As shown in the diagram, the second line in this file
|forward ${LOCAL}@mailhub.refugee.ngo
uses qmail's forward utility to send any message to a refugee.ngo addressee not found locally, rewriting the domain part of the address as mailhub.refugee.ngo. In our example, the envelope address is rewritten as lois.lion@mailhub.refugee.ngo and returns to qmail further processing.
Again qmail checks the locals control file, and this time finds no entry for mailhub.refugee.ngo. But now there is a matching entry in the virtualdomains control file. This particular entry, with the first field empty, is a form of qmail wildcard expression and has the effect of treating any domain as a virtual domain. In this case the entry
:qrelay-ppp
tells qmail to forward all messages for any domain to the user qrelay, prepending the string qrelay-ppp- to the original address. Now our outbound message will be addressed to qrelay-ppp-lois.lion@mailhub.refugee.ngo. (And any other internet-bound addresses would be rewritten likewise, such as qrelay-ppp-guinix@yahoo.com.)
Similar to the user qturn on the radiohub congo, qrelay is a special user account we set up on all field hosts, such as kenya, to receive all mail destined for outgoing delivery via HF radio modem. The qrelay user has a home directory of /var/qmail/qrelay, and the dot-qmail instructions in its ~/.qmail-ppp tell qmail to deliver to the Maildir mailbox ./.QMAIL.PPP/. Here the messages for all outbound mail will collect until our next PPP link over the radio. Then we will again use QMTP and serialmail, this time in the other direction.
When the link is up, the command sequence that sends the batch of collected mail to the radiohub in Conakry is:
# cd /var/qmail/qrelay # maildirqmtp .QMAIL.PPP qrelay-ppp- 10.0.10.241
As in the previous example, the first argument to maildirqmtp is the path to the Maildir directory containing all outgoing mail. The third argument is the IP address assigned to the radiohub end of the PPP connection. And the second argument is the prepended string to remove from each envelope address. Now when maildirqmtp sends the example message, it will be restored to lois.lion@mailhub.refugee.ngo. (And any e-mail to guinix would be restored to guinix@yahoo.com.)
Bounding wirelessly out of Kissi, following rainbows and hot harmattans, the message gracefully traverses the wilds and dangers of earthly terrain to arrive safely in Conakry on the radiohub congo. And here it won't linger. When qmail doesn't find any matching entry in the locals control file, it immediately queues the message for remote delivery. The entry in the smtproutes control file then has the wildcard effect of relaying the messages destined for any domain back to the mailhub server. This configuration purposely segregates the radiohub host as a server only for HF links, while using the mailhub host for all other e-mail services and administration.
When qmail receives the message on coyah, it now finds an entry in the locals control file for mailhub.refugee.ngo and initiates local delivery. In what by now should be a familiar sequence, fastforward matches lois.lion in the /etc/aliases database and passes the message for delivery to the llion user account. The message is then delivered to llion's POP3 mailbox, the Maildir named ./.QMAIL.POP/. Hungry for word from Ms. Rabbit in the field, Ms. Lion pounces on her mouse—and lo the message is deposited right to her own desktop.
What if lois.lion is reassigned to the field office in Nzerekore? The administrator would make the change in mailhub's /etc/aliases file and run the fastforward version of the newaliases command to update the fastforward database. Now qmail will turn messages for lois.lane around for delivery to lois.lane@nzerekore.refugee.ngo via HF radio, as in the rhonda.rabbit example. In this way Conakry serves as the central hub for all e-mail traffic, and field offices need never make links with one another directly.
Each of the radio e-mail servers in the field run headless, controlled from a simple command-line interface via Telnet session from the operator's desktop PC. The basic interface consists of four commands, usually run in the following sequence:
ppp.start mail.get mail.send ppp.stop
Simple shell scripts perform their respective tasks, each providing the operator with a modest amount of feedback about what is happening at the time. The functions could be further collected into a single command, such as mail.run, but we want to enable the operator to maintain some discretion over radio access, depending on the demands for voice communication. For example, if getting the mail takes more than 15 minutes or so, the operator may stop the session, reestablish voice communications for a security check, then start a separate session a few minutes later to send outbound mail.
The command interface shows that we don't try to get and send mail simultaneously. Rather, first we do one, then the other. This is another accommodation for the anemic, half-duplex bandwidth of the HF radio link. As far as network traffic goes, this link is like a one-lane backroad. More than a little traffic creates a long skinny parking lot.
It should be pretty easy to figure out that the mail.send script is a simple wrapper around the maildirqmtp command described in the second example. What may be less obvious is how we get the mail. That is, how do operators in the field run the maildirqmtp command on the radiohub server in Conakry?
As with most things UNIX, TMTOWTDI. You may choose, for example, to set up ssh for remote execution of the maildirqmtp command shown in the example. Alternatively, you could just go ahead and set up a POP3 server on the radiohub and use fetchmail over the link (with the—qvirtual option to strip the prefix added to each envelope's address), though you would then lose the benefits of QMTP.
For the Radio E-mail project, we decided to take advantage of Dan Bernstein's daemontools and ucspi-tcpi packages, already installed as part of our standard Life with qmail installation. The tools in these packages make it ridiculously easy to set up special purpose servers. Ours is called qturn and is modeled on Bernstein's AutoTURN comments described in the documentation with his serialmail package. With qturn we keep the significant efficiencies of QMTP, while avoiding the time-consuming overhead involved in establishing and authenticating an ssh connection.
The components of the qturn server running on the radiohub are provided in Listings 9 - 11. The principle is simple. The qturn run script (Listing 9) is set up as a daemontools service listening on a specific port of your choosing (here we use 55210). Anytime a connection to the port is made, the qturnd.sh script (Listing 11) is executed. This script reads the $TCPREMOTE environmental variable passed to it by the tcpserver invocation and gets the static IP address we have assigned to the incoming connection. The script then matches this IP address to the Maildir directory used for that field office. The maildirqmtp command is then run, sending the collected mail in the Maildir to the QMTP server on the remote host, while directing status output to filehandles that can be read by the client process.
Now the mail.get script on the client is simple, as is shown in Listing 12. It calls tcpclient to connect to the qturn server created with tcpserver on the radiohub and redirects the stream returned from that process to standard output. That's it! Client/server programming just doesn't get any easier.
As we may have mentioned, the HF radio link is a tad on the slow side. Yet we do manage to move a decent amount of mail with it nonetheless. On an average day, over 300 messages travel the airwaves between Conakry and field offices, over two to three brief connections per office. And as is typical with all internet technologies, every taste stimulates even greater appetite.
Given the limitations inherent in radio e-mail, we try to maintain a usage policy that is as open as possible. For example, staff are free to use radio e-mail for personal correspondence with friends and family anywhere in the world, and there is no limit to the number of messages any user may send. Our only explicit policy restriction is the request that users not subscribe to mailing lists.
To prevent the radio links from getting choked-up for hours on huge attachments, such as large documents and graphic files, all qmail servers connected to radios (that is, the radiohub in Conakry and each of the field office servers) are run with a message size limit of 8,000 characters. This is sufficient for three to four pages of text and is configured with qmail's databytes control file. We advise users of this message size limitation, suggest they stick to plain text for their correspondence and configure their mail client software accordingly. But whatever can be squeezed into the 8,000 byte limit by way of attachment and file compression is free to go. (Users in Conakry, which has a full-time, broadband gateway to the internet, are not quite as restricted. Here the databytes control file is set to one megabyte.)
As one would expect with our software selection, the system has proven extremely reliable. Despite the intermittent power outages typical in Conakry, we do try to keep the mailhub server coyah running at all times through the use of generator and battery back-up. So far these measures have kept this machine serving flawlessly since it was first installed, with a continuous uptime at this writing of over three months, no reboots.
Yet this reliability would mean nothing if the system were not sustainable over the long term. Two months before we installed the first radio server in the field, we formed a Network Users/UNIX Group among interested and capable IRC staff. This group meets regularly and enthusiastically to learn Linux/UNIX and to develop network administration skills. UNIX is a linguistically rich operating environment, and I think it finds a natural affinity among Africans whose language abilities seem inherently prodigious. In any case, the group now has a number of functional production systems on which to work and play, using mostly recycled hardware. While this article has focused on qmail and serial communications, the Linux servers installed for this project also host a typical range of other servers and services, including DHCP, DNS, natd, Apache, FTP, Samba and PostgreSQL. The IRC-NU/UG provides a human network that will continue to sustain and grow the technical network over the years to come.
The successes of this project are readily duplicated anywhere in the world. Schools, government ministries and other NGOs can easily build remote networking solutions over HF radio where access is otherwise not available, and at minimal cost. Once installed, these systems are almost trivial to administer and may be quickly adapted to alternative TCP/IP carriers. Maintenance of the e-mail system itself involves only the routine adding/deleting of user accounts, while keeping the /etc/aliases files up to date.
The current result of our own Radio E-mail project is that we are now serving mail to over 50 desktops and 150 staff in four offices throughout Guinea. The entire wide area network is serviced behind a single public IP address, at a total ISP cost of $150(USD) per month. Based mostly on existing hardware, the Radio E-mail project has leaped boundaries and opened dialogs for its users that were previously not possible.
Best of all, the system has deployed standard network and internet technologies throughout the organization and throughout Guinea utilizing the freely available, best of breed, borderless open-source technologies that underlie all global connectivity. Not only does this plant grass-roots networking infrastructure where there is yet no Internet, it helps build the core competencies and capabilities essential for Africa's connected future.
Dan Bernstein's qmail and tools
Andre Opperman's “The Big Qmail Picture”
Bushmail is a commercial provider based in South Africa offering a publicly accessible radio e-mail network to many locations in sub-saharan Africa.
The Mission Aviation Fellowship provides additional information on the use of HF radio modems, particularly oriented to the use of the E-Smith Linux distribution.
The International Rescue Committee
Wayne Marshall is a UNIX programmer and technical consultant living in Guinea, who never intended to become an e-mail administrator. He and his wife Paula enjoy traveling and African skies.
