SPF Overview

SPF is an emerging antiforgery standard that aims to prevent worms, viruses and spam from forging arbitrary e-mail addresses as the envelope sender in SMTP. SPF has two parts: domain administrators need to publish SPF records in the DNS, and e-mail administrators need to install SPF-enabled MTAs to read those records. SPF records indicate the servers from which a domain sends outbound mail. Mail coming from anywhere else is considered forged.

This article, the first of a two-part series, explains the concepts and trade-offs involved in SPF protection and shows DNS administrators how to set up SPF records. The second article is aimed at showing e-mail administrators how to activate SPF protection in their MTAs. This article was written in early January 2004 and reflects the state of the Internet current at that time.

I got spam from myself today. I founded pobox.com, and I'm an e-mail guy. So I pressed H for headers and read the Received lines. Just as I thought: like much of the spam I receive, this one came from a broadband machine. It's probably an old PIII running Windows 2000 unpatched, used for gaming and MP3s, quietly humming at the foot of someone's bed, draped in dirty underwear. Maybe it lives on a potato farm in Idaho; maybe it looks out over Central Park. Either way, it's probably infected with a variant of the Sobig virus, written under contract to a spammer. The machine's rightful owner has no idea he's infected, no idea his machine has been sending a few hundred spams and viruses every hour since that forgotten day long ago when he clicked on that weird attachment that didn't open.

Spam messages disguise their origins. Spammers use compromised machines to send the spam. They forge message headers. They fake Received headers to throw off the scent, make up bogus Subjects to trick Bayesian filters and forge From lines pretending to be PayPal or eBay.

Spammers also forge the return path. When messages are undeliverable, they bounce back to the sender whose address is in the return path. Not the From: address in the message headers, but the return path of the SMTP envelope, the RFC2821 MAIL FROM. Often, spamware uses lists of old addresses, or they simply guess common user names or launch a dictionary attack. The result is a lot of bad addresses and a lot of bounces.

Spammers don't want those bounces. They'd rather somebody else receive them. So, they pick an address at random or use the recipient's address. That's how they made it look like I got spam from myself. Sometimes they choose a hated enemy and maliciously forge his address so he gets flooded with thousands of bounces.

In 1997, a spammer forged a return address at joes.com, which then was flooded by so many bounce messages it went down for ten days—and gave the world the term joe-job. Hotmail and AOL get joe-jobbed every day: a lot of spam pretends to be from AOL but doesn't really come through their servers. Under conventional SMTP, AOL can't do anything about it. If you put the AOL logo on a T-shirt and tried to sell that shirt, AOL's lawyers would have you ceasing-and-desisting in a heartbeat. But spammers forge @aol.com every day. They can get away with it because they use SMTP.

The Simple Mail Transfer Protocol (SMTP) was designed more than 20 years ago—a kindlier, gentler time. The entire Internet was only a handful of research institutions. SMTP has served us well since then, but it's beginning to show its age.

SMTP is open and trusting. Its rules are relatively lax. You can assert any envelope sender and make up all the headers you want. You could argue today, though, that a protocol that lets joe-jobs happen is a little too open, a little too trusting. That's where sender authentication comes in. SPF tightens the rules.

When you send mail to a domain, your MTA does a DNS lookup (an MX query) to find out to which server to route the mail. Such a server is called a mail exchanger (MX). Small domains tend to have only one MX server. Big domains tend to have more. Mail to a domain goes to its MX servers.

Now for the big idea. In 99% of all cases, when a domain sends mail, that mail originates from a relatively small set of servers controlled by that domain. The domain could designate those servers using the DNS, then announce that any mail not received from those servers probably is forged. That's called a designated sender scheme (Figure 1).

Figure 1. With SPF, one mail server can check whether another server really is associated with the address the mail claims to be from.

Designated sender schemes are useful because they help fight forgery and are easy to set up. After all, domain owners already know which servers send mail from that domain. When I say send mail from that domain, I mean originate an SMTP transaction where the MAIL-FROM envelope sender shows that domain. I'm not talking about the From: header. This is an important distinction.

Mail from a domain tends to come from a small number of servers. That's true for domains large and small. Mail from aol.com comes from AOL's servers. Mail from my personal domain comes from my personal servers. It certainly doesn't come from a machine covered in dirty underwear.

Many ISPs already are implementing these kinds of rules in a haphazard and often slightly broken way. The problem is, one ISP doesn't know the insides of another ISP, and it's easy to guess wrong. Maybe aol.com's mail servers also originate mail for aol.net or vice versa. Wouldn't it be better if AOL themselves announced their designated servers in a simple, flexible, extensible, open format that everybody could use?

Well, they do. SPF is a standard, flexible, extensible, open format that everybody can use. At the time of this writing, AOL recently had started publishing their SPF record.

MTAs can interpret that record and use it to tell whether mail that claims to be from @aol.com is a fake.

Figure 2. SPF performs a simple DNS-based lookup for each incoming message.

All this rule tightening is purely voluntary: domains that don't publish SPF records can continue to send mail as before. Some unusual domains might be served better by not publishing SPF; it's up to them. But most domains should want to use SPF.

To publish SPF, a domain has to add only one line to its zone file. That line is a TXT record, and you can publish it today. Let's see what the TXT record looks like.

Suppose example.com wants to publish SPF. It expects MTAs everywhere to read its SPF record and use it to reject forgery attempts. It hopes SPF reduces the volume of joe-job bounces and bogus abuse reports. So it adds the following line to its zone file:

example.com.  IN TXT  "v=spf1 a mx ptr -all"

The v=spf1 version string identifies this as an SPF record. The -all means reject all mail by default. Domains that don't send any mail, such as altavista.com, can get by with simply v=spf1 -all. But if the domain does send mail, it declares mechanisms that describe how legitimate mail should look. Mechanisms go in the middle, before -all. The first mechanism to match provides a result for the SPF query. -all always matches and so belongs at the end.

Mechanisms are interpreted left-to-right. Using v=spf1 a mx ptr -all first would check whether the connecting client was found in the A record for the domain or, failing that, in its list of MX servers. Then the MTA would check to see whether the hostname of the client matched the domain. If none of the mechanisms matched, -all would be evaluated, the result would be fail and the MTA would be justified in rejecting the mail.

A, MX, PTR and IP4 are enough for the overwhelming majority of domains. The setup wizard at spf.pobox.com/wizard.html can help you configure SPF for your domain. But if your situation is complex, you can use the mechanisms described in the “Advanced SPF” sidebar.

SPF has a number of built-in mechanisms. The basic ones let you designate the hosts that send mail from your domain. This works well for almost all domains out there, because each domain's mail comes only from a small set of hosts. But if mail from your domain is distinguished in some other way, say you always sign it with S/MIME, instead of typing a or mx you can type smime.

Using designated sender mechanisms (A, MX, PTR and IP4) is one possible approach to sender authentication. New sender authentication methods are being developed. SPF is extensible, though, so it can work gracefully with them. SPF plugins that understand future extension mechanisms will be able to interpret them correctly. SPF plugins that don't understand those mechanisms will return unknown, and your domain will be treated as though it did not have an SPF record at all.

Today, spammers forge domain names. Tomorrow, they might forge hostnames. They might try to joe-job your laptop by making up username@ibook.example.com. It's a good idea to protect your subdomains as well. You should start with your MX servers and move on to other hosts with A records. Here's why.

Bounce messages are sent with MAIL FROM: <>. The null sender address ensures that bounces don't themselves bounce and create a loop. When SPF sees the null sender address, it falls back to the hostname given in the HELO command. When your MTA sends a bounce message, it announces its hostname in the HELO command it sends. If that hostname has an SPF A mechanism listed, the message passes. So SPF prevents HELO forgery as well.

SPF was designed to give the greatest benefit for the least cost. It tightens the rules in a way that makes it hard for bad people to do bad things, while not bothering the good people who do good things. Even so, some power users who have taken advantage of SMTP's lax rules may be inconvenienced by SPF. This section describes the two problems SPF causes power users and offers ways to work around them.

Most end users relay their outbound mail through their ISPs' SMTP servers. Most modern clients also support SASL authentication or POP-before-SMTP for users who need to phone home from outside the ISPs' networks. Users who always send mail through their ISPs' SMTP servers are automatically SPF-compliant and don't need to do a thing.

But some power users with an MTA on their laptop are used to originating mail from random IP addresses, bypassing their ISPs' SMTP servers entirely. SPF accommodates these users: the advanced mechanism (see the “Advanced SPF” sidebar) is a way to exempt certain users from being required to use their ISPs' SMTP servers. They can keep doing what they want.

Some power users have a dozen or more addresses that forward all over the place by using entries in /etc/aliases or .forward files. In classical forwarding, the envelope sender remains unchanged while the recipient address is rewritten. This becomes a problem, though, when the message arrives at the destination—it still has the original sender address, and SPF tests fail.

The workaround is easy, however; you simply need to switch to remailing, where the sender address changes as well. There are many ways to accomplish this. Read the SPF FAQ (spf.pobox.com/faq.html#forwarding) to pick up the one that's right for you. Most end users have nothing to do with forwarding; only power users need to implement this workaround. If you have third-party service through an alumni, vanity domain or other commercial forwarding provider (such as pobox.com), you should expect them to implement remailing for you.

The primary goal of SPF is to stop forgery. I don't want to get any more spam from myself, and I certainly don't want you to receive any spam that claims to be from me. Worms and viruses tend to forge the envelope sender, too, and we can block them with SPF. And, stopping forgery carries a bonus. When spammers are forced to use their true names, we can figure out which domains are legitimate and which are spammers. People already are doing this: a right-hand side block list (RHSBL) is the domain name version of a DNS block list (DNSBL). Spammers who aren't afraid of using their own domains end up on RHSBLs quickly, and they can be blocked that way. In an SPF world, RHSBLs will become more important and effective.

Big domains, including ISPs, banks and well-known brands care about controlling their trademarks. They have an obligation to protect their names. Altavista.com publishes an SPF record as do AOL and Oxford. More domains get on the bandwagon every day. Smaller domains publish SPFs simply because they don't want to be joe-jobbed.

On the receiving end, ISPs upgrade their MTAs and turn on SPF simply because it means less forgery—less spam, worms and viruses. Their bandwidth costs go down, too, because SPF lets them cut off the spammer before data is transmitted. They don't have to perform any cryptography or verify any signatures. SPF saves money.

By the time this article is published, SPF support should be either bundled in or available as a downloadable plugin for the latest versions of SpamAssassin, Postfix, Sendmail, Exim and qmail. Commercial antispam vendors have committed to support SPF; Declude JunkMail, for one, reports that SPF is successfully blocking spam in the field.

If all goes well, the SPF standard will be published as an RFC in the near future. But thousands of domains, including some quite large ones, already publish SPF records. There's no reason to wait; you should publish SPF today.