Why you shouldn't jump on the SPF bandwagon.
Posted 13 Jan 2005 at 14:02 UTC by dwmw2
SPF is a very broken anti-forgery technique
which you should not implement. You should neither
publish SPF records, nor check SPF on incoming email.
If you use SPF, you will be causing genuine email to be
rejected. There are much better alternatives which address the
forgery problem without throwing the baby out with the
OK, so what is SPF?
SPF is Sender Policy Framework, formerly known as Sender
Permitted From. It makes the assumption that mail from
a given email address will only ever be seen when sent directly
from certain IP addresses which are associated with that address,
and it gives the domain owner a way to specify which IP
addresses are 'acceptable'.
So, for example, I might state that mail from the
infradead.org domain will only be seen from hosts with
hostnames ending in 'infradead.org', from the IPv4
subnets 220.127.116.11/29 or 18.104.22.168/30, or
from the IPv6 subnet 2001:b80:10b::/48. The idea would be
that if you ever see a mail claiming to be from an address at
infradead.org which is coming from a different
host, then you should know that it's a forgery.
(That's about as far as the pretty graphical
but mostly content-free official
explanation goes, but if you take a moment to look closer or
think about it, there's a little bit more to it than
Unfortunately, this assumption is false. You
do see perfectly genuine mail from my domain, from
machines other than mine. This happens due to mail
forwarding. People tend to change their ISP quite often,
but don't want to have to tell everyone that they've changed
their email address. So they have an account elsewhere, at a
vanity domain or on another computer, and they forward
mail from that address to whichever is their current ISP, or
There is a proposed solution to this -- the SPF advocates
say that the whole world should 'upgrade' to make the initial
flawed assumptions on which SPF is based come true.
Their idea is that when a server forwards a mail, it shouldn't
just use the original sender's address as has been done for the
last couple of decades. Instead, it should munge the sender's
address as it resends the mail, faking an address at the domain
which is actually responsible for the forwarding. Then, if a
bounce is generated, that faked address receives the bounce and
the bounce needs to be forwarded on to the original sender of
the original mail.
This forwarding of bounces could be easily abused if done
naïvely, so the idea is that we also include a timestamp to
time-limit it, and a cryptographic hash which can only be
generated locally. And of course you have to include the original
sender's address; both the domain and the username. So when Bill
Gates sends me an email to my vanity domain, for example, and my
computers forward it to my current ISP, they'd actually end up
rewriting the sender address to:
SRS0firstname.lastname@example.org instead of
leaving it intact.
This rewriting scheme they call 'SRS', and they need it
to be implemented everywhere before they can use SPF
without throwing away genuine mail.
Problems with SPF
You may have managed to spot one or two potential problems with
this scheme all by yourself, but some people do seem to have
trouble so let's spell them out explicitly:
- SPF is not compatible with today's
SRS is not common. If you publish SPF records, you are going to be
asking people to throw away genuine email which you did actually
If you check SPF records strictly for anything larger than a
toy domain with a handful of users, then you're likely to be
rejecting genuine email which is sent to you or your users.
- ...and won't be compatible with
There is little incentive for people to deploy SRS. It's a
cumbersome and convuluted workaround for a broken
Those who believe SPF to be an insanely stupid idea will
strongly resist, and those who are simply uninterested in SPF
have little reason to want to use it. Yet SPF require these
these uninterested third parties to upgrade too -- it's not
just limited to those who are participating.
People are very slow to deploy new ideas on the
Internet, and especially with respect to email, even when
those ideas are good. Many people haven't even
managed to deploy Extended SMTP (ESMTP) yet, even.
There will be a porcine implementation of RFC1149
before SRS is ubiquitous.
- SPF is not an anti-spam
Some people claim that SPF directly combats spam. It
doesn't. SPF attempts to address forgery. In fact, a
large amount of spam rates an SPF 'pass' result, because
rapidly adopted SPF for themselves.
You still need a blacklist or other kind of trust database,
to tell you which domains are trustworthy and which are
not. But we already have lots of blacklists; it's
just that we list the IP address instead of the domain name,
to tell you which hosts are trustworthy and which are
not. We don't need to break email and force everyone to
upgrade to some bizarre new scheme just for that.
- SPF is easily duped.
Although the uptake of SRS on forwarding sites will be low,
it can still be a common spammer trick. Any spamming host can
do the SRS trick to 'take responsibility' for forwarded
mail. If your ISP receives mail from the address I used in the
above example --
then they have no real way of telling whether it really did
come from Bill Gates via my servers, or whether it's a
fake. SPF would just accept the mail, depending on my domain's
reputation in the trust database. It's all about how much you
trust the one server which is offering the mail -- it's not an
So SPF is really not any better than blacklisting by IP
address or HELO name. Again, the breakage it imposes is not
- SRS obfuscates useful
The original sender address is useful information, and can be
lost if an intermediate host mangles the mail by using
A sender address may be present in a blacklist, or may fail
SMTP callouts. Yet an intermediate host 'takes
responsibility' for the mail in question, causing it to be
Sender addresses are also used for mail filtering. The
sender address is the most reliable method of filtering
mailing list traffic into its own folder; using Cc:
and/or To: headers has both false positives and
false negatives, and using the List-Id: header has
false positives too. Again, SRS mangles this information.
- By using SRS, you can be vouching for
How good are your spam filters? By rewriting the address of
mail you're forwarding so that it appears to come from your
own domain, you put your own reputation on the line. You could
be blacklisted for mail which you claimed even though
you didn't send it and you have no real knowledge of the
- SPF offers a whitelist, not a blacklist.
In the simple case without forwarding, SPF does manage
to give a clear indication that a mail is valid. But that's not
really very useful in practice. What we need is a
blacklist but SPF gives us only a
whitelist. We can't safely use that to reject mail --
we can only reject mail if it's clearly invalid, but SPF can
only honestly say 'valid' or 'unknown'.
In summary, SPF doesn't really achieve what it sets out to
achieve, and it breaks far more than it's worth. The costs far
outweigh the benefits.
Some SPF advocates would claim that you can't make an omelette
without breaking eggs; that the spam problem has got so bad that
we have to break things and switch to a new email
That's silly for two reasons. Firstly, if you're going to break
the world and make people upgrade, you might as well just switch
wholesale to something like Internet Mail 2000
rather than pretending it's still compatible with SMTP. SMTP has
enough other problems (like the way bounces are done, the
difficulty of implementing variable preferences for different
users, etc.) that we might as well fix it all at once,
if we find it necessary to make such an incompatible
Secondly, and more relevantly, there are alternative schemes
which give a measure of protection against address forgery but
without the flawed assumptions, the loss of genuine
mail, and the need for the rest of the world to 'upgrade'. You
would have to be mad to choose all the problems of SPF, when
other schemes are far saner can and offer the same benefits:
- Yahoo's DomainKeys,
These schemes (and some others) are all fairly similar in
nature. They offer a cryptographic solution to the forgery
problem, where a sender adds headers to the mail with a
signature, and that's verified by the recipient by checking
against a published key for the domain or user in
Google, Yahoo and Earthlink, amongst others, are already
using DomainKeys on outgoing email. AOL are reportedly going
to be doing so in the early part of this year (2005).
These solutions don't suffer from the forwarding problem;
they're compatible with email as we know it today, and offer
the same protection as SPF does but without the false
There is an effort under way to merge these into a single
proposal for approval by the IETF. See the MASS web page for more details.
- Signed Envelope Sender (SES), Bounce Address Tag
These proposals are based on the same idea, originally
inspired by the concept of using SRS on your own
outgoing email instead of on forwarded mail.
For example, I never send email with MAIL
FROM:<email@example.com<; it's always rewritten
to appear to be from a time-limited SRS-encoded address.
This means that I can instantly start rejecting bounces
targeted at the raw email address, because I know they're
bounces to mail I did not actually send. And this, in turn,
means that anyone doing SMTP
sender verification callouts will reject faked mail
claiming to be from that address, because a bounce cannot be
Since SMTP callouts are unpopular in some quarters, SES
takes the concept further and provides alternative methods
for validation of reverse-paths, and also allows for a
message digest to be included in the reverse-path, to
prevent replay attacks in the event of an encoded
reverse-path being discovered and abused by a spammer.
By implementing SES or BATV, you can instantly stop
accepting bounces to mail which you didn't send.
- Certified Server
CSV is a very simple method for identifying servers by their
HELO greeting instead of just by their IP
address. It uses SRV records in DNS to specify
which IP addresses may use a given hostname in their
HELO greeting. It also offers a method of storing
the reputation database in DNS.
Since SPF can only really be used for evaluating the trust
level of the host which is actually sending the mail, CSV is
sufficient to use as a viable alternative.
"When banks start DomainKeys or S/MIME signing all
outbound mail, I promise to give up SPF and Sender
-- Meng Weng Wong, inventor of SPF.
How about just a .45 to the back of the head for all spammers?
But seriously, folks...what happened to the promise of PGP encoding for all SMTP traffic that we heard about in years gone by? Aren't any of the simple solutions viable? Do we need proposals for changes that will impact the entire Internet and won't be backward compatible? Please.
Where are all the simple solutions? Like ISP enforcement of bulk email quantities? Or the ICANN's monitoring and DNS-blacklisting of all those crappy Taiwanese SMTP servers? Or any of the sixty-seven jillion other proposed solutions to crushing spam? Oy.
Simpler solutions, posted 14 Jan 2005 at 10:30 UTC by dwmw2 »
But seriously, folks...what happened to the promise of PGP encoding for all SMTP traffic that we heard about in years gone by? Aren't any of the simple solutions viable?
PGP isn't really the answer in general because it requires MIME and doesn't reliably survive mailing lists. You'll get broken signatures and hence false rejections.
However, the crypto solutions described in the 'Alternatives' section above improve on that idea by hiding the signature in the header and being applicable only to the most recent sender address -- which would be the mailing list itself in the case of a mailing list, so you don't have to worry about breaking the original sender's signature.
Do we need proposals for changes that will impact the entire Internet and won't be backward compatible? Please.
No, we don't need such proposals; that's my point. The alternatives given above are much more compatible with the real world of today than SPF is. Take your pick of those instead.
The article makes a lot of presumptions over what exactly SPF does and is.
Yes, SPF allows you to specify rules stating that a certain server can send mail from your domain. Yes, that can break forwarding. Yes, SRS sucks. No, that doesn't mean SPF is useless or broken.
First, you should *not* be publishing -all SPF records. A -all command in an SPF record means "deny all" and would usually be the last item in the list of SPF commands. The problem is, of course, that doesn't handle forwarding. However, you *can* publish a ?all record. That means that "any host that doesn't match a previous rule might be valid, or maybe it's not, I don't know."
What's the use of this? It eases policy. For example, many organizations have started blocking mail from free mail hosts like Yahoo and Hotmail because the domains are spoofed so very often in viruses and spam. (Note: that *also* breaks forwarding, along with all other mail from those domains.) Using SPF, Yahoo and Hotmail can let you know what the real servers are. You can then add a rule to your domain that will throw away Yahoo and Hotmail messages only if you can't guarantee they're valid. In this case you'd still dump forwarded messages, but the alternatives for these organizations is even worse.
Some mail systems use *very* resource-intensive anti-spam measures. Or use cumbersome sytems like white-lists with reply verifications. Using SPF, you can let mail bypass these systems if the SPF record verifies as guaranteed valid.
The above scenario does have a hole: spammers can use SPF records, too. The answer is something that *ALL* long-term anti-spam solutions need: domain ratings database. Namely, a way to check if a given domain is a spammer domain, with checks for things like the age of the domain and whether it has a web site and so on, in addition to more concrete ratings. So then you have mail in four classes:
* Explicitly denied. (Yahoo knows that 22.214.171.124 spoofs their mail and isn't a forwarding service/user, so they deny it explicitly. If you get mail from 126.96.36.199 with a yahoo domain, you should drop it.)
* Unknown. (Not explicitly granted or denied; the default for most hosts.)
* Verified, possible spammer. (It is definitely from the domain owner, but the domain *might* belong to a spammer. The mail should be scanned with your stronger, and likely slowest, content scanners.)
* Verified, not a spammer. (It is definitely from the domain owner, and the domain owner isn't a spammer.)
So in the first and last case your mail system's life just got a let less over-loaded. ;-) In the second and third case your system's load didn't really change at all, save for the SPF lookup itself, which is pretty lite-weight and will be cached if you run a caching domain server.
Now, that's not all. SPF can do far, far more than just check the sending machine's IP address and the envelop domain. SPF can tie into other authentication systems as well.
Let's say you use DomainKeys. (*gasp* it also breaks things - that's what happens when you fix a flawed and insecure protocol, you have to change it.) You can use SPF to describe your DomainKeys setup, specify rules for when it should be used, and so on. SPF becomes your globally-readable configuration file for your DomainKeys setup.
Let's say you use PGP keys, in which each user of your company has their own key. (Unlikely setup, but possible.) You could define a method for specifying a key server and use SPF to tell mail servers about that key server. Mail servers that support this system would get a mail, check the domain, see that the SPF record tells it to check a PGP key against the server, and does so. If the mail is not signed or the key isn't valid according to the specified key server then the receiving mail server knows the mail was forged. No, this PGP mechanism doesn't exist yet (that I know of), but you could help form the specification for it, and simply utilize SPF in that specification to greatly reduce the amount of code that implementations need to implement (as they will likely already have SPF reading code).
SPF is the Sender Policy Framework. It's a framework for defining policy. While the most common and simplest policy the framework is capable of using does indeed break forwarding, and while some people may think that SMTP can never be evolved to handle that problem, it is simply untrue that this weakness will actually affect SPF's over-all usefulness. There are other policy mechanisms that SPF supports and new policy mechanisms will likely be added with time. In the future we are likely to see different organizations using one of several different popular mechanisms based on whichever best fits its needs.
Understanding, posted 21 Jan 2005 at 17:03 UTC by dwmw2 »
, some of what you say is true. Just as with most flawed technologies, of course there are
some limited ways in which you can use SPF records which aren't entirely broken and stupid. In the interest of simplicity, I was referring to the use of them for their intended purpose, as a flawed anti-forgery mechanism.
Yes, you could publish a record ending in '?all'. That's what I meant by my reference to whitelists. And yes, you can use whitelists to avoid time-consuming spam checking in some cases. But that's not what the SPF advocates are getting people to do -- in fact some of the SPF advocates are even treating '?all' as if it were '-all' in some cases. Without the ability to safely publish records ending in '-all', it's basically useless for its intended purpose, although there are still ways you can abuse it to achieve interesting results.
One possibility you didn't mention is that a record which says only '-all' for a domain which never sends mail is actually sane. But that again is beside the point. Since people can't safely reject mail for a 'fail' SPF result, I don't see the point in doing that even for those domains. And you don't need SPF for that anyway.
The example of an explicit 'fail' for a known spoofer is tenuous at best. Yahoo doesn't keep track of these -- other people do though. And the offenders don't spoof only one domain either. This is what IP-based blacklists are for -- again you don't need SPF for that.
Your example of a "globally-readable configuration file for your DomainKeys setup" doesn't seem to make much sense to me, and again it doesn't bear that much relation to SPF as it is advertised and advocated. Even if there's a practical way to do as you suggest, it's a private thing. I happen to use DNS TXT records as a globally-readable alias database for my virtual domains -- but the globally readable nature isn't really a feature; the fact that DNS allows easy replication across all my MX hosts and DDNS allows domain-owners to modify their own aliases is the feature. If you want to use SPF-like syntax as some kind of private configuration method, you gain nothing by using actual SPF records, surely?
Yes, you're also right that SPF can be theoretically be extended with new mechanisms which aren't based on IP addresses, and aren't entirely broken. But that hasn't happened yet, although the 'exists' mechanism can already query for the existence of a fairly arbitrary host name, and hence be used in conjunction with a stunt DNS server to achieve some interesting results.
But again that's not really the point. Since a recipient can't trust that an SPF 'fail' result is sufficient to reject the mail, the flexibilty of potential extensions to SPF grammar is in vain. Unless we're going to see a new 'reallyfail' result code, perhaps? :)
You can abuse SPF records in interesting ways to achieve some vaguely useful things, I agree. But SPF as it is presented, as an anti-forgery mechanism based on IP addresses, is not workable. Furthermore, because so many people are publishing records ending in '-all' when -- as you correctly point out -- they shouldn't, you can't trust a 'fail' result. So even the interesting abuses like stunt DNS servers aren't really a useful way to publish information to third parties in a way that allows them to safely reject faked mail.
When I first heard of SPF, I thought it was useless and misguided. Their misleading-by-omission marketing doesn't exactly help the situation. However, I am now fully in favour of SPF, with the following stipulations:
1) It is nothing but a whitelist. Anybody who thinks otherwise is crazy. However, it allows the sender (that's me) to define the rules the receiver will use to tell if the mail is really from me or not. In other words, if my rules are sufficiently strict, you can guarantee that the message you're getting either is, or is not, from me. If my rules aren't sufficiently strict, then you can't (and I get my flexibility to use mail forwarders and things). But if you know for sure that the message is from me, and you know for sure you want messages from me (eg. you've sent messages to me in the past) then you can have a 100% reliable whitelist for messages from me; I can completely bypass your normal spam scanner, and messages from me will never be lost. In short: I, the sender, control whether you'll incorrectly reject messages from me. If I claim I'll never use a mail forwarder, and then I use a mail forwarder, then I'm an idiot.
2) It's unarguably better than the current system. That's a very strong statement. Why is it unarguably better? Because it will work fine (no worse than the old way) if nobody publishes SPF records, if you publish "infinitely permissive" SPF records, if you check SPF records or if you don't. It's a general solution that includes the old system as one of its allowed implementations, so if any variation of SPF deployment is potentially better than the old system, you can't lose.
3) Reliable whitelists permit strengthening your blacklists. If you know for sure when a message is from me, and you know that you want messages from me, then you'll be assured that messages from me are never, ever blocked, until the end of time. If you send a message to me, your anti-spam system can auto-whitelist my email address. (Without SPF, mindless address-based whitelists are pointless because spammers fake messages from people you've communicated with in the past.) And if everyone you've ever corresponded with is auto-whitelisted, then you can turn up the insanity level on your spam scanner: only messages from people you don't know could ever be blacklisted by accident (false positives). If that happens, they can always send you a less-suspicious-looking email next time.
4) SRS sucks but is orthogonal so we can ignore it. Well, yes. Enough said.
, if we can make those stipulations then I'll be behind you in supporting it. Please let us know when your takeover bid is complete and people have stopped thinking they can use it to reject mail.
In the meantime, I don't agree that it's better than the current system. If it could only be used for whitelisting, or if there was even a way to publish records which could stipulate that they are only to be used for whitelisting, then that might be true. But people are using it for other things, and some are even rejecting mail if they hit '?all' at the end of a record.
Leaving your From: address as one which you forwards to you but you can no longer even access the mail server for is broken behavior, and trivially fixed. Reasonable accomodations can be made, but that one isn't reasonable.
Your premise is up their with the old arguments that newgroup commands should be issued only after a long beurocratic process on usenet because the awfulness of some nntp servers combined with the awfulness of some file systems caused them to melt if there were too many newsgroups. Please fix your own personal setup before demanding that the entire internet be architected so you don't have to bother.
Que?, posted 10 Feb 2005 at 10:44 UTC by dwmw2 »
, it's entirely unclear what you're talking about, or to whom your comments are addressed.
You speak of "Leaving your From: address as one which forwards to you but you can no longer even access the mail server for", but I don't see how that is relevant to the discussion. SPF isn't about the From: address, and you don't specify what type of access you mean -- do you mean access to send mail by SMTP AUTH, access to read mail from a local mailbox, or access to change your .forward file on the host in question?
Like many who support SPF, you seem very confused. Please could you try to be more specific about what you mean and why you think it's broken, and then we can help you to understand.
Whatever, posted 15 Feb 2005 at 06:27 UTC by Bram »
dwmw, I was referring directly to your original post.
Oh, I'm 'confused'? If you had attempted a rebuttal I'd respond, but your complete lack of one strongly indicates an attempt to get the conversation bogged down in technical detail, a highly disingenuous tactic, although one I find unsurprising given how overtly destructive the behavior you're recommending is.
The fact remains, you only have one example of functionality which SPF damages, and that functionality is highly dubious at best and clearly worth sacrificing. In fact, many admins of machines whose former users are abusing forwarding by not re-subscribing to mailing lists at their new address view breaking that functionality as very much a feature.
Original post?, posted 15 Feb 2005 at 14:00 UTC by dwmw2 »
Yes, I do think you're confused. Many people who support SPF are confused, and your responses in particular have certainly lacked coherency.
My original post didn't refer to the From: address at all in the context of SPF, and neither did it mention anything about not being able to access the mail server. You seem to have gone off into the weeds -- I can't respond because I can't actually work out what you're trying to say or how it relates to SPF.
I'm sorry that you feel my request for clarification is a 'highly disingenuous tactic' and that you don't feel able to explain your meaning more coherently. I'm only vaguely surprised that you seem to consider a technical discussion to be a bad thing -- many SPF advocates seem to take that point of view, in my experience.
If you change your mind in future, please let me know. But I am only really interested in technical discussion, I'm afraid -- the purely emotional knee-jerk response of "spam is bad, and we must force the world to change and throw away valid mail to help protect against it", and adopting the first hare-brained scheme which comes along instead of the saner alternatives, isn't really something which can be fruitfully discussed in a forum such as this.
Btw, if forwarding sites don't want to continue forwarding for old users, they have no obligation to do so. It's bizarre for you to say that the admins of such sites may view SPF's breakage as a 'feature'. Partly because they can stop forwarding for themselves if they really want, and partly because they can't actually publish SPF records which prevent forwarding through their site -- SPF would be published by the original sender, and checked by the ultimate recipient. Again, what you say just doesn't really make sense.