Mail Bounces & Gmail/GApps users: The ugly truth of DMARC in open-source mailing lists
This is a slightly edited copy of an email I send to the mailing lists for my local hackspace, VHS. I run their mailing lists presently for historical reasons, but we're working on migrating them slowly.
Hi all,
Speaking as your email list administrator here. I've tried to keep the logs
below as intact as possible, I've censored only one user's domain as
being identifying information explicitly, and then two other recipient
addresses.
There have been a lot of reports lately of bounce notices from the list,
and users have correctly contacted me, wondering what's going on. The
bounce messages are seen primarily by users on Gmail and hosted Google
Apps, but the problems do ultimately affect everybody.
67.6% of the vhs-general list uses either gmail or google apps (347 subs of
513). For the vhs-members list it's 68.3% (both of these stats created by
checking if the MX record for the user's domain points to Google).
Google deciding that a certain list message is too much like spam, because of
two things:
- because of content
- because of DMARC policy
Content:
We CAN do something about the content.
Please don't send email that has one or twos, containing a URL and a
short line of text. It's really suspicious and spam-like.
Include a better description (two or three lines) with the URL.
This gets an entry in the mailserver logs like:
delivery 47198: failure:
+173.194.79.26_failed_after_I_sent_the_message./Remote_host_said:_550-5.7.1_[66.196.40.251______12]_Our_system_has_detected_that_this_message_is/550-5.7.1_likely_unsolicited_mail._To_reduce_the_amount_of_spam_sent_to_Gmail,/550-5.7.1_this_message_has_been_blocked._Please_visit/550-5.7.1_http://support.google.com/m
+ail/bin/answer.py?hl=en&answer=188131_for/550_5.7.1_more_information._mu18si1139639pab.287_-_gsmtp/
That was triggered by this email earlier in the month:
> Subject: Kano OS for RasPi
> http://kano.me/downloads
> Apparently it's faster than Rasbian
DMARC policy:
TL;DR: If you work on an open-source mailing list app, please implement
DMARC support ASAP!
Google and other big mail hosters have been working on an anti-spam measure
called DMARC [1].
Unlike many prior attempts, it latches onto the From header as well as
the SMTP envelope sender, and this unfortunately interferes with mailing
lists [2], [3].
I do applaud the concept behind DMARC, but the rollout seems to be
hurting lots of the small guys.
At least person (Eric Sachs) at Google is aware of this [4]. There is no useful
workaround that I can enact as a list admin right now, other than asking
the one present user to tweak his mailserver if possible.
There is also no completed open source support I can find for DMARC.
Per the Google post above, the Mailman project is working on it [5], [6],
but it's not yet available as of the last release. Our lists run on
ezmlm-idx, and I run some other very large lists using mlmmj
(gentoo.org) and sympa; none of them have DMARC support.
The problem is only triggering with a few conditions so far:
- Recpient is on a mail service that implements DMARC (and DKIM and SPF)
- Sender is on a domain that has a DMARC policy of reject
Of the 115 unique domains used by subscribers on this list, here are all
the DMARC policies:
_dmarc.gmail.com. 600 IN TXT "v=DMARC1\; p=none\; rua=mailto:mailauth-reports@google.com"
_dmarc.USERDOMAIN.ca. 7200 IN TXT "v=DMARC1\; p=reject\; rua=mailto:azrxfkte@ag.dmarcian.com\; ruf=mailto:azrxfkte@fr.dmarcian.com\; adkim=s\; aspf=s"
_dmarc.icloud.com. 3600 IN TXT "v=DMARC1\; p=none\; rua=mailto:dmarc_agg@auth.returnpath.net, mailto:d@rua.agari.com\; ruf=mailto:d@ruf.agari.com, mailto:dmarc_afrf@auth.returnpath.net\;rf=afrf\;pct=100"
_dmarc.mac.com. 3600 IN TXT "v=DMARC1\; p=none\; rua=mailto:d@rua.agari.com\; ruf=mailto:d@ruf.agari.com\;"
_dmarc.me.com. 3600 IN TXT "v=DMARC1\; p=none\; rua=mailto:d@rua.agari.com\; ruf=mailto:d@ruf.agari.com\;"
_dmarc.yahoo.ca. 7200 IN TXT "v=DMARC1\; p=none\; pct=100\; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com\;"
_dmarc.yahoo.com. 1800 IN TXT "v=DMARC1\; p=none\; pct=100\; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com\;"
_dmarc.yahoo.co.uk. 1800 IN TXT "v=DMARC1\; p=none\; pct=100\; rua=mailto:dmarc-yahoo-rua@yahoo-inc.com\;"
Only one of those includes a reject policy, but I suspect it's a matter of time
until more of them will include it. I'm going to use USERDOMAIN.ca here as the
rest of the example, and that user is indirectly responsible for lots of the rejects we
are seeing.
Step 1.
User sends this email.
From: A User <someuser@userdomain.ca>
To: vhs-general@lists.hackspace.ca
Delivered to list server via SMTP (these two addresses form the SMTP envelope)
MAIL FROM:<someuser@userdomain.ca>
RCPT TO:<vhs-general@lists.hackspace.ca>
Step 2.
If the MAIL-FROM envelope address is on the list of list subscribers, your
message is accepted.
Step 3.0.
The list adjusts the mail to outgoing, and uses SMTP VERP [7] to get the mail
server to send the new message. This means it hands off a single copy of the
email, as well as a list of all recipients for the mail. Envelope from
address in this case will encode the name of the list and the number of
the mail in the archive.
If it was delivering to me (robbat2@orbis-terrarum.net), the outgoing
SMTP connection would look roughly like:
MAIL FROM:<vhs-general-return-18094-robbat2=orbis-terrarum.net@lists.hackspace.ca>
RCPT TO:<robbat2@orbis-terrarum.net>
And the mail itself still looks like:
From: A User <someuser@userdomain.ca>
To: vhs-general@lists.hackspace.ca
Step 3.1.
I got this email, and if I open it I see this telling me about the SMTP
details:
Return-Path: <vhs-general-return-18094-robbat2=orbis-terrarum.net@lists.hackspace.ca>
I don't implement DMARC on my domain. If my system bounced the email, it
would have gone to that address, and the list app would know that
message 18094 on list vhs-general bounced to user
robbat2@orbis-terrarum.net.
Step 3.2.
Google DOES implement DMARC, so lets run through that.
The key part of DMARC is that it takes the domain from the From header.
_dmarc.USERDOMAIN.ca. 7200 IN TXT "v=DMARC1\; p=reject\; rua=mailto:azrxfkte@ag.dmarcian.com\; ruf=mailto:azrxfkte@fr.dmarcian.com\; adkim=s\; aspf=s"
The relevant parts to us are:
p=reject, aspf=s
The ASPF section applies strict mode, and says the mail with a From
header of someuser@USERDOMAIN.ca, must have an exact match of the MAIL
FROM transaction of @USERDOMAIN.ca.
It doesn't match, as the list changed the MAIL FROM address.
The p=reject says to reject the mail if this happens.
This runs counter to the design principles of mailing lists, so DMARC
has a bunch of options, all of which require changing the mail in some
way.
Here's the logs from the above failure:
> 2014-03-19 11:19:50.783996500 new msg 98907
> 2014-03-19 11:19:50.783998500 info msg 98907: bytes 8864 from <vhs-general-return-18094-@lists.hackspace.ca-@[]> qp 32511 uid 89
> 2014-03-19 11:19:50.785359500 starting delivery 211352: msg 98907 to remote user1@gappsdomain.com
> 2014-03-19 11:19:50.785385500 status: local 1/10 remote 1/40
> 2014-03-19 11:19:50.785450500 starting delivery 211353: msg 98907 to remote user2@gmail.com
> ...
> 2014-03-19 11:19:58.713558500 delivery 211352: failure:
+74.125.25.27_failed_after_I_sent_the_message./Remote_host_said:_550-5.7.1_Unauthenticated_email_from_USERDOMAIN.ca_is_not_accepted_due_to_domain's/550-5.7.1_DMARC_policy._Please_contact_administrator_of_USERDOMAIN.ca_domain_if/550-5.7.1_this_was_a_legitimate_mail._Please_visit/550-5.7.1__http://support.google.com
+/mail/answer/2451690_to_learn_about_DMARC/550_5.7.1_initiative._ub8si9386628pac.133_-_gsmtp/
> 2014-03-19 11:19:59.053816500 delivery 211353: failure:
+173.194.79.26_failed_after_I_sent_the_message./Remote_host_said:_550-5.7.1_Unauthenticated_email_from_USERDOMAIN.ca_is_not_accepted_due_to_domain's/550-5.7.1_DMARC_policy._Please_contact_administrator_of_USERDOMAIN.ca_domain_if/550-5.7.1_this_was_a_legitimate_mail._Please_visit/550-5.7.1__http://support.google.co
+m/mail/answer/2451690_to_learn_about_DMARC/550_5.7.1_initiative._my2si9389106pab.76_-_gsmtp/
[1] http://dmarc.org/
[2] http://dmarc.org/faq.html#s_3
[3] http://dmarc.org/faq.html#r_2
[4] https://sites.google.com/site/oauthgoog/mlistsdkim
[5] http://www.marshut.com/qskkv/adding-dmarc-support-for-mailman-3.html
[6] https://code.launchpad.net/~jimpop/mailman/dmarc-reject
[7] http://en.wikipedia.org/wiki/Variable_envelope_return_path
Syndicated 2014-03-29 22:35:20 from Move along, nothing to read