Cyrus, Virtual Domains, malware-free exim4 debian config

Posted 12 Feb 2007 at 09:33 UTC by lkcl Share This

I love exim4 - and i cannot get on with either postfix or sendmail. The level of sophistication that is required these days to set up a trouble-free email server is, i believe, matched only by exim4's sophistication and ease-of-setup. postfix is great - and secure - and lovely - but if you need a rocket wizard to configure it, then it's not much good to anybody. And sendmail is just... my hat off to anyone who can configure sendmail because it's just... just... absolutely fine for anyone who has an IQ of above 160. So, finally, having got a setup that i am finally happy with, that can cope with several domains, i thought it was time to let people know about it.

(Original report is available at lkcl.net reports which will be updated as-and-when: advogato has no article edit function - sorry!)

This setup is very fussy, and all of its checking is done at MTA time. As a result, it is particularly important that you also have your backup MX records do the same level of sophisticated checking (I had one client for whom i did a similar setup, and they used some idiots for their MX backup, whose server accepted the email immediately and then tried to deliver it 60 times per hour, and then gave up).

I've spent about four years trying to track down issues to get this configuration right. Finally I have solved the one thing that has been bugging me: messages from the <> "bounce" address from spammers to random addresses on my cyrus server causing me grief.

The issue is, I believe, that there is a bug in exim4 (which the developers will not acknowledge as I pissed them off. as usually happens when people don't know me very well, i went from trying to learn what to do, asking for help and demonstrating both my ignorance and my willingness to learn and help, to swearing at them in under 48 hours. so, if you want to investigate this issue, please do so, because i certainly can't). The issue is, i believe: in the LMTP transport, which will, I believe, affect every single exim4 system in existence that uses LMTP to deliver to the mailboxes (cyrus, courier etc.)

I believe that this bug stops you from being able to use LMTP for sender-verification, as anything that is attempted to be delivered (or tested/verified) returns immediately, and any errors, such as 550 mailbox nonexistent, are ignored. The logical result of this is that spammers can send you email to any address, and it will be accepted by exim4 into its queues, but then it cannot be delivered! (The symptoms are that you keep getting undeliverable messages sent to postmaster).

Finally, I found the excellent 'virtual domains' concept, by steve. Now, this addition to exim4 is extremely useful, and i said so in a bugreport to the exim4 debian maintainers, who managed to justify not adding it, because they believe that it over-uses the word 'virtual'. This is, in my opinion, pure laziness on their part, and if you also believe that having, out-of-the-box, the simple capability for exim4 to do what postfix has been able to do for years (handle multiple domains in a very simple fashion) then please do contact debian-devel, the 408467 bugreport and the debian exim4 maintainers to express your views.

So - in short: I have moved even my main domain lkcl.net into this format (/etc/exim4/virtual/lkcl.net), and made the exim4 "standard" configuration responsible only for localhost (/etc/mailname is localhost). Any email addresses e.g. lkcl@lkcl.net are now listed as entries "lkcl: lkcl@localhost" which is the only way that email will get into the cyrus22 server - addresses to localhost.

Now, i have had to create entries for postmaster: postmaster@localhost as well, and root, and webmaster, which is a bit of a pain, but there you go. But it's still... it's the last irritating thing that i've had to fix.

The rest of my configuration is a probably-quite-extreme-and-probably-broken sa-exim setup, which is so extreme low spam settings that virtually nothing gets through. certainly, it's pretty hit-and-miss as to whether hotmail users, with their insistence on sending HTML by default, will get through spamassassin (which has both razor and pyzor _and_ dcc _and_ clamav to back it up) but to be honest, I couldn't care less, and if i do happen to care then i add them to /etc/spamassassin/local.cf as a whitelist_from {email address} entry.

The only other thing is that i found someone who recommended adding these two lines to /etc/spamassassin/local.cf:


        score MIME_HTML_ONLY 3.0
        score HTML_MESSAGE 3.0

the reason why to push the scores up a bit is because many messages which contain those irritating viruses-in-an-image are HTML only messages with a single MIME attachment.

Razor and Pyzor are very good distributed real-time spam checking programs. DCC is a fuzzy spam checker that I found randomly and thought it might be nice to try. I added the testmsg-whitelist which I found on the dcc client web site, because treating test messages as spam is kinda rude.

Exim4 with anti-virus was easy because i'm using exim4 4.64: see main/20_clamav and also see acl/40_exim4-config-check-data near the end, the 'This message contains a virus:....' This is absolutely brilliant because the only opportunity to correctly inform people that they have a virus on their system is at MTA time. anything else is risky, and you shouldn't do it (send to the 'From:' address after receiving the email) - as you can pretty much guarantee that it's going to be faked up.

Also, i installed greylistd. greylistd for debian has a script that automatically reconfigures the ACLs in your exim4 config to make it do greylisting. Unfortunately, there's a bug in the maintainer's config, which turns off greylisting for anything that comes from <> which is the key source of problems that I earlier encountered.

Of course, it's very _sensible_ to disable greylisting for <> addresses (if you do not do proper distinction about where the incoming message should to be sent) because if you get two servers doing sender-verification using the <> "bounce" address, you end up in a 'please try later' bun-fight over which nobody wins... :)

but here's the thing: only local mailboxes should be told 'please try later', whereas messages which are forwarded on to another SMTP server should not be rejected at this stage.

so, I commented out the '!senders = :' line in acl/30_exim4-config_check_rcpt and i hope like hell it was the right thing to do :)

Also note: I don't know where it came from but I have a section main/09_exim4_local_options with smtp_accept_max = 0 to accept an unlimited number of connections (for teergrubing) but the load average must not be above 10: once it goes above 10, it stops accepting incoming connections.

Overall, this is an extremely sophisticated setup, where, especially since I installed dcc, and i set the spamassassin scores for those two HTML thingies a bit higher, I am literally receiving no spam whatsoever. Prior to dcc being in the mix and those two extra scores, I would occasionally get three to four spam messages in one go, every few days.

At the moment, I'm receiving about 1,500 spam and virus delivery attempts per day: with the combination of dcc, pyzor, razor, spamassassin, greylisting and clamav, absolutely none of them get through.

Neither, being absolutely honest, will some email messages especially from certain ISPs - but at least, unlike non-MTA-time-setups, the sender is guaranteed to receive a bounce message pointing them to some advice...

Here's the exim4 config files.


Spam, Spam, Everywhere, posted 16 Feb 2007 at 10:13 UTC by dfenwick » (Journeyer)

Luke:

As in your case, I've had the same email address for a long, long time. Over 15 years now. It's virtually flooded with spam. However about 15 months ago I outsourced my mailer to the guys at fastmail.fm. They give me virtual domains, manage my inbox, manage the greylisting, what have you. This wasn't a choice I made lightly. I preferred to run my own mail server. However the combination of daily hack attempts, the cost of getting a static IP address, and the fact that I simply didn't have the cycles to manage my mail server anymore pushed me to these guys. For $40/year, they manage my mail for me. They even give me a static web site, up to 15 domains, and up to 150 aliases within those domains. I did have to spring for another account for my wife, but it's peanuts compared to managing the server myself.

That's not to say I **like** their webmail interface. I don't. In fact, it kind of sucks. They give me a secured IMAP interface, but I really like to be position independent and there's not always a fat IMAP client laying around at the places I go. So rather than fart around with that, I have an interesting setup.

I use OfflineIMAP to synchronize my IMAP local IMAP server at my house with all of my mail accounts that support IMAP. I then run Horde on my web server (the one I still have a static address on) and I connect to my own Horde instance that's connected to my own IMAP server to read my mail. The outbound mail was a bit tricky to setup, but in the end it was a simple script to map my local username to my mail provider's username and keep the passwords synchronized.

But wait, there's more! I also use SpamBayes to filter my messages right after the IMAP synchronization runs. So I have my mail provider with their virus scanners, spamassassin, greylisting, and what have you, running out in Internetland(tm). OfflineIMAP synchronizes every 5 minutes. After the synchronization is complete it then passes itself off to SpamBayes, which then classifies all of my mail based on my own tokens. So the spam that DOES get through my mail provider (it's usually less than 100 per day at that point), get classified with my own Bayesian filters. In the end, I end up with about 3 spams per month in my inbox. I do have it configured somewhat pessimistically, and those messages with scores that are borderline go into an 'unsure' folder. I then go through those and move them either to the 'not spam' or 'spam' folders, depending on which they are, and the classifier then learns from them again.

Outsourcing has probably saved me 50 - 100 hours of tinkering with sendmail, spamassassin, dealing with hacking attempts, etc. If you don't **have** to run your own mail server, don't. There are plenty of really cheap providers out there that will do the work for you.

Rejecting all bounces?, posted 19 Feb 2007 at 07:38 UTC by dwmw2 » (Master)

Oh dear, you really did make a muppet of yourself there, didn't you?

You turn up with a bogus 'problem' caused by your own configuration accepting messages to all users without first verifying that they really exist, rant fairly incoherently about it, then disappear in a volley of abuse claiming that you're going to 'fix' your configuration by rejecting all bounce messages, in violation of RFC2821.

I wish you the best of luck sending email to anyone to does sender verification callouts or uses the dsn.rfc-ignorant.org blacklist.

OK then, posted 19 Feb 2007 at 18:41 UTC by ncm » (Master)

dwmw2: Exim people congratulate each other on how self-documenting their system is, but it really is quite opaque without a great deal of background. It would be more helpful to all of us if you were to suggest what you would do instead, and how.

What I'd do instead..., posted 20 Feb 2007 at 16:38 UTC by dwmw2 » (Master)

I don't think I've ever heard anyone claim that Exim is self-documenting. It's relatively well-documented but that work is done by people, not the system itself.

I'm not entirely sure what lkcl's problem is -- it seems to be that he configured his system to accept incoming mail to arbitrary non-existent users, by removing the 'check_local_user' condition which causes only existing usernames to be accepted at the local domain(s) -- and this means that he's accepting mail to bogus users which Cyrus then refuses, not unreasonably, to deliver.

As documented, the 'check_local_user' condition only checks against system users, and Cyrus can have a completely different set of users, so perhaps what lkcl wants is to accept mail only to users known to Cyrus, rather than to all system users. Simply removing the existing condition without adding any replacement for it was never really likely to work though, was it?

This was of course pointed out in the first response which he received on the mailing list, along with an example of how to express the new condition (which involved having a list of Cyrus users in a text file for lookup purposes). A real-time method where Exim calls out to Cyrus for verification when the mail is received is slightly more complex because Exim can only currently do such callouts over TCP rather than UNIX sockets; it's shown at at http://www.exim.org/eximwiki/CyrusImap.

I'm not entirely sure how lkcl settled on the idea that it was only bounces which were getting through to these non-existent users. I suspect that he only saw bounces on his queue, but they were bounces generated by his own system when the mail could not be delivered.

What I'd do instead? Read the documentation, listen to the advice people give me when I ask for help, and in general apply a modicum of common sense to what I'm doing.

Oh and, in particular, I'd fix the 'local_user' mail router in the configuration so that it only matches mail to existing users, letting the rest get rejected at SMTP time as it should be. As was suggested in the first email response lkcl got on the mailing list.

callouts via LMTP, posted 24 Feb 2007 at 09:45 UTC by lkcl » (Master)

dwmw2, yep - i had great fun completely bolloxing everything up.

the thing is: i _did_ put in sender-verification, using LMTP.

the 550 mailbox error coming back from the cyrus server was totally ignored.

this is what pissed me off: receiving advice that basically indicated that i must be completely wrong, because the issue had been discussed to death already.

and yes, you are absolutely right about seeing bounces on my own queue but there were so many of them they were irritating me beyond belief.

finally i settled for not having _any_ local users and everything virtual domains, and utilising the virtual domains concept to effectively implement the suggested solution of putting usernames into text files, utilising existing work by someone else.

the advantage of using the virtual domains config is that i can now offer cyrus mailboxes to any of my domains, rahther than having lkcl@lkcl.net _and_ lkcl@travellinggourmet.co.uk being the same user.

mmfckndem nlc.n, posted 24 Feb 2007 at 09:46 UTC by lkcl » (Master)

cf4fdte

katie-baby :), posted 24 Feb 2007 at 09:47 UTC by lkcl » (Master)

my baby's first post, dribble-dribble :)

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page