A circuit-switched email network

Posted 22 Sep 2004 at 19:16 UTC by ahosey Share This

Right now the global email network is pretty much a large scale packet-switched network. Packets (emails) are passed from node to node, on a potentially dynamically changing path, until they reach the endpoint. Individual hops are synchronous but the system on the whole is asynchronous. (Deferrals.)

But what if we switched from a packet-switched model to a circuit-switched model? What if every node on the message path (including forwarding addresses) holds open its connection to the previous and next nodes until the final destination MTA accepts the message? This solves a lot of problems. SPF works without SRS. Joe-jobbing is gone because there is no "bounce message" - the origin MTA is just told "nope, sorry." How many messages does your site MTA have queued right now? 40000? 80000? And 99% of those are queued because they are bounced spam but the sender address is bogus. Imagine that those are all gone.

I'm not suggesting we totally do away with retries. If the original sender's TCP connection is broken before he receives acknowledgement of final delivery, this is exactly analogous to when the connection is broken during a DATA transmission in the current model. All of the intermediate MTAs are allowed to drop the connection and pretend they never saw the message. It is the responsibility of the original sender to resend (if the break was an accident) or screw him (if the break was malicious.)

It might sound crazy but I think it would work because the Internet on the whole is much faster and much more reliable than it was 20 years ago. The timeout on the entire circuit wouldn't have to be too long - 5 minutes maybe. I think the increased TCP overhead on an individual MTA would be offset because that MTA would no longer have dozens of TCP connections sitting idle trying to deliver bogus bounces. MUAs do not necessarily have to be penalized in this model because we're headed for a world where all sites are going to have SASL-authenticated central MTAs anyway. The MTA can accept the message from an authenticated MUA and allow the MUA to disconnect, then start the virtual MTA circuit. If the message is given a temp deferral the origin MTA - not one of the intermediate MTAs - queues the message and takes responsibility for trying again later. This, combined with SPF, would push negative impact and accountability back to the origin sites of the messages, where it belongs.


Nice idea but ...., posted 23 Sep 2004 at 14:17 UTC by sarum » (Journeyer)

a) your average MTA would need to be on a machine where the real limit of open ports was very very high.

b) I thought that the source MTA was suposed to only talk now to the destination MTA, ie relaying was a bad thing and is mostly switched off, so this idea should not be needed now.

c) if you did do it you are still trusting the fact that the proxy MTA trust for the source MTA is valid.

d) such a change of the SMTP protocol is unlikely to become common place.

e) there is no such thing as a relable network, to assume it will just make process spin waiting for network connections to come up.

SMTP could be circuit switched, posted 23 Sep 2004 at 20:43 UTC by richdawe » (Journeyer)

sarum wrote:

d) such a change of the SMTP protocol is unlikely to become common place.

SMTP typically works in a store-and-forward fashion. Basically as soon as you return a 250 success code to the DATA command, you have accepted responsibility for delivery of the mail. If you can't deliver it on, you generate a bounce.

But there's no reason why you couldn't proxy the mail. This could get a bit hairy in the circumstance where the final server has 250'd the DATA, but some connection in the middle gets broken. What do you do then? Send the mail again. What happens if it's 250'd again, but an intermediate connection breaks? Retry. Etc. Soon you end up with a bunch of duplicate mails. The original poster (ahosey) mentioned TCP here - some mechanism would need to be added to SMTP to avoid duplicates. There's already the Message-ID header, but that would require SMTP servers to parse the mail, which is horrible. Perhaps some ESMTP extension could be added to add the Message-ID after the DATA command? Then the destination server could say &quote;already accepted that" - that might not scale too well.

b) I thought that the source MTA was suposed to only talk now to the destination MTA, ie relaying was a bad thing and is mostly switched off, so this idea should not be needed now.

c) if you did do it you are still trusting the fact that the proxy MTA trust for the source MTA is valid.

Not all mail is sent from originating SMTP server to the destination SMTP server. There are various reasons: firewalling, content filtering gateways, virus scanning gateways, scalability (e.g.: large companies may have multiple SMTP hops before mail leaves their network), etc.

If you don't trust your internal network (big company example), you could use SMTP over TLS with client certificates, to authenticate clients to internal servers.

e) there is no such thing as a relable network, to assume it will just make process spin waiting for network connections to come up.

I agree with you. Chaining a bunch of TCP connections together will probably makes things much less reliable.

A good comparison might be with web proxies: how much less reliable is web browsing through a proxy compared with a direct connection (software bugs aside)? I guess there you'd typically have two HTTP connections chained. Having said that, HTTP is slightly different - it's pull rather than push; the proxy could keep the client's connection open, while it retries. Anyway, now I'm waffling.

exim4 teergrubing at MTA time (using sa-exim), posted 24 Sep 2004 at 12:47 UTC by lkcl » (Master)

now where did i see it. marc.merlins.org somewhere on there look for sa-exim.

sa-exim is a little c-code hack which is run at MTA time from an exim4 server. it does a lot more than just run spamassassin: it does reverse DNS lookups for the MX record, makes back-connections to the sender's SMTP server to verify it actually exists.

sender verify i think this technique is called.

it's problematic because some people send email messages from fake locations: some send from badly configured ISPs who don't give a shit about actually maintaining their DNS records - all of these things will stop email that you _might_ actually care about from being delivered.

teergrubing is german for "tar-pitting".

you can do a _lot_ of abominably nasty things - and quite a lot of good ones - if you do spamassassin checking at MTA time.

i had this running on my machine for a while: i managed to get the spamassassing "reject" score down to 1.2 (!!!) as a result, and had a whitelist of about 30 friends' email addresses / domains which were a bit iffy.

Re: SMTP could be circuit switched, posted 28 Sep 2004 at 14:11 UTC by sarum » (Journeyer)

richdawe wrote:
Not all mail is sent from originating SMTP server to the destination SMTP server. There are various reasons: firewalling, content filtering gateways, virus scanning gateways, scalability (e.g.: large companies may have multiple SMTP hops before mail leaves their network), etc.

Sorry, I had kind of assumed that when I was talking about a source SMTP server that it would be understood to mean that it was the 'authoritive' source for that domain which would be the gateway SMTP server in the case of a company sending to the general internet. Inside a domain it is possible to have relay SMTP servers that forward to the gateway one but this does depend on the size of an organisation. In our company in fact all users can only send email to the gateway server for delivery, if you get email from our domain and it does not come from this machine you can assume it is spam.

If you had each relay SMTP server having to proxy through to the gateway SMTP server and then on to the detination STMP (an who know this might be a proxy SMTP server itself) richdawe says this would be a big problem and the protocol would need to be change. This is why I said that there was zero chance of this change catching on. The number of SMTP servers that would need this change would now be too great and the fix is so marginal, if there is any.

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!

X
Share this page