The author is still pretty far behind the curve with this info. Unfortunately while these policies are great to have (well these days they are more or less necessary), the simple fact is that they are so often misused or improperly maintained they don't really stop that much; a SPF mismatch is simply treated as another item to score the likeliness that a message should or should not be blocked. A DKIM signature is almost completely useless in this system since all it can do is prove a message was handled by a specific domain holder and not modified. So if you want to modify it, just remove the signature. If you want to send a forged message, don't sign it. There's nothing in DKIM itself to tell a recipient "my messages must be signed."
The new hotness is a DMARC record, and it finally allows mail senders to basically say "Here are my suggested handling policies for DKIM and SPF, and please actually enforce them because they are properly maintained and tested." Best of all it has a feedback loop, so you are able get reports of how receiving hosts are treating your messages and forensic reports of forgeries which can help with detection of several types of phishing threats.
It's still a long way from perfect, but it's better. It doesn't help deliverability though. For that you will want to stop emailing shit that people don't want.
In short: ARC builds on top of DMARC to fix some cases that DMARC didn't handle well. (And DMARC, in turn, builds on top of DKIM and SPF.)
DMARC is overall a very good thing, and has significantly cut back on the effectiveness of forged email for spam/phishing, at the expense of mailing list/forwarding headaches.
Hopefully it's been good for other people... I haven't seen any benefit personally. :\
On a related note: do you know if DMARC feedback leaks information to the sender on whether or not you opened an email, or reported it as spam, or similar? If so, do you know why this isn't considered a privacy or security issue?
DMARC feedback is only based on the MTA (Message Transport Agent) handling the mail. I don't have much experience with DMARC reports but Senders typically use it when they are first Setting up DMARC. It is up to the receiving MTA to send the inbox/spam result according to the RFC. I think that the reports are aggregated so single inboxes are not able to be detected. I don't think many people actually send / follow up on DMARC reports imo.
DMARC is for the sender ARC is for the receiver/intermediary. The problem with DMARC (really it's not a problem with DMARC but we'll play like it is) is that if you do something like set up a Gmail account to forward mail to your "real" yahoo account then yahoo might not accept the message from Gmail since from yahoo's perspective Gmail is trying to dump a forged message into your box that they didn't originate. As a sender I don't really give a shit that the other end is broken by some lazy person.
A message can still be considered a DMARC failure since the check depends on both SPF and DKIM. You can have various failures like recipient rewrites and general SPF/DKIM failures which will fail DMARC a significant amount of the time.
Basically mailing lists break DKIM signatures, and ARC fixes that. It also fixes certain types of automatic message forwarding, e.g. when you have account forwarding turned on, not when you hit the forward button.
I've been thinking for a while that there should be a movement to allow people running mailservers at their home again.
"Again" as in get other actors in the field to allow/ease that: spamlist managers should stop blocking residential address by default and make it easier to appeal. ISPs should make it possible and easy to get a reverse-ptr with every fixed ipv4 allocation.
In this monitoring age I want to be able to handle my private conversations (and my data) on my own iron (running somewhere in my house).
All this impediments to running your own mailserver and take back control of your own data really look like a push towards state-/corporate-backed espionage of individuals.
It's not 1995 anymore, most of us have an internet connection well capable of sustain enough mailboxes for a whole family.
I'd absolutely be all for it - I've been running my mail server since the 90s, and used to run it out of my closet.
There are two major, somewhat interlocking problems, though: home users with compromised machines currently represent the population of home "mail servers" (spam malware) at the moment. Selling this involves convincing mail administrators that not simply blackholing all of what is currently a cesspool is a good idea. That's not easy.
The second problem is that fixed-IPs, open well-known ports and DNS have become a product differentiator for ISPs. Those are "business" features; dumb ole' consumers don't "need" them.
I don't see any compelling argument that would convince either prong of that trap - hell, I agree with you, but as a mail admin both professionally and personally, the first one gives me hives, thinking about the years of work it'll take to re-stabilize a functional antispam infrastructure to handle it.
And after that, you need to convince Comcast to change their revenue-extraction plans and generally be less dickish.
If just you want a mail server now, your best bet is a cheap VM somewhere. It can be really low-end and cheap.
Other way I've thought of effectively achieving this is by using VPS for static IP outside of residential ranges and home machine connected by WireGuard VPN. WireGuard handles roaming easilly, so dynamic IP is no problem, IPv6 is no problem, and the mail will stay on your home machine. You'll also bypass your country's/ISP tracking somewhat if you place your VPS outside of the country.
I run home mail sever for a while. I used to have smtp at home but a few years back I switched to a smtp proxy/forwarder on AWS ($5-6/month) connected via TLS SMTP on custom port at home (which also serves IMAPS on a custom port). Once a year I have to update DNS when my Comcast IP changes. Otherwise it is completely trouble free setup.
Dynamic IP is a bummer, but outright closing ports is a direct violation of net neutrality. It's baffling that it's tolerated at all.
An ISP's job is to forward IP packets. Not read them. TCP/UDP ports number are the content of those IP packets—not the meta data. Filtering based on such content is already a form of discrimination. What's next, deep packet inspection?
Some ISP filter by default, but they do this at the router level, and you can either configure or change the router. That's okay, possibly even beneficial. Blocking at the backbone level however is just evil.
Counterpoint: Unless you've been in network engineering for an ISP with tens of thousands or more residential customers, you have no idea how much spam/abuse/trojaned PC traffic comes from residential users. There's a very good reason why ISPs block some of this, for the protection of their own users. It's even necessary to block port 139 windows file sharing or people will connect PCs with totally open file shares straight into their cable or DSL modems.
It's a real headache. The average HN user is in like the 99.9th percentile of residential internet users. HN users, give them a proper DIA feed, sure. Everyone else, I'm wary.
I've done it at the scale of a few hundred and I still feel I've seen more than enough to 100% agree with you.
I've seen customers max out gigabit links with outbound spam from a compromised desktop. Contacting the customer leads to ignored emails and unanswered voicemails at best, angry Facebook rants about "getting in my face" at worst.
Blocking ports by default is okay, and will address the clueless user problem (clueless users won't change the defaults to begin with). It's blocking them no matter what that's evil.
Now I live in a country where most ISPs lend you a "box", a router, which is capable of doing all the blocking they want, and is configurable by the user. One of them uses that box to block the SMTP port by default, and let the users who want to change that setting.
> The average HN user is in like the 99.9th percentile of residential internet users.
And we have the right of doing this.
Anyhow: the dynamic ip problem might be solved with ipv6. The spam problem... Imho graylisting + spf + dkim + reverse-ptr should be enough to kill spamlists once and for all.
IMO, we passed the point years ago when gr[ae]ylisting was still a useful technique. For anything within the last decade or so, I would say it probably does more harm than good.
Good luck getting the average residential ISP to edit rDNS records for you. It's a special thing they will only do in certain circumstances and as a special one-off thing. Automated provisioning tools and customer service are not set up for it. I'm not saying that is a good thing, or satisfactory, but that's the current status quo.
I'm all in favor of ipv6 everywhere but the current reality is that a huge number of organizations presently have v4 only smtpd in their MX records, if you try to do a pure v6 approach you will break delivery to/from something like 65% of the internet.
OT: most developers have probably not been in network engineering for an ISP. If you've written anything about your experiences, or if you could be persuaded to, I believe that a lot of people on HN would be keenly interested (myself included).
I'd be totally willing to answer any specific questions. Not sure if I have the free time or motivation to write a full guide to network engineering, the best I can really do is point people at the right reference sources. I'm honestly not nearly as qualified to do so anyways, compared to far more senior people who work for huge transit ASNs.
I probably mean more in the "human interest" relation of experiences sense than anything like a comprehensive guide to the profession. I'm not wholly ignorant of what happens to bits after they leave my router, but I know nothing about the people involved in making that happen, and I don't get the impression that it's widely written of. Anecdotes about daily challenges tend to be interesting; I would probably have gone quite a long time without ever knowing about port 139 otherwise I'm sure. But, I certainly wouldn't want to try to compel anyone to write anything they're weren't going to write anyway, so please don't take this as anything other than an expression of interest.
> If just you want a mail server now, your best bet is a cheap VM somewhere.
How do you make that work? In my experience, the mail will just be blocked because the IP has no reputation and the range is probably grey/blacklisted.
> The second problem is that fixed-IPs, open well-known ports and DNS have become a product differentiator for ISPs. Those are "business" features; dumb ole' consumers don't "need" them.
I understand the fixed ip thing, but closing well-known ports by default should be considered a form of censorship.
It is like selling a mobile phone, except the user find out that they can only listen but cannot talk.
It seems like there might be an opportunity here, for a hybrid service comprised of "client" software installed on your own hardware, which actually hosts and runs the email inbox, and central server-hosted software that takes care of configuring the client and all the associated authentication schemes.
The server-hosted software would never see the actual emails; it would just manage the provision of email service. Perhaps the server could store locally-encrypted inbox backups too.
If the service was responsibly run, it could even "vouch" somehow for the email inboxes it provisions. So even though the emails come from, like, Comcast ISP address space, other ISPs would recognize a trustworthy source of email. (Dynamic IP provision would complicate this part.)
But the big question is... what's the addressable market?? It would likely be even more expensive than something like ProtonMail. You're not going to build much of a business off of 347 neckbeard paranoics. (I joke with love.)
> The server-hosted software would never see the actual emails; it would just manage the provision of email service
...
> If the service was responsibly run, it could even "vouch" somehow for the email inboxes it provisions. So even though the emails come from, like, Comcast ISP address space, other ISPs would recognize a trustworthy source of email. (Dynamic IP provision would complicate this part.)
If you are willing to to let the server-hosted software have some visibility into your mail traffic (but not the content), there's a way to handle this that would not have problems with dynamic IP addresses [1].
Briefly, whenever your email client needs to send a mail through your Comcast outgoing SMTP service, it would first contact the vouching service and provide a hash of the message ID of the message it is about to send (and maybe the from address?).
The vouching service would verify that it is talking to one of its legitimate customers, and then return a line for the client to put in the header of the outgoing mail. This line would contain a copy of the message ID hash (and from address?), a timestamp, the client IP address, and would be signed using a key of the vouching service.
The client puts that line in the header and sends the mail through Comcast.
The SMTP server that Comcast relays it to can look at that line, check the signature, and if it verifies it can compare the information in that line to the corresponding information in or derived from the rest of the header. If the information all matches and the timestamp is in the recent past, accept that the mail really came from the person it claims to be from.
This would give the vouching service get a good idea of how much email you send and when you send it, and also give them hashes of your message IDs, so it is not ideal.
A problem with this is this it depends on every SMTP service that Comcast might relay your mail to (such as the SMTP server for every domain you send mail to) being set up to verify the signature of the vouching service you use. This would be a pain in the ass.
This could be addressed by having the verification of the signature and header data done at your ISP's (e.g, Comcast's) SMTP service. That way you just have to pick a vouching service that Comcast recognizes.
Comcast would then do the check on your outgoing mail to see that you have used a vouching service they recognize and the data matches. They can then add a header that says they believe the mail is verified. For any mail that they cannot verify thusly, they can add a header that says it is not verified.
Now instead of having to know about and trust your vouching service, the SMTP servers of the domains you mail to just have to trust Comcast. That seems much more likely to come about.
If Comcast (and other residential ISPs) were willing to do this, why bother with third party vouching services? Why not handle the whole thing themselves?
One reason would be that providing vouching service requires getting more involved with the details of the customer's setup than an ISP might want to deal with. They might not want to have to deal with support questions dealing with how to get the vouching plugin or whatever working with the various email clients and operating systems customers use.
[1] Maybe...I've not sat down and tried to actually design the protocols for this. This is just off the cuff.
> I've been thinking for a while that there should be a movement to allow people running mailservers at their home again.
This will be really hard, because unless you're willing to give home users actual permanently assigned static IP addresses and treat them like medium sized businesses (IP space SWIP to customer, etc), any given /27 to /23 sized v4 DHCP pool of residential customers is going to have a ton of abuse coming from clueless people running infected routers, virus/worm/trojan infected home PCs, etc on it.
Getting the average residential ISP to do proper rDNS for your IP address is impossible in many places.
Therefore when recipient SMTPd are scoring the IP space reputation of your residential netblock, your home mail server is going to be mixed in with all the shit.
I did run mailserver(s) at home for a while in early 2000s.
Proved difficult when I once went on holiday as I switched it off and then it got flooded with emails (mostly spam) when I got back (from backup mxs run by friends). Had my ISP call me as worried my PC was on a botnet due to sudden traffic. Explained I was running Linux, and they immediately said: "thats fine then have a nice day". Thought perhaps they would need more detail but no. ¯\_(ツ)_/¯
As others have said. Today's cheap nano-esque cloud instances (with swap disk enabled) on AWS and similar is easier to have running 24/7, and less risk it will burn your house down...
I'm curious as to what privacy you think you'd gain from storing your mail at home? Unless everyone encrypts everything they send you, most likely most of that mail will go through a government monitored connection at some point. It wouldn't be that hard for them to reconstruct your mailbox if they wanted to.
Also, why not keep it on a hosted server somewhere? You could encrypt it at rest to prevent anyone coming in and getting the data, which would make it as private as storing it at home. And then you don't have to worry about deliverability or backup power as a nice side bonus.
You would need to convince your ISP to allow you to modify your PTR. There are many servers that implement one form or another of the S25R methodology [1]. That is, simple regex against your FCrDNS to see if you are a generic network device or generic home user. The page I am linking to is an example of the concept, but many anti-spam tools implement this in varying degrees.
It would probably be easier to just build an AMI for one of the existing mailserver-in-a-box systems to proxy email via an AWS t2.nano instance. You can run a t2.nano with a 3-year reserved instance for $30/yr, and then proxy all your mail to and from your home server on non-standard ports.
In my experience, spam blacklists have significantly decreased in efficiency over the last 10 years. I think the biggest e-mail providers stopped contributing to them, so the user-reported lists are almost unused. The honeypot lists lag behind the spammers by a few days, so plenty slip through. They do trim out 85% of my incoming spam, but that last 15% is still a lot. Back in ~2013 they cut out more like 99%.
Today, the single most effective thing you can do if you run your own mail server is to completely block all gTLDs. Screw 'em, they are 99.9999% spam.
Plenty of spam has valid SPF and DKIM records. They are sent through legit services, either through cracked credentials of real users or rotating through new accounts.
It also doesn't seem like anybody cares about abuse@/spam@ reports anymore...
> completely block all gTLDs. Screw 'em, they are 99.9999% spam
[citation urgently needed]
Anyway, in my own experience with many years of self-hosting mail - until giving up and going Fastmail a couple of years ago - the real problems were in sending. No matter what rigorous level of DKIM'ing and ip-hygiene and whatnot, Google and Microsoft - Microsoft to a grotesque degree - would randomly ditch incoming mails from my server. Would sometimes happen in the middle of a conversation thread, and for the most part without warning.
The kind of person using Hotmail is typically not someone you can convince that the error lies in his end.
> completely block all gTLDs. Screw 'em, they are 99.9999% spam
That's for _my_ e-mail. Your experience may differ. Perhaps you communicate often with people on .loan domains.
E-mailing Microsoft accounts is just not an option. They have no process for fixing incorrectly blocked IPs. I take the same approach with Microsoft as with gTLDs...
Never had a problem with any other mail provider. Google has never blocked or spam-holed me.
Three years ago I investigated the effectiveness of some realtime black lists (RBL) [1]. I didn't find it worth implementing.
I also looked into the result of SPF [2] and again, found it not worth implementing. I rechecked the results for SPF earlier this year [3] and the results were the same.
Been there, done that. And then Microsoft answers my complaints (after my mails never showed up at my brother in laws outlook.com mailbox):
We have reviewed your IP(...) and determined that messages are being filtered based on the recommendations of the SmartScreen® Filter.
Email filtering is based on many factors, but primarily it's due to mail content and recipient interaction with that mail. Because of the proprietary nature of SmartScreen® and because SmartScreen® Filter technology is always adapting and learning more about what is and isn't unwanted mail, it is not possible for us to offer specific advice about improving your mail content. However, in general SmartScreen® Filter evaluates specific words or characteristics from each e-mail message and weights them, based on their likelihood to indicate that a message is unwanted or legitimate mail.
Unfortunately, after reviewing the information you provided and in compliance with our mail policies, we are unable to offer immediate mitigation for your deliverability issue. However, we have some specific recommendations for you to consider that can help you to improve deliverability over time.
Yes, it's annoying because mailbox owners have zero control too. If your brother in law complained to outlook that some emails are missing from his inbox/spam because outlook decided to reject them outright, he would get the same respnse. "We have smarter systems(r) than your judgement, sorry." (Remeber, you can't click "not spam", if the e-mail was not delivered at all)
The problem is the burden of not upsetting a mail filter got somehow shifted to the sender in case of big services filtering mail, which is ridiculous. Customers should talk to their providers if they're not receiving mail, because providers don't deliver it. But the correcting feedback loop is not there or/alternatively is much longer with more steps than in case of sender getting a bounce.
Much more important than SPF and DKIM is Forward-confirmed reverse DNS.
You need SPF and DKIM, but before that you need FCrDNS.
Without FCrDNS, your server will look like a dynamic IP address.
It's also not enough to monitor your own dedicated IP address. You need to monitor your entire /24 IP address neighborhood at your hosting company. You might be sharing the same /24 IP address space as a hacked Wordpress installation and find that Outlook.com will block you just because of that.
After that, you might run into issues with large SpamAssassin setups hosting hundreds of thousands of mailboxes blocking your email because of arbitrary rules such as too little text combined with a large image etc. Plain text email will improve your deliverability for these setups.
Yes, Microsoft blocked me sending email from my shared hosting to my Hotmail. The email address was whitelisted, and established email back and forth to that domain, and no spam ... but another server at the same host had previously been blacklisted in one blacklist.
SPF nor DKIM helped, I don't own the mail server nor want to pay Microsoft's service provider to verify me.
In the end I had to bounce the emails from a third-party address. Which you'd think shouldn't work -- certainly not when whitelisting doesn't -- but solved the problem.
So to avoid being flagged as a spammer, do the spammiest thing possible: set up a new, free mailbox and bounce your mail from there. SMH.
rDNS is of course extremely important. That was important even long before the existence of SPF and DKIM.
The neighboring IP space and the general IP space reputation of the ARIN/RIPE/APNIC/AFRINIC (or whatever) block assigned to your ISP is extremely important. Your neighbours can ruin everything for you. The only way to ensure that this doesn't become an issue is to host your smtpd at an ISP that has a highly attentive, clueful NOC and responsive abuse policy that actually deals with network abuse (real abuse, not DMCA notices).
If you're looking for an easy way to test your outgoing mail setup, I recommend mail-tester.com. They give you an address to send an email to, then analyse it and tell you how you can make it more deliverable.
The great fallacy of spam filtering is that access control SHOULD be
probabilistic whenever it is easy to implement that. Let's look at the
physical world where implementation is harder. It's very unlikely that
I'll arrive home from work at 4:30 AM, AND that I'll be driving a
rental car instead of my own car, AND that I'll be wearing new shoes
with a sole pattern that my smart walkway hasn't seen before. So,
ideally my home security system would automatically call the police.
Right?
I don't think that's how it works. You train your spam filter both with spam and ham and the sets need to have comparable sizes for it to work reliably.
So in your example, you'd train it on your patterns of behavior AND patterns of burglars. So unless you had a clear burglarly behavior, the system would probably not flag you out for just behaving differently.
I've spent many, many hours playing with my mailserver and adding all of these extra features and security settings to try to keep Gmail and others from denying mail from me.
It turns out that no matter what I do, I'm pretty much out of luck because I'm sending from a non-standard (.link) domain which gets treated as spam by Gmail as well as others. I get a perfect score on every email test site I've tried, but I still go to spam (or worse) regularly which is a big problem if I ever want to use my personal email for anything important.
For anyone looking to host their own email using a personal domain, I'd suggest sticking to something like .com to avoid the hell that I've been through with this.
Would love to get feedback from the HN community. We're a decently sized sender following what we think are best practices (sending to engaged, dkim, etc.) however our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards. (Wouldn't a double opt-in result in the same situation). What are the most impactful levers to pull in your opinion?
> ...our gmail deliverability is rock bottom and it's been difficult to improve. Every other provider is to benchmark or better. ... What are the most impactful levers to pull in your opinion?
I'll be blunt, and maybe others will disagree with me, and say that I don't think there are any impactful levers you can pull.
In my experience, Gmail frankly does not give a shit about receiving e-mail from any senders except ones that fall in one of two categories: a) very large so users will complain in large groups if receiving from them is impacted (put your Outlooks and your Yahoos and your huge residential ISPs in this category) or b) technically-savvy senders that aren't quite as large but Gmail employees routinely interact with so they're otherwise "trustworthy" (Fastmail is an example here).
I've ranted about here but I've had terrible success being smaller than a multinational ISP and sending e-mail to Gmail users. Messages will silently disappear, even though Gmail's SMTP server claims to accept the message, and the postmaster tools are less-than-useful (that might be my one suggestion, sign up for them on behalf of your domain and see if you get any reports; I didn't, 'too small'), and no amount of tinkering or asking would help.
I admit I'm just a SMB org whose business isn't sending e-mail--it's "just" a communications tool for us--so I had this option: I threw in the towel and switched to Fastmail and have had no problems since.
> That might be my one suggestion, sign up for them on behalf of your domain and see if you get any reports; I didn't, 'too small.
I had the same issue, what I found though is that DMARC reports give at least some insight into the process, but that's about all the information I've managed to get from Google.
Gmail is all about user engagement. If you are sending mail that users are not reading or are reacting negatively to, you will eventually be blocked. Have you used the posmaster tools they make available for delivery insight?
> One issue may be high hard bounce rates on our very first email sent, but we don't send emails to bounces at all afterwards.
This is a big red flag, though. Are you saying it just happened one single time? In that case, change your IP and start over.
Or are you saying that you regularly trigger a lot of hard bounces? Because if that is the case, that might explain it all right there. High rates of hard bounces are taken as evidence of spam or other nefarious activities, since the typical reasons for high hard bounces are a) a really old list (implies it has been transacted somehow, therefore not opt-in), or b) enumeration attempts.
Make sure SPF, DKIM, and From domains are all at the same root/parent domain. AKA "alignment" in DMARC - this can help even without a full DMARC record.
Tighten your engagement segmentation for Gmail. Repeat every 2 weeks if you don't see improvement. If you get down to 30 day actives with no improvement, the problem is almost certainly elsewhere. Occasionally it can be related to external factors - 3rd party email traffic which might use your domain, say if you buy advertising in other newsletters, bad SEO practices (think 'blackhat' rather than 'mediocre'), or poor customer experience with your business/website.
Hard bounce rates on your initial email are always a little elevated vs known recipients, but if you're above 2-3% that's too high and you should work to fix problems with the signup process.
c) the IP block your smtpd is in isn't in any RBLs?
d) You're not sending commercial bulk email?
e) Your mail server identifies itself properly to the other smtpd?
f) Your mail server properly implements TLS1.2 opportunistically?
I run my own mail server and have zero delivery issues to gmail. But I do not run marketing mailing lists, nor bulk communications of the same mail content to many customers, it sounds like you're mass-sending mail.
Nice article. It's worth noting that it's not just the reputation of the sender domain you should be looking at - the reputation of the links in the body also plays a big part. In fact over at https://emailoctopus.com, this is the number one issue we see for legitimate senders landing in spam.
The new hotness is a DMARC record, and it finally allows mail senders to basically say "Here are my suggested handling policies for DKIM and SPF, and please actually enforce them because they are properly maintained and tested." Best of all it has a feedback loop, so you are able get reports of how receiving hosts are treating your messages and forensic reports of forgeries which can help with detection of several types of phishing threats.
It's still a long way from perfect, but it's better. It doesn't help deliverability though. For that you will want to stop emailing shit that people don't want.