Definitely exciting to see another very promising modern mail server option that seems to really be developing nicely. The major pain of self-hosting email for a long was ensuring delivery, but with SMTP relay services also getting quite slick that doesn't seem as much of a blocker anymore. Also means being able to clamp down even more tightly on the mail server at the network level as well as its own security since it only needs to talk out to the relay service and nothing else. In an ideal world there'd be a real solid secure DNS and in turn full e2ee email-like standard itself, and authentication could at least be one by certs in DNS.
But in the meantime email still fills an important role, and perhaps enough layers of options will get us close. After the Gandi.net sellout most recently this feels like propitious timing to me, I really dislike the typical email service pricing models. Paying just for the domains and relay, both of which are trivial to swap around at will, and then otherwise having that on my own infrastructure certainly feels attractive to try firing up again at least for a handful of domains.
> pain of self-hosting email for a long was ensuring delivery, but with SMTP relay services also getting quite slick that doesn't seem as much of a blocker anymore
Has the definition of self-hosting email evolved to include using a third-party SMTP relay service?
(It's always been a slightly fuzzy definition, and maybe the last time it shifted was when it included running on cloud servers/VPS rather computer hardware that you own.)
Can you recommend any SMTP relay services? I've been interested in self-hosting for a while, but building reputation to get reliable delivery seems like a full-time job.
I've looked into this extensively and there are essentially four options:
1. Self-host on a reputable VPS provider. Deliverability is usually not a problem if your provider actually takes action against spammers on their network. They tend to work pretty hard to keep up the reputation of their IP space. I have self-hosted my own email for well over a decade on providers like these with no serious issues. It's not generally too hard to test the reputation of an IP/domain before putting it into production.
2. You can use a dedicated SMTP relay service, but these are usually quite expensive and their customers tend to be bulk email senders for blasting out marketing wank. You probably don't want your domain to be associated with these anyway.
3. Another option is to buy a full-fledged email account from any of the common providers and just use their SMTP servers. (Make sure to set your SPF records appropriately.) Not expensive but typically far from free. And it seems silly to self-host your mail if you're literally paying a company to provide the same service...
4. This is less "self-hosted" but the cheapest and most reliable way to get your email out is to sign up for a cloud account at any of the major providers and just use their SMTP relays. Most will allow you to send from outside their network, after proper authentication is set up. Unless your monthly email volume is north of 4 figures, it will likely be free or cost pennies per month.
Don't forget that amongst email providers reputation is a thing, and if you go out and purchase a domain, it may be some time before you can actually use it. Lots of providers specifically penalize domains younger than X months old, they will "graylist" IPs that haven't talked to them before, and a few will flat-out blacklist entire gTLDs known to be heavily used by spammers.
Also, email reputation and deliverability only applies to sending email. While many of the advantages of self hosting only require receiving email - eg better control over your root of trust for account auth, different address per account to avoid your email address being used as a join key for cross-company surveillance, etc. Self hosting doesn't need to be an "all or nothing" affair. Set up your domain with whatever server setup you think you'd like and start switching account identities over to that. Then only after you've gotten comfortable running it and tested deliverability, start switching over your personal correspondence.
Utilizing fancy gTLDs can still prevent you from receiving mail. Not due to a decision to explicitly block it but because it doesn’t match whatever regular expression they use to validate. Notably, .email fails consistently due to it being >3 characters. I tried to convert to using first@last.email and there is a significant minority of sites that didn’t allow it.
Good point! That's still kind of orthogonal to deliverability though. In fact in line with my point, you're better off finding this out before you start transferring personal correspondence to that domain.
The only similar problem I've experienced is sometimes companies will get uppity if you put their company name in the email address you give them. But it's easy enough to just make up a difference nonce for those cases (or start your scheme based on opaque nonces for everyone). I'm still waiting for the other shoe to drop and surveillance companies to start discriminating against non-surveillance-company email addresses the way they do against VOIP phone numbers.
I've been using gTLDs for an email for several years now (about the time gTLDs came out). It was really rough going for a while but these last couple of years it's been rare for me to have an issue.
> 2. You can use a dedicated SMTP relay service, but these are usually quite expensive ...
I don't typically give endorsements, but I've been using DuoCircle.com since back when they were part of "dyn.com" and I've been very pleased. Apparently they have a free tier, but their current pricing is very reasonable to my eye. (I'm on an old annual plan that doesn't appear to be offered anymore...)
I've been doing a combination of 1 (good reputation IP) and 4 (using AWS in my case). I switched VPS provider a couple of years ago - before that I had deliverability issues to anyone using MS 365 (despite jumping through Microsoft's various hoops etc) - since then everything has been fine.
I also have my outbound SMTP server set to send via AWS if my email includes a particular custom header (which it also then strips out before forwarding on) - which means for domains I think might have deliverability issues I can deal with this without needing to make a huge effort.
I have been running my own services for years, all self-hosted, with different VPS providers: A2Hosting, Digial Ocean and now Contabo. I use ISPConfig and host about 25 domains and 200 email addresses. You have to configure SPF, DKIM, DMARC, reverse PTR address, but nothing impossible. Encountered some problems with some providers blocking the IP range of my VPS provider (Microsoft in particular), but if you send an email and explain you have a new server configured form scratch and carefully configured, they add an exception for your IP. Check your IP addresses are not list in any black list and you are done. I use Rspamd and I had to add some domains to the whitelist ... There are many servers out there (even of medium sized companies, very badly configured) that don't even respect the minimum requirements like a public registered and valid server name for the smtp server. I avoid any external SMTP service ... they can read all your emails ... There are many ready-to-use solutions, such as mailcow, which simplify the management of the mail server.
Forgot : I configured the compression with LZ4 algorithm (it saves a lot of space) and encryption of the mails. The encryption uses a master key. It is possible to encrypt every single mail file using the password of the account, but if the user forget or lose (and it happens sometime) his own password all the mails are gone ...
Another option to explore is to autmatically encrypt every mail with the GPG public key of the owner, again if he lose the GPG private key or the password, the mails are lost forever, but unfortunately I don't see any interest in this by my clients.
I think one just needs DKIM, SPF and DMARC. I had to go through that configuration change sometime after 2008. I have no idea if domain reputation counts. I certain get no special consideration from google and my domain might be older. I guess one needs an ssl cert for clients on iOS.
panix.com hosts my start of authority for DNS and they probably have a product that might fit your needs.
I've tried a few of the free or low cost ones and have found direct from my mail server provides better deliverability in all cases. I'm not sure how well the more expensive ones would do, although one would hope it would be better.
As far as senders, from what I've seen AWS SES is still probably the basic go-to for the HN type, pay as you go pricing looks to be quite good in this context and most of us are familiar with navigating AWS. Without any major experience, it seems to work in my light kicking of the tires so far. When I tried Postmark more heavily like a year and a half ago it seemed truly excellent for a more full fat flat per month service, and that's what I'd planned to move to already following the legacy GSuite sunsetting. Unfortunately bad timing for me, they finally decided it was time to move on and sold last year to a marketing company [0], and since then there have been significant price increases, elimination of non-subs, and a few concerning events. I think they were the last of a big grouping of '09/10 email startups to do the acqui-exit. Doesn't mean they won't still work and aren't mostly still fine, but something of note. Mailgun, Sendgrid and so on are all farther along the post-acquisition curve there. Last month there was a new one announced on HN called Resend [1] which is more development oriented but still of interest.
On pricing/ROI: most of the paid tiers for monthly plans seem to start $15-20/month now though with free tiers to experiment with first. I think self-hosting tends to pay for itself best if you fall into certain now neglected niches and have existing infra, or else are willing to pay some premium ideologically. Most email services now tend to squish a bunch of the actual underlying stuff into a specific payment model: mailbox (email address) is 1:1 with a person, and also covers storage, while people don't really think about sending numbers. Whereas underlying storage is actually dirt cheap particularly in the context of email, mailboxes are effectively free, but sending emails costs. So for example I have a bunch of domains and lots of email accounts at them, I was always in the habit of making heavy use of separate mailboxes for basic utility usage like a server sending a status alert (and that also means the server email address can be restricted and not have credentials fro my personal or work email etc). Low volume, tons of mailboxes, occasional big messages with logs and such is an absolutely awful fit for most mail services and getting worse. I also have reasonably solid self-hosting infrastructure already that I've amortized for other things, so at this point essentially adding another VM is quite efficient. For someone who falls into the general bucket, just going somewhwre like Fastmail or even GSuite or the like would almost certainly make more sense. $15/month would buy 3 of Fastmail's standard "users" (ie, mailboxes/different addresses). But I have way more than that, lots of which only send a handful of emails. Doing that with Fastmail/ProtonMail/Gmail/etc type pricing would be hundreds of dollars including $5/month accounts that receive nothing and might not send more than a handful of emails per year.
Anyway, that's my thinking and what I've been experimenting with so far. But ultimately part of the point/value of it all is that on the "difficulty of change" scale, moving to a new email address entirely is the worst though cheapest, owning your own domain and being able to point at a new email provider then is vastly easier but costs domain/year (this mid level is probably best for most people), and having merely to change relays on a server costs the most but is the most transparent. So trying to get out of the habit of thinking of these things as needing to be long term relationships. If a relay service isn't working for me with self-host or someone offers better I'll just move. I'll probably keep one or two addresses traditional too as fallbacks.
Amazing I was just looking for a good mail server to configure for my demo. Which reminds me since you folks have mentioned LiteStream, have you tried Marmot (https://github.com/maxpert/marmot); I recently configured Isso with Marmot to scale it out horizontally (https://maxpert.github.io/marmot/demo). I am super curious what kind of write workload on a sub thousand people organization will have and if Marmot can help scale it horizontally without Foundation DB. I always find the the convenience of SQLite amazing.
Congratulations! Very interesting project !
I have been running my own servers with ISPConfig with PostFix and Dovecot for many years, but this modern all-in-one solution seems better integrated and complete.
As others have commented, rspamd integration would be a great addition, although I think using Proxmox Mail Gateway instead would give more control with more ease. Proxmox doesn't need any special integration. It works like a firewall for mail.
ISPConfig, unless they've changed it, is a very dangerous control panel. ISPConfig3, at least, runs the control panel as a vhost on the same Apache instance as the users, which is Very Bad. I actually wrote an article a few years ago that roasted it pretty hard. Try Virtualmin for a more robust, secure control panel.
The problem is that the server can be configured from userspace. There's even a (horribly outdated) WordPress plugin for setting up email address from within WordPress. Super insecure, no matter how much SSL you throw at it.
I'm sorry, but I don't really follow. This tool, like many others, was created for exactly this purpose: to configure the server from userspace. The ISPconfig web interface doesn't write directly to the server. It creates a series of tasks that are executed in the background by another process with different privileges (and you are limited to a list of predefined tasks, you cannot do an "rm -rF /"). So... I don't think it's possible to run arbitrary code through the ISPConfig web interface, because it runs with the limited privileges of a PHP FPM process that writes a task to the database queue, which is later read and executed by another process.
The main problem for me is when the web server (Apache or Ngnix) fails to start or restart for some reason. In this case you are cut off from the ISPconfig interface and forced to fix things by hand.
Not sure if Stalwart recently got funding, but the number of participants shot up in the last week. Stalwart's popularity (stars/watch events) also shot up like crazy about 5 weeks ago.
Note, I'm not indexing the code history for both projects right now (they are queued but they probably won't be indexed for another hour or two) so the community insights is incomplete.
EDIT: Missed a section of the readme. Ignore the following.
Maddy mentions actual security features (DMARC, MTA-STS, DANE, DNSSEC, DKIM) in its overview and compatibility with rspamd etc. (which is rather vital if you want to properly handle spam).
Maddy (also) has a single maintainer. Development activity seems to be low, but the couple PRs I submitted (documentation fixes and cleanup) did get accepted.
This is the chicken and egg problem. It is far easier to write a good JMAP client than a good IMAP client, but we need servers and providers to support JMAP before it makes sense for app developers to write clients with JMAP.
Why is this not set up to filter spam out of the box? It's not really an optional thing at this point, and writing Sieve scripts that work well is not necessarily easy.
Given it's all-in-one thing, MDA/LDA is integrated, to best of my (very shallow, just ~30 minutes haphazardly checking docs and source code) understanding there is no separate delivery agent program sitting in-between the components, it's all a single process, but it has places where you can hook arbitrary external filters/transformers (https://stalw.art/docs/smtp/inbound/data#content-filters)
It is not directly in the README but there is a link in it to a getting started guide [0] which covers this. The user "zie" probably got it confused with the README.
Looks interesting, but something I always look for, is what it doesnt do and then try to find out why, partly because I'm not up to date with all the RFC's, so I couldnt tell straight away if those RFC's are the latest, proposed or deprecated.
It is new, but it would be interesting to see a list of customers using the system.
One concern is that Email is designed to be highly modular in the UNIX sense. Does this eliminate some of this modularity? For, instance can I still use Dovecot for IMAP, POP3 if I want?
I’m just about to set up a couple of new domains, and was procrastinating because email. This looks perfect, something new to play with and get email set-up done at the same time :)
Those are definitely “step two” once you have managed to get the relevant services running. Fortunately, spam and basic phishing filtering isn’t too hard at small scale. You can get free access to very high quality blocklists for non-commercial use and you can use rspamd’s fuzzy hash API and various phishing URL data sources for free as well.
The really hard thing is to self host outbound email delivery. You almost have to use a relay service to get mail delivered these days. Most IPs at cheap hosting services are in a bad neighborhood and will be treated poorly by association. On the other hand, most transactional email services have a generous free tier that would work for a lot of self-hosted setups.
Relaying through MailChannels is free with no volume limits if you do it via Cloudflare Workers. Would be nice to see someone merge that with this project as an option.
In particular the DATA (or BDAT) stage configuration; it runs a command and send the headers / data to stdin and receives back the modified message from stdout.
But in the meantime email still fills an important role, and perhaps enough layers of options will get us close. After the Gandi.net sellout most recently this feels like propitious timing to me, I really dislike the typical email service pricing models. Paying just for the domains and relay, both of which are trivial to swap around at will, and then otherwise having that on my own infrastructure certainly feels attractive to try firing up again at least for a handful of domains.