I feel like this solution is optimizing the wrong problem.
The bulk of work with managing a mail server (these days) isn't software setup and admin. On the receiving side, it's all the work dealing with abuse and attacks. On the sending side -- and this is the tough one -- it's getting sites to accept your email. When I finally gave up managing my own mail server (about two years ago), I found that about every six months I was involved in some panic where some large mail provider (Microsoft and Google most frequently) decided they didn't want to accept email from my server. Solving these issues is neither easy nor quick.
These days I'm very happy to pay somebody else to run email services using my provided domains.
Spam is a solved problem, thanks to SPF and DKIM. But despite doing all the right things, Microsoft and Google continuously block and rate-limit delivery.
Case in point: we deliver 20,000 booking confirmation emails every day, all requested by users and not spam. We have perfect Postmaster Tools metrics: absolutely zero reported spam, 100% IP reputation, high domain reputation, zero feedback loop spam, 100% encryption. But Gmail still rate-limits us daily, so their customers get their booking confirmations several hours late.
It's getting to the point where running an independent mail server is impossible. Yet another part of the open internet bites the dust.
How is spam related to SPF and DKIM? Those prevent forgery, but if a spammer actually owns a domain, they can send you whatever they want. That's where the majority of spam comes from, so it's far from a solved problem.
SPF and DKIM allow using domain reputation instead of IP reputation. Especially with IPv6 IPs are fairly easy to aquire and can be "stolen" either by spoofing/highjacking the network or more easily by compromising some hosts and sending from there. Hijacking a domain is generally harder as you need to compromise a mail server to get both the IP and DKIM key. And there are less mail servers than WordPress instances to hijack.
Sure, you can buy domains but that leaves some paper trail and generally you will see added spam scores for new domains.
So SPF and DKIM don't improve spam scores on their own, but the give a much more reliable path to building reputation.
Spam is only solved by the large mail providers (who have enough data to act on it within a split second).
After the great Gmail exodus two years ago (when they killed then unkilled legacy domain accounts), I moved to MXRoute — and the amount of unfiltered spam I get is insane. It's a daily nuisance. I have added about 200 filter words now, blocked hundreds of e-mail addresses, but it's next to impossible to filter out sometimes.
From conversations and threads online by the MX Route admin, it seems like this is a nearly unsolvable problem, still. Spam evolves so fast and uses throwaway one-time approaches only, so once you've filtered for a pattern, the next campaign looks completely different.
So, yeah, sorry about the long response, but I don't think Spam has been solved, at all, at least not if you're using an e-mail that's been around for longer than a few years (and appeared at least in one leak).
I am using a small German email provider and I can't remember the last time I got spam.
What's even crazier is that they are so confident in their spam filter that they simply reject mails classified as spam. But that hasn't been a problem since the few years I have been using them as well.
I have a wildcard email on my domain. In fact multiple domains. Multiple of these have emails listed in bug trackers, git repos and in plaintext on my website.
The only filter I have is a couple of regular expressions to block the most lazy of spam at submission time. Even without that I got less than 50 spam mails a day, usually quite a bit less - and almost all of those are automatically sorted into my spam folder. How long does it take you to glance over a low two digit number of emails each day? Not a lot for me.
I think the "spam" problem is overrated. Do you get more spam than me or do you just have different expectations?
Not only are the blocking. They also send bounce spam although they know with SPF record, that the original mail was send by a server that's not allowed in that domain.
We have a Microsoft email subscription (all our mail is hosted with them), and their smtp server rate limits how much mail we send to our customers. We can’t even send 30 emails at once. We had to implement retry logic with back off.
Have you spoken to an account manager there? I send way more than that on a regular basis. I do it sequentially for simplicity, I wonder if that makes a difference.
Really hoping ENS can step in here and fill this gap. There's a lot of work being done on social and messaging infrastructure. Using that as an email and chat handle and social handle would be huge for maintaining open messaging. Unfortunately this relies on adoption and people living past their bias and admitting there's use cases.
I’ve been in the Ethereum ecosystem for over six years.
I’ve heard “there’s a lot of work being done on…” literally thousands of times. Yet outside of the crypto-sphere I’ve only met a handful of people who’s life would be impacted in the slightest if the entire thing disappeared.
Ethereum is eight years old and debates about use cases, etc aside I think even someone towards the middle of the debate can readily acknowledge progress, utility, value, and adoption has been incredibly slow.
Consider ENS - the last time I registered with ENS the process was just as confusing and convoluted as ever, only in this instance with gas fees I paid around $300 for the name. Oh, and the various transactions kept failing - leaving me wondering (per usual with crypto) if my funds went up in smoke, off to some scammer, etc.
Like so many times in crypto before I just assumed the funds were gone but like I said I DO follow this space and DO find it interesting so I consider burning crypto left and right a relatively cheap education/hobby. Plus I get to tell cool stories like this one on HN ;).
I can’t imagine how creaky, bug ridden, impossible to scale, nearly impossible for mortals to use, and inevitably completely overrun by scammers and frauds an Ethereum (or any crypto) based mail and messaging system would be. No. Just no.
With every day, week, month, and year that passes hearing “We’re building! People are working on XYZ!” when meanwhile the only thing people actually see is the latest headline for some crypto criminal, fraud, scam, collapse of last years “use case” (NFTs anyone?), etc.
I’m not saying it’s dead but I think it’s pretty safe to say the window for anything approaching mass adoption, credibility, and real utility has closed.
“Stop trying to make fetch happen! It’s not going to happen!”
100% agree. Been in crypto for 12 years. I’ve held this view for the last 7 years. So much delusion in the space. So much “technological orgasms” with no grounding in real world business dynamics and economics.
Precisely. I follow because there is some interesting technical work being done. There is a lot being built but even after tons of time, effort, and resources (at least 10s of billions of dollars) the only thing that ever comes out of this work is another incredibly dense and nearly impossible to use "technological orgasm" with an extremely thin use case utilized only by (seemingly) the team that built it and a minuscule circle of hardcore enthusiasts.
MetaMask is by far the most popular wallet used for all things Ethereum and Web3. They touted "30 million" MAUs[0] during the peak (3/22) of the last crypto craze/bubble. Let's put that in perspective - there are an estimated 5.3 billion people on the internet. 30 million MAUs represents 0.06% of worldwide internet users. Facebook (which has been in rapid decline) has at least 2.6 billion MAUs. One social media platform vs an entire ecosystem (MetaMask can easily be characterized as the official onramp to Web3). Literally a half of a half of one percent of the TAM (internet users).
Want a (somehow) even bleaker picture? According to DappRadar the top 10 most popular Dapps[0] COMBINED have a lifetime total of 888k wallets (all numbers rounded up) that have ever interacted with the Dapp in any capacity. That's 0.00167% of worldwide internet users and I'm sure the MAUs for these Dapps are even more pathetic. Additionally, it's widely known that wallet != user (many users use multiple wallets and addresses). Looking at these Dapps none of them have any utility other than "DeFi" (swapping tokens around), play to earn games, and straight up gambling. To look at it another way, an absolute best case (impossible) scenario for MAUs of these apps worldwide is roughly the equivalent of the entire population of a small US city. Have you heard of Lee's Summit, Missouri (to pick one)? Yeah me neither and the same could be said for these Dapps.
All of the building in the crypto ecosystem is by crypto zealots for crypto zealots which the MetaMask and Dapp numbers show is an absurdly tiny (so small you can barely wrap your head around it) population. Crypto (other than gambling on exchanges) is an extremely obscure fringe religion.
Things are being worked on but nothing changes, crypto is still just a speculative asset for people to try making big bucks quickly with, people use monero to buy drugs. Those are the two usecases, it doesn't matter how much people work on cool crypto things if the usecase stays the same.
Having been a part-time postmaster for more than a decade by now, I fully agree, and would even go further: Ingress spam is pretty much a solved problem if you play your cards right. ChatGPT et al. might change that again - but the mechanisms you can deploy today are very effective against the current UBE landscape.
The _real_ problem is reliably getting your 100% legit mail into your consenting recipients' inboxes.
There are absolutely no shortcuts and yet most of the problems I diagnose regarding email delivery will find a missing PTR record or a miss-configured (or non configured) HELO. You cannot be lazy when it comes to email. SPF + A + AAAA + PTR and (E)HELO is a minimum.
There is one thing that you cannot generally, personally mitigate and that is being on a deny list due to your IP address. This one is a bit more tricky to deal with. You might have to use a relay. Another mitigation might involve IPv6.
There is no design committee, they just keep adding stuff over and tell us is definitely the one that will fix security/phising/privacy/spam issues SMTP suffers.
Sorry for naivety, what is HELO in this context? I've been running a mail server with the other aforementioned things set correctly and tested; I've been able to reliably deliver mail to all of the big senders for years now, albeit on a small scale.
Am I missing something not knowing what HELO is ??
As a user, I'd rather my receiving email provider (eg. Gmail, iCloud, Outlook) be less restrictive in their filters if it means individuals can host their own mail servers and we can have a more egalitarian internet.
Extremely filtered emails is practically the same as a social network determining what I can see at this point.
Web standards have advanced to a level that walled gardens are replacing email. Email itself is still stuck on IE6 level standards. And SPF, DMARC and DKIM are a confusing mess to deal with.
I've had a company tell me the other day that you can't do DMARC without using their services. It's just too complex. Why is that - we don't run our own mail servers and we use Google mail for our organization. I'd think Google would do that?
Sounds like a salesmen who doesn't understand DMARC himself was trying to sell it by labeling it too complex.
From a purely technological point of view DMARC isn't that complicated. It just specifies how to treat DKIM and SPF results (with a bit of rather simple configuration). SPF is basically a list of ip's you own published on your domain (if the email was sent by that ip, you vouch that it was sent by you) and DKIM signs the email with a private key and you publish your public key on your domain so everyone can verify that this email was signed by the domain owner. SPF might fail if your mail gets proxied (as now its a different sender ip that you didn't vouch for) and DKIM might fail if the mail got modified including headers (because the signature can only be verified for exactly the original headers+content). So if you're sending email for someone else it gets a bit tricky, but for your own emails it's certainly not "just too complex" and boils down to a few line long configuration file, a list of ip's and a private/public key pair for signing emails.
Yeah, I need to do more research on it, thanks for your advice. It seems like this must surely be possible to set up on our corporate Gmail admin account.
I moved a Gsuite client to Zoho because once or twice a week, an email from one person at the custom domain to another person in the same custom domain (company) would have their direct email get put into spam in Gmail.
You would really think that Google would whitelist emails within the same custom domain which is paying for Gsuite service. Maybe that has changed, but it was definitely a problem 5 years ago.
And we're not talking about an email which had some copy/paste of spam, we're talking about a one or two line sentence giving an instruction or asking a question to a colleague.
It is solved by the product. You tick a box, and internal mail is bypassed.
The issue here is that developers and IT folk seem to think that email is easy because the know IT. Email is a complex, and old protocol that has many nuances and, believe it or not, phishers are smart; end users are naive, and often reckless. Spammers, be they benign or otherwise, can be incredibly lazy too, but don't think for one second that you, as a postmaster, are cleverer than they are. Spam and phishing are getting harder to beat. Businesses like Proofpoint, Cisco and Mimecast, along with the likes of Google and Microsoft, are investing heavily in trying to beat these guys, but the reality is that they are always at least two steps ahead.
At scale, that’s not an option. If you have external services that send mail to your Workspace tenant, there is a risk of compromise to the sending service, especially if it is outside of your control. The sending service could have SPF records that permit sending mail on your behalf. That mail needs to be scanned. There is also the threat from bad actors inside your perimeter. Sure if you’re a small operation, or using Workspace as a personal mail host, this may seem like overkill, but I can assure you that the majority of orgs that use Workspace (it a business too after all) would prefer the status quo.
I’d guess the main issue is, the big guys don’t care about you. Like, at all.
If they pass their log through a scanner, come up with a few dozen small servers that look fishy (e.g your IP is contiguous to some other spammer’s IP), and yours is rope in as false positive, you’re banned.
And from there you’ll have to convince Google that you’ve done nothing wrong.
> And from there you’ll have to convince Google that you’ve done nothing wrong.
Imagine the conversation with their support when you ask why bob@domainx.com's email was put in spam folder of jan@domainx.com's email, as both are hosted by Gsuite and presumably entirely within the Google network.
The answer, after two levels of support, was, "We don't understand why, but we can open a ticket with the developers." In the business world, you don't have time to wait for that, especially when non-technical business users are making mistakes or losing business because they can't function properly.
Even with DKIM, all you need is the recipient of one email from one user on one domain (I have hundreds of domains) of your mail server to file a spam report, and WHAM you are blacklisted. So yes, it is a problem even with DKIM. If you have a solution, I would LOVE to hear about it.
I file this under “you can’t have a technical solution to a social problem”. We can do all we want to protect e-mail, but when it comes down to it, someone is going to figure out a way around it and ruin it for others.
The current situation is that we have technical solutions for authenticating smtp sending domains. But there will always be someone who flags an email too quickly or just wants to spite you. Or someone that will send spam regardless (or hack an account, etc...). And so we’re back at square one.
This is especially true for situations (like email) that aren't subject to market forces. Because sending email is so cheap, the return rates can be very small to still justify sending spam. There just isn't any market pressure to keep it contained...
One of my clients is a construction firm specializing with churches. It is not infrequent for them to be communicating about a project with a church - often to a role account (e.g. info@church.org), which is step 1 towards being filed as spam - where the role account is shared with a dozen or more people. The building manager will check the email on Mon, Wed and Fri and every other day there will be a number of well-intentioned volunteers at least one of which will click on the spam button dooming my client's email into the abyss. Weeks later we find the church has gone with a different contractor, as they "could never get a response" from my client.
We will go so far as to donate a new IT/Cloud system to the church (small $$ compared with the project) just to ensure reliable communication. But then they think we are just trying to buy them.
Now my client has a blacklisted domain/mail server IP, and bids they send out are rejected as spam by the providers to other churches.
Makes no difference if we are hosting the mail server or if it is a 3rd party mail service. As soon as a customer is sold, we try and move them off email into a web-application framework for ongoing legitimate communication. Again a lot of resistance.
Does this happen even if you move your client to Google Workspace/Office 365? Using one of those two should eliminate the problem. Was that your experience?
If the alternative is that your construction firm fails and you and everyone else are out of a job - well your email hosting choice is a weird hill to die on.
Presumably most of the firms clients are themselves hosting email on Workspace or 365. And Google/MS might treat email originating from their own systems differently than inbound email originating elsewhere.
Also I'd expect all the firms competitors to be using Workspace or 365, so if using them gives no apparent protection, presumably they should be suffering from this as well?
Hashcash was originally proposed to add some form of cost to sending email. Something similar could be a great way to get mail from legitimate people through. Spam wouldn't scale but the average person would only pay in a bit of CPU time/cost.
“spam wouldn’t scale” - unconvinced on this: spammers already mostly use other people’s compromised machines to do the sending; there is no cost to them here.
If you charge some cost per mail (whether that's CPU time or actual money), users/teams would check their spend and optimize accordingly. They'd notice runaway spend and act on it. The only reason why mail servers become compromised and nobody notices is that bandwidth for mail is generally too cheap to meter. On any 10s of megabits connection, sending a deluge of spam is trivial.
Not even that, you can become the victim of a "noisy" neighbor if someone on the same IPv4 /24 sends too much spam (some of the common blacklist providers will do entire netblocks if they get enough complaints)
That would demand maintaining an address book. Most mails I receive are from services which I have never written to, and I don't see a reason to extra maintain their data when I already have their mails.
For sure. I’m contemplating a scenario I’ve had multiple times “No I didn’t get it” “Okay let me try again, did you check spam?” “Yeah” “Maybe add me to your address book” “Nope still nothing”
Well, ChatGPT could write a mail server too, so what are we even trying to achieve any more?
Just let ChatGPT send emails to itself, using mail server and client software that it wrote itself, and the rest of us don't have any need to handle email ever again.
I believe what the parent meant was that SPAM blocking relies heavily on keywords, and strings. With the advent of chat gpt we go beyond a/b testing and push it to into the every infinite way of someone conning you to click on something through email.
Not my experience. I have run a personal mail server for about 15 years. My mail has almost always been delivered just fine. The network my mail server IP has been in a blocklist less than handful times, but this has always been temporary and resolved without my intervention (perhaps my hosting provider is on top of this).
On the other hand, I'm also sending email for a client through AWS SES, and those emails are regularly blocked due to their IPs being on a blocklist. And indeed I also get spam delivered through AWS SES. I understand that it is a cat & mouse game for mail service providers to prevent spam being sent through their services. My point: switching to cloud providers doesn't magically solve your mail problems.
Aonther issue with large mail account providers (gmail at least), is that they will accept & deliver your message, but just "deliver" it to the spam folder. It's so annoying if mail is accepted but you don't know if it is actually seen by the recipient. This erodes trust in email. I understand mail servers don't want to tell spammers whether their email is accepted/rejected, but I would prefer a few more spams in my inbox if I at least won't miss emails.
Any any well designed website will work with javascript disabled unless it is absolutely needed for the task (i.e. we are talking about an actual app).
There is also no guarantee that the image being loaded means that the user opened the mail and it wasn't just preemptively cached by the server/client. There is also no guarantee that the user opening the mail means they read it.
have you seen the web lately? javascript is increasingly being used on backend. not just for apps, for static one page sites too. those who disable javascript do not have a pleasant browsing experience and are forced to enable it often
I'm not here to guarantee anything, just mentioning the technology exists and is still widely employed. there are no guarantees with messaging app ticks either
I am always reading and never experiencing this (knockonwood).
I'm self hosting email servers for multiple domains for now nearly 25 years. When moving to a new hoster with new IP addresses (especially Hetzner, as it is cheaper and seems to attract more malicious people), I have to contact Google, Microsoft, Yahoo and one or two other big providers to clear my IP address range. This has to be done only once and usually is resolved within hours (faster for automated sites). The bigger ones have a web page to do this, others can be reached via email and are always replying quite quickly. So after a day, everything works and once it works, this never changes and mail gets accepted.
These servers are being used by multiple users and businesses and none have reported any problems. For decades.
I am only using dedicated servers though and using everything in the book: DNSSEC, DMARC, SPF, DKIM, proper DNS records, TLS with valid certs and a correctly behaving SMTP server...
No, that kind of software optimizes a very important problem. It’s quite cumbersome to set up all components of a mail server by yourself. At some point you start hosting a domain for a friend. Then then friend wants to create some mailboxes, forwardings and so on by themselves. So you just give them SSH and tell them to edit the postfix config? Having a web interface that does it all and doesn’t break things is very important.
simple-nixos-mailserver... One step deploy. Hasn't broken in years (it's never broken for me). Extremely stable. Can handle all your use cases. Declarative config, so no messy state.
I hate to be an elitist jerk, but if you can't understand a man page describing a single file format then maybe running an email service is something best outsourced.
Giving out free email addresses to friends and family? Me.
Hey, we are starting this charity and need email for 15 people, what should we do? - order domain, create admin-account in the web interface, pass it on, done.
> Last year I moved my mail to an old fashioned shared webhosting account at Hetzner. Very happy with it!
How exactly is that solving the problem? If anyone does something remotely spammy from that ip, your mails are spam again too. And you probably got lucky that the ip you're sitting on was warm and trusted to begin with. You didn't really find a solution, the problem simply hasn't occurred yet for you or you are not aware of it yet.
GP is right. Self hosting email sending (and by that I mean any solution where you control the mail server) doesn't work unless you accept that you will randomly end up in spam folders and sometimes not delivered at all.
From my experience shared web hosting accounts have the worst trust. You get the IP range of a public cloud provider, but you also share it with random people, that will send spam at some point.
So either get a package at some of those smaller providers with a dedicated IP, or go to one of the big providers (Google, Microsoft) and use their highly trusted IPs, that everybody whitelists by default.
I think you misunderstand. You cannot have a mail service on shared webhosting. It’s like 1999: Upload PHP using FTP, the end.
Instead, the hosting provider will operate a mail service for you, much like Microsoft 365, G Suite and the like. They will also take care of the IP reputation, SPF, DKIM and all.
Maybe they do, but I would be surprised seing managed email sending in Hetzner's portfolio as part of shared web hosting. It's just not what that company typically has to offer, that's why I expected they just provide you the infrastructure, but ip reputation and so on is your problem. Taking care of IP reputation and successful delivery is a massive undertaking, especially if you let customers send whatever they want.
However, I dunno why people only mention the cloud providers, while services like MailChimp, Sendgrid and others offer this for mass emails and companies like fastmail for personal users. The cloud isn't really necessary at all.
The business for the web hoster is to provide me with some MBs for a website, plus a database, and also to provide me with some mailboxes on my domain.
Many small busineses have their 'online' presence like that. If I have mail deliverability issues, it's not my problem any more, I can call Hetzner. And before I started using it I was sceptical because if Hetzner did not fix the reputibility of their IPs or the spam from their other customers, it would be a problem for me. But so far, it worked quite well!
> These days I'm very happy to pay somebody else to run email services using my provided domains.
I was in a similar boat, though I bailed much longer ago than you. I think "Big Email" is happy to let enough spam flourish in the world so that it presents enough of an excuse to tighten the screws to make it effectively impossible to run your own server... just so that more and more mail goes through their servers to be mined for personal data. That's why I don't use Gmail any more, but it wouldn't shock me to find out that Apple is doing something with my email data too.
Counter point...
Make email hosting as easy and abuse free that many more people can host their servers( for themselves or for community) breaking the big tech monopoly.
Some big mail carriers who are more FOSS-friendly should create a service where they will deliver your private mail-server mail to Google and Microsoft inboxes for you. So you can use your own mail server for every other mailbox with more reasonable filtering protocols, and when you have to send mail to a Microsoft inbox you just send it to them and they’ll forward it.
They could also provide Cloudflare-like spam and DDoS protection for receiving mail
You would still deal with the deliverability challenges of getting your private email server trusted by everyone. It will work 95% but there are a bunch of weird edge cases due to old IP blacklists, bad spam rules, etc that will cause you problems and basically require a greybeard email expert to diagnose and fix. Sadly, I think it's just too hard to run your own email server these days.
Why are the email providers so bad about this? When I check my "spam" filder in Gmail, around half the email is not spam. Is it because humans are so bad at classifying spam?
This is an honest question, in my opinion the only way to make Google more friendly with self-hosted mail servers is their users complaining about or leaving Gmail.
google has one of the worst spam filters. It can be done much better, and google could do it much better.
But I suppose they are just big enough now. Can't reach your contacts that use google? Well, just sign up to google as well, then you can reach them.
People like you are the reason google can behave like that.
We use Mailgun as outbound relay for self-hosted email stack, and no more problem with delivery. As a home user you can even stay within the free tier and never pay a dollar for it.
you are allowed to run unlimited count of instances for your own use only
you can't sell or distribute container images to third parties, every mailserver operator needs to have its own license
Rather take the 3 hours to just set up all that FOSS software myself and give it away to everyone for free but thanks anyway
I'm not the OP/creator and have no affiliation with it. On the first two points, I thought the snark in this comment could be accepted had it been followed with more useful information and pointers along with the criticism. A one line criticism and attack isn't helping anyone and doesn't help us have "curious conversation" that HN asks users to promote.
At least they are hashing and not storing encrypted passwords.
But even a baby framework with may be 10s of deployments have switched to bcrypt, etc. Im not sure why they're boasting about SHA512. But I am a little lost on the RFC thing. Could you enlighten me. I thought they were standard ports for legacy,TLS, and SSL ports.
Technically yes, but for the last decade I've seen only one instance where 587 was explicitly STARTTLS (Fastmail), everyone else just running TLS on it.
First thought: oh, huh, a self-hosted CVE generator.
In seriousness, installing Roundcube on my own server circa 2006 was the cause of the first and only time I’ve had a server hacked. It’s probably improved since then or it wouldn’t still be around, but it put me off ever hosting my own email. The risks only get worse the further away you get from personal/hobby use.
I've used https://mailu.io. It works well, but your biggest problem is going to be getting over the spam filter hurdles of the email giants of the world. Even if everything is properly configured (including dkim / spf / whatever else they've added) your messages will get plopped in the spam folder.
My experience is, that the „quality“ of the mail server‘s IP really matters. The worst experience I got was with digital ocean. A lot of providers just don’t accept email from their IP ranges. Some of them just completely block all DO IPs on router level, and refuse unblocking.
For my current server I had to switch IPs a few times, until I got one that was not blocked by any of the major providers. Unblocking a once blacklisted IP seems to be practically impossible.
And hotmail or outlook.com just mark a lot of email as spam. I see it now as a problem of the recipients. Office365 just accepts the same emails, it seems to be a strategy of the free mail providers, to give their non-paying customers a worse experience.
We got a /24 at our data center and the reputation was, unfortunately, poor. I went through all of the public reputation lists and asked to be removed. It took about three months of incremental effort, but the reputation for the entire /24 is clean now.
This is with a "real" mail server, and not mailu.io, but the idea is the same.
I just went to my cloud provider of my choosing and started to add floating IPs. After a few tries I got a good one. I went through the unblocking process once, and I decided not to do it again. Especially Microsoft gave me a hard time, they started to request documents and then let me wait a few weeks until they replied: we don’t unblock, and we don’t tell you why.
Is it possible that Microsoft distrusts your IP range? Some providers are known not to be very responsive to abuse reports. And then whole IP ranges get (soft) blacklisted, and individual IPs can’t be unblocked, without having a very strong case.
>>First thought: oh, huh, a self-hosted CVE generator.
Haha, same. I've run my own mail servers, got the tshirt, and don't want to have to do it again. Point your domain to one of a bazillian email services instead.
There is also basic forms of protection you should put in front of everything you make public, in order to reduce the attack surface. Firewall that blocks everything by default, strip all headers unless you veto them manually, aggressive rate-limiting you increase the limit only for specific IPs and so on.
Putting up any type of software on a unprotected server even in 2006 is begging for trouble.
Define “unprotected”. The particular server had a firewall and fail2ban along with other measures. But Roundcube is a webmail service, so you’re leaving 443 open in any case. No amount of firewalls or rate limiting will help you if the thing you’re running is a web service that turns out to have a SQL injection vulnerability in one of its endpoints.
Email servers in particular are going to be under attack all day long just from normal email activity, and that’s before you throw in any kind of web interface. It can be a big help to point your MX records at some other filtering service, but at that point why are you bothering hosting your own?
I use http basic auth in front of every https internet exposed service.
The services may have their own auth system on top of that, but htpasswd in front solves the vast majority of problems. Can’t exploit an SQL injection vulnerability if you can’t reach the endpoint in the first place.
I’m less concerned about apache2 and nginx http basic auth vulnerabilities. They’ll get fixed much quicker than random webapps.
That's what I do. Mailcow on an isolated machine, 25/587 open on firewall port forwarding to it, the rest of the various services it offers are only accessible via my home network (https, imaps, there's probably more). Then, I am always on my home network.
I started out with a different variation of this that was the same, except instead of using my (thankfully static) home IP in my MX record, I got some cheap hetzner/lightsail/whatever, then routed the incoming 25/587 across a 2 node wg network to the real mail server. It worked fine but ultimately I decided I'd rather expose my real IP in the MX record than pay $5/mo not to.
Of course, the secret to making this work without tearing my hair out is that my outgoing mail server only delivers mail to the relay I pay to deliver my mail to the 3 or 4 corporate behemoths who have taken over a once great decentralized service. I have no interest in tending to my deliverability or making appeals to Microsoft or whoever. Also at a personal mail volume with 0 transactional mail, it's very inexpensive.
Some https services are internet exposed with http basic auth as a first line auth requirement. Some services are available to friends, or I want access to from devices I can’t VPN from.
Funnily enough Roundcube isn't even one of the mail protocols. Its software connecting to the mail servers via a GUI over HTTP(S).
You don't have to give the entire world access to your web server. You could even use something like AuthPF to allow yourself to access it. Or a VPN like Wireguard. I do the latter now, but I used to do the former. Although back then I just used Mutt over SSH usually. Way faster than the web software I ran back then (probably Apache with Horde). What remains is all the stuff required for sending and receiving email. SMTP, IMAP, and the stuff to deal with spam (some kind of tarpitting as well as SPF/DKIM). In fact even the IMAP server could run behind Wireguard. So its only SMTPd and SPF/DKIM. There are some very secure SMTPd written, with great track records. Back in the days I ran Qmail with Courier-IMAP but I don't think SPF and DKIM existed back then.
I’m hosting mail servers for over a decade now. They are all very low frequency, so probably not a lot of attackers find them. I try to enable as many automatic updates as possible, because I don’t operate them professionally. Just every few months I check if all updates are installed, and if there is something wrong. So far I only had two hacked accounts (probably the users got phished or used compromised public PCs while logging in to webmail - the country of the attacker was the same where they were on holiday).
So far no break-ins that I noticed. But it is for sure possible that somebody broke in without me noticing (and did nothing worth noticing).
Hm. Been running mailinabox since 2012 or so, no issues. I like the idea of consolodating executables a bit and simplifying the system, so I'll have a look at poste.io.
I _love_ playing around with (and sometimes actually) self-hosting stuff. But email is something that I will HAPPILY pay someone like FastMail or ProtonMail ~$5/mo to handle and avoid myself the hassle. It just works. I can add whatever subdomains/addresses/sending profiles/etc., and I don't ever have to think about data backups, blacklists, spam reputation, or any of the dozen other issues I've heard about since I became aware of how email hosting works in ~2006.
Kudos to those of you in this thread who live the dream, though (really).
> All passwords are by default stored as salted SHA512 hash (5000 rounds). Attackers will have hard time to crack your passwords.
SHA512 isn't a good choice for this, because it's optimized for fast low-memory computation. Why not use bcrypt or argon2, which are industry-accepted best practices for password hashing?
Their rationale is probably because those two don't scale very well when you want to make their efforts count, whether bcrypt's hunger for CPU or Argon2's hunger for CPU and/or RAM. Bcrypt is very capable at bogging things down when you have lots of users authenticating very frequently, which is often the case with a POP3 server. A mere 100 e-mail clients authenticating every 2 minutes on average to check for new mail incurs a significant load even with a mild bcrypt work factor. On the opposite end PBKDF2 with 5000 rounds is much leaner, and if you enforce long passwords - which is immensely important no matter what password-hashing you use - then even fewer rounds are needed.
Yes, and enforcing long passwords is the primary and most important way of dealing with it. Enforcing ridiculously high CPU/RAM use for authenticating is a cost that both sides have to pay, but in itself it doesn't solve the problem at hand.
I'm wondering, if pop/IMAP auth is so costly, why POP3/IMAP community not allow user to get session token and auth with it? Like JWT or anything with similar security properties.
Both protocols are easily extensible...
It's only costly if the software (or its administrators) decides that it has to be. This particular product's approach (5000 rounds of salted SHA512) is not very costly, nor is it inherently insecure.
It is not. Even long/"complicated" passwords usually do not have much bits of entropy, which is why you need an expensive hash to make bruteforce attacks difficult.
If you want to host email on your raspberry at home, your main issue is usually that your IP will be in a known „dial-up“ IP range, that is blocked by all major email providers. And most home internet providers block port 25 too. And you need a fixed IP, it’s a nightmare if your mail server‘s ip ever changes.
This is also a problem for ARM servers. Oracle's free tier is quite generous, especially if you use their free ARM offering (4 * 1 core with 6GB of RAM, no time limit on the free offering), but you can't run Mailcow on it.
This has been flagged on Github and the dev's response was "well, don't run anything important on a free server" and that was basically the end of that. I was pretty disappointed with that, but on the other hand there's nothing that prevents you from building the images yourself on ARM.
What is solid about exim? Not only does it have more CVEs than any other mail transport agent (including four, all critical or high, just last year), they tend to respond by doing infamous things like releasing security patches on Christmas morning. I'm not the biggest fan of Haraka, but exim is easily the biggest security hassle you can ask for in an email server.
I generally recommend replacing Roundcube with SnappyMail (https://snappymail.eu) -- not having to deal with a database by not maintaining much state is a win.
I was expecting to see Postfix instead of Haraka. I wouldn't have been very surprised at exim.
Ugh. Docker only. So again, people are basically rejecting any engagement with the [Net|Open|Free|Dragonfly]BSD world.
iRedMail has exactly the same components, and from what I can see almost exactly the same flow logic, except its available on BSD platforms as well as linux.
I think the fact that it includes SoGO with Cal/CardDAV and active sync was the main reason, poste.io doesn’t seem to provide a solution for contacts and calendars.
I’m still very happy with mailcow. And they include all features in the free version.
Offtopic, but have people had any success maintaining a personal email domain as forwarding to the major email providers?
I have a vanity domain, and used to be able to use GMail to send email as that domain [1], and forward received mail from that domain back to GMail.
But with SPF, DKIM, and DMARC (or something), this has broken and such received email gets marked as spam and/or phishing.
I don't know how to fix the forwarding/receiving flow [2], because GMail will see the message coming from an arbitrary source, but being forwarded by an intermediate (my hosting provider's forwarding email server) which will fail its integrity checks.
I have not been able to understand how to solve this. Clearly, forwarding servers aren't a well-regarded use-case in this new era of verified email flow.
Also, I'm not just talking about GMail. I used to do forwarding for my family who used a variety of providers, so I can't just switch to GMail (via Google Domains) for my entire domain.
[1] GMail still offers that feature, under Settings->Accounts and Import->"Send mail as"
[2] the sending flow is easy: just add gmail's SPF to my own domain's
You can't forward email to Gmail - they'll blame your server for forwarded spam regardless of everything else being correctly set up. This is true to an extent for other providers, too.
The simplest solution is to set up IMAP on your mail server and have remote mail accounts fetch from it. Most email providers can be configured to fetch via IMAP.
The problem is that there is no such thing as mail forwarding. It is mail re-sending. So, you get a piece of spam and re-send it to Google, and the From address isn't the spammer, it's you. They'll eventually block the email address, and then the whole domain it's on.
Don't forward. SMTP. SMTP is authenticated, which solves the issue.
There is such a thing as mail forwarding - it just doesn't work with SPF. In fact, since submission and outbound connections are both SMTP pretty much all mail servers to happily forward mail from your client to another server - only with domain and authentication restrictions, but those are not required by the protocol.
> Validating an ARC chain only makes sense if the receiver trusts the ARC signers. In fact, an ARC chain can be counterfeited,[3] so ARC processing applies when receivers trust the good faith of ARC signers, but not so much their filtering practices.
Is there any documentation on how GMail and Hotmail onboard a ARC-using domain? Their documentation is very... sparse.
I used to use forwardemail.net for this and recently switched to cloudflare since I already use it for dns and domain registration. Seems to work fine. I use the same flow as you for outgoing mail in gmail
I'm a happy user of ImprovMX.com - used it for free for a long time, now paying some 30$/year. Delivery is pretty much instantaneous, and I had only very few mails not being delivered (and there was a legitimate issue on the sender side, so ImprovMX was rejecting it).
But I'm not really satisfied with the sending side of things - my plan doesn't include the feature, and sending via Gmail is enough most of the time, but setting up each alias manually in Gmail is quite annoying.
I have for many years BUT you have to pay Google to make it work. It used to be free but I guess it was abused too much for spam. Now you have to pay for Google workspace and it should work.
I've used all three of these actually (over the years -- more recently I've moved a bunch of my imapsql workload to S3-compatible storage on Backblaze) and they work great.
If you're sending and receiving gobs and gobs of email maybe think twice, but for someone who is dipping their toes into self-hosting email maddy is one of the best choices out there.
As usual YMMV, and back things up, if they are important to you.
>> User database is stored in SQLite database - in file
How much of the configuration can be data-driven from SQL sources? Just the users? What about multiple domains? Aliases? etc. Something like the MySQL interface with PostFix [1]
I looked it over. If I were starting out today I might try it. I disagree with the usual horde of "ohh there be dragons in there" since I generally do not have blocking issues with deliverability.
Gmail from my gmail to my personal email account the past couple of months can take anywhere from a minute to over an hour, that's been odd. Gmail from my gmail to one of my other gmail's or from Hotmail to my personal or from Yahoo to my personal are all fine.
Delivering from my personal to my gmail has been fast and consistent. It's odd that from my own gmail to my own personal can sometimes be slow the past couple of months.
Other than that though, I've found running my own server to be liberating to have the option. Probably doesn't mean anything any more but I feel good to be able to do it.
I'd like to try it, but the landing page writing doesn't inspire confidence that this is a polished product.
If the developers are reading this comment, I might be able to help tighten and smooth that out. I have 99.9999% English fluency and write at the highest calibre.
I used poste years ago and I enjoyed it. It doesn't magically fix DKIM/all that other crap but it was nice having something run perfectly "out of the box" from all the daemon perspectives.
I think a focus on transactional email would be more rewarding, as landing in spam doesn't really matter(just tell the user to check their spam folder).
Currently the best I have found is Postal.io, but its not really light.
You should've said that you want libre software instead. A custom license doesn't really make an open-source project just source-available, you're granted plenty of rights as an user to use that source. Source-available would be much more restrictive than this.
Bold claims and assumptions. If you want OSI open-source then say so.
In addition to the fairly generous interpretation of that case. You make the assumption people should even recognize OSI definition or the US courts. The OSI definition is very narrow, someone forbidding military use doesn't make open-source software just source-available. Say what you want, one group doesn't get to decide a term, especially not that black and white.
Significantly more precise from you would be to say OSI open-source or libre software if you want that. Open-source as a generic term is much more liberal.
> Restrictions. You agree not to: (i) sell, rent, lease, distribute,
sublicense, loan or otherwise transfer the Commercial Features to any third party;
(ii) alter or remove any trademarks, service mark, and logo included with the
Commercial Features, or (iii) use the Commercial Features to create a competing
product or service. Postalsys is not obligated to provide maintenance and support
services for the EmailEngine Software licensed under this Agreement.
This isn't a small change either, I don't plan on selling it, but calling it open source is a stretch.
Even MongoDB, which has arguably less strict restrictions, have given up on trying to label the SSPL as open source.
If I was running a service that required parsing emails from external sources, could I easily write a script that could parse inbound emails and then do $SOMETHING with them easily? If so, where would that sit in Poste?
If I want to send unlimited emails from my domain (not 500 max or 2000 per day) and I don't want to worry about hacks AND don't want to pay for overages, simple shared hosting is the (cheapest) way!?
I know of no shared-hosting provider that doesn't throttle emails. The minute you're talking more than about 200 an hour, you're going to either host your own mail server or route through any of the dozens of commercial SMTP providers.
MailCow has CalDAV/CardDAV/ActiveSync support. They leverage Nextcloud for DAV support; you could theoretically set up a minimal Nextcloud setup to get *DAV support into this but you'll have to do it manually (and for some platforms you may need to add and configure an extra reverse proxy).
Agreed the real problem is getting other servers to accept your mails.
While I can understand the paranoia, bottom line is that the number of domains that can successfully send mails is now seriously limited to a few big players who now have a virtual monopoly on emails.
Which is not really good in my opinion...
The bulk of work with managing a mail server (these days) isn't software setup and admin. On the receiving side, it's all the work dealing with abuse and attacks. On the sending side -- and this is the tough one -- it's getting sites to accept your email. When I finally gave up managing my own mail server (about two years ago), I found that about every six months I was involved in some panic where some large mail provider (Microsoft and Google most frequently) decided they didn't want to accept email from my server. Solving these issues is neither easy nor quick.
These days I'm very happy to pay somebody else to run email services using my provided domains.