Yep, after over a decade of trying to run independent mail servers we finally broke down and switched to SendInBlue and SendGrid last year. Even if you do everything right, it's just not possible to get gmail to reliably deliver mail unless you use one of the big services for smtp. It's really disappointing.
Edit: We can at least keep using our servers for incoming mail, and just use those for sending.
I ran my own email server for many years. It was fun, easy, and gave me a lot of freedom and power. Gmail was never an issue for me, but I put a lot of effort into it. Others have not had such promising results.
For me, it wasn’t Gmail. It was Proofpoint and Microsoft. During a very email-critical time I couldn’t get emails out to certain Proofpoint customers or to Hotmail.com addresses (specifically). And it was important enough for me to switch rather than try to fix.
Microsoft was likely arguably fixable but Proofpoint wanted me to prove ownership and control over the entire IP allotment from which my mail was emanating in order to lift the restriction on my single IP. That wasn’t possible because I didn’t control the entire space my IP was in.
It’s become increasingly difficult to operate email independently without a lot more money and time at the least. This worries me.
No one will do anything because free and independent email is less valuable to the general public than spam free email. HN loves to downplay the existence of abuse on the internet. Gmail isn't dropping your emails as some big evil corporate plan, it's collateral damage in a war on unimaginable amounts of spam.
Pretty sure newsgroups were killed by spam and lack of moderation. I spent a lot of time on them back in the day, and by the time Google got in the game they were already dying. By the time they digested Deja, Usenet was a tiny fraction of original size (if you discard spam).
(Not to mention lack of basic features. I've got a laptop at some point and wanted to synchronize read status between my devices. I ended up writing NNTP to Maildir converter so I can use my IMAP server for this. Not pretty.)
Google continues to hold the pillow over Usenet text newsgroups. Google Groups is by far the largest enabler and gateway of spam onto Usenet and they don't respond to abuse complaints.
And then ironically they're blocking access to some newsgroups (and more annoyingly especially their archives) because they contain too much spam. Gee, thanks for nothing…
For all email. I disagree that marketing email in general is a problem though. Eg. we send a monthly newsletter that is explicitly marketing, but it's opt-in (not opt-out; users need to explicitly choose to receive it), so I'm not sure what the harm is there. And yes, it's easy to unsubscribe too.
Unsolicited marketing email is a problem, and I don't think we send anything that a reasonable person could categorize as that. We send
-replies to user support requests
-new vehicle listing alerts that users have explicitly signed up for (eg by running a search for a car on autotempest and then entering their email address in the "get alerts for this search" field)
-monthly newsletter, again with explicit opt-in only
Users are stupid, they just hit spam because they can't be bothered to unsubscribe or are too dumb to realise it's from someone they have an existing relationship with - people love giving end users the benefit of the doubt but they really don't deserve it.
I don't think so. The post claims there's ways to deal with spam without trusting single authorities to police it.
Additionally, the linked post explains that if you get flagged as spam or "sending to many emails" there's literally no way to fix it without sounding the alarm in a post like this.
Microsoft is even worse. They are blocking their own customers from replying to emails because their outgoing smtp server rejects it as spam.
Despite all the advancement in machine learning, spam filtering seems to keep getting worse. And both Google and Microsoft have absolutely laughable support for when your emails are getting blocked. Could we just dial things back a bit? Some spam is fine if the alternative is having an unknown amount of legitimate messages not delivered to you.
Even if you have SPF and DKIM working correctly, they will block your email to outlook, hotmail, and other services unless you fill out their form. If you fill out the form, then you still need to wait months before they decide whether to accept your email.
On my own servers, I warn users who try to use hotmail or outlook that my email to them may not work and they should use a different email address.
Having SPF and DKIM implemented says little about the email coming from a server. But it should allow filtering of spam from domains based on the history of emails received from them, and how users have interacted with them. They should allow the keeping of trust scores at a domain, rather than server or IP, level, and that should be feasible with even relatively few emails involved.
Yet Google and Microsoft don't do that. I can send replies on a personal domain and server to people I've talked with often in the past, and quite often they'll go to spam with Gmail. I can send emails to universities running Office365 from the domain of a prominent university, and sometimes they'll go to spam. I've seen Google and Microsoft email at universities have emails from mts-nature.nature.com, Nature's manuscript tracking system, something no academic would want sent to spam, get sent to spam. SPF and DKIM should allow these things to be rather simply avoidable. But they don't.
They say "Both email servers have PTR records set up, and SPF [...] DKIM, and DMARC records[...]".
Yes. Great. Thing is that this is such a trivial barrier to entry that guess what? Spammers do it too! Email has become so utterly corrupted with spam that the reality is that an independent provider who has no existing reputation is, 99% of the time, going to be a spammer.
It would be wonderful if we could fix this - but so far noone's come up with a workable solution.
It's not that the solutions are unworkable: The problem is that they are unpalatable.
There are outlier Mail-wonks who maintain pay-to-send wouldn't "work" but they are somewhat in a minority: If you forced senders to pay even tokenistic per-mail sums, the attractiveness of mail would disappear.
People don't want to monetize Mail for complex reasons. The "won't work" aspect in spam pushback has always been debatable.
(there's a well known checkbox list of "your proposed anti-spam mechanism won't work because.. which is a huge antipattern to having a rational debate about it)
The problems are regulatory: who sets the price, who collects the price, whats done with the money, and what it does to the ecology of email internationally. But, it would within some definitions of the term "work"
Any pay-to-send model destroys mailing lists and the ability for large sites to use email to send notifications (e.g. Amazon package shipments). It's a complete non-starter. I get that you're frustrated with the famous checkbox list but it exists for a reason beyond intellectual laziness - the "won't works" really are won't works.
It's a matter of opinion. It doesn't destroy them, it demands something like patreon. It's a complete non starter because the community does not want to explore it.
I don't expect to convince you any more than I expect to be listened to really: I've held this view since the eighties, earlier list paradigms I used in the 70s met cost, and were policed by list administrators accordingly. This topic has been dominated by a very few loud voices who basically prevent rational discourse.
They pay to send model, at something like 0.1 penny an email does not destory amazon package shipments, but would properly force mailinglists to require payment to be on them.
But more importantly: the list didn't give technical answers, it gave political tradeofs that we never debated. Meanwhile the Gmail team went ahead and did what they did, and now we are stuck with one particular system that people then complain about.
Spammers are the most willing to pay negligible amounts. Way more than regular users. They pay for tools, servers, services, etc. because they have some expected return on email.
A global nonprofit with a curated “whitelist” which takes vetting to get into (and maybe an application fee like $10). And once you’re in, monitoring or some other system to ensure that your emails aren’t spam (and maybe a monthly fee to pay for that too).
That isn’t “true decentralization”, but close enough: a global nonprofit organization with strict policies is hard for companies buy and governments to influence. And it’s not truly free either, but those who can’t afford $10/mo sadly have bigger issues than hosting a private e-mail server.
It doesn't work if you don't tick the boxes and it's not obvious which boxes should be ticked. I could see a couple that would apply but wouldn't be dealbreakers.
Also this is not a proposed solution to spam. It is a proposed solution to aggressive anti-spam.
I agree. It really sucks that a handful of big corps control email now, and we can't all just have our own email servers like in the old days, but the spammers really ruined it. And I don't see how any technical solution could change this: anything that's free and open-source can be easily used by the spammers just like anyone else, and then subverted. The solution to spam isn't technical, it's legislative and judicial, but that's just not possible because they operate across national borders and no one's going to start a shooting war over spam.
Closed platforms have an advantage of being policed by an owner, open platforms can survive only when there’s a governing body and some entity with the function of enforcement. People problems are solved with people solutions, not with technical ones. As soon as spammers become liable for every message in the cross-border platform jurisdiction, i.e. the email police can seize their collateral or block their servers by invalidating their certificates, the problem is gone.
It doesn't seem to matter. Actually getting domain ownership information is generally quite difficult now, because anyone with real DNS contact information will get spammed (even spammed physically). Domain registrars have no requirement to care about spam from domains registered with them, and so generally don't care. Server owners will actually sometimes care, but it seems easy enough for spammers to find companies that don't.
To add to this, if the spammers are in the US, unsolicited spam is essentially legally protected there. I've tracked down the actual US offices of some companies sending me clearly unsolicited spam with database-harvested information (usually sketchy loan offers to email addresses they could have only gotten through inappropriate means), but they can say they have an unsubscribe link and thus comply with CAN SPAM, and even if they don't, the act doesn't actually provide any recourse to individuals or small companies being spammed.
> an independent provider who has no existing reputation is, 99% of the time, going to be a spammer.
I suspect that this is the actual reason; “School Interviews” seems to be a new thing, and anything new is viewed with suspicion by large e-mail providers.
At this point if I had to allow-list all source domains (with choice of unblocking whole domain or just the sender) in exchange for no delivery issue as well as no spam.. I’d probably say yes to that. :-|
Heretical opinion: the government does a good job with postal mail. Let them handle electronic mail.
Most of the problems of spam have to do with who's an approved sender and who's abusive. Regulate the mail, it's much harder to be abusive.
You're in this country and sending spam? We arrest you. You're in another country? We rate limit your mail, report you to foreign authorities, and flag anything that looks like your mail.
I think there's room here to have a 'public option' open source implementation with appropriate governance / rules. Give everyone a basic 10 digit @ email.usa.gov.
But I also think there should be a government regulated digital 'town square' for each level of government, with township / county being optional but the rest funded by taxes. Then have those groups determine what is allowed and what's not, through people they nominate in elections. This is like the 'public option' for Facebook.
But mostly I just think this because I want more tax dollars to go to open source projects trying to solve problems that are very real and very much caused by the incentives of private industry.
USPS EDDM/ECRWSS is one of the most persistent sources of fully unblockable spam in my life. No opt-out and nowhere to escalate it. Just a perpetual stream of trash from the mail truck to my recycling bin.
Okay, instead of taking this at face value (with a big acknowledgment that it is genuinely frustrating that both Google and Microsoft don't have good contacts to just know what went wrong), I'll analyse this carefully.
This website is operated by Virtual Industries Group (https://www.vig.co.nz/), which is according to their website operates three services: School Bookings/School Interviews (https://www.schoolinterviews.co.nz/, focused on school scheduling and the service discussed here), Care Bookings (https://www.carebookings.co.nz/, which is the same service as above but focuses on day-care and other similar functions), and MessageMyWay (https://www.messagemyway.com/, which according to their website "is the communications hub for your community. It is your emergency communications plan, your telephone tree, and your email broadcast system all in one").
From a cursory glance, these three services shares this set of outbound MX servers. While it is very unlikely that someone who uses School Interview and Care Bookings would mark their message as spam, if the messages relayed by MessageMyWay is sent on the same outbound MXes then I could immediately see the problem. A large part of MX operators knows this and separates "marketing" and "operational" messages into separate servers to prevent this exact thing from happening. While I understand this dev's frustration, maybe the messages relayed by MessageMyWay are the ones marked as spam by frustrated parents who are receiving irrelevant school marketing which are sent to same MXes as their important operational messages?
If I were the developer (and still insist on using on-prem email), I'll operate three groups of servers:
Set A: purely for company-initiated messages, never for the customers
Set B: "operational" customer-initiated messages: School Bookings, Care Bookings, and MessageMyWay mails which are marked "critical" by the users (which is apparently additional cost)
Set C: "marketing" customer-initiated messages: "normal" MessageMyWay mails
I take my hat off to you - an impressive amount of research went into this!
MessageMyWay is opt-in and has a robust opt-out mechanism - and is sadly moribund. It sends a maximum of a couple of hundred emails a day.
And we are registered with GMail spam feedback loop and Microsoft's SNDS, both of which tell us when someone marks a message as spam. This happens less than once a week, so this isn't the signal that triggers the rate-limiting.
If you're being rate limited your mail is probably going to the spam folder for a lot of people anyway, and/or are going to accounts that are no longer active. It's not the absolute number of spam reports that matters it's the ratio.
Unfortunately if that's not the reason then shrugs I guess. Another usual reason is too few emails - this is worse for Microsoft's but I also heard that Google's filters prefer higher volumes.
I can't even get Gmail to deliver my 100% genuine, properly authenticated, sent-from-my-own-domain-which-has-never-sent-spam firebase sign-in emails - and it's the same parent company!
The get sent to spam or (worse) silently have the links stripped out a decent percentage of the time
What I don't get is, what is Google trying to achieve with this? A rate limit that doesn't increase daily or check the spam rate or provide a support rep to lift it?
Gmail has been around a long time. Is this the first we're hearing about this?
I have a more positive experience than the OP, I worked with a news outlet that has several newsletters and sent about 200K emails/day, and we use our own servers/domain and an open source/self-hosted tool to send them (I think it was Mailman [1]).
We did all those technical bits like SPF and DKIM, put the one click unsubscribe link on the message and also on the header of the message so that clients like Gmail can put the unsubscribe link on their own UI [2], all the recommended practices. I remember using the tool Mail Tester [3] and the results where all green.
We doesn't have issues with being marked as SPAM by Gmail/Outlook, and have average open rates of 50%, which is a lot higher than the industry standard who is around 20% [4].
We have a good UX and an ethical way to treat our users, like all users have to opt-in to their desired newsletters when creating an account (or choose to receive newsletters without creating an account), one-click unsubscribe link in big text at the bottom of each newsletter, but also a one-click-no-need-to-be-logged-in link to opt-out of all the newsletters that the user was subscribed, and more stuff like that.
But one thing that I think give us a lot of reputation for the Gmail algorithm, it was that we designed a feature that if the user doesn't opened a newseltter for about 3 months, we started to send the newsletters with an alert at the top saying like "Seems that you aren't reading this newsletter anymore, you will be automatically unsubscribed in 30[n counter] days. Click here to disable the auto-unsuscribe." (the disable the auto-unsuscribe also was for people who have images disabled and we can't track the openings, but is small percentage). So with that feature, we get sure that our users where engaged with the newsletters, and we have a system to avoid sending messages to "dead" emails, maintaining a fresh and healthy database of emails, and seems that Gmail/Outlook knows and likes that.
It is important to point out the several unusual things that went right in this story:
1. Your decision makers recognized that the important number to pursue is "engaged users", not "number of emails in the list".
2. You have a large enough subscriber base so that Gmail/Outlook realize you're a good player. This wouldn't happen if you only sent a few thousand mails per week.
3. You have the technical expertise to properly maintain the list and the surrounding (unsubscribe, etc.) infrastructure. Usually the budget is allocated to _create_ the list/service and then assumed that it will continue working forever with 0 investment.
IF you can get these things right too you'll (eventually) have a similar experience. If you fail on any of them... you're screwed like everyone else is saying. Of course, having to send large volumes of email to ensure that "the big guys" treat you fairly is why many people running small/personal email servers complains about them.
> We recommend choosing a word or phrase that means something to you, and then adding one or two numbers or symbols, like "!" or "+". So something like squash:club! makes an excellent password.
Ever run a dictionary cracker like hashcat? You really really should, with modern GPU hardware. Read the documentation too. Your example is a pretty early match. letter/number cycling, single case substitutions, symbol concatenation - GPUs can cycle through those in parallel, fast.
It's a guessable password + guessable ruleset. You've introduced far less entropy than you've imagined. If you want to have protection against attacks and use your technique, you have to come up with a ruleset you (1) don't see in the documentation (2) that's computationally unreasonable.
For instance, combine multiple languages. Strawberry Octopus Sundae, which is pretty memorable, can become
FragolaOctopusEisbecher combining Italian, English and German. And with that we just went O(N^3) where N is dictionary size (actually more than that because you've got a wide choice of latin scripted language).
That gives you far more protection then say sTr4b33ry+0ct0pu5!5und43. You aren't fooling a GeForce RTX 4090 with those tricks.
That's an example I just came up with though. There's lots of things like that. But really run a cracker. Try it out.
I have finally just started using completely and totally random upper/lowercase letters, numbers and special characters. No rhyme or reason. Usually 15 characters or more.
The example I gave isn't that guessable. It's combining things from different popculture into one new phrase that won't be in any dictionary. An even better attempt is to insert a whole word between two sides of a broken up word, e.g. `4mAzonij-0rdan4Npr7nces7`. And yes I use hashcat frequently.
Using something like a 20 character password following that formula is going to be fine for most people - it isn't going to be brute forced before someone gets an alert that passwords are compromised and can change it.
It's a much better solution IMO than using something entirely random and writing it down, or using something like LastPass.
Really? Pop culture wordlists are insanely common and small - sometimes they're run Before general ones because they're fast to go through and have so many hits.
This response isn't for you par se, I just don't want people to go off and think they're protected with beyonceMadonna because they read it on hacker news.
Put your needles in the biggest haystacks you can find, not the rarefied air of celebrities.
Also don't think you're protected with "old" names like GreenDayEminem. Using older celebrity lists to preference forgotten or abandoned accounts is another common technique. Sometimes you're just a stepping stone and not the target.
Find a bigger crowd to get lost in. For instance, say an address + phone number of a place you remember that's not yours. 419FSt#201 is an old friend and 8184093100 is a friend I had in childhood. Those are two big crowds. Phone numbers have way more structure than you probably think but it's still a fairly big haystack.
> Really? Pop culture wordlists are insanely common and small - sometimes they're run BEFORE the general ones because they're so fast to go through and have so many hits.
That's the thing though, I'm talking about combining pop culture words in a way that they are not in any list.
beyonceMadonna is quite a bit different from `b3Y0nm-Ad0nn\ac3`
That's using two words from pop culture, merged together with numerous substitutions. It's easy to generate, easy to remember, and won't be in any lists, and shouldn't be bruteforced (well, I'd increase it to at least 20 from 16) in the time a breach is reported and a reasonable window for changing a password after being notified.
Note the part where the example says "squash:club", not "n!mGgl342r0nin"
Password guessing and cracking is part of my job. GP is right: the advice is absolutely terrible. Your example looks good but the secret sauce is the way by which you create it. Maybe it's just leetspeak for something a million fans know, in which case it would be in my dictionary by virtue of downloading Wikipedia and other cracked password lists and the leetspeakify rules will find it in a matter of hours or days for a typical leaked hash (this looks like a domain admin password we might find on a workstation they logged into).
Eyeballing mGgl3etc., it doesn't look like something you'd find in anyone else's password, which makes it unpredictable, which makes my job hard and your account safe. But you can't determine a password's strength from the generator's output (only if it's terrible) so idk.
Password manager. The password for which you're best off randomly generating and memorizing if you're able (one of my grandparents can't remember a 4-digit banking PIN so that's a sticky note on the back of her card... do what one can).
Also memorize, if possible, a few other important things like your bank password, or rely on 2FA for that, e.g.: we have a system where you log in with a reader for your card itself, so your password being stolen and your chip+pin being stolen is a risk I accept. If someone is after me specifically enough to get physical things out of my pocket, I'm not confident about resisting a rubber hose for that password / they can also install a keylogger.
For generating a password, it just needs enough entropy if you calculate log(pool^elements)/log(2):
- pick 6 actually random words (like in the xkcd) from a 7776-word dictionary like diceware, or some other combination that gets you at least 72 bits of randomness (so here it is log(7776^6)/log(2)),
- or use a much shorter but slightly harder to remember set of random characters (a-z, A-Z, 0-9 would need 13 characters. If you don't want to bother with shift at all but include a few (10) symbols that don't need shift, add one character.
Depending on the password manager, it would strengthen with a KDF, but that depends on details and remembering just a handful of 72+ bits of randomness should be attainable for most people if they use it regularly (at least once or twice a month after the memorization phase).
I should really find a canonical source for this so I don't have to write this up every time. Not sure if Schneier has this somewhere but I looked at e.g. the Wikipedia on password strength (a source people would trust) and it's less explicit and also contained a pretty bad mistake for more than a year not so long ago...
That's really good advice if you use the same password for everything. It becomes untenable with the, what, hundred+ websites I have a login for, each with their own password.
> Note the part where the example says "squash:club", not "n!mGgl342r0nin"
Sure, the example I gave was more complex, but I was specifically referring to the "We recommend choosing a word or phrase that means something to you, and then adding one or two numbers or symbols, like "!" or "+"." being terrible advice.
It's not great advice, but they are on the right track.
> Password guessing and cracking is part of my job
Same here.
> GP is right: the advice is absolutely terrible.
It was just worded badly.
> Maybe it's just leetspeak for something a million fans know, in which case it would be in my dictionary by virtue of downloading Wikipedia and other cracked password lists and the leetspeakify rules will find it in a matter of hours or days for a typical leaked hash (this looks like a domain admin password we might find on a workstation they logged into).
Sure, there are tons of domain specific dictionaries, but that's really more useful for people that are targeting an individual and know something about them, or are just going to go through every dictionary they can. Most attackers are not doing that.
And besides, it's not about ensuring the password can never be cracked, that's not a great goal considering the compromise it would take - the goal is to delay brute forcing enough so that you can change passwords after you have an alert that a site has been breached.
Alternate case, make some substitutions for letters with numbers/symbols, add in additional numbers/symbols, and you'jj have something that isn't trivial to bruteforce.
Once you know the core words you are using, you only have to memorize the substitutions you have made, which is much easier.
This is why Proton is extremely aggressive with its free users and close accounts seemingly arbitrarily. Any abuse of their service could cause them to be banned because they are small.
Google doesn't have to care, who is going to ban Google SMTP servers? That would be suicide.
The promise behind SPF, DKIM and DMARC is that they allowed positive reputation mechanisms, which cannot work if any domain is easy to spoof. So if you're careful, only send good email and stop sending when the recipient asks, that should be a golden star on your review, destination domains will happily let your email through and nobody will be able to freeride on your good reputation because your domain is authenticated.
However, this is still hard to establish, because it depends on the destination domain server to decide whether you're sending good stuff or not. I would like to have a mechanism by which the user can decide.
I imagine something like this: each email provider, say Gmail, issues to its users a number of single use codes like "Sor7xeik". When the user wants to subscribe to a newsletter (say news@interesting.com) it gives its own email address and one of those codes. The first email from news@interesting.com contains some header like
Authorized-Sender: authorize Sor7xeik
When Gmail receives it, scratches the code and marks @interesting.com as an authorized domain. From that point on, all (DMARC validated) emails from @interesting.com having some header like
Authorized-Sender: yes
are deemed to be interesting for that specific user, and accepted without further spam filtering. The user can revoke the consent at any time on Gmail's web interface, at which point emails from @interesting.com (with that header) will be rejected. The sender at @interesting.com will see the rejection and disable mail sending for that user.
With this mechanism bad practices like address harvesting and selling become much less useful (because an address alone is not that useful, if the sender is not authorized; and the authorization must be initiated by the user).
BTW, I am not saying that all emails should be sent with this authorized sender mechanism. I don't expect individual users to collect authorizations for each of their contacts. Email without the Authorized-Sender header would still be subject to the usual spam filtering, but agencies that often send legitimate mass emails can have a mechanism to prove that they're doing it with the user authorization.
It's very easy to get legitimate emails false-positived as spam by gmail/gsuite or office365 MX even if your sending IP address is not in a bad-reputation IP block, your rdns, spf, dkim and dmarc are impeccable.
I know people who've been running mx/smtp servers on the internet since 1994 who have now given up on running their own, not because they're technically incapable or unwilling to continue to do so, but because they've been forced into themselves using gsuite or office365 because of the monopolistic practices of the huge email-as-a-service providers.
> I know people who've been running mx/smtp servers on the internet since 1994 who have now given up on running their own, not because they're technically incapable or unwilling to continue to do so, but because they've been forced into themselves using gsuite or office365 because of the monopolistic practices of the huge email-as-a-service providers.
This isn't FUD and I still run my own, I'm just saying that some people have given up dealing with the hassle. These are even people who have effectively free zero dollar 1U rack spaces for colo, bandwidth at the small to mid sized regional ISPs that they themselves run engineering operations at.
For the record I do not endorse bowing to the pressure of the gsuite/office365 monopolists.
One of the important parts that's hard to solve for a person who doesn't work at an ISP these days is that you absolutely don't want to be in a /24 netblock (and parent /22, /20, etc) that also contains $5 per month VPS/VM/hosting customers.
Because the inevitable result of having shared hosting IP space where anyone with a pulse and credit card can buy a VM for one month for 5 dollars is that your IP space will end up on some blacklists/spam lists/opaque and not publicly available spam-blockage lists run by gmail, because other peoples' server IPs in the same adjacent ranges announced into global BGP tables by the same ASN have historically been a source of spam/UCE within the past 12, 24, 36 months.
I've been running my SMTP server since 2004 and just last week, I was finally whitelisted by Microsoft. It's the first time in 19 years that I am able to send email to @hotmail.com addresses.
But other than that I have never had a big problem, it just works and I spend little time on maintaining the system.
I’ve used Fastmail for email since 2015 or so, with a custom domain, and have never had any issues with email delivery to Gmail, Office365, or any other email host.
Data point of one. It’s something I occasionally worry about actually but it has never happened to me. So I probably just shouldn’t worry.
not really. providers have a lot of tools that monitor IP reputation, and also human contacts never hurt. Fastmail and other providers maintain their reputation by taking spam seriously and having prudent rules in place.
source: i worked at fastmail and regularly got oncall alerts to go deal with some rbl issues
The OP describes carefully monitoring reputation and is their own self-hosted mailing provider (not a service-provider) so presumably isn't hosting spam...
The difference may be the human contacts? Or just that self-hosted mailing providers are so rare that it's hard to keep their patterns from looking like spam to the filters? Or other?
Well the human contacts were great when something was really weird, like Apple calendars changed something and our code broke, or something kept getting flagged and you needed someone at Yahoo or such to go bang on something with a hammer.
His complaint seems to be mostly about rate limiting an IP that doesn't normally send in bulk volume. It's a weird behaviour. I would recommend he not use fastmail or gmail, but an actual provider for bulk delivery.
"backing off" people submitting mail, mail resending when failed, these are all really good useful parts of the email system that help you recover when things go wrong, even though it feels like a slap in the face - it's much better than the alternative of whitelist only, or blocking people permanently.
It was frustrating for me sometimes.... free.fr was often on our rbl list, and if I recall it was mostly because a customer (different ones at different times) would send an email to bob@free.fr an that person was actually like bobby@free.fr, so free.fr thought you were sending crap, or spam or whatever. Usually it was just an undeliverable mail.
Another thing is - IPs yes - they have "warm up" as you start sending mail out of it. In fact, providers like fastmail have a bunch of IPs, that way if one was flagged, and they figured out what was going wrong (you wouldnt just flip to the new IP, because a outbound spammer would muck up your new IP instantly), you flip over to the other IP which wasn't on any RBL.
anyway, I'm probably botching the explaination, one time our anti-spam guy drew a giant whiteboard of how it all works and it gave me an instant hangover.
Just to say - yes it's more complex than 'open up the port and start sending mail' but certainly not approaching 'no independent providers are allowed'
From your further description, it definitely sounds like something hard to do without significant dedicated staff -- like I could believe it is not realistic to just "run your own mail server" anymore, when mail isn't meant to be your business. For better or worse, you need to use someone where mail is their dedicated business, with the scale to devote significant resources to keeping on top of it. (an "anti-spam guy"!) Which didn't used to be the case in olden days. But, as you say, that doesn't mean it needs to be one of the big three.
Is my conclusion at this point.
But a "bit" (lot) more is required than one interpretation of "just taking spam seriously and having prudent rules in place" might suggest!
You don't need to be a big three, but yeah, you need some staff if you want things to "just work". Fastmail and other providers deal with it day in and out and have enough reputation that a single spammer from their IP doesn't impact everyone else.
Keep your domain, outsource your mail service.
Plus fastmail is just all around a cool company so, yeah I mean, who wouldn't want to support small local business that love open source and do 'the right thing' when it comes to their customers and really care about how email works, and how it's evolving.
I tried to move off fastmail back to gmail last year because I lost a job opportunity due to a random fastmail-origin email i sent getting blocked by my interviewer's google mail. the first three went through fine, it was only the fourth one that didn't make it. they thought i ghosted and had already chosen another candidate because of it.
ultimately i didn't make the move because email is unusable without custom aliases i can completely bounce email from. but i think i might attempt again, but again I recently found myself insanely paranoid about my interviewing situation and had to make the awkward transitioning to my gmail account and explain why to my interviewer
We are registered with Google's spam feedback loop, and Microsofts SNDS, so we get informed about any messages marked as spam. We get maybe one a week, so this isn't what triggers the rate-limiting.
OP here. For those thinking that Gmail MUST have valid reasons, and that we abusing the system in some way, consider...
1. We are registered with Gmail, Microsoft and Yahoo to provide feedback when someone marks our messages as spam (we DO do everything right). We get reports less than once a week, so if I've got my maths right, less than 0.002% of recipients think we're spammers.
1. Our two servers send around 10,000 messages a day, spread fairly evenly over time. That works out at three or four messages per server per minute. If we were spammers, we'd be ashamed of ourselves.
OP here: thanks for poking your head over the parapet. But we've submitted that form three times over five months, with no discernible effect.
I have wondered if a pattern-matching algorithm is penalising us for having a similar IP address to a spammer. But given that practically all servers are now located at hosting companies, IP address is a completely valueless signal - and an arbitrarily damaging one. We have no control over other servers near us in IP space, and can't even ask our hosting company to deal with the spammer because we have no idea which IP address they are on.
If you have access to an internal directory, could I prevail on you to pass this concern on to the Gmail spam filter team? I think there is a genuine bug here, and if they fix it and let me know, I promise to post a followup article saying that fixing this shows GMail isn't killing independant email on purpose.
For me at least, existing completely without GMail / Google SSO is unrealistic, and the million email corpus that is attached to it would be painful to part with as well.
But I've been trying to slowly de-tangle myself, and for it's faults ProtonMail has been working out pretty OK for me as a compromise between usability and true digital freedom.
This is what all fediverse protocols and specs gaining traction today should be designing for. Email was lucky and got their foot in the door before the dotcom era figured out about walled gardens. Now we're slowly crawling back out of them, but if we're not careful we'll be out of the pot and into the frying pan.
Unfortunately, the fediverse likely will have many of the same problems as email and RBLs today: It's less automated, but groups of admins share servers to block en masse, and not all admins bother to vet the claims before doing so. Once your server is on enough servers' block lists, it's near impossible to get off of them.
One of the most useful block lists people upload is generated based on how many other servers have blocked it. So once you're there, even if the situation has changed, you may have to petition a dozen or so servers to remove a block on you before you come off the list everyone else is using.
Botspam is an increasing problem, servers can get blacklisted because too many spammers get on it faster than the admin addresses the issue. And there's really no automated tools for handling all of it yet.
I am running my own server @entropy.be for last 20 years. Maybe I'm lucky but it just works? The secret is only to have trusted users that do not send spam.
OK I get the occasional spam through spamassasin, but I can live with archiving 5 spam emails/day.
Unfortunately it's difficult because Google is beholden to nobody. If you have an issue with a telecom blocking legitimate communications (including email), you can complain to the FCC, and it will get fixed shockingly expediently (hours, not days), meanwhile Google will happily ignore all regulatory bodies. Google is terrified of being classified as a telecom because telecoms are held accountable.
And yes, I have direct personal experience here. :)
If you’re being rate limited by IP, then I would guess you need more IPs to spread the load. Maybe even stagger these mass emails to send over 1-2 minutes.
We have two email servers for redundancy (like when Gmail is rate-limiting), not load. We send about 10,000 messages a day total, or about seven per minute, spread fairly evenly across the day. No mass email.
I suppose it would be an interesting exercise to slowly increase servers/IP addresses until rate limiting stops, but servers cost time and money.
Only way I know of is to pay a company that provides SMTP service and is too big for gmail to ignore. Some have free tiers if your volume is low. I switched my personal domain email over to sendgrid for that reason.
i wound up developing a pretty complex email sending infrastructure because of situations like this. it involves putting a message in a queue and then sending to a list of predefined proxies (mailjet, mailgun,sendgrid, etc, etc) and using backoff algorithms per proxy per hour....
It's what I use, but it means I have to have Thunderbird running all the time or I see all the spam on my phone. It's just kind of kludgey compared to a server-based solution.
Yes, email has evolved from an open standard to a closed system dominated by only a few big corporations. I don't see how is this situation acceptable - such a fundamental functionality is out of reach to all but a few of us. Well, Gmail and such are free and convenient (until you get banned for no reason), so it's good enough for the most. Trying to get back to the open decentralized ecosystem is a waste of time.
But we do it anyway - and we start with the identity layer. Email based on blockchain identity, free and open. I've been working on this for a while, still WIP but check it out: https://ubikom.cc
Edit: We can at least keep using our servers for incoming mail, and just use those for sending.