Google has no incentive to fix these kinds of problems.
It's big enough that when someone complains that a message sent wasn't received, the intended recipient will say, "I never have problems with my Gmail account. It must be you." And the sender has to switch to Gmail to reliably communicate with the outside world.
I wish this was just paranoia, but we've seen multiple discussions on HN about Google programs and policies that alter the internet in ways that only benefit Big G. It's like we're heading back to the days when people didn't know the difference between AOL and "the internet."
I'm one of the PMs for Gmail and hang around HN quite a bit. This is my personal take, not an official reply.
It's simply not true we have no incentive to fix this. Here are a few:
Firstly, Gmail's success is entirely predicated on the health of the global email ecosystem. Gmail does not, inherently, have any network effects (unlike FB, messengers, any other comms tool of Gmail's scale). Email itself, of course has a huge network effect, and that is because you can email anyone in the world, regardless of what email system they use. It's because email is open. If we lose an open, healthy ecosystem with many providers, we'll destroy the base we stand on.
Secondly, we care deeply about having positive relationships with developers and all our users. I can tell you it definitely makes me sad to see articles like this. There are going to be false positives, we will make mistakes, but we certainly care a lot about fixing issues like this when we hear about them.
I agree Postmaster tools has been underinvested in and we could do much better there.
I appreciate the response. From someone who has operated a mail server since 2002, it comes across that Gmail does not care about cooperating with small but legitimate mail servers.
SPF, DKIM, reverse DNS, no blacklists, no open relay, longtime ownership of IPs, etc etc. Using various mail testers returns a 10/10 deliverability score.
And yet, messages sent to Gmail always go into the spam folder, or are never delivered at all. These are everyday regular messages, I have never used mailing lists or sent bulk automated messages.
The issue is, there is no recourse, no fix, no acknowledgement of the problem with false positives. There is no tool available to me to understand or correct the "problem". Hint: this comes across as Gmail not giving a shit.
Gmail has a responsibility to be more accountable, even if these problems are unintentional, because Gmail is such an enormous node in a federated network.
> If we lose an open, healthy ecosystem with many providers, we'll destroy the base we stand on.
Correct. Gmail is contributing to the erosion of email reliability. Please course correct.
I've had gmail send the following to the spam folder:
- Legitimate class action notices related to Amazon purchases.
- Email coming from addresses to which I had already sent email. (!)
- Email from my landlord.
- Email coming from Google itself.
Based on the contents of my spam folder, which I have to check fairly often because of the extreme overaggressiveness, I would be vastly better off if nothing ever got filtered at all. [1]
>> There are going to be false positives, we will make mistakes, but we certainly care a lot about fixing issues like this when we hear about them.
This doesn't sound honest, or at least not complete. People have been complaining about this for years. I have personally been complaining about this for years. The loss of obviously legitimate email is completely outrageous.
It doesn't look intentional (look at that fourth category!), but it certainly doesn't look like anyone is trying to address the problem.
[1] Yes, if spam filtering was disabled, more spam might get sent.
Before I switched to fastmail, gmail would not believe me when I marked an email as not-spam. They would still get flagged. All I wanted was updates from one band.
That didn’t stop me from repeatedly getting junk from every other record label my email was sold to. It was an endless procession of shit I never subscribed to.
"Inboxing" with Gmail & Outlook is a baroque, hellish process for mailservers.
When either provider decides your small email server is sending spam (eg: sending an email with an attachment, or any kind of form email like a daily report) you won't get through to user inboxes, and instead you'll be routed to spam, or for Outlook.com hosted addresses they will accept mail from your server and send it to /dev/null. Gmail's process is bad, but Microsoft has decided to accept emails and throw them away (which is ridiculous).
At my workplace we have two email addresses, one from google mail (gmail for business) and one from MS Exchange.
I never had a problem with gmail for business regarding spam. I regularly receive mails from smaller businesses (some of them hosting their own mail server) and never had a complaint from anyone yet. Since i can also be contacted via phone i'd know.
On the other hand, MS Exchange constantly delivers obvious spam mails and (quite seldomly, but still) swallows legitimate mail.
Anectodal, i know. And disclaimer: the behavior depicted in the article is as bad as it gets, if everything is as described.
Yep. I missed an invitation to a Google-hosted event at a conference I attended because the email (from an @google.com address, no less) got caught in Gmail's spam filter.
This is a problem with ML approaches, right? Instead of water boiling at "100C" it boils at "99.98C +- 0.04C". Normally this is ok, but sometimes it isn't!
I imagine most humans have error rates worse than that. And what does 'error rate' even mean in that context? A small delay or a catastrophic failure ending in death and destruction?
I would think that would be quite a good error rate. Especially when considering people in the hospital are often not in good health, possibly making their veins more difficult to find.
That is assuming by error you mean missing the vein. If error is defined as a fatal complication, then 1/1000 is terrifying.
If there are 50,000 people and 50,000 people experience problems, that's bad.
If there are 5 million people and 50,000 experience problems, that's fine?
Isaac Asimov's comments about world population increase involved something about this; the more people there are, the more each individual is dehumanised and rendered irrelevant (my paraphrasing).
No I don't think it's fine at all. I think Google, twatter, Facebook, et al don't care because 50 people and who they represent don't matter to them compared to the money they make.
When all rounded up it isn't even a single penny on the balance sheet. The owners of these businesses literally never even know from the their only view into the companies.
I have no idea why I'm being downvoted on this. Hackers can't do math or what?
I think your position is a little unrealistic. 50 people experiencing problems out of what, a billion? is pretty good. Do you think that if those billion people were served by 20 million small business e-mail providers, that none of those 20 million e-mail providers would ever make a mistake and affect their 50 customers?
Yup, I've had the same thing. Just kinda amusing and ironic since I didn't happen to care about that event but it makes one nervous about relying on spam filtering.
On the other hand, once you train it a bit, it is mostly remarkable good. For me, switching from fastmail.fm (which was pretty good itself) to Gmail gave me a big improvement in spam control.
Super curious about this response saying on gmail you have more control, because from where I am the “mark as spam” button does nothing but move things to the spam folder. In theory it should learn from that but when someone used my email address to sign up for AT&T no amount of marking things as spam will stop their emails landing in my inbox.
As in signed up for AT&T service? If so it's because it's not spam - it's misdirected mail, but there are tens of thousands of other Gmail users who think that messages almost identical to those are things they absolutely want to receive.
My point is that absent information that Google simply does not have no matter how creepy they get, there's literally no way they can identify such messages as spam - exactly the opposite in fact because probably 99.999% of such messages that they process are explicitly not spam.
The only way Google would have to identify that this message was not for you would be to get the subscriber information from AT&T and cross-reference it with name and address information they had for you - and even then most of the time they'd probably be wrong (e.g. if the email is coming to you but the account is actually in a family member's name).
I just cleaned out a little over 100 emails in my Gmail spam filter yesterday. About 80% of what was in there were emails from YouTube giving me notifications of new videos people have uploaded that I am subscribed to. These emails never used to go to spam, but slowly over time more and more of them would end up in spam. It's at the point now where almost all of them from YouTube go to the spam folder.
It doesn't make any sense since they are emails from Google, they are emails I even have a filter applied to so that a label is applied to them. Yes I can adjust the filter and choose "never send to spam" but the messages will still show a warning on them saying "This message was not sent to spam because of a filter you have applied".
Sure false positives makes sense, but I don't get how the majority of what is in my spam folder would be emails sent by Google.
> It's at the point now where almost all of them from YouTube go to the spam folder. It doesn't make any sense since they are emails from Google
It makes a lot of sense... people use the spam button as a lazy man's unsubscribe. Youtube adding the bell button, making mail opt in is probably a response to that.
I actually give Google a lot of cred for not simply white-listing its own domains. Though spammers would probably find ways to abuse it and make them look bad anyway.
> people use the spam button as a lazy man's unsubscribe
This is the small mail server crux right here. If you’re a small mail server and a few of your emails have been spam binned instead of unsubscribed, it would likely lead to your whole server getting shit canned.
You know what's really funny? That even with that overly agressive spam filter, once in a while (once a quarter maybe), it somehow manages to miss obvious BuY@@NiGERiAn@@Vi@gRa-Cia1is type of emails... which, by the way, my morally outdated spamassasin marks as spam.
I would make an educated guess that "arpa" is from EX-USSR and their native language is Russian. It's direct translation of idiom "морально устаревший" which literally means something is:
* Available for decades.
* Far from being top notch technology.
* Sometimes of course it's literally mean outdated. Like if you run older CentOS or Debian with decade-old packages.
So it's doesn't mean SpamAssassin is bad, but it's very far from state-of-the-art ML technologies that Google might have.
I've recently started checking quite regularly my spam folder as I've noticed more and more legit emails ending up there.
One of them being support emails from TradeMe (one of New Zealand's biggest sites), keep getting put into spam, even after multiple "mark as not spam", along with some other kinda important emails from TradeMe. I've had to put in manual filters to force an email from TM to skip the spam.
And yet, in my own none Gmail hosted email (fastmail), I currently have 2/55 false positive spam emails. I very rarely ever check it. To note I usually get one actual spam (non newsletter blog spam) to my actual inbox, a month.
A spam filter can't whitelist government email. My personal SpamAssassin filters out spam from government servers all the time. The latest was from somewhere in Quebec.
You'd think the various governments would put more effort into computer security. They appear not to care, though.
Ditto, Multiple missed emails from factory owner in fuzhou I'm on site doing business with and actively emailing back and forth with daily. Despite the back and forth communication some of the direct messages were in the spam folder. This could have caused me some major issues.
I don't think that has anything to do with Google needing special treatment. It shows that the company who made the rules and has every privilege to follow them isn't able to.
Valid point. They've become so large and synonymous with email that many people I encounter are actually unaware that there is other email besides "Gmail."
It feels like Google no longer has any incentive to follow the rules, and they feel that they are going to be the ones to make the new rules. The rest of us end up having to implement workarounds.
> You've made a lot of good points, but I don't think that's one of them.
I disagree. I think in the absence of that point, it would have been hard to say this:
> It doesn't look intentional (look at that fourth category!)
But also, I think special treatment for trusted actors is a completely appropriate way to handle email delivery, and I also think it's appropriate for gmail to trust themselves to be sending legitimate mail. Blocking their own email makes them look totally incompetent. They absolutely should whitelist themselves. And they should have a way for you to be whitelisted too, if you want to send email.
I've also had email from Google recruiters (@google.com email addresses) go to spam. I considered this a high level indicator that Google spam filtering is incompetent, not malicious - if it was malicious, they would at least be able to get their corporate emails through.
- Email coming from addresses to which I had already sent email. (!)
That is understandable. It is hard to validate if an email is authentic. SMTP has no authentication built in. Gmail can't just blindly accept all emails from addresses that you have already sent an email to.
Not if you use SPF/DKIM/DMARC they can't, that's the whole point of those various additions.
All those "I hacked your email and send you a message from you account" I don't get, because I have a DMARC policy that says if you don't pass SPF/DKIM then you get rejected. So try as the spammer might to connect to my mailserver and pretend to be me, they can't, because my mailserver sees they're not authenticated, and the mailserver they're sending from isn't in my SPF records, isn't signing the message with my DKIM key and therefore it gets rejected at the SMTP level.
Fair comment. But that issue is pretty simple: there is no good reason for them to fail to deliver email under any circumstances. (This has happened to me too -- someone tried to email my gmail account and gmail completely refused to deliver it. It was pretty embarrassing.)
Messages they think you won't want to receive are what the spam folder is for.
> there is no good reason for them to fail to deliver email under any circumstances
Yes there is. They don't want to carry traffic from anybody from the major email blacklists. If a mail server is on a real, very transparently-managed blacklist, no large provider should be accepting their smtp traffic.
Yet all the people here trying to administrate servers from residential and VPS blocks of IPs are telling you they're caught up in this list you're lauding as all-knowing and safe.......
>> I would be vastly better off if nothing ever got filtered at all.
There are layers of filtering beyond what appears in your spam folder, layers that block obvious spam long before it gets anywhere near your account. If every email ever sent to your address wound up in your spam folder you'd beg for filtering.
Before switching to shared hosting from my VPS (one reason being that i didn't want to bother with email maintenance and didn't want to have a separate service just for email), i had my mail with (badly configured) Spamassassin that wouldn't delete mail, just add a "SPAM" prefix in the topic. My mail isn't exactly commonly known, but still is one i have for years and was made public thanks to me releasing an Android app once (after which, spam increased dramatically).
Even after running it for years, Spamassassin never marked a legitimate mail as spam, so i'm pretty sure that if i wasn't too lazy to configure it to move it to a spam folder, it'd work fine. Stuff did pass through it (at a ratio of one every four or so) but i was fine with deleting those.
What i'm trying to say is that from personal experience, i'd be fine with a spam filter that errs on the side of not marking stuff for spam and me deleting whatever goes through manually. Having to see a bit of spam mail is small cost for losing mail i'm actually interested in.
I'm using my e-mail pretty much everywhere (me@vbezhenar.com you have it in clear text, every bot will crawl it now, not for the first time, though). I'm too lazy to setup spamassasin yet, so I'm getting a notification for every spam message I got. The only thing that I'm trying to do is to click "unsubscribe" even from obviously spam mails (may be it'll make more harm than good, not sure). So I'm getting around 20-30 spam e-mails per day. I don't think that it's THAT bad. I'm spending may be a minute every day to delete it. And I'm sure that with absolutely minimal spam filtering I would achieve almost perfect filtering.
I'm prepared to believe this. But it's not a defense of gmail's policies -- this suggests that in fact nothing would be lost if gmail eliminated the spam folder and delivered everything that would have gone there to your inbox instead. So why are they doing this?
Normally I wouldn't +1 a message because it doesn't really seem to add much to the message. But in this case I will (even though it might earn me ire from some). If I could highlight the parent in bold I would. I've run mail servers since the mid 90's and the experience above exactly mirrors my experience. And my takeaway is the same, GMail just doesn't seem to care. I understand that GMail can't give away too much info on why a particular email ghosted (opsec and all that) but GMail doesn't need to do a better job of explaining to postmasters what good actors can do to avoid being ghosted, and more importantly have that reflect actual real life experience.
After reading this thread I checked my spam and it seems google has gotten way more aggressive in the past ~ 6 months when it comes to spam. They are mass mailers and advertisements but they are moving items that I want -- legitimate subscriptions and people I do business with are ending up in spam. Never experienced such a high failure rate before and I have had gmail for about 15 years.
Early 2000's for me, and yes, similar experience. What was really interesting to me is that my home domains had no problem emailing my work and side bit gmail based mail service a few weeks ago. Then, suddenly, this stopped working.
I changed jobs as well, and now work is using a MSFT based hosted mail service, and I am getting delay messages.
Seriously, GOOG, MSFT, and others broke mail. This is not an improvement.
I've not looked into speaking with MSFT mail folks about their breakage yet. With GOOG, you have really no mechanism of reaching out to someone there and getting attention for the problem they are causing.
This is the much bigger problem with GOOG actually, in case any googley people are reading this. They just don't get customer service. At all. It is near impossible to be able to report a real problem across the spectrum of their services. Unless you are one of their bigger customers, you don't have access to even a telephone support number. Their online help is a crapshoot, with you getting useful information less than 50% of the time.
So where I am now is with locked down, long time existing domain mail servers, which send maybe 5-10 outbound messages per month, that suddenly and inexplicably, have a bad reputation. Well, no they don't have a bad reputation, they can send email just fine to other services.
That's a great idea. I've run my own mail servers since 1999. I'd happily join a union of independent mail operators with the purpose of lobbying Google, Microsoft, and Yahoo to treat us as first-class citizens.
I think you should have a web page where you list your issues, show how you've done all the normal things to be considered a well-behaved mail domain. And then perhaps report your experiences with iinteracting with gmail support on the matter.
Even if it isn't deliberate, even if gmail give a fuck, as can be seen from the first (and so far only) response from anyone at Google, they're so far in the Google "We're better than everyone else and always get it right because we're geniuses" RDF that it literally doesn't matter. They can't believe that they're the problem.
I will chime and validate this experience. You will not fully understand or realize the hostility google has towards open email unless you have working experience with email protocols. This is by design.
It is my belief that this is intentional and I would love to be corrected if it’s not.
A few years ago, I was dealing with a few small business sites which were self-hosted and always had deliverability problems with the large mail services (mostly, back in the day, AOL and Yahoo).
Large providers tended to have a world view dividing all senders into two categories:
1) Bulk senders who are clearly mailing the same spam to a list of a billion addresses
2) Non-commercial individual hosts which should be sending five messages a day or less in total.
It felt like there was a huge missing third category for transactional emailers nobody wanted to acknowledge. They are probably difficult to score fairly. A hundred "Order details" emails are going to have the same level of randomness/templatedness as the old Viagra spam which had a random block of Project Gutenberg text pasted at the end to trip up filter math. You're not going to have a clear history of "this address bounced twice, let's stop sending newsletters" when most of your messages are to first time customers or once-every-few-years return ones. A lot of the messages will look generic because they use default shopping cart templates.
To the extent they provided sender guidance, it was focused around use case 1) -- sign up for feedback loops and deal with greylisting (because people really love waiting 18 hours for an acknowledgement)
This is totally my experience as well. It got so bad that I gave up operating my email server.
Anyway, Gmail is getting some heat in this thread and rightfully so. We should however not forget that Microsoft and Yahoo are just as bad if not worse in this respect.
I also have a 15+ years old set of mail servers and I host mail domains for 10ish friends and family. In fact, I once wrote how to set this up (http://flurdy.com/docs/postfix)
I no longer use Gmail myself but half of my users relay some aliases to their main Gmail account. No problems with that, except my servers continuously get rate limited by Google:
Our system has detected an unusual rate of 421-4.7.0 unsolicited mail originating from your IP address. To protect our 421-4.7.0 users from spam, mail sent from your IP address has been temporarily 421-4.7.0 rate limited. Please visit 421-4.7.0 https://support.google.com/mail/?p=UnsolicitedRateLimitError to 421 4.7.0 review our Bulk Email Senders Guidelines.
I have the normal anti-spam features set up (SPF, DKIM, DMARC, Postgrey, Spamassassin, etc) so most spam gets rejected but not all. But several of my users have very common English names as their aliases that are already guessed and added to many spam databases.
This leads to family phoning me all the time to say my server is broken when it is just Google throttling everyone's emails. I frequently have to check Postfix queues and clear some obvious spam or just pointless Facebook update emails which seems to be the majority... (Yes your email admin can read your email...)
The only way to improve is to constantly remind my family to not sign up to all crap, use not so common aliases, and try to keep tightening my anti-spam configurations. But we really are not talking about a lot of emails. Probably less than a hundred per day spread across 5-ish end accounts of which 95% is probably legit. Yet Google is treating me as some totally open relay. (╯°□°)╯︵ ┻━┻
I do understand Google's challenge with handling probably the largest amount of email traffic in the world, and then the most spam in the world, and that it is a continuously moving target.
But it does feel like they are treating nearly all minor relays as spam relays. We can not block 100% of spam before we relay onwards to Gmail as that would mean too many false positives emails get blocked, but most try to block as much as possible.
In my case, my servers probably block 99% of spam, but some will get relayed, and most of those ends up being handled by Gmail's even better Bayes scoring and filtered to end users' spam folders.
I don't know what the automatic threshold limit is to be on their naughty list, but it must be very low, as in double digits per day.
Now in your case, you failed to mention DMARC records. ALL of the big cloud email services have required that for years. Your SMTP server checklist is straight out of 2005.
This entire thread reminds me of the guy who pulled out of a desk drawer an Analog Startac cell phone and blamed AT&T for it not working. Email has moved on. It not Google's fault that people are configuring SMTP servers like it 2005.
He's wrong about what this tool provides – namely information for people who want to use GSuite for their email. In that context it makes sense that you have to allow Google "spoofing" your domain, of course. But it has nothing to do with what is being discussed here.
Just to be clear, your comments are refering to problems with Gmail's Spam filtering. Isn't that a different issue than what the OP is saying--that his mailserver is being blocked? In other words, emails from his mail server wouldn't even make it to Spam folder.
Gmail spam filtering and it's mail server are two sides of the same coin. The mail server itself is blocking believed spam messages as a first layer, then the spam filter acts a second layer that marks messages as spam. I guess they don't want to save messages that they are completely sure are spam, the problem seems to be their false positive rate or simply their approach against small operators.
I _think_ this is because G doesn't want people gaming the system.
> or correct the "problem".
I regularly fish mail from my spam bin. From mailing lists, and other important stuff -- and indeed Google's own mail!
One thing you can do is to get people to add you to their contacts list.
It's a hard problem. There is one "solution" you probably do not want. Have Google (and other companies) give their imprimatur to certain mail senders.
Have you looked into WildDuck.email? I find it very nicely made for people who wants to run their own email. Still in Beta. FYI, I'm just a js developer not related to them.
I sent mail from my personal gmail to my company email (different user) for expenses
Since money was involved, I paid attention and followed up
Our finance person found many emails in Spam that were important and should not be there
On the other hand, my personal Spam folder is certainly full of crap I never want to see. But now I don’t trust the system, so I have to scan them anyway
Datapoint: Ive run a small exchange server with fewer than 500 users for 5 years and have never had an issue delivering to Gmail except a couple times when we had legit issues on our end.
I have everything set up correct and gmail even accepts 90% of my emails but every now and then it randomly marks one of my emails as spam. A gmail user emailed me first and I replied with a text email no links and that gets marked as spam.
> It's simply not true we have no incentive to fix this
I just happen to have set up an email server and encountered the same problems with Google as described in the article. I own the IP since quite some time, it is not on any black list, reverse DNS is set up etc. but Google rejects email as spam.
And this even happens when the gmail account has added the sender in his address book and has send the first email to which I replied - thus there is a message id that should already be known on Gmail's side.
Use your AI to put email into the spam folder. Refusing it outright is a case for the European Commission which hopefully will slap you another few billions of fine onto the wrist until you remember to play nicely with the other kids.
There is no excuse to refuse SPF, DKIM, reverse DNS, proper MX, no blacklist, sender in recipient address book and reply-to msgId email.
It's not just small fries getting hit by this. Google regularly refuses to deliver mail from backerkit.com to my account, by far the most popular Kickstarter fulfillment support service. I frequently have to contact Kickstarters and ask them to manually send me mail because Google is refusing to accept it, which is annoying and wastes their time.
I wish Google had a way you could tell it "messages from these people/domains are never spam (or I'll deal with it myself)".
Google Apps (or whatever it's called now after the last dozen name and service changes since "Postini") does offer a whitelist function, but as far as I can tell it's there only to placate users.
Whitelisting domains or individual email addresses globally, or per user seems to have no bearing at all on whether they will show up in your inbox...
Postini used to work flawlessly, it's hard to imagine how they could have screwed it up so badly.
It’s AI above all else. It seems instead of a rule that says all emails from these people are ok, they’re too reliant on ai filtering based on the content. They’re probably trying to prevent spoofing.
I use gmail and they know who is sending. I got an email today with a gmail flag saying “this user sent from a different email address previously”
In gmail, you can create a filter to skip the spam folder. I regularly receive emails where, at the top it says "this would have gone to spam, but you told us not to send it there".
I doubt this affects things when messages are rejected at the protocol level, but if your problem is emails showing up in the spam folder, this might fix it.
One of the problems is the pure mechanics of how SMTP works: anyone can send an email with any "from" address, and filtering on people/domains in the "from" field is basically pointless if you want to catch spam.
What Google is likely doing is checking the domain of the originating IP in the SMTP "envelope", but that also gets tricky with outsourced email services or internal IPs.
I do not use Gmail, so these are just wild guesses, but I do run my own mail server and frequently get my email not showing up for people.
Problem with SMTP bounces is that it may take a week for the final bounce to show up in my inbox (because again, that's how SMTP protocol is designed, to expect nodes to be down and retry a number of times).
> anyone can send an email with any "from" address, and filtering on people/domains in the "from" field is basically pointless if you want to catch spam.
Isn’t this the exact thing that DKIM is designed to fix?
This was a response to a complaint about not receiving emails from some "people/domains" even when whitelisted, and I highlighted how this can be hard for a service to detect reliably.
So yes, DKIM will help a receiving server know for sure, but a receiving server still needs to accept emails from non-DKIM-enabled servers, and perhaps that's why whitelisting didn't work for the parent.
Bounce handling depends on the error code, 5xx is usually a permanent error so the bounce is immediate. Mailbox doesn't exist etc. Its not going to exist later either. IIRC 4xx is effectively try again later because something is wrong right now that might get fixed. Mail server out of disk space, you're greylisted etc. So try again later. Those can take a few days to give up retrying.
> Use your AI to put email into the spam folder. Refusing it outright is a case for the European Commission which hopefully will slap you another few billions of fine onto the wrist until you remember to play nicely with the other kids.
I prefer getting a bounce than ending in the spam folder by far. Lots of relatives on Gmail never look into the spam folder so I never know if they get to see my email or not and I end up having to ping them on eg. WhatsApp.
At least with a bounce, I know they did not get my email.
But if it ends up in the spam folder there's at least a chance that it'll be trained out of the spam folder by people marking it as "not spam", and it provides a workaround for the people who really need it to work _now_.
I wonder if the correct solution is "send an informative pseudobounced-as-spam message to sender AND put it in the spam folder", for borderline spam (which passes the other checks). Downside is it provides a deliverability oracle for spammers, but since gmail accounts are free, that's not too hard to establish anyway.
I believe that the majority of the users are trained to ignore g-mail spam since it's been working good enough for most people. Unless we condition the customer to always view the spam folder it's asking them to do something on top of already checking their e-mail.
My mother would just simply ignore the spam and often times spam catches majority of the phishing e-mails too. So it's a double edge sword educating and conditioning the users to review the spam folder. Why is it the end user's job to determine what is spam and what is legitimate ?
> Why is it the end user's job to determine what is spam and what is legitimate?
Who else could determine that? I could hire you to send me emails about Viagra sales. They would not be unsolicited, because I specifically asked you to. They wouldn't be commercials, either, so they wouldn't be Spam. An automatic filter can't determine if it's spam or not, it can only take an educated guess.
That's as a sender (even that reasoning is questionable).
As a receiver, would you rather never received an important email from some org instead of finding it in your spam folder?
But you can mark it as "not spam" (and hope that it does something, I guess) while you (as the recipient) have no options whatsoever with a hard rejection.
I'm genuinely sorry to hear that. Our analysts are looking into the issue described in the article, hopefully we find something we can fix that will resolve this.
I struggle to see how your analysts could do this without looking into specific instances of the problem. Perhaps you intend to contact the author of the article to ask for this. That might work in this case, but in the general case there is absolutely no way for people to report this problem to Google!
Yes, I'm yet another person with the same problem as the author of the article - running my own mail server for years, only I send mail from it, very low volume, everything set up properly, etc. The difference is that my emails go to spam, rather than being rejected by the SMTP servers, which to me is even worse, since I never know whether an email I sent to GMail has been delivered or not. Since GSuite is so popular now, I never know this for any unfamiliar domain, unless I do an MX lookup.
You say all the right words, but Google's actions on this issue (or lack thereof) speak much louder. It's very difficult to believe that the situation will improve and many comments here reflect this skepticism.
Since you profile claims you are from Google, you should know by helping here you really doing disservice to everyone else having same problems. Hacker News is not a “gmail customer support hotline center”. You have billions of dollars to setup a system or heck the whole state of the art department to help people with their gmail problem. So you will help two people on HN bitching at gmail.. what about all those that never heard of HN? Sorry friend you doing this only because at some degree google and gmail obviously does not want to look bad to tech society. /rant
Not sure why you are being downvoted, you have a point. There is no excuse for the way Google is operating now - poor customer support, the only recourse twits and HN posts.
I think it also hurts Google themselves. There is a reason they have trouble competing with AWS. Why should developers trust Google that there will be a person on the other side helping them out when things go wrong? And they always do, at worst possible time. Amazon, for all its problems, has excellent customer service.
So your parent's attempt to fix this is a PR stunt at best. If they (/Google) care about this, they should fix the problems in the process:
- define, hardcode and publish rules that will lead to successfull e-mail delivery (SPF, DKIM, history,...)
- establish a gmail technical customer support service
And they should stop with preferential treatment of those who shame them publicly. Or do they want all of us to start doing the same?
> Or do they want all of us to start doing the same?
An increase in public shamings probably means that public shamings become less news- or interest-worthy. So an increase in shamings might not have as much effect as you might expect.
That’s true, but I don’t think we're anywhere near there yet. Especially when it comes to the mainstream — most people outside of tech don’t have any idea this is a problem.
(FWIW, I don’t seem to have this problem in a significant way, but that might be because I have DMARC set up.)
I don't think you can have an open, healthy email ecosystem, when the parties involved (the people running and managing email infrstructure) can't communicate with each other and get help.
Their customer support is just fine. Only trouble is, we're not the customer, we're the product. (And no, I'm not interested in hearing about how that's a hackneyed worn-out cliché, or whatever, given that it's a true statement. Downvote and move on.)
A big part of the problem with Google taking everything over is that only in very limited situations -- the service formerly known as YouTube Red, for instance -- are we given the ability to actually conduct business with them as a paying customer. Conventionally, Google users don't even rise to the status of sharecroppers, since we're the "crops" being sold to advertisers. They expect us to depend on them for the everyday conduct of our personal lives and careers, yet the only way to appeal for help is to start a shitstorm on Twitter or HN and hope somebody notices.
> I'm not interested in hearing about how that's a hackneyed worn-out cliché
Well, it's a hackneyed worn-out cliché. Without the users there's no viable product. That's what makes them the customers and that's why they need adequate customer support. The fact that you pay for the product with ad impressions rather than dollars doesn't change anything about that dynamic.
Well, it's a hackneyed worn-out cliché. Without the users there's no viable product. That's what makes them the customers
No, that's what makes them "users." The customers are the advertisers who actually pay money to Google. You can rest assured that they don't have to post a cri de coeur on social media to get a response from Google when something goes wrong.
IMO, a company that does its level best to act like vital public infrastructure needs to be held to standards appropriate to vital public infrastructure. If that's a controversial point of view, then so be it.
> No, that's what makes them "users." The customers are the advertisers who actually pay money to Google.
I understand your idea. But in my opinion this is a useless and unconventional definition of the word 'customer'. The customer is the one who receives a service in exchange for compensation. In this case the service is gmail and the compensation is ad impressions. The fact that Google can sell those ad impressions is secondary to that dynamic. If there were no users willing to provide that compensation in exchange for access to gmail, then there'd be no product, and there'd be no secondary market for selling those advertisements. Meanwhile even with no advertisers there would still be a product, and there would still be other avenues for monetizing it.
> You can rest assured that they don't have to post a cri de coeur on social media to get a response from Google when something goes wrong.
Are you sure about that? A quick google for "adwords support experiences" gives quite a number of telling stories to the contrary. Besides, there are several orders of magnitude more users than there are advertisers, so obviously it's going to be much easier for users' issues to get lost in the noise. And obviously no advertising agencies are going to be getting any sympathy by complaining on Twitter.
This is not doing what you think it does. And it's the second comment where you are spreading your misinformation I've seen in this thread now. :( This tool is for people who want to send their emails via GSuite and their own domain. Not a lot of people in this thread want that, and it certainly has nothing to do at all with what is being discussed in the comment threads where you replied to.
That all depends entirely on whether the "fix" is white-listing the two responders somehow, or flagging the larger issue to the correct team and figuring something more general out.
I agree with your message except the motivation part - you can't question that particular person's motives. It may very well be they care just like many other people at Google do, even though the management doesn't.
> Refusing it outright is a case for the European Commission which hopefully will slap you another few billions of fine onto the wrist until you remember to play nicely with the other kids.
This is a very interesting idea that I completely missed.
> There are going to be false positives, we will make mistakes, but we certainly care a lot about fixing issues like this when we hear about them.
Oh really? Then what is the mechanism by which an affected user can report an issue like this and receive individual, accountable response? Because that is what a company that cares about fixing issues does.
As far as I can tell, the only redress available for any issue like this with Google is 1) be a big enough name like Jamie Zawinksi (he has been posting about struggles with this exact issue for several months) that your blog post gets attention, or 2) hope your blog post makes the front page on a site like HN.
This is the part that's going to bite them when antitrust comes around.
They are largely unaccountable and there's no real way to get a hold of a human being unless you shame them on social media and it gets enough traction.
If you're a paying customer, you definitely can get a human being to look at your problem.
The difficulty in this case is the people operating SMTP servers are not paying customers, the free Gmail users are not paying customers and the GSuite paying customers don't think it's their problem (after all, the spam filters works quite well these days and lots of people are on Gmail, making the problem very small in the receiving end of this situation).
It seems an external force will have to push Google in the right direction (of having systems in place to deal with the exceptions). If that's going to be public outcry, legislation, etc... We will see.
On the automation front, it seems Gmail could do a better job at tracking sender reputation over time (i.e. you haven't send spam in a while, we'll be more lenient with our spam rules).
>If you're a paying customer, you definitely can get a human being to look at your problem.
Based on the number of hacker news posts from people who pay them money and are locked out of their accounts by automated systems with no recourse I'd say that's utterly false.
Maybe you meant to say "if you pay them enough money" which is true. But most people don't have that much money to pay them.
I'm not saying Google's support is stellar but there's a lot of misinformation going around. It's become a meme to say you can't reach a human at Google, they will shutdown new apps after an year, etc.
Have you had direct experience with Google support? My experience has been pretty regular but maybe I'm an outlier.
> If you're a paying customer, you definitely can get a human being to look at your problem.
Doubt. We're paying ~$10k/month on ads. The Ads API is rate limited, but it's either broken or has arcane logic, so about every 4-6 weeks, the usual workload will trigger it and we're shut out for 12-24 hours. Can talk to ad support whenever I want, but all they are good for is optimizing campaigns and explaining the UI. No option to get to technical support, no option for ads support to escalate.
Of course, we might not be a large enough customer, but it certainly isn't about "paying".
Thank you, but I and pretty much everybody at the company has given up months ago. We've tried everything we could (well, I guess we should've tried to go viral on HN or twitter...) and it got ignored again and again.
It has become a bit of a rite of passage when somebody new arrives, they see the error ticket for the first time and are eager to dig into it and solve a long standing problem. It's like a trust fall, only the lesson is not to trust, and you learn that lesson by falling.
> If you're a paying customer, you definitely can get a human being to look at your problem.
I may not be giving them money directly, but my use of their products contributes to their billions in ad revenue. Without us free users providing eyeballs for their ad network they would be significantly less rich.
It's upsetting that the default response to complaints about Google's terrible customer service is "if you don't pay them you aren't a customer." Most of their value comes from us, it most certainly isn't a one-way relationship.
It feels like Google hopes that you just give up on self-hosting your own email and simply move over to Gsuite. It certainly feels that way. Especially when it feels like Google/Gmail/Gsuite are commonly recommended nowadays.
His blog is very easy to find, but also is full of all sorts of off-topic NSFW stuff which I don't really want to send people into unawares if they're not familiar with who he is. He has several recent posts on this topic which I don't feel like tracking down as their content doesn't really add anything to the conversation -- it's basically exactly what you see in the OP here, with more colorful expletives.
Glad you're here and listening. I'm another poor schmuck who runs his own email server for family members (you can look it up from my HN profile -> mx record) and Gmail deliverability has been a problem for me for over a decade. I do DKIM, SPF, reverse DNS, had the same IP for many years, etc etc., and it's a total crapshoot whether Gmailers get my message in their inbox or spam folder.
Postmaster tools have been useless for me because apparently my volumes aren't high enough.
I'm inclined to think Gmail's approach to this problem is fundamentally flawed, because Gmail has been by far the worst at binning my server's emails. I don't have this problem with Hotmail, AOL, Yahoo, etc., etc. Now, feel free to argue that their approaches are too lenient, but if I have to choose between false negatives and false positives in my spam filter, I absolutely know which one I'd choose.
> Gmail does not, inherently, have any network effects
This is wrong. The current situation is "if you want to reliably contact gmail users you need a gmail address". This is no different than Facebook or messengers.
And that may actually be the incentive for Gmail being such a bad player compared to everybody else.
(also @reaperducer)
Something I didn't fully appreciate until I got here was:
1. Work (aka enterprise) sends much more email than consumer, and
2. Outlook dominates enterprise. Their market share is pretty astounding.
I assure you there is a lot of email coming from Outlook/Exchange.
My employer is one of those enterprises; we host our own Exchange server on our own hardware on our own IP address.
I suspect that is not super relevant to this conversation, because there are a lot of signals that Google can use to see that we’re a big fish, and therefore automatically tread more lightly on the SMTP bouncing. We’re running Exchange, for one thing, which is not cheap or easy to use. We’re in the IP address of a big enterprise ISP. We have other enterprise services running on adjacent IPs, like OWA and websites.
Small personal email servers have none of those sort of “ambient” signals; they’re probably running open source email server software on a single IP coming from a consumer ISP or general data center IP space.
So the question might be why Google seems to react more to those ambient signals than other email providers. Because we have no issue at all getting personal emails into a Gmail inbox.
This is unfortunately technically untrue, as Google has beaten Outlook and Yahoo and Mail.ru into submission and into supporting it. It's still a huge security risk, and an inherently proprietary email spec Google designed and implemented without community input and deployed despite significant community resistance.
Perhaps the solution is to have two e-mail accounts: a GMail account to communicate with GMail users, and a non-GMail account to communicate with non-GMail users.
Ideally, also a client that handles all this automatically, i.e. chooses the right sender account depending on the recipient, so that I don't have to think about it.
Just like Pidgin was a common solution for AIM and ICQ, this would be a common solution for e-mail and GMail.
As a "small fry" myself, the hoops I've had to jump through in order for Gmail to accept legitimate mail from my mail server are rather large. Like the op, legit email appears in the trash of some recipients for goodness knows why. Like OP I've spf, dkim, & have never been on rbl. I (& services on my machine) are the only senders, and the recipients of those machine messages want them (about 10ppl in total & email have no marketing content). I communicate on port 25 using tls where possible.
It's only ever Gmail that sends mail to spam. Every test I run on that email marks it as clean (generally spam assassin)
Gmail is certainly NOT a part of the open email system you mention, but is a constant thorn in open communication.
Ironically, I use Gapps for one domain and because of my spf settings with "include:_ghs.google.com" I get every cat and his dog trying to send as users from my domain, which thankfully end up in that domain's gapps spam. (hint: let us use geo located includes, like _ghs-us.google.com or _ghs-au.google.com, so that there's a smaller include list!)
I have a small mail server and fortunately, as long as my server is well configured gmail has always been well behaved for me. Microsoft, on the other hand...
I had a similar reaction to that statement. Google tends to make the happy path quite pleasant, but the moment you stray from it, things get kafkaesque very, very quickly.
Saying this from firsthand experience as a customer of Google's B2B offerings, Google earned its bad reputation on the enterprise side.
Saying this after a good number of years as a PM: when you spend 40-50 hours a week thinking about your product, it's really easy to get lost in the weeds and forget that your users see your product from a totally different perspective than you do. I see this all the time because it's very, very difficult to avoid this problem.
I think (some of them) may think that's true, but their internal reward structures and top-down priorities are clearly set up such that it's not, in practice. There's no other reason it could be so bad.
I live in Argentina. Few years back, an ex government official from the previous government was caught in the act of hiding 8 million dollars in a church.
My partner's auncle was also an ex government official from the previous government.
He was in charge of the country's office for regulation work in rural areas.
Up to that point, he was a complete believer that the government he was part of, was there to really change things and put the country in the right track.
When the news about that other guy's 8 mill broke out, he was completelly and uterly devastated.
I think the same can be applied to a lot of the people that works in Google regarding the way they perceive the company.
I just recently read in the news that there has been some retaliation exherted on googlers because of organizing some walkouts.
To think that your company just does good because you try to do good, and not take a look at what is really going around in the world related to what the company you work for is "a little" naive.
Also, I think that right now companies like Google and FB should be treated and reasoned about like states. To think that they will not abuse their power in one way or the other - just like countries do - is also "a little" naive.
>Also, I think that right now companies like Google and FB should be treated and reasoned about like states. To think that they will not abuse their power in one way or the other - just like countries do - is also "a little" naive.
That companies this big ought to be reasoned about just like Nation States is a very interesting idea.
Alternatively, we could reject the notion entirely and revive the antitrust movement. Centralized private power is just as, if not more, dangerous than centralized public power and we aught to have a discussion about whether nation-state size companies are really a desirable feature.
I think both the approaches in your comment and the one above (Or any comment to that matter) are worthy of a lengthy discussion.
But my point was more directed towrds understanding how companies this big are starting to behave.
I do think antitrust is a way to look at it, but the anti trust laws in the US I think are difficult to point towards Google for instance (Probably the same about European laws)
> we aught to have a discussion about whether nation-state size companies are really a desirable feature
I think it goes a way longer than that. Obviously private companies are subjected to a lot less scrutiny than states, but states do miss behave and do not run into the risk of eternal intervention, unless they really fuck up, and even then.
Forcing mega corps to assume the responsibility of statehood is one of the key plot points in the "Poor mans fight" series of books.
(some of the best sci-fi I've read in a while too).
This seems to be common with the product teams I’ve interfaced with. Good intentions, but bad prioritization and little incentive to do anything other than cater to the big revenue drivers.
How many people don't experience problems sending to gmail? Do you think blog posts criticizing and praising gmail deliverability are in equal proportion to users' experiences?
I don't really care about how many people it is, I primarily care they are fucking up my mail, with absolutely zero recourse or transparency whatsoever, just a total black hole.
Until recently, I didn't think I have a problem with GMail spam filtering.
Then I looked at what was in my Spam folder. I haven't looked into it for a long time before, because, well, Google has trained me that their spam filtering "just works".
I now try to check it at least once a week. The amount of false positives was staggering, and some of it was important.
Key point is that words don't matter unless they're backed up by some concrete action.
The worst thing about this blocking is that you often times don't get to see what's wrong. So you're running around in circles, trying to use various mx tools on the internet to find the problem.
This is not just gmail issue. Though with many smaller providers you'll at least get something like "your blocked by XYZ blacklist".
> This is not just gmail issue. Though with many smaller providers you'll at least get something like "your blocked by XYZ blacklist".
I would much rather get this than silent discard. At least when I've seen things in the logs in the past (yes, I do read my server logs), I have been able to follow up and fix them. I get the sense that Google is even going beyond using 550 and discarding email silently. I just did a grep through my logs to see if this message came up and found it nowhere, yet I have users on my mailing lists claiming they don't get emails and they've checked their spam folders. They're on GMail.
> Gmail's success is entirely predicated on the health of the global email ecosystem
That depends on what you define as "health".
> Email itself, of course has a huge network effect, and that is because you can email anyone in the world, regardless of what email system they use
If that's the case, then why does Gmail system explicitly thwart this objective, as described by this very article?
> If we lose an open, healthy ecosystem with many providers, we'll destroy the base we stand on
And that's exactly what Gmail is doing when it does what is described in the article.
> we care deeply about having positive relationships with developers and all our users
You personally might, but your company does not. What your company cares about is advertising revenue. If your company really cared about users, it would figure out a way to let them pay directly for your company's services (not just Gmail but search, maps, etc.) so you could see directly from users how valuable those services were, instead of having any benefit to users be a side effect of trying to capture their eyeballs for advertisers.
It's too bad that Google has no incentive to allow a premium/ad-free experience as that implies having ads is a negative thing, which basically undermines their entire business model.
I would pay quite a lot for google-class services with a binding contract that limits their options for spying on me and is 100% ad free. Instead, I spend an inordinate amount of time configuring my computers to avoid accidentally using their free “services”...
No, they haven't; they've been given no opportunity to vote. Such an opportunity would be, for example, Google offering its basic search service for pay to users. Of course they would have to offer some incentive, some benefit that free users (more precisely, ad-supported users) don't get, but we already know how that works: the obvious benefit is no ads. I would certainly pay for an ad-free Google.
> They have G Suite, and they also tried a more consumer oriented version but no one wanted to pay.
I have paid them for years for photo storage. Looking for somewhere to move it in case their AI suddenly decides it doesn't like me (I've already went through a reCaptcha party after trying to dig up some documentation for some special error messages that I felt should be out there somewhere.)
> I would love it if more of the Internet were based on payments and micro-payments but consumers have voted decisively against that.
Didn't have a chance at all to vote.
They could have earned a lot more on than than than they do from me not clicking on the dumbest ads I know about.
> They could have earned a lot more on than than than they do from me not clicking on the dumbest ads I know about.
Then they would have - they are a for-profit company.
My assertion that consumers are not willing to pay if they can get something 'free' (ie. with targetted ads) is hardly new or controversial.
How many news sources do you pay for? I pay for my news and take a very keen interest in its quality, but the media outlets have an extremely difficult time convincing enough people to care enough to pay even for their news - the most important thing of all.
>> They could have earned a lot more on than than than they do from me not clicking on the dumbest ads I know about.
> Then they would have - they are a for-profit company.
I doubt they ever considered me as a person, only as male 25-65 => show ads for scammy dating sites.
> How many news sources do you pay for?
2 newspapers, +used blendle a while ago, ready to pay for more when I can pay pr read.
I also have paid other things like an tech/art channel on a video site etc.
(I guess I'm not the only HNer that does this?)
> but the media outlets have an extremely difficult time convincing enough people to care enough to pay even for their news - the most important thing of all.
Around here that might be because my choice is either a two hour drive to somewhere that sells that paper printed, -or to sign up for an auto-renewing subscription.
I could of course sign up and cancel but I already has too much on my plate (more than two kids, trying to be active in my communities etc.)
> Firstly, Gmail's success is entirely predicated on the health of the global email ecosystem. Gmail does not, inherently, have any network effects (unlike FB, messengers, any other comms tool of Gmail's scale)
Google initially federated with XMPP, until you had critical mass and shut out the rest of the Jabber/XMPP ecosystem.
Of course now you seem intent on killing your _own_ messaging product so it's hard to divine any self-consistent intent here.
> we certainly care a lot about fixing issues like this when we hear about them.
And there's the problem. The only effective way for OP to contact someone who likely has the ability to fix it was to write an article, post it on HN, get upvotes, and hope you happen upon it.
That's not a viable solution for everyone who runs a mail server Gmail falsely identifies as a source of spam. You don't have a good way to hear about issues like this.
> I'm one of the PMs for Gmail and hang around HN quite a bit.
Can you bring back the old web UI?
I experience obnoxious interaction and interface bugs in the new one on an hourly basis, it has horrendous interaction latency for many operations that used to be snappy, it looks worse, and has basically nothing new I want.
Actually, can you bring back the Gmail of like 10 years ago? Basically every change since then has been negative for me.
New UI really makes Gmail barely usable, basically on any kind of connection. Since it came into being, I avoid checking gmail because its unbearably slow and buggy.
I have 20 years of XP in this industry and build large gov services, and I am astonished that Google allows itself such a failure.
If you want, I will make a gif of Gmail unresponsiveness in default Chrome browser on 50MB connection - taking more then a minute to load and then some more to stop glitching. Then I will show you ProtonMail, Zimbra etc. having no problem whatsoever.
Its embarrassing and affects all their cloud tools in some measure. Google was once a good company. Today, most of the IT people I know or work with try to replace anything google.
Above post from the google employee which "cares" makes me angry - what do you think m8 ? That we are squirrels ? Shame on you. If you have some moral decency that you propagate here, go and work for somebody non-evil, take as many people you can with you and write a blog about it so anybody can know.
Just use the "basic HTML" view. When the new design is loading, there will be a link at the bottom to click to use basic HTML, and then there will be a button to set it as default. I use this and have (almost) zero complaints. It is an excellent mail client.
Was never fun of 1995 Internet design really.
I do use it, but it looks terrible, organizational capabilites are limited and ... did I say it looks terrible ?
Not to mention it always returns to default view, so you need to click html view each time.
I know its not politically correct and I am sorry that some people will have their feelings hurt, but lets pretend that we speak to grown ups for a moment and not 13 year old teenage girls: the GMail is gigantic pile of shit and due to the fact that personal or other non-shitty mails will be blocked because they do not fit Google agenda, there is really no easy way out of this lock-in. Making it euphemistic and diplomatic counts in my world as dishonesty so if you do so, go freck yourself too because you are contributing to destruction of this nice planet we live in.
At least I have freedom to express and say here that Google can go fuck themselves (minuses are welcome, counting morons is my interpretation), and please Google
PMs, take those words as my default response any time when Google Something asks me on a phone 'Hey, you were at XYZ, how was it?' during those 3 seconds I keep GPS on. Please make default answer - go fuck yourself, thats how.
>Firstly, Gmail's success is entirely predicated on the health of the global email ecosystem. Gmail does not, inherently, have any network effects [...]
That's ... unconvincing to the point where it sounds like PR spin. How does Gmail not benefit from crappy email service outside of gmail? How do more gmail users not accentuate this?
>Secondly, we care deeply about having positive relationships with developers and all our users.
Frankly, talk is cheap.
I don't mean to be rude, but your response is either (a) completely missing the point or (b) disingenuous. Assuming the former, can you substantiate your claims?
Hey, thanks for replying. Just sticking your head over the parapet is pretty brave.
I have a contra story. One of no delivery issues to Gmail or Hotmail, none, zero, nada. I've run a private email server (friends/family/small business/private mailing lists) for two decades. It's kept pace with every possible factor for reliable delivery - SPF/DKIM/DMARC/ARC, valid client SSL, IPv6, correct PTRs, DNSSEC etc and have no sketchy affiliates. In that time the IPv4 address changed exactly once and has never been RBL'd. Our mail gets delivered AOK.
And yet, even though I think my compliance level is good, I still feel like the blind man groping an elephant. I'm hoping I'm perceiving things correctly; I have no idea if I'm missing something. It helps that I'm an old-school ISP engineering inmate and contributor to well-known MTAs and MDAs, but few folks are lucky enough to have exposure to so SMTP radiation.
My take on the Postmaster Tools is that they've been created entirely to serve Google's purposes, and thereby serve no-one well because (as you point out) it's ecosystem engagement that makes a difference. If you sincerely have an incentive to improve, there is an awful lot of work to do there. It's okay to push the burden of compliance back to the sender, but the Postmaster tools offer only the most rudimentary levers to pull and provide almost no useful information, particularly for smaller/indy senders.
The message that comes through is that Google only really gives a shit about other large scale entities and struggles to see other points of view. This stands in quite stark contrast to Google's effort level over HTTP certificates and webmaster tools.
"Gmail does not, inherently, have any network effects" - you do and you can't help it.
I've been doing email for quite a while (20 odd years) and I don't find Gmail refusing my customer's email for silly reasons too often.
I have had some odd rejections from Gmail (int al) and no-one to talk to. You opine that Postmaster tools are under-invested in but actually miss the real point:
You (G) seem to under-invest in people and put too much faith in magic (AI/ML/nonsense). I really am not a Luddite (I'm giving the tyres on my smart new Node-RED home IoT thingie a good talking to via Javascript right now) but please don't forget ... "memento homo")
Small anecdote: The acceptance email to my universities study abroad program landed in my Gmail spam folder. An email that I had 5 days to react to or someone else gets the spot.
I thought it's great that I have so rarely stuff in my Gmail spam folder that I don't even have to check it regularly until I nearly missed a very important email. Apparently I was lucky that ot didn't get outright rejected.
I rather have to deal with some spam than to miss important, or even regular, emails.
I use email hosting from my domain name provider (email in my profile). I regularly find out that personal emails from me end up in people's spam folders on gmail. As far as I can guess from digging into email headers, gmail is upset because my domain name provider's email server doesn't use whatever the latest mail-server signing mechanism gmail expects is. There's absolutely nothing I can do about that, and no obvious way I can get gmail to stop treating my mail as questionable and throwing it in people's spam folders. And it's not at all obvious how to debug this.
I get why gmail isn't transparent about all their spam-filtering mechanisms. But when personal email to gmail users is being regularly classified as spam, that produces a serious usability problem for non-gmail users.
>Firstly, Gmail's success is entirely predicated on the health of the global email ecosystem.
It absolutely does not. It is so large now that what the parent poster said is the real case. It has such dominance that people are pressured to switch to Gmail or Google apps to get stuff reliably delivered to the massive percentage of email users.
This is exactly like an IE6 PM coming on extolling the virtues of open standards and how they are critical to the success of internet explorer.
Seriously think about it (assuming you are even posting in good faith). If Gmail's success was "entirely predicated on the health", do you really think Postmaster tools and other interoperability efforts would have so little investment?
> Firstly, Gmail's success is entirely predicated on the health of the global email ecosystem.
That simply is not relevant anymore if 99% of email is on Gmail.
I want to echo the exact same issues as the original author. Running my own email servers since the 90s I recently just “gave up” - no single indication about spam issues. All green on several tests. But Google rejects mail. Zero help or tools from google to get off their filter or whatever it is that makes these kind of decisions.
Friends and family all have Gmail accounts for those important emails that Must go through. Enough said.
I’ve moved my email to another provider and stopped self hosting.
>> but we certainly care a lot about fixing issues like this when we hear about them
It's sheer chance that you happened upon this HN post. Google needs to have an open, healthy customer service infrastructure in place to hear about, track, resolve, and follow up on reported issues. This is antithetical to "The Google Way". That is how you get posts like this.
But then, how can a simple heuristic like "hey, this guy has sent and received messages from this non-google address before, we should most definitely not drop incoming mail from there" not be in place? I gmail is pretty much the only google service I still use. In pretty much all regards, google has repeatedly disappointed me over the last couple years. A couple months ago, gmail suddenly started dropping incoming mails from a mailing list I've been subscribed to for 8(!!!) years, three times so far. Exactly like in the linked blog post here.
To sum it up: sorry, but what you're saying here doesn't sound like it's true, and if it is let me tell you that you're failing at it, miserably. The only reason I didn't move away from gmail is that it's just a lot of work. But the time will come, eventually.
My properly configured DKIM/SPF/etc mailserver, with a 10-year old IP, still gets email redirected to gmails spam folder more often than not. Even replies to emails from gmail addresses, with reply-to-id's.
That's the box I use for my personal email. Out of principle. I believe in a distributed/federated net. It just means that I can't expect that emails I send actually get delivered. Sigh.
My experience for business has been that you simply have to bite the bullet and be on outlook.com or gmail.com. Or you can't expect that regular email always arrives. What everyone in the corporate world expects of course.
I've heard this story from everyone I know that operates their own mailserver. No hyperbole. I would be tremendously happy if Google would spend some serious (and visible) effort on this. Thanks Paul.
Keep trying to make yourself feel better. The company you work for is ruining the internet which means you are complicit in just that. Perhaps you could find something more constructive to do with your life.
> Gmail does not, inherently, have any network effects (unlike FB, messengers, any other comms tool of Gmail's scale)
That's not true at all. I've noticed when sending from Gmail to Gmail it always gets delivered and quickly. I've noticed with other email service people, even ones using big providers like Yahoo, it's really hit or miss if they can receive from me, send to me, and it takes a long time to show up in my email. So there's a really strong network effect to use Gmail because Gmail doesn't work well outside Gmail. So much so that I keep a bunch of address (personal/Gmail, work, school, my personal web site and email server) and try to pick the one that works best based on who I'm sending to.
Every single person inside of an institution can care deeply about something, but that doesn't mean the institution as a whole will act as if it cares. Think of Google as a huge animal made up of individual people. The people can perceive and act on their immediate surroundings, but only by forming larger structures (like teams or orgs) are they able to act on the world at Google's scale.
Really, it's not about incentives; it's about perception. Just like you can't act at Google's scale on your own, you can't perceive the effects of Google's actions on your own. You care about fixing issues when you hear about them, but how do you hear about them? What can the Google-animal see, and what is it blind to?
I very much want to believe that Google cares about small email providers and self-hosters. I believe that you care personally, but have a look at the documentation. All the help articles target bulk-hosters, the documentation doesn't even mention humans! The case that a human wants to send an email to another human, which is the very reason email was created in the first place, is not even covered or planned, so I'm pretty sure Google doesn't care about collateral like us, who have fun caring about their own email servers, playing by the rules and doing everything to not be on blacklists.
Why not count the amount of emails coming from those domains and give them a "softer" filtering algo if it's below a certain threshold? Or open a whitelisting program where you can go full Google on people who violate the terms? I think there are (automatable) ways to solve this problems, but only if Google understands that emails are for human interaction, not to receive ads.
I'm someone in exactly the same situation as the author of this blog post. I appreciate your well-meaning response, but the bottom line is that Google will have to offer a way for a real two-way communication channel for email administrators or be considered as acting in bad faith.
There needs to be a way to inspect what caused mail delivery so we are able to fix it. There needs to be a way to provide feedback to Google about mail that was wrongly rejected. If you can push this forward internally, please do it.
This is a far too regular occurrence to just ignore: it is a rule that delivering email to gmail from outside is unreliable, not an exception.
This is your personal opinion. And there is the experience of those people out there. I believe you that you personally stand for these but I seriously doubt your employer cares. There are many examples of the same issue, like that one with Firefox.
It is kind of funny to me that other email vendors could implement SMTP and co properly and Gmail _more recently_ started to have issues. Just because of this I started to look for alternatives because I will not stand and watch how an ad-tech company destroys the open internet, even if this will be very inconvenient personally.
Wow since you're so concerned and helpful, here's a tidbit: Everyone who has tried to run a personal mail server in past five years can't because of gmail.
N=1, but my small email server actually hosts emails for a business, and the only problems I've ever had were with Microsoft, and they went away after i jumped through some hoops for them. I'm using a linode and dkim/spf/dmarc/tls/rdns.
Personal email server will send ~100 emails a month (maybe 1000 if they are extra prolific). Roughly half of those emails will be in reply to emails that originated on your mail server (you can link those with basically unique message-ids that you generated). There will be a bunch of extra safety measures on the server (reverse DNS, SPF records, no open relay...).
Spammers, on the other hand, will send thousands of messages every hour. And have none of that "other" stuff.
Spammers can buy cheap VPS and setup all this stuff with scripts. So I don't think that it really helps. It's good to have it (to prevent spoofing), but I'm not even sure that Gmail will use wrong SPF record to deny e-mail, it's not really part of SMTP, those are just optional additions.
But volume is a good argument. I don't think that spamming could be cost-effective with something like 100 e-mails per day from a single IP-address. So why not just let small servers pass every filter in the world (may be except reverse-dns record) as long as they are small? The danger should not be very big.
Of course I might be missing something, I have no idea how spammers really operate. Or may be Google just don't care about personal servers at all.
So apart from the hang-wringing going on here. What can the original poster do?
They have literally shown what they have tried to do and it is not enough.
If this poor guy is a "false positive" then what is his path to resolution? It looks like he doesn't have one and saying "Yeah, that sucks" or "Because you made it on hacker news we will investigate" isn't helping all the other "small fry" out there.
But the problem with all Google services is that there's apparently nobody listening. Unless you're a paying customer, and even then, I gather that it's iffy. It's all automated, in extremely unhelpful ways.
I do appreciate, however, that it's largely a scale issue. Google is just so big, and operates on such small margins, that it's arguably not economically feasible to provide individualized support.
Unless you make it to the HN front page, or whatever.
As a practical matter, if you want to send email to Gmail accounts, you'd better be using Gmail. And apparently, even that doesn't always work.
Thanks for sticking your neck out. When these kinds of articles happen, it often feels like a witch hunt.
How extensively does the Gmail team test interoperability with other mail systems? From my personal experience, one sin committed by many teams who in say they play nice with the ecosystem is to neglect to test integration against other players in the ecosystem. It's annoying to test and slows down development pace, so frequently no developer wants to do it. This leads to a situation where it may be unintentional, but there is definitely a "favored client."
It really got worse. In the past years (always the same IP, good reputation, DKIM), I had a few mails land in the spam folder of gmail users, but it did not happen very often.
This week I encountered the following situation the first time:
1. I received an email from a Google apps (or whatever it is called) user. My response to that email landed in the spam folder.
2. I exchanged one or two mails more with another person in the same organisation without problems.
3. Suddenly my response got rejected. I tried sending via various means such as via Thunderbird, via mobile on 4G, via mobile on Wi-Fi, changing the content slightly... No chance. My email got rejected. In the I ended up calling the person from number 1 on his mobile to get the second person's mobile number...
Needless to say, I have been less than pleased.
Person 1 was very surprised about their spam filter misbehaving, but at least I can now offer an excuse by pointing to this blog post...
> but we certainly care a lot about fixing issues like this when we hear about them.
What then should we make of the fact that Google practically ensures that they won't hear about such issues by making it next to impossible to report them?
How can you seriously argue you have no network effects when Gmail has repeatedly added (to be fair, somewhat useful) features that basically only work on Gmail, like AMP emails or expiring/deletable emails? Sure, other vendors could go out of their way to reimplement AMP and shove it into email but really?
I'd love to have a discussion about gmail but offline, feel free to mail me. I've just transitioned a small company over to gmail and the experience so far has been nothing short of terrible, this can't be how it is meant to be. If you're up for it my email is in my profile.
However, as one of the PMs in charge of one of the most important messaging applications on the planet, you cannot claim the following without stating your reasons:
> Gmail does not, inherently, have any network effects
Yes, compared to facebook, and within the context of messaging apps, gmail's network effect is weaker. But the fact that the very existence of the gmail monolith can kill off small mail servers (not implicitly, but by failure to deliver messages), is a real problem.
I understand it's a difficult problem. It's not a gmail scale problem, but the effects, times the number of small mail servers is important, because those small mail servers will be what is left if any of the big players in this ecosystem collapse.
"It's because email is open. If we lose an open, healthy ecosystem with many providers"
Don't you see? This post on HN means you have created the CLOSED unhealthy ecosystem. You built what you just spoke against when you shut out email from small providers. Shame on you for speaking idealically and not confronting the reality of the situation.
I've had emails sent to gmail users from our paid MS Exchange accounts blackholed. And not just randomly once or twice. My wife was a contractor for someone and despite having years of previous conversations with them, one day they just stopped getting her emails altogether. She had to create a gmail account just to keep working with them.
You are a PM, you may “theoretically” care when you see something in HN or some other social media, but the number of people having problems with your product are magnitudes larger than that. Everything is completely filtered out by customer support, even for companies that have good customer support unlike Google, and never reaches your ears.
I am not taking a stab at you personally. I truly believe that you care when you read the story here. But if you really cared enough there would be no story to be reported here.
Have you guys considered an automatic system with collateral. I'll gladly sign a contract stating that I will be under X% bounce rate for the next month and put some collateral (depending on the email volume requested). At the end of the month you get your collateral back.
It's a purely economic and automated solution that should work well enough for small operators (collateral for a small email volume should be small since spammers really need high volume)
I call bullshit. Google has flaunted their complete disregard for RFCs time and time again. Try to send an abuse complaint to abuse@ any domain that's hosted by Google. You can't. Try to correspond with a human. You can't.
Content filters should never be applied to abuse addresses, yet Google happily ignores this and never even replies to abuse complaints sent to @gmail.com or @google.com.
> Gmail does not, inherently, have any network effects
In an ideal world, yes. Unless Gmail makes it harder for anyone using a different provider to communicate with Gmail users. In which case you've effectively "encouraged"/forced everyone else to use Gmail to take advantage of the Gmail network. This seems to be exactly what's happened to the author. This is very similar to the complaint voiced recently by the Mozilla leader as well.
I don't doubt that you as an individual have very good intentions. But individual intentions do not translate well to organizational priorities. The best way to prevent Gmail-lock-in in the long term is to have more diversity and competition in this marketplace.
it's cool that you hang around HN, but this kind of response just exacerbates the big problem with Google these days: unless you can get to the top of HN, or picked up by TechCrunch, you have no hope of getting any resolution for any of the bad things that Google does. Whether you have incentive to fix it or not, the reality is that you refuse to acknowledge these "false positives" or give people any recourse to fix them.
I appreciate the response and the evident care at the personal level that you apply to this.
It is also an excellent goal to use AI to scale as many processes as practical to provide ever-broader services.
Yet, it is obvious that, at the corporate level and despite being one of the most massively well-funded companies worldwide, Google is famous for abysmal customer service and most particularly, providing NO way whatsoever for a person who has been damaged by a "false positive" to every correct a real person at Google to get a resolution.
Yes, providing an avenue to real people who could resolve the issue would be expensive. I seriously doubt that it would be so expensive as to make any serious dent in Aplhabet's profitability or stock price (unless, of course, the problems are far more massive than anyone knows).
Moreover, providing the ability for people who have been "false positive" damaged by some spam, account violation, etc. to reach a person for resolution to a problem would also provide large amounts of the fine-grained detailed data that would allow Google to fix its issues at the algorithm level, thereby reducing the need for the support service as well as the frustration.
Why is this not done? Is it that the problems are actually massive but hidden? Is it that management does not care the way you do? Is it that mgt actually wants to drive off all small services in order to dominate? Obviously all speculation, including bad speculation. But leaving zero ways to contact, as well as zero information about the issues, leave the field ripe for all the speculation. This is the sort of reputational problem that can fester for years, seen only as a minor issue -- until it grows so massive and passes a tipping point, where the reputation and market position is unrecoverable. I hope Google does not go that route.
If Google really cared about the health of the global email ecosystem maybe you guys should consider putting a GPL on Gmail. Maybe if we could study the code and come up with a solution that benefits everyone maybe we could make email better. Instead you guys at Google just want to maintain your near monopoly on a vital part of our online existence.
Just signed up for two newsletters today, both “Please confirm your subscription” messages went to spam. Corrected it then hit the confirm links. One “subscription confirmed” email went to inbox after that, the other still went to spam.
Guess I’ll find out whether correcting it a second time applies to the actual newsletter messages.
Thank you for the reply. Wouldn't it, somehow, be possible for Google to provide a place where one can go when such problems occur? Maybe paid (in order to scale). As is one feels completely powerless and negative emotions will be caused.
Google the giant will only bend when lawsuit happens. Assuming Google's AI put an urgent email to spam by mistake and made a huge loss to some party, Google should be immediately sued, probably by class action, only then it will respect small and well-behaved players like the original post here. Whenever I read post like this, I recall its once popular(now withdrawn) slogan "don't be evil".
Email is not an appropriate method of communication for highly urgent, impactful, and time-sensitive messages. So the party should sue themselves for negligence.
GMail is placing email from other users in the same google suites domain into spam which makes 0 sense since google knows they all belong to the same gsuite account.
If you downvote, please reply and explain your position. I simply asked if this Gmail PM is going to fix the problem with Gmail in the comments about that problem. So who is upset with my comment and WHY.
Apologies for unsolicited advice, but since you’re newish...
Downvotes don’t mean anyone’s upset, there’s a large variety of reasons people downvote. Same is true of upvotes, there are lots of different reasons for them. Try not to take either personally.
Downvotes don’t require explanation, just like upvotes don’t require explanation. It’s a great idea to be curious and seek ways to avoid downvotes, always good to assume you might be wrong. Less so to demand you were right, even if you are. :)
Thank you for the advice! I've always seen HN as a great platform for thoughtful discussion. I don't see downvoting a comment that doesn't have any replies yet as a way to participate in that discussion. I was genuinely curious what view point a person could have to not find my comment beneficial to the discussion. I'll try to make these queries a bit less complaint-sounding in the future! Thanks again!
Great to hear the positive response! FWIW, initially on HN I felt pretty frustrated with what I called ‘drive-by downvotes’ with no explanation, and also just generally upset when I’d get downvoted. That’s partly because it didn’t happen that often, so it felt more serious when it did. And sometimes it feels unfair, for sure, especially when I don’t know why and it doesn’t seem warranted to me. But over the years, I’ve become more okay with the vagueness and uncertainty that comes with it. The votes are communicating something, so I try to assume I did something wrong and just maybe don’t get what it is yet. But you never know other people’s interpretations of your words, or their moods or mental states either, so sometimes I just have to let it roll off. It’s fascinating to see something I write get misinterpreted after I thought I was really clear. And I don’t see this happen that often, but because votes are used to rank the comments, sometimes people are just moving the interesting discussions up and the less interesting ones down, so there might even be nothing at all disagreeable with a comment, it’s just that there are other really good discussions nearby. Believe it or not, I actually don’t ever downvote because it bothered me to get them myself, I just ignore threads I don’t want to read and upvote threads that are interesting. I usually upvote people who reply to me as thanks for taking the time to read & reply to me, especially when they disagree with me, so that’s an example of using votes for something other than liking or agreeing or ranking. Anyway, HN does have some pretty thoughtful discussion even with the occasional downvotes, so just seek out the good stuff!
While I know those guidelines, I still hate it when people get downvoted and there is not a single response (unless the comment obviously offends the guidelines). I mean, especially because people are voting for various reasons you can't even be sure that you broke the rules when you get downvoted and if the person has no clue what is wrong with his comment he doesn't know what to change.
So thank you for explaining the situation to craftinator :-)
a) thanks for the response
b) do you have any clues why this failure is so sporadic? That is to say, why does an email from domain/server x get blocked on one day but get through a few days later (repeatedly)?
Google does not have incentive to fix this, because they can simply not fix it, and get along just fine.
That's the point.
It doesn't have to be conscious dereliction, it can just be 'big, dumb bureaucracy'. Interestingly, operational ineffectiveness can happen within well run organizations, particularly if there's no poignant reasons to change.
These kinds of things tend to happen when one part of the organization is mining Oil or Gold and throwing vast surpluses around the rest of the camp.
You have $100 Billion dollars to fix the problem - a hoard of the best and most highly paid talent on planet earth.
If you deeply care what is the proper way of letting you know about it? I'm sorry but calling these relationships caring is rubbish. If Linus himself said that no one in the mail team listens, then what do the devs that are not the most popular name in the open source world have to do?!
Google could fix this rather simply. Have an opt-in system whereby anyone sending you email is charged a penny. (Of course, this will initially mean the sender will need to get a google account, but as this catches on more email providers will provide this service.) Split the revenue between the recipient and Google.
I don't mind spending a penny to send an email, I don't send enough to make any difference. But to spammers, it makes abusing the email system unprofitable. And if I receive half the revenue from people sending me email, it'll likely work out to costing me about nothing.
I wholly agree with your statement. I would add that, sadly, its like this with most things Google: 1) vague descriptions of the problem, 2) impossible to get support or talk to anyone, 3) severe consequences (because of the ubiquity of Google’s platforms i.e. search, Gmail, Android, Chrome). Anyone who’s had a safe browsing problem, an App Store issue, or, heaven forbid, an AdSense issue will attest.
As far as it relates to email, this is why platforms like Sendgrid got so popular, they manage the relationship with Google and others for you.
It's interesting that you bring up AOL, because I remember dealing with this kind of crap with them. It was annoying, but didn't feel like it was as big of a deal.
I guess the only real difference is that with AOL it was less insidious... You knew it was only for aol.com emails and that you weren't dealing with business info. Not that emails to personal addresses aren't important, but at least it wasn't suddenly SomeRandomFortune500Customer.com that sales is trying to email a proposal to right before the deadline that "is make or break for the entire company!"...
AOL had a highly competent mail administration outfit, at least compared to other big shops at the time. It could be annoying to deal with them at times, but they had fair, mostly sane processes and I could always get in touch with a technical person with authority there. (They may still be this good, I just haven't had reason to deal with them for quite a while.)
Google - well, I haven't had to, but can get in touch with a competent person there, but not officially. Which is not the case with ATT, with whom I have had a mail issue for over a decade. At this point I'm thinking either their postmaster died in his closet office back in about 2006 and nobody noticed, or they simply don't give a damn.
Didn’t AT&T outsource all their email to Yahoo! a decade or so ago?
Not sure I agree about being able to reach a tech person at Google, even informally. Bitching on Twitter can get some results, but that’s hardly the way to operate a business...
Didn't mean to imply that Google's stance is a good thing.
It absolutely should not be the case that you need a pal in the right department to get something done, but that seems to be where we are - our Politburos are just privatized.
Edit - My mail issue specifically is with WorldNet. I believe that's run by an ATT spinoff call "Maillennium", but maybe that changed. I don't especially care from a practical perspective; there is exactly one person whom this affects and we worked around it forever ago. I just see it as a barometer of ATT's interest in being a competent network operator.
> It's interesting that you bring up AOL, because I remember dealing with this kind of crap with them. It was annoying, but didn't feel like it was as big of a deal.
I do too, because there was an actual useful URL that had contact info in the bounces and SMTP errors. IIRC, I actually managed to get a human being on the phone and got this sorted right quick!
He he, this. E-mail rised in the 90s and was hugely popular very soon, because it brought real benefits. I was under impression that in today's world the number of emails is going down (on account of social networks and whatnot).
Well, I am almost 40, so not really a millennial at all, but a gen x'er. And I remember being in college and going days, maybe weeks without checking email. I also remember my university giving me an a .edu address, which I was technically supposed to use (supposedly there was important information sent to it from professors), but never logged into once.
The bigger problem is how to define what Google is and split it. Many of their platforms post losses like youtube, because they can afford to spend their ad money to prop it up. If you split away youtube, youtube would likely fold. Similar thing with android, Google props up android so they have an ecosystem they can get data from to use for ads. If android was split off, it would have no income stream and likely fold.
It wouldn't necessarily, but Google doesn't really provide help to customers who pay for services with their attention and data, largely because they have a substantial enough market position that they don't have to worry about losing their customer base.
Breaking them up would be a remedy to the general attitude that big, concentrated companies often develop toward the average consumer.
Google Ads support is incredible. I’ve been shocked. You can tell where their real business is. You call and you get a competent human in less than a few minutes.
Also, Google email deliverability is good, inbound and outbound. I’ve been running an email service and have had no problems with them, unlike Hotmail/Outlook.
Interesting. I've called 3 times, to give you a sense of my sample. They seem to have an interesting program going there. It's been at least fun to play with and run experiments.
> IMO what "you can tell" is that they have good support where you're actually a paying customer, and no support for their free products.
I've heard a few examples here of people having trouble getting support with both GSuite and GCloud.
Sometimes I've got support from GCloud. Even once got two blokes onsite because they'd messed up something badly, but at that time I was working on a project that was seemed to be big time bragging rights for anyone involved on both our side and their side.
Other times our requests would be handled by someone who was more or less clueless && who clearly didn't care at all IMO (this was also the same project.)
For smaller customers it seems to be hopeless from what I read here: - Someone getting their account (GSuite or GCloud) closed for no good reason and with no way to get it back seems to be a thing that happens from time to time.
Rule #1 if you're applying for jobs or graduate school - Don't use gmail for your email address.
Source: Led a hiring committee this year. Interview invitations and job offers went straight to the junk mail folder. Also serve as Director of Graduate Studies. Admissions and funding offers, as well as general inquiries about availability, all went straight to the junk mail folder. We're not going to change our email service.
This may be a fair sentiment for an educational institution (you're not directly making money from email sending / hopefully admissions marketing doesn't have this problem), but as a candidate I'd be extremely concerned if a company I was going to work with A) couldn't deliver to gmail and B) held the view that it was everyone else's problem. As it is you're statistically leaving half the candidate pool at the door, which is recoverable. A biz that can't deliver to gmail is a terrible smell though.
You have it the wrong way. Noone can make it so that gmail accepts their e-mail. It's the gmail's job after it receives the connection to process the SMTP commands and deliver the message to you.
I mean, assumming that the sender doesn't have broken SMTP client or doesn't follow standards.
It's gmail's user's problem if they don't accept mail or check their Junk folder or use an e-mail provider that they can't contact to solve issues on their side.
In the old days, if I didn't receive mail because my mail provider was rejecting it, it was me who had to talk to them. Free has a cost. And it's gmail users who should be paying it in their time and fruitless attempts at communicating with google. At least they'd realize what company they're enabling and how much it doesn't care about their e-mail.
You can absolutely make it so that gmail accepts your legitimate email if you're delivering email at high scale. A lot of people here running their own email servers would have no problem with the same emails getting delivered if they were hosted on any decently-sized platform, because part of the cost / value of those platforms is that there are human beings doing relationship management with counterparts at Google, fixing deliveribility issues when google is hard-bouncing, or maintaining reputation of IPs (which you can't do -- thats what the original article is running into, it doesn't matter if you've been delivering off the same IP for years if you're only sending small amounts of traffic).
> In the old days, if I didn't receive mail because my mail provider was rejecting it, it was me who had to talk to them. Free has a cost.
In the old days average email user was being inundated with low-quality spam and getting mired in fraud. Google fixed that problem for their end users. In doing so, they made it hard for a small % of people who want to run their own email servers, but they've given all of these people an out -- go sign up with someone who knows what they're doing, and google is going to effectively outsource the fraud/spam management to those companies.
You can tell me I have it wrong but I'm literally just describing the landscape that actively exists and how to navigate around it for anyone who wants to, but you're not taking out google.
> thats what the original article is running into, it doesn't matter if you've been delivering off the same IP for years if you're only sending small amounts of traffic
That suggests a simple solution to the problem that can be done on the gmail side: for any small mail server (to pull a number out of my behind, say less than <200 emails a month to any @gmail address for the last 12 months), white list them if they satisfy the rest of the usual requirements.
Even if someone attempts to game the system and creates a number of servers, for an effective spam campaign it means a large number of servers, costs go up.
It might be worth experimenting with "abusing" this behaviour to put your small server on the Gmail whitelist. Start sending a large number of generated emails to a @gmail.com mailbox, log in and ensure none of it ends up in spam (ideally automate that too :), and there we are. Anyone have any idea how many emails that is? :)
IP churn is a huge problem with email that everyone has to fight, these are the emails you’re getting from like list sellers and the like. And you don’t have to even have to spin up new hardware or vms, you can be in something like aws where you’re attaching new network interfaces, go and do your damage, spin it down and up goes the next one. This actively happens all of the time right now and it happens because you can profitably do it.
We are a relatively small company and we operate our own mail server, mostly out of a shared belief in data privacy among myself and our staff. We self-host most of our tooling (mail, internal chat, Gitlab, CI/CD, and so on). It turns out this is all pretty simple given modern virtualization infrastructure and, in my estimation, has many upsides versus the alternatives.
That said, like the OP, despite having DKIM, SPF, proper DNS, and every other measure applied, it's conceivable that gmail might reject some emails from us. I don't have any memory of that actually happening, but I have heard stories like the OP's several times. It's a fairly common topic in conversation forums about self-hosting email servers. I don't expect the situation to change any time soon—I don't expect Google to change their ways and it would require significant pain for me to surrender to the gmail hegemony.
Gmail's spam filters always err on the false positive side. It's always been like that. Add the opaque nature of google operations, and you can't in good faith place the deliverability burden on third parties. It's like trying to hit bullseye in darts in pitch black darkness.
Best to assume gmail is flawed, which it is, as the parent poster did. At some point Google will realize the image problem and act.
Judging by what ends up in my gmail, very few people "can deliver to gmail" in any kind of reliable fashion. It depends entirely on how messed up the spam filtering is for individual accounts. The only reason I'm still on Gmail is that it's so entrenched in my life that tidying things up to move is a massive pain and will take a long time (and no, forwarding is not a reliable solution; my experience is that even forwarding from one gmail account to another will invariably cause some mail to go the spam folder of the account you forward from - yes there are options that are meant to prevent this; no they don't work reliably, and haven't in the years I've been using it, so it means one more place to check mail).
Getting started now will make it easier though. I just bought a domain, and started emailing from it and using it for new accounts. I still use both emails, but I always reply from the new one. It's a slow transition, but quite a smooth one (I have email clients that can show an unified inbox).
I still have quite a few a accounts under Gmail, but I'll probably do a full change this year.
Having control of my email is very important for me. Google could close my account for any number of reasons (billing dispute, hacked account, etc), and I would lose my actual address since the domain isn't mine
Absolutely, it's gone from "maybe I should move somewhere" to working on untangling things because I know I have to move.
I do have a very old address that is on my own domain, and though that is currently routed to Google, at least that simplifies things a lot, but unfortunately for many years I was not consistently using that address for signing up to things etc.
> A biz that can't deliver to gmail is a terrible smell though.
If anything, it means that teams are probably too independent and that there is no central approval process for things like sending emails.
For example, a lot of companies use products like Greenhouse. I am sure this will send email from your domain on your behalf. If you don't know what you have to talk to the DNS administrators to add a bunch of TXT records to enable that, you will just notice that some percentage of emails never get delivered (or if the product doesn't tell you that they're bouncing, you may never know. how could you?)
If you use something like Zendesk, you'll note how many people have been burned by this. By default, they end up using something like support.yourdomain.com because despite detailed instructions on how to set up the necessary DNS records to send email from youdomain.com, people still fail to do it right and then complain "nobody ever sees my support tickets".
My point is, email has been abused so heavily that it is somewhat difficult to set up a working system. That was my experience when I ran my own email server. Although I did OK with delivery, I also aggressively filtered messages and used greylisting. This broke a lot of broken email systems, whose administrators immediately blamed me. (I had a long back-and-forth with some company that wanted to hire me. Their email system was super broken. They blamed me and said that they weren't interested in a programmer that couldn't set up a mail server. LOL.)
> They blamed me and said that they weren't interested in a programmer that couldn't set up a mail server.
Tangent, but this kind of stuff is super annoying and probably happens far more often than anyone wants to admit. A few friends and I have been commiserating re the incompetence of potential employers who've rejected us because of their own misconfigured environments or fundamental misunderstandings of the systems they run. We had one interviewer close out a candidacy because the code sample "didn't run" on their interview's system -- the traceback showed he had a broken half-Py2/half-Py3 install.
Yeah, you hate to see it. Sometimes you are so far behind in hiring that you can't even get out; there is nobody to ask about that half-Py2/half-Py3 install. It is unfortunate.
In the past year or so, pulling non-spam out of my gmail spam folder has become routine. Previously it was very rare that legitimate mail ended up in my spam folder, including personal mail from my father from an email address he's been using for decades. A biz that call's gmail's problem gmail's problem is not a terrible smell.
While’s gmail’s filtering is obviously neither your responsibility nor under your control, you also can’t really fault people for using one of the most popular email services on the planet.
I would hope that your department follows up some other way (you surely have a phone number!). Searches are crazy time consuming and expensive, and grad student recruitment isn’t exactly free either.
> While’s gmail’s filtering is obviously neither your responsibility nor under your control, you also can’t really fault people for using one of the most popular email services on the planet.
Why not? Popularity does not imply quality. If it is a poor service, you can (and should) fault people for using it.
> If it is a poor service, you can (and should) fault people for using it.
Agreed. There should be some give and take, some minor reputation at stake. It's healthy to have some people applying back-pressure against popularity. I like the idea that laypeople might hear whispers among their tech-savvy friends that it's not considered "cool" to have a gmail address. Much like having an aol.com email address was seen as uncool in years past. It's perfectly reasonable for some people to be working to give gmail.com a bit of that same reputation.
And actually, more seriously, the reputation push-back on Facebook is finally reaching levels where laypeople are picking up on it. There's still a huge amount of ground to cover, but laypeople are now hearing the criticism of Facebook. You can see many switching from a state of ignorance to "I don't care; I have nothing to hide." Maybe in a few years we'll see more people saying "you know what, I do find these privacy losses bothersome." As with many reputation matters, it's a gradual process with give and take.
With Facebook the average person doesn't really lose anything though, whereas with Gmail is like the mailman decides to drop some mail in the garbage every now and then (and it is on you to also check the garbage in addition to your mailbox in case the mailman decided to drop some mail in there too).
People's choice of email provider has very little bearing on their ability to do (e.g.) biology or economics research. Plus, most career centers actively advise students (i.e., many applicants) to sent up a professional-sounding gmail (or similar account) accounts. In short, it makes no sense.
More broadly, grad school admissions--and, even moreso, faculty hiring--is already full of unwritten rules, backdoors, and tacit understands of "how things are done." It emphatically does not need more of those, especially petty ones related to email.
You seem too focused on the exact scenario of applying to graduate schools, when the OP's comment wasn't circumscribed to that, and my comment certainly was much more generic than that.
People's choice of poor quality service provider X impacts their ability to do their tasks when the service is poor. X can be email, insurance, banking, anything you externalize in your life.
The fact that a service is popular is a poor indication of service quality, if there is a correlation at all. Examples abound for popular services and products that are poor quality. Usually, the best services are a bit up, from the popular choice point, in the price/quality ratio. More so if there are free (as in beer) offerings in the market.
> It emphatically does not need more of those, especially petty ones related to email.
What's the alternative? Making sure that the recipient gets the message by sending a courier to his home address?
I remember when a buddy of mine, back in the pre-email days, applied for a job at a company. About a week later, he received a letter, asking him to call them. Turns out, he forgot to put his contact info into the application, but they were interested enough to look up his name in the phone book (they also have/had street addresses were I live) and write him a letter, hoping it would reach the right person. "Always put your contact into into an application" isn't a petty rule, and neither is "use an email-provider that doesn't randomly reject messages" in my book.
A courier is clearly a little over the top (though maybe not totally absurd: a faculty search costs tens of thousands of dollars).
I would expect both sides to make a bit of effort. Applicants should check their spam folders and follow up, especially if there's a known timeline (American grad school offers need to be extended and accepted by the 15th of April, for example).
Institutions should maybe not rely on only an unreliable, unacknowledged form of communication. Since the applications usually collect phone numbers and addresses, a phone call or letter would be a very reasonable follow up; neither costs more than 50¢ to use. I think all of the PhD programs that accepted me did call and it seems like a good way not only to notify, but to actively recruit.
Indeed, the spur for me to finally sign up for Fastmail was that I was applying for a job and I can't trust Google with crucial email. You get what you pay for.
I had similar thoughts, but it took a while until I found something like Netlify to handle the domain end of things. My personal site was on DreamHost, who threw in Google Apps for free (this was before the bottom tier went paid).
At the very least, I knew that if mail ended up in that inbox, it was usually important.
More to the point, Fastmail allowed me to use Mail.app on macOS and get away from Google's web UI which, despite best efforts on their part - read: none at all - never worked correctly on the desktop with regard to message deletion in IMAP clients.
Fastmail's web client is absurdly excellent compared to Gmail. It's as if, instead of copying all of the bogus stuff Google piled into Gmail, they just made a really good version of webmail the way it was in 2003. Instead of scrapping webmail and starting over, they evolved it into something better. The result is fast, clear, easy to use, and no-nonsense. It's as if all the features that bloat Gmail at this point are just there to solve problems caused by Gmail's bloated feature set. You really don't appreciate how bad it's gotten until you switch.
Even if you pay for it and don't use g, you can't know with absolute certainty that your email is going to get to the person you are sending it to.
There's a high 9's probability that it will arrive safely, but there's several ways that things can go wrong, simplest of which is that you are deleted by a misconfigured filter.
There are US ISPs (cough sbcglobal) that routinely throw away mail from some arbitrary IPs despite there being absolutely no SPAM blacklist entries for the IP, even if you are replying to a mail sent from one of their clients.
outlook.com once, (and maybe still does) blocked email from an entire C-block for some reason.
> Even if you pay for it and don't use g, you can't know with absolute certainty that your email is going to get to the person you are sending it to.
And there's the core of all of this. SMTP is a best-effort protocol that does not guarantee delivery. Until someone comes up with something better, we're condemned to seeing "Help! My very important email disappeared in transit" threads on HN every several months for the rest of eternity.
I'd suggest that regardless of your email provider, you should obsessively check your spam folder if you're expecting a job/internship/etc. offer. Every spam filter has false positives, and those sorts of messages can trip a lot of the heuristics tuned for actual spam.
True, but this does no good if your email system doesn't accept it at all (the sender gets a bounce) or worse, if your email is silently deleted (not sent to the spam folder).
If I send you an email and gmail doesn't get it to you for either of the above reasons, you won't get a second chance.
Seems like if you don't change something you are potentially losing out on a lot of great students. Maybe you do not need to change your e-mail service, but I would hope you are at least warning applicants about this issue somewhere in the application process.
When I try to be sure that I will get a answer, I always try to write to the company email instead of filling a web form. When Gmail sees that I wrote to a company, it won't put its emails in the spam folder.
The right response to this problem - for gmail users - would be to switch to a different email provider. After all, if you are not receiving important mails because Google decides to block them for inexplicable reasons, you are getting an inferior, faulty service.
(For me gmail is inferior regardless as my privacy is violated)
I recently switched (mostly) from gmail and realized what insidious lock-in mechanisms the "unlimited inbox" and "inbox categories" really are. I hadn't been giving it much thought, but I've gone well over a decade just casually archiving everything, on the presumption that I can always find anything important I might need to reference by searching for it down the line. And I'd just been ignoring those junkier "category" tabs for months on end, presuming I can always go sort through them later to find those handful of actually-important messages that end up in there.
Maybe this is just my personal organizational problem, but it was like a waking nightmare when I really started to look with fresh eyes at how unstructured my email management strategy had become with the tools gmail gives you. It took me weeks of nights and weekends to sort this all out (finding messages I actually want to save, identifying senders I actually want manually sorted to other folders, unsubscribing from lists I've been ignoring for years because they were conveniently out of site in "promotions") to a point where I felt ready to migrate to a more traditional mailbox setup. I can easily imagine many people seeing how daunting this task is and just deciding to stick with gmail.
Free is not possible. Someone needs to pay for email, servers need to be repaired/replaced every few years, the power bill needs to be paid, someone needs to ensure that the latest exploit doesn't compromise the server. You are paying for your email, the only question is how.
I use fastmail because the payment is direct: I give them a few bucks each month.
With free email (gmail) the payment is not direct. They are doing something to get money, but what is unknown.
There are a lot of free services. Sure, they are getting money from someone, but small personal accounts often are free. Like they hook me on free account and then I recommend it or upgrade as I grow or just spread information about it. For example I never paid for DNS hosting. Currently I'm using Cloudflare for free.
ProtonMail has a free tier. It's definitely not the same level of service as Gmail, but I prefer it to being so dependent on the big G juggernaut for a critical service.
Downsides:
* spam detector is a bit aggressive, sometimes legitimate emails make it in there.
* features are limited unless you pay.
Full disclosure: I pay for their Plus product. My rationale is that I pay twice as much per month for Netflix yet email is hugely more valuable to me than Netflix is.
More full disclosure: I used to work for Google. That's part of what made me move from Gmail.
I'd argue that if ProtonMail's spam filter is too aggressive then basically you're in the same place you were with Gmail, aren't you? (Except now you're paying?)
The issue described in the OP is email not even making it to the spam folder. If it at least makes it to ProtonMail's spam folder, that's an improvement.
I'm hosting my mail on $1 VPS right now. It's hard for me to justify paying more. I hate to setup that mail stuff, it's too tricky and takes a lot of time, but still not worth paying extra.
This tip is good for almost any service. Gmail could ban you any minute. I doubt that they'll allow to get backups after that.
May be if I'm paying some money to some local company in my city, I can have some confidence, because I can come to their office and yell at them until they return my data to me. But they can just be incompetent and lose it, so even in this case I better do backups.
I would love to pay FastMail for my family’s email. I have myself, my wife, my four children, one family address and one other generic address. 8 emails would cost me $240 a year at a minimum.
If they could solve for my situation and not charge me that kind of money, I would switch from Gmail immediately.
I find it astonishing how much people devalue software and information services. The same people who have to have a $1500 phone and seem to think nothing of dropping $150 a month on Verizon or AT&T won't spend $5 on any cloud service.
This is why Google keeps winning and everyone else is losing, particularly the people who get the "free" services but don't have any say about those services.
The real numbers might be different, but I heard that Google gets about $100/yr in ad revenue per user. That is, they are getting $800 a year for all of those people. Everything you buy is $800 more expensive and there is no way you can say no to that!
You seem so unsure about your own point that you immediately downplay a users concerns about spending $250 a year by reducing that number to $5. Why the need to fabricate a theoretical number when someone provided you with a specific value?
If I spend $300 dollars on a phone every 2-3 years and $40 for service a month, am I then permitted to raise concerns about spending hundreds of dollars a year for email accounts?
I can just tell you that I've seen difficulty at selling things at the $5/month price point. In particular I've seen people bridle at the costs of running a very small system in the AWS cloud that costs about that much.
Not everyone spends money like that. I personally have a 6yr old phone that cost me $20, and since I'm usually on WiFi i have a $10/mo cell phone plan from an mvno. Most of my friends pay more than I do, but still are far from your example.
Fastmail allows for email aliases. These aren't those addresses with a `+` on them, but rather completely different addresses. You can put the family address and generic address in your own or your wife's account and setup a filter so that email to those addresses go to separate folders automatically. That saves you $60/year. If your children don't need privacy from one another (I've no clue how old they are or how well they get along), maybe you can also group them into 1 account to save you another $90/year.
By the way, Fastmail prices are rising. One used to be able to get a free account, and the standard plan used to be $40 a year, so might be good to get them sooner rather than later to get grandfathered-in in case of upcoming price risings.
Can you honestly not afford it? Or do you just think it shouldn't cost $20/month? I'm the sole income earner in a family of 5 and switched us to fastmail.
A lot depends upon how frugal you already are, but I found it trivial to find some extra savings to pay for it. There's a few things I like to treat myself to, and doing that one less time per month pays for it. Eating out one less time per month pays for it too. Ordering slightly cheaper meals for myself when we eat out would also pay for it.
Hell, when I turned off my home server, which I rarely used, I saved $9/month in electricity. If you live in a high power cost state, do a personal power audit. Our family was pretty lazy with turning things off until I figured out how much it costs to run any given thing.
With a family of 6, if you haven't crunched your finances, there are lots of very minor behavior changes that can save you enough money to pay for something like fastmail.
Migadu is the place to go--they charge for traffic, not mailboxes. I'm in a similar boat with lots of domains, aliases, and mailboxes. My family comfortably fits in their "mini" plan, $48/year.
Indeed, I am very glad they have no free tier. It keeps bad apples out and let's them focus on actual paid customers. Great products don't need a free tier.
Does Google allow you to use your own domain for free?
They removed the free GSuite tier a long time ago, and they've obscured some the options you could use to configure a similar setup in gmail. I'm not sure if it's still possible.
I don't believe it's possible with non-Google Domain hosted domain names but I own a number of domains that I use to send and receive from my root @gmail.com account for free (Google is my registrar for these). They come in to @gmail.com from @mydomain.com and when I reply it goes out as @mydomain.com. I can also send email from any of my @mydomain.com emails by opening compose and clicking an arrow on my from address.
There is the huge possibility that this will cause issues with some mail servers because I'm assuming it's doing a send on behalf of (I could be totally wrong). But so far (1yr+) I don't recall any issues.
I switched to Proton for a bit but I just didn't enjoy using it at all. The final nail in that coffin; I was in the middle of selling a house and receiving documents and tons of back and forth and it was an absolutely terrible experience. Every realtor/contractor/etcs signature pictures, etc showed up as attachments and the search was awful so having to dig up contracts and everything else that goes along with a house sale was just an absolutely miserable experience. Every single email looked like it had tons of attachments. I had to go through every single email in a thread to find the actual documents.
I'm now regularly seeing search results bury political pages which seem to run counter bay area political views. I'll search the terms on Bing and DuckDuckGo and the page I'm expecting will be the first hit, but on Google it'll be on page 3 or 4.
Clearly they keep the hit in the search results for plausible deniability, but they're trying to influence people by suppressing speech. Given that Google is becoming synonymous with the internet, like AOL, this is a very concerning trend.
>I’m now regularly seeing search results bury political pages which seem to run counter bay area political views.
This has been a well known occurrence. I’m think about all those pictures of “Hillary Clinton is” [0] typed into each search engine and Google was polar opposite of all others.
I think one needs to be pretty obtuse to not see Google is “curating” results to fit an ideology.
Everyone should be mad at that even if they believe in the same things.
EDIT: Well... apparently some people don't believe me, so I just re-tested. "Hillary Clinton i" in four engines.
I couldn't use "Hillary Clinton is" because "for some reason" Google guessing completely stops working when you type "is" after her name - can someone find me another phrase besides "Hillary Clinton" that "breaks" the search fill when "is" gets added? No? Well.. That sure seems like Google is "curating" then.
One example is the story of the sons of John Kerry and Joe Biden partnering together to form a private equity fund which received money from the Chinese government, and then using that money to acquire strategically sensitive assets, in some cases partnering with firms accused by the US government of hacking US companies for intellectual property:
One interesting thing to note is that most of the websites which picked up this story are right-wing, and that Google appears to simply be blacklisting non-mainstream right-wing websites from its search results. Whether or not this is due to any implicit ideological bias on Google's part is uncertain.
I see an article covering this story with both links. Google serves The Hill first, while Bing serves NYPost. On the whole, Google results seem to be better, although this may have something to do with the more detailed profile they have on me as their user.
The evil part about this is that it needs no one at any step to say it out loud. It just “happens” as a result of Google being big.
Whatever lead to Microsoft being sued for pushing Internet Explorer in Europe and whatnot? Shouldn’t that kick in already for at least some of the recent Google bullshit?
I think it should, or some sort of oversight should be in place. Their filters are extremely aggressive. I have a specific GMail rule in place to ensure that all mail gets through and nothing gets sent to spam, but even with that the occasional message gets sent to spam. It's generally advertising, but never actually spam.
There's definitely for GMail to become the new "Windows", with everyone else fighting to stay connected.
In an attempt to avoid google, I kept a non-gmail address that I don’t administer, and instead forwarded it elsewhere.
The administrators of the account decided to outsource the smtp relay to google. It is ridiculously unreliable, and regularly sends “no such address” responses to my synology NAS (which emails me periodically).
Similarly, when I used gmail’s IMAP gateway for work, it was regularly down.
I’m pretty sure Google will succeed at killing email. I hope gmail also goes down in flames if they succeed.
> It's like we're heading back to the days when people didn't know the difference between AOL and "the internet."
I hate to say it, I really do, because I used to respect Google and liked their products, but if you imagine combining the perception of "AOL is the internet" with the way Microsoft acted up to the mid-2000's and I think you have Google.
we're definitely there in the K-12 market. The only email kids know is Gmail. The only browser they know is Chrome. The only OS a lot of them now know is ChromeOS, on the Chromebook their school was basically given "for free." their editors are Google Docs, Google Sheets, they use Google Hangounts to chat. Their phones are Android with all of the Google services. They know YouTube, Google Play, Google Music, etc. YouTube again for music.
Some have iPhones
.. and their Apple ID is a @gmail.com address.
It terrifies me just how deeply Google has entrenched itself in the K-12 education market. At least when Apple was everywhere, we still had Microsoft Office vs Appleworks/etc.
Google is now more "the internet" than AOL could have dreamed of.
The question now is fight them or join them. On Microsoft people had chosen to fight, because the tech was horrible in effect, and Linux made much nicer server platforms.
On Google, I'm not so sure. The tech is nice, and everything works beautifully for 99% of the users
They don't even have to be doing it deliberately; as you say, the lack of incentives to fix such issues will cause a gradual evolution in that direction.
I feel like as they leave more and more up to the algorithms, we suffer. I've been having this ever increasing in frequency issue where Gmail Search just doesn't working. Not that it doesn't return search results but rather that it returns what I know to be incorrect results.
I have an email with some account information that I access a couple times a year at most but will consistently need at least once a year. Just infrequently enough that I can't remember the exact contents.
I know exactly what the subject line of the email is and it's a suggested search query in Gmail but about 2 years ago Gmail just stopped being able to find it. I can navigate to the email via label or star just fine but I can no longer search for it.
Is it because it's an 8 year old email? idk what the reasoning is but Gmail as deemed it irrelevant and stopped returning it as a search result. This isn't a platform issue either, it happens in App, or browser on multiple devices.
> I feel like as they leave more and more up to the algorithms, we suffer.
With all the hand-wringing about what happens when the "computers" take all of our jobs, I think you crystallized how it all ends for us. It won't be a cinematic extinction of humanity. We will just be ignored--each of us trying to talk to a machine with a human voice, and each of us perpetually on-hold.
> I feel like as they leave more and more up to the algorithms, we suffer.
Seems like it ends up working somewhat like China's social credit policy, but without specific intent. As long as you always conform enough to fall into the normal area of the algorithm you're good, but get off towards one edge for any reason and you quietly get [effectively] booted off the 'net.
You get hired to be the subject matter expert on economic models, then you remind executives who consult with you that their plans don’t get better just by adding computers (or fancy math of any form).
Eventually, even when you get fired because they tired of you being a nag, you write down all the times your nagging was right, and someone else hires you to be the math nag, because corporations have accepted they need them — even if they’ll also eventually fire you for being right.
There’s easier ways to make money with the same skill set.
I too self-host and have done for decades. When I had a problem that may perhaps have been like this, Google disabled the relevant part of them spam filter and made an issue in some internal tracker. I write "Perhaps" because my report was a lot more specific.
Google sells mail+spam filtering+more as a product, with a seven-digit number of paying customers, each of which pay per month and user. What more incentive could they have?
That bit is easier for me than for you, perhaps, since gmail implements about four RFCs I've written ;)
That bit isn't the MSB, though. The MSB is that they do care about fixing interop problems with everyone, they just don't read mail from randoms. I think they assume is that if a problem is their fault it will show up on their monitoring graphs, and watch their error graphs instead of their postmaster inbox.
These days many people don't know the difference between Facebook and "the Internet". We've handed the reigns to our communication and culture into the greedy palms of about five mega-corporations that put profit above all else. Much better to use community software and platforms - things that people run for people, not companies running for profit.
Totally. I think that we need to start conceiving of email as the first widely used social network, in terms of the tendencies of large companies to try and balkanize the internet, and what a decentralized social network can look like.
Funny you mention AOL. There used to be a magic formula for how many concurrent SMTP connections, how many emails/minute, etc, you could pump at them before you silently went into spam.
I worked on a team with a large (real double opt in, not spam) email list. We had to pay for a bunch of AOL accounts to figure out the trigger points, so that we could get our email updates out overnight without tripping AOL's spammer logic.
And there was a time when they had dominant email share like Google has now.
You don't have to switch to gmail, you can pretty reliably switch to a myriad of service providers. You could even solely switch your SMTP server to something hosted and retain message storage on your end if you wanted.
It has become very difficult / unwise to host your own SMTP server, though, but that's been true for ages now. Gmail's marketshare is going to make that veeeery obvious, but it really has been a bad idea for the better part of a decade, at least.
It is a bad idea to host your own SMTP, but only because you need to keep on top of maintaining it. Most of that effort is the same if you have one user or a million, so it is more efficient to outsource that maintenance. However if you are willing to pay the price in time there is no reason you shouldn't be able to maintain your own server.
I use fastmail. They have humans that respond quickly when I have a problem (twice in 10 years)
Difficult to host smtp?! That's absolutely not true and has never been true. It's difficult to configure and set up in the first place but after that it's just updating. Source: personal experience for decades.
I though the issue was with getting your mails accepted. Even gmail has no problem sending email to a server that isn't properly set up as long as it is willing to accept.
Can you mention (or link to) situations where gmail refuses to send mails to another domain?
But it's not just postfix or sendmail you need to configure correctly and then put apt-get update/upgrade -y in cron to avoid the damage caused by making a mistake or falling asleep at the wheel. Casual self-hosters have other vectors of attack to mitigate, for example making sure sasl or whatever they may be using for imaps auth is solid, that passwords are strong and routinely changed, that php is also tight and so on. The stakes are relatively high, the efforts to compromise systems for this and other purposes are voluminous and incessant. So I'd say "it takes a village" to do this safely enough, not individual enthusiasts who cannot commit to administering a server diligently.
Any tips for somebody looking to move to my own smtp server? It's been on my mind a few months, and I want to move away from gmail to my own server or to another service, preferably one I pay for.
Surely someone has to run their own SMTP servers, or there won't be any SMTP servers. So how is it decided who can run SMTP servers and who can not, and is this a good method?
> It's like we're heading back to the days when people didn't know the difference between AOL and "the internet."
Why is this no longer true?
IMO, other competitors came along offering a better service/experience and AOL just couldn't compete. At the time, you could not know what this better product looked like. I believe the same will happen with the internet giants of today.
You're over-thinking it. GMail's market pretense is large. The number of edge cases like this in the entire world combined is a rounding error to them. It would likely be easier to fix problems to minimize bug reports and keep community good standing than increase GMail's user base by 0.001%.
Define 'people'. We're already there IMO. The kids today don't have any particular understanding of what the internet is and how it works. They view it through walled gardens via the app store.
> I'm sure there is no malicious intent behind this
How can anyone be sure of this? This is only one of Google’s practices that seems to follow a pervasive pattern of eroding open internet standards while presenting Google’s own proprietary implementation as somehow superior. Eventually, the open standard loses all meaning because the most popular implementation does not actually adhere to it. Meanwhile, Google reaps enormous benefits in the form of additional signals for its advertising business. How can this not be grounded in malicious intent?
> and that there are some very smart people working on spam prevention at Google.
There are some very smart people working on advertising at Google. The rush to forget the primary nature of Google - it’s an adtech firm - is why they’ve been allowed to skate for so long. Gmail’s spam filtering is just a pretext for passing all email through a machine learning system. Sure, one possible signal emitted by that system is whether a message is spam or not. Perhaps this determination is conflated with wether the message is useful for ad targeting: after all, when viewed from Google’s own perspective certain e-mail messages contain no information which can be used for ad targeting, so they must be spam. The user’s interests are clearly secondary to this.
So, back to the “smart people” working on this: at what point do we begin judging engineers for working at Google? There’s a lot of highly vitriolic criticism that emanates from Google’s workforce on a variety of subjects, but how many of them would actually pull the pin and leave their employer? I don’t have any statistics to offer, but it seems to me that we still have a ways to go before Google has become completely drained of engineering mindshare.
Indifference that causes a business problem turns into not-indifference. Indifference that does not cause a business problem remains indifference. No one ever calls a meeting.
There is a lot of evil done in the world that comes from this simple and straightforward principle. You don't have to have an evil plan to end up doing evil.
Personally I see this as just more Google incompetence. Gmail has just become bad. If you want to get all your mail you have to use something else.
> Indifference that does not cause a business problem remains indifference.
More generally, indifference to small "trivial" problems eventually normalizes the deviant behavior[1]. Evil behavior doesn't happen in one step, it's end result of regularly being indifferent to "minor" problems[2].
For a very good explanation of why this happens, I recommend Richard Cook's excellent talk, "Resilience In Complex Adaptive Systems"[3].
Personally I see this as just more Google incompetence. Gmail has just become bad. If you want to get all your mail you have to use something else.
It's like we're going back to the days when each e-mail service only trusted messages from its own users, and if you wanted messages to go to another network, you had to use an e-mail gateway.
I don't think this is a general problem with e-mail services. If you use an e-mail provider that's a paid service rather than a pipeline into an ad company, you actually get good service and a customer support team that has an incentive to fix your problems. (I've been very happy with Fastmail for many years.)
>> I'm sure there is no malicious intent behind this
> How can anyone be sure of this?
I know this would sound crazy to an alien learning English but the phrase, "I'm sure there is no malicious intent" in this context actually means "I don't presume there is malicious intent".
It does not mean, "I have a positive reason to believe there is no malicious intent."
Actually, I interpreted it as exactly "I have a positive reason to believe there is no malicious intent." and I presume the positive reasons are, e.g. most people not being malicious. "I don't presume there is malicious intent" is significantly weaker than the original.
I have been thinking a bit about the anti-competitive effects of Google's behavior in the email space. It is something I'd like to research further. One interesting part of the research would be looking at email provider market share over time (or even just a current snapshot).
Does anyone have suggestions on how to go about this? My first thought was to use a password breach like Collection #1 to get a distribution of domains used, then query each domain to see which provider they are using or if they are self hosted. But I would certainly appreciate other suggestions!
I find it mind-blowing that Gmail's algorithm doesn't lean more on users to train the spam filters...instead of top-down arbitrary keyword filters & rules that constantly result in false positives.
Apparently human users are trustworthy enough to train 4,000 lb self-driving cars to not kill humans via CAPTCHA. Why the hell can't we rely more on humans to train the spam filter?
It could solve literally every issue with email deliverability today. It's clear the current algorithms are simply not sophisticated enough to do this job properly. Hell, gmail can't even properly discern the difference between an "Update" and a "Promotion." Every dev-related newsletter I'm subscribed to (all of which I want to receive) randomly alternates between those two tabs each week.
To be honest, I highly doubt Google is using any sophisticated Machine Learning algorithms inside Gmail. It doesn't seem like a big priority for them. The space is ripe for a competitor to come in.
I would LOVE to have an email address where I was 100% confident in the spam filter, didn't hide a large portion of my emails behind disappearing tabs, and had more control over what categories my messages go to. Any alternatives out there I should be using?
Just because Google stopped scanning email for ad targeting two years ago doesn't detract from the fact that it was their practice for ten years previously.
> Gmail’s spam filtering is just a pretext for passing all email through a machine learning system. Sure, one possible signal emitted by that system is whether a message is spam or not. Perhaps this determination is conflated with wether the message is useful for ad targeting
And well, that's clearly not the case if Google moved from doing both ad targeting and spam to just spam. The evidence is literally Google moving in the polar opposite direction from the nefariousness that they suggest.
Organizations are often best modeled by an optimizing AI system. Often there is no intent behind behavior, simple that behaviors that result in propagation of the meme are preferred by the system. Never ascribe to intent what emergent behavior and individual's apathy can explain.
You’re saying that the system’s apparent behavior is by design, but wasn’t that design put forth by employees of the company in question? These aren’t natural phenomena we’re discussing. When a puma tears apart a rabbit, it does so without any influence from the agency of man. Gmail’s “spam filtering” is no such thing: it was deliberately designed to work this way by paid professionals who knew what they were doing.
the next step of analysis is that they behave this way to capture more growth and market share, which they must do under capitalism or lose out to any more ruthless company.
this is basically equivalent to pretending that as long as no one says "this is my evil plan" before doing something, it's not evil. which even google doesn't bother to try to do anymore.
I'm also one of those oldfarts that insists on hosting my private emails for me and a few friends. I've owned the ip since early 2000's and have had zero "spam"/blacklist incidents. One trend I've been noticing is that Microsoft/Outlook have regularly started to block me. I have to contact their abuse which takes a day or so to get unblocked. To date, I've done so >10 times:
hotmail-com.olc.protection.outlook.com[104.47.1.33] said: 550 5.7.1
Unfortunately, messages from [<redacted>] weren't sent. Please contact
your Internet service provider since part of their network is on our block
list (S3150). You can also refer your provider to
http://mail.live.com/mail/troubleshooting.aspx#errors.
[VE1EUR01FT028.eop-EUR01.prod.protection.outlook.com] (in reply to MAIL
FROM command)
Reporting-MTA: dns; <redacted>
X-Postfix-Queue-ID: 14A41FEB66
X-Postfix-Sender: rfc822; <redacted>
Arrival-Date: Thu, 14 Mar 2019 14:07:42 +0200 (CEST)
I host my own email stuff too (private stuff and for a couple of friends) and sending to microsoft domains (hotmail, outlook.com) consistently gives me the most problems. Right now it seems to be ok again but I always make sure that people using my domains are fully aware of all these problems.
Edit: not long ago I used a gmail account to contact microsoft support to get them to remove me from their blacklist again (couldn't use my normal email for obvious reasons). Ironically enough their reply got marked as spam by gmail (something in my included message parts might very well have triggered that but I had to laugh out loud at the situation).
I'm trying out purelymail (basically one dude's AWS hosting of email for $10/yr) and I'm having the same problem with my custom domain: google stopped having issues after the first few emails I sent got marked as "not spam", but microsoft, despite the purelymail guy filling out all the requisite forms, still eats all of my email, not even sending it to the spam folder of the addresses I'm sending to.
Don't worry, you can fix all of this by joining the ReturnPath Protection Racket™^W^W "Sender Certification Program" that will let you pay to guarantee your mail ends up in the Inbox at all the major players.
Is it illegal to pay for one of the captcha solvers? Ones like this are the easiest, captcha solves cost something like 4 cents each, and all you do is send the image to an API, which responds with the solved captcha.
As someone who runs their own personal mail server, this annoys me to no end.
I set up DKIM, SPF, and reverse-DNS records and resented every moment of it. Even after all that, there's some chance that an email from my server will be marked as insecure/spam or otherwise just not be delivered because Google has come up with some new brilliant mail security/auth/permission scheme that the world has to adopt tomorrow or be cut off from all Gmail users.
It's not just email, it seems like any service is at risk of being ignored through malice or just being too tiny it gets neglected if it doesn't help the walled garden somehow.
At one point I heard a rumor that no human had worked on Google Finance in over a year, simply because nobody was interested. Not their biggest service of course, but still a major site that presumably generates a lot of revenue. If they think that global finance is "too boring" for their amazing engineers then support for everything is liable to be canceled at any moment.
> but still a major site that presumably generates a lot of revenue
That's very possible! It's definitely an amazing site and it's absolutely clear that it could drive revenue.
With that said, perhaps it might be like Google News, where it not only doesn't generate profits, but doesn't generate any revenue at all? Is it possible that being a major site and generating revenue might be entirely distinct?
Please accept my heartfelt apologies! I can see I was unclear in my genuineness.
Let me restate. It is, in my opinion, possible that Google Finance is in a position like that of Google News and thus generates no revenue. Have you considered this possibility? Do you think it might affect the willingness of people to work on a product? If I'm utterly wrong, I would love to know so that I can avoid being wrong this way in the future.
Kalium, I'm going against my better judgement here, but I'm hoping that you are trying to apologize for real. As such, more feedback may be useful to you.
You tend to do this a lot and it does come off as very sarcastic, mocking, and/or un-genuine. As in, like, every sixth comment you leave.
Usually, you say something like : "You're right!" or "You're totally correct!" and then you exaggerate the point the person was trying to make, so much so that it comes off as mocking the comment and the person.
If you are mocking them, stop. It doesn't add anything to the conversation.
You are a pretty quick and sharp person, and you make a lot of very good points. So that's why I'm leaving this comment at all. You may just not realize that this is what you are doing.
I'm hoping that you just haven't seen that your style of commenting is coming across very badly.
But if you are trying to do this on purpose, stop it.
I have seen it go over badly. I've also seen it go over very well. The running total so far seems to favor the latter.
I'm working mainly from Dale Carnegie. The core thesis is this: never tell someone that they're wrong. Instead, start by finding a way to tell them how right they are on a point you genuinely agree with. Once you've satisfied someone's ego, it becomes possible to engage with their intellect. This can be tricky, especially when someone's main point is something you disagree with.
I sincerely doubt that "What if all your stated assumptions are wrong?" would have been well-received. That's exactly the kind of comment that tends to get a defensive response rather a thoughtful one.
My apology was genuine. I had no reason to be nasty, mocking, or sarcastic to this person. I was hoping that they might be willing to re-examine their stated assumptions. I failed to sufficiently hedge against a sarcastic reading. Which is admittedly a difficult task in a text-based medium centered mainly around a language and culture that uses sarcasm heavily. I've yet to find a way to reliably do this.
I do appreciate your feedback. It's a work in progress.
I've reread it a couple of times, and I fail to see the malicious intent. You assume Google Finance makes revenue. The reply simply asks, what if it doesn't?
It's understandable. It's very easy for people to read questions challenging them as malicious and pointedly sarcastic, implying that of course the person is wrong.
>Even after all that, there's some chance that an email from my server will be marked as insecure/spam
Don't I know it. This happened to us but not with Google. We use Exchange Online (Office365) and have DKIM, SPF headers configured. At some point our emails stopped being received by bell and cox emails addresses ... sporadically but sometimes for days at a time. It turned out that the spam filter provider (used by bell and maybe cox) correlated one innocuous phrase in our confidentiality notice (which is appended to all outgoing email signatures) with spam and therefore marked the entire email as spam as well.
Especially the part about how "if you are not the intended recipient, you are ordered to destroy all copies of this message."
It's my computer. You sent the bad message. You're in no position to "order" me to do anything.
I suspect they're mandated by self-important Lower Middle Manager of the Year Award nominees, not actual legal departments. The sort of people who never do anything but push papers around, yet put a bunch of meaningless initials after their name in e-mail signatures because they went to a conference.
> Especially the part about how "if you are not the intended recipient, you are ordered to destroy all copies of this message." It's my computer. You sent the bad message. You're in no position to "order" me to do anything.
Best case scenario, someone listens and destroys the message.
Worst case scenario, the company is no worse off than if they hadn't included the disclaimer.
Forget it, Jake. I have a doctor who has a big old footer, "If you have any questions, reply to this HIPAA-compliant email." I explained the problem with this in detail, but it's still there.
I'm not surprised Microsoft's IP range is known by google. Sending from sendgrid, mailchimp, and a few others that send a lot of mail also works. It's when a new player wants to get in the market (or god forbid, an individual with their own nerdy server) where the trouble arises.
To clarify, I wasn't arguing that sending email to Gmail is easy especially if you're using an independent provider that is unknown to Google - which is how you're framing my response as (why?).
I was acknowledging the general issue of dealing with modern email problems and how opaque the underlying systems are.
Postmark requires a minimum of 50K emails a month for a dedicated IP address, and they state that minimum is for quality reasons. You won't get a good enough reputation on your IP without it.
That should give people an estimate of the required volume to get decent delivirability rates to big providers.
Uh, SPF was being developed long before Gmail was a thing and was pushed for by at least 1 other (Yahoo) mass mail provider, as was reverse DNS verification. If you're going to be salty at someone, it should be the spammers.
I think the point still stands. As annoying as any of these technologies can be for someone maintaining a mail server, the unfortunate reality is that we don't live in the early web utopia anymore. There are people seriously trying to bypass and break these systems, which is why these protocols are needed. Being angry at them doesn't solve the underlying issue.
I'd rather turn on encryption/signing/security for my mail than contribute to a system like the telephone network where bank accounts get emptied by transoceanic criminal networks on the regular.
Asking whether anyone resents using https is just being snarky. You know the answer and the obviousness of it makes it condescending. We all agree that https is good, so comparing that to spf and dkim is like saying GP is stupid for not having enjoyed it.
Even ignoring spf and the other anti-spam tech that doesn't do anything remotely like https, dkim only signs messages. Https is more like pgp or s/mime, not dkim. It's like posting the hash when you share a binary online: the recipient can verify the integrity and sender (namely, whoever put up the hash), but I don't think anyone considers a signed binary to be private like bank transactions done with https.
I do actually resent the way google imposed HTTPS everywhere, but my issues were with the rollout as opposed to the tech. Rant for another day :)
But with the security/encryption/signing for email... yeah, if it was just an "on" switch somewhere, I'd be all for it. Unfortunately, setting up SPF for something like exim isn't very straightforward at all. It's flipping LOTS of "on" switches in lots of different places, generating and copying keys around, setting permissions on all the files properly, testing etc. It turns into a legitimate project fairly quickly, and it's hard to prove that this is a better method than just really well-trained spam filters. So, basically, it ends up being by Google fiat that I had to do that project.
Are you thinking of DKIM? SPF only requires configuring a DNS entry. Also DKIM did not start with Google. DKIM was originally the work of Yahoo and Cisco. Yahoo, now OATH, has a patent covering it.
Nitpick: you appear to be describing dkim; SPF is a single setting in a single DNS record (per sending domain) and requires no modifications on the sending server
>I've been running a small SMTP and IMAP mail server for [...] around 15 years [...] I have SPF records and DKIM message signing setup on the domains I use. The server is hosted on commercial static IP space (with the very same IP it first went on-line) and I've made sure with the ISP that correct reverse DNS records are in place. [...] Being a good administrator and a well-behaved player on the network is no longer enough [...] Now every time I write a mail I wonder whether Google's AI will let it through or not. [...] So far Google has blocked personal messages to friends and family in multiple languages, as well as business mail. I stopped guessing what text their algorithms deem suspicious.
Yep, I know the author's frustration very well. I made a previous comment[0] trying to warn others of personal email servers' outgoing email being spam-holed -- and yet some of the replies still argued I was overstating the difficulties.
Everybody's risk tolerance is different. Personally, I just don't have the bandwidth to administer my own private email server and constantly worry if recipients are receiving my emails.
The last time a company screwed with e-mail at scale was Earthlink.
Remember for a couple of years, every time you'd send a message to someone at Earthlink, you'd get an automated rely demanding that you verify yourself before the message would go through?
Now I can't remember the last time I saw an @earthlink.* e-mail address.
Google has apparently learned from that and put the "error" in the 550 messages, where they can't be seen by the end user, and lead to non-helpful resolutions for sysadmins.
The result is that the blame for missing messages goes to the sender, not to the recipient's email service.
If Gmail at least notified the sender that there was a problem, then a pattern of responsibility could be established. But this is just another dark pattern.
If enough expected messages don’t show up, users are going to start thinking gmail is broken. The problem for Google is that they are not the only trusted brand consumers interact with. I’ve seen a lot of stuff start ending up in spam that shouldn’t be there. If enough password resets, customer service responses, and order confirmations aren’t delivered, it looks like Google has a problem. If it doesn’t even make it to the spam box it makes Google look really bad.
Google has definitely been changing things, everyone presumes it’s some rogue ML system. My company had solid Gmail delivery for almost 10 years, and at a fairly large volume. Now we can’t get it delivered any more. Very close to showing users a message not to use Gmail.
> The problem for Google is that they are not the only trusted brand consumers interact with.
The solution to this is to start tarnishing this trust by word of mouth when talking to non-technical users. I take every opportunity to put Google in a bad light until they change their behaviour.
Here's the problem with inline filtering of SMTP transactions.
1) The RFCs mandate a server to accept multiple recipients--100 minimum per RFC 5321 (inherited from RFC 2821 and RFC 821).
2) A server can only accept or reject a message for all recipients.
Theoretically a global rule shared across all recipients could be implemented in a way that rejects at the transaction level. But because you also have to implementing lazy, per-user filtering (or more importantly, per-user whitelisting), in practice the architectural and engineering focus is on filtering after accepting the message and closing the transaction.
Without the ability to provide immediate and accurate failure responses upstream, filtering issues are left to metastasize and fester. (Because of spamming and spoofing mitigations bounces are next to worthless in terms of proper integration into the end-to-end software stack.)
One of two things needs to happen with SMTP.
1) RFCs drop the requirement for multiple recipients per transaction.
2) An extension mechanism is added that permits per-recipient transaction responses.
I think the only practical option is #1. Both cases require some amount of infrastructure patching, but for #1 it's extremely minimal and in most cases none at all. #2 is infeasible, at least without #1 preceding it (i.e. once you can no longer rely on batch sending as part of the basic SMTP command set, you'll have to support the extension.)
Once that's accomplished vendors who don't implement inline filtering can be rightly criticized and shamed. As inline rejections become more prevalent, mail being effectively black-holed will become a thing of the past and we'll finally begin to see proper back-pressure (literal and figurative) return to the e-mail network. There'll be more pressure on large vendors like GMail to improve their systems and policies once the fog of complexity and plausible deniability lifts.
unrelated:
i was going through a pile of CDs in my archive this last week, and i noticed an earthlink email address on one of my indie electronic CDs, and i still lament that i can't identify some of my old CDs because there's no discernible artist/album names on the CDs themselves, and neither I.. nor anyone I know locally.. own a CD drive.
I run a mail monitoring service [0], and we hear this complaint every now and then.
A couple of things that regularly seems to trigger false positives in spam algorithms:
- no or misconfigured SPF and/or DKIM
- no or misconfigured reverse-DNS
- automatically included footer texts (confidentiality, copyright, safe a tree don't print, etc)
- regular automatic replies from the domain (such as out of office notifications)
- the use of embedded images (logos, human signatures, etc)
We sometimes joke that these triggers were built in by the algorithm developers as a means of punishing those who litter their email with pointless texts and images.
You can satisfy all of this, have an IP with an absolutely spotless reputation that you have personally controlled for years that has never sent spam, and still be arbitrarily shitlisted with zero recourse whatsoever.
IP addresses are really insignificant in spam detection because spammers can easily switch to a new IP address.
Like with Google's search algorithm, the spam detection algorithms are secret and evolve constantly. So I'm not claiming I know how the algorithm works, but it wouldn't surprise me at all if IP addresses aren't even used anymore in Google's spam detection.
So no, you are not 'shitlisted' as you called it. There are no lists, there are only algorithms.
I know it's going to be an unpopular opinion in this thread, but my opinion is that email is one of the things that most businesses shouldn't run themselves.
- Email is hard to get right. There are many subtle configuration mistakes you can make, and it's pretty hard to detect those mistakes (hence one of the reasons why I started Mailhardener)
- Your users will have high expectations. For email to be used effectively users expect email to work on many devices, have a webmail client, calendar integration, a good search function, good spam detection, infinite backups, ability to have huge attachments, etc, etc.
- Your spam filter is never going to be as effective as that of the big email services. They have ML based spam filters that are constantly evolving. Even if you were to have access to their ML models, you wouldn't have enough data to effectively train that model.
- Hosted email solutions are almost always more cost effective, regardless of the size of organisations. Running and maintaining an email service is labour intensive.
Of course, there are also many legit use cases where you want to run your own email server. But for >99% of organisations, it just won't make any sense to run their own email.
> - Email is hard to get right. There are many subtle configuration mistakes you can make, and it's pretty hard to detect those mistakes (hence one of the reasons why I started Mailhardener)
It's not that hard if I've done it right. ;-)
Jokes aside, yes, it's a bit overwhelming at the beginning, but I don't find it that cumbersome to keep it running. That is, it wouldn't be if Google was not actively being malicious in this regard.
> - Your users will have high expectations. For email to be used effectively users expect email to work on many devices, have a webmail client, calendar integration, a good search function, good spam detection, infinite backups, ability to have huge attachments, etc, etc.
From my experience, these aren't that hard to set up.
If you're a corporation, your users are your employees. This is a bit different than if you were an email hosting company where users were paying for your product. In the former case, it is your responsibility to handle this well to serve the company's needs. It either serves you well or it does not.
> - Your spam filter is never going to be as effective as that of the big email services. They have ML based spam filters that are constantly evolving. Even if you were to have access to their ML models, you wouldn't have enough data to effectively train that model.
I'm not convinced by this argument. As others have stated, I've also had very bad experiences with Gmail's spam filtering. It has a very high false positive rate which often makes email go unnoticed until it's too late. I have a basic SpamAssassin setup for my self-hosted email and it works marvelously. It very rarely lets spam through and I haven't had a single false positive yet.
> But for >99% of organisations, it just won't make any sense to run their own email.
Perhaps, but the solution is decidedly not going to the monopolist. Hiring an email hosting company instead of doing it yourself is reasonable, though.
Same here. I love email for a variety of reasons, one of them for being an open standard and protocol.
So, naturally, as many of you, I went the mail/postfix, DKIM, SPF, etc way. And all is fine until you start receiving random hard bounces with no real debugable answer for Google.
It got me deeply sad and questioning my decisions: since you can’t really ignore Gmail, email isn’t in practice “open” anymore. So I might as well sign up for Facebook, WhatsApp and the likes. It’s been years and I haven’t yet, but it’s getting harder and harder.
The real issue here is that if you tell some gmail user to contact their e-mail provider to resolve delivery issues on their side, like this, they'll just stare at you, not knowing what you're even talking about.
It's like there's an assumption that gmail is perfect, and the problem is with the sender. Even if that was true, a normal mail hosting company would at least tell its customer why the mail is not being delivered, so that the customer can tell the sender what to fix.
The gmail recipient is never exposed to this side of google. So they don't know what a nightmare comapny it is to communicate with.
Why should everyone and their dog be solving gmail users's problems with receiving messages? It's such a demented system. It should be the other way round. Recipients, via their provider should be solving their issues with spam filtering and blocking.
If the gmail user would be blocked by my mail server, I would not tell them to go guess what's wrong, fix gmail, and to have fun. It should not be acceptable the other way round either.
I'm in the same situation as many people here. Started as a small personal email projects many years ago just to learn. I really enjoyed it, reading all the specs and making it always better year after year. Dmarc felt like a total achievement.
I started to host emails to many friends, small businesses and even a SaaS I developed. The subscription needs an email validation and I'm aware that the activation email ends up in the Spam folder for the new customers using Google emails. This activation email has everything from dkim, spf, dmarc, to unsubscribe link, full physical address of the business, etc and still I can't hit a good enough score.
I was thinking to start using Google service to send the activation link and hosting my personal domains, but seeing that I am not alone, I will continue to improve my little email projects.
Thanks all for cheering me up on this. I'm sure we can come up with a solution and I would be happy to help. When do we start?
I have logwatch logs that I email to my gmail address via Google's own SMTP servers using my own Google credentials for authentication.
Hilariously, Google will flag these emails sent to myself using my own credentials and their infrastructure as spam. I have no faith in them ever getting this right.
Thanks for the link. I think the tool you linked specifically checks whether DKIM is set to allow Google's servers to send mail on the behalf of your domain.
These checkers for example don't find any errors with the DNS record:
One thing that this second one does find is the lack of a version number in the record. I see that the RFC [1] got updated since the last time I checked and having that is now recommended instead of optional. I'll add that. Thanks!
EDIT: after adding version record, Google's tool now shows green checkmark for DKIM as well.
In any case, it's impossible to test a DKIM setup just by looking at the DNS records. The true test is to verify the actual signature in the message. Last time I checked, the mail on the receiving end (Gmail), did have "dkim=pass" in the Authentication-Results headers.
Not set up to send email through google. Unless they are using gmail as an smtp relay (which they aren't), that shouldn't be a problem - otherwise he would be giving google authorisation to send emails on his behalf.
Recently Google has bounced replies to emails from a gmail.com address to us, with a message about it being "suspected unsolicted email" (for a reply???). That's just egregious (we're on a dedicated server at a reputable supplier, same IP for years, proper email processess, not on any blacklist and only low-volume usage - certainly no spam).
Yup, I literally lost jobs because of responses to enquiries hitting GMail's spam trap, before finally giving up and moving from self-hosting to Fastmail.
This also mirrors my experience, as I host my own email and it's solely for me. At one point, I couldn't successfully send any email to gmail, but then it changed for some reason and simply gets sent to spam folders instead. Perhaps it's entirely related to some people emailing me first that it gets through at all.
I don't send much mail to gmail, though. Sans that, my only issue has been a mail server that uses Reverse DNS, which I don't have set up, and entirely ignores my email without it.
I suppose I can understand this if some people get a great deal of spam, but requiring so much of this on an unencrypted message seems more like useless reassurances than anything. I'm not criticizing email for being unencrypted, but this seems more like another hoop to jump through than anything.
Also of note, almost all of the spam I receive is from gmail addresses and I wouldn't be surprised if the invalid addresses that send messages demanding bitcoin are also from gmail, but with fake From fields.
I'll just chip in with everyone else who is also self-hosting:
I'm in a similar boat; been running email servers since before GMail existed. My personal one I've been running out of a home server closet since 2001. I've also done everything I can to guarantee I'm not running an open relay and not sending email unsolicited. Have been mostly lucky so far, but occasionally I will have people on mailing lists I manage (people I have met IRL and put them on the list to organize group meetings IRL) not get email. Used to be other stupid mail providers (AOL comes to mind), but these days it appears to be Google, sometimes.
I've had this domain nearly twenty years and run email on it for that same amount of time. I'm not going to "just switch", especially to a privacy invading ad-spewing "alternative" that doesn't give me as much control. Fix your damn servers, Google.
Google seems to have entered terminal senility. In my circle of friends googles search engine is pretty much useless, they have killed all the useful applications we used and now this.
I can't help but think that we are seeing the google transition to late 90s microsoft now.
I mean that artists are unironically using bing as a better alternative for image searches, academics have gone back to tracking citations + libgen/scihub to find information and programmers are using duck duck go to debug their code.
This is a huge problem since these people used to be the core of googles search user base.
I have run a private mailserver for over 20 years, and I have the same problem. There's no telling what Google will let through. I have a second account at a medium-sized email provider, basically paying them for access to the email deliverability cartel.
I fear this will lead to the same problem: If it’s interesting enough for Google, they will offer a free service, gobble up a large number of users, at which point they can hold the new protocol hostage, add their own extensions, and everyone else must follow.
It’s called XMPP, and Google ran a public gateway federating for Google Talk. Basically nobody used it, and what little use it did see was mostly third parties sending spam to Google users over the gateway, so they shut it down.
Centralized systems seem to give users what they want. This is why iMessage, FB Messenger, WhatsApp, Snapchat, and Instagram are huge, and email is dying (and XMPP failed to get any real traction in the first place).
There's nothing wrong with the basic principle of email (other than its lack of built-in security, arguably).
The problem is that certain large organisations now claim to offer an email service, but don't actually make it work properly. It might have been better for the rest of the community to block those services until they did work properly to force everyone to play by the same rules, but unfortunately we have reached a point where some of them are too big for that to be a practical solution.
The obsession with fighting spam and malware means some mail services are now far too ready to accept false positives where they block legitimate mail, in order to reduce the risk of false negatives where they let illegitimate messages through. This seems strange, because often the false positive will be more damaging. In the cases where it is not, such as messages being sent with known malware attached, the receiving mail service could still deliver a short replacement message to the intended recipient instead, notifying them that something was blocked because of malware and providing essential details like the sender and subject line, so the recipient could take further action rather than miss something important.
The real issue is a preexisting "dancing hippo" one where what people want doesn't line up with other higher minded ideals like security or freedom that others presume should dominate. (Whether said ideals are right is irrelevant for better or worse - the phenomenon can happen for good or ill.)
Google cares about spam and malware because the users do. If nobody gave a crap about spam they received or did their own filtering on their "raw" local copies then they wouldn't bother with filtering beyond infastructural reasons.
Spam and malware are the ultimate enemy of openness as it gives reasons to shift to centralization - analogous to how raiders helped lead to the rise of feudalism although the stakes are obvious vastly different.
I think most people would also care about not receiving important messages that someone was trying to send them, but since they often don't know what they're missing in this scenario, they don't tend to object or make decisions accordingly.
I have a similar concern about a lot of privacy and data processing issues. My experience has been that many people aren't OK with what happens if they are told about it, but in most cases either they didn't previously know or perhaps they suspected but didn't think there was anything they could usefully do about it short of becoming some sort of digital hermit.
That is not the only reason people eschew federation. Think about all the federated protocols in common use today: email, SMS, phone. All of them are overrun by spam. 90% of the time when my phone rings, it's a robocall. That, my friend, is federation in action. It ain't great.
Add another perspective to this. Once I updated my phone, and google would not accept new google Authenticator codes after set up. Locked out of 15 years of email.
They specifically provide 10 one use codes for this type of situation that should be saved somewhere. Also, email should be backed up locally if it’s important, just like anything else.
I’m not sure what else people want from a secure email service you don’t have to pay for. Also, any work around the 2FA by a human simply means less security for everyone.
I had a fright last time I changed phones. After restoring my backup onto the new phone, I was about to wipe the old one; I did a final pass through my apps just in case... and saw that Google Authenticator configuration did not carry over to the new phone (which makes sens from a security perspective but is a PITA to manage). I had to re-enroll my device with all the services on which I use MFA.
After this episode I made damn sure I had recovery codes stored in a safe place.
When a family member had a phone stolen not long ago, the only major functionality on it that they couldn't quickly and securely disable was the Google stuff.
The procedures for doing so seemed to be unnecessarily complicated and difficult to find when starting, ironically, from a Google search on another device.
Worse, the security policies seemed to be fundamentally flawed, because they kept insisting on some form of authentication based on a trusted device when the purpose of the transaction was to notify them that the trusted device had been stolen.
There has been an unhealthy trend recently of assuming that everyone has a mobile phone and that communications to that phone/number are a good method of authentication, without adequate thought to what happens if the physical device and/or the associated phone number are compromised, or to whether protocols like SMS are really suitable for this sort of application. And some of the really important things, like banks and government services and email providers (which are in practice a gateway to everything else you do online) are often among the worst offenders. I don't know what to do about this, but certainly raising awareness of this kind of problem would be a good start.
> Worse, the security policies seemed to be fundamentally flawed, because they kept insisting on some form of authentication based on a trusted device when the purpose of the transaction was to notify them that the trusted device had been stolen.
Are you suggesting any random person without any authentication proof to be able to just sign people out of their devices ? That would be a broken security.
I'm suggesting that a security policy should actually be practical. There are any number of viable ways to handle this. Requiring someone to possess the device they are reporting stolen is not one of them.
You can also have more than one device set up for TOTP - phone, previous phone or tablet, desktop using WinAuth or similar. Authy and the password managers will also track those seed values, though it's best to store them separately from your actual password storage.
Another thing that will not migrate phone to phone is Signal conversations if you're inclined to keep those.
Yes, I lost my Signal conversation history as well and had to be re-added to all group conversations (another PITA).
How do you store seed values in password managers? More specifically: how do you export them from Coogle Authenticator? (I’ve not found that option). And how do you import them again?
ISTR at least one service (AWS or Google) asking for the first 2 codes after scanning the QR code, probably to sync. So I always assumed simply re-scanning the QR code wouldn’t be enough but maybe it is this simple.
The underlying technology behind (almost?) all of these is TOTP (https://en.wikipedia.org/wiki/Time-based_One-time_Password_a...) which pretty much just depends on both systems having clocks that are reasonably close to synced. The initial value is basically a random number generator seed, and given the seed and a number of iterations (based on the time differential from a set starting point) calculating a code is fairly simple.
There would be problems on fully-isolated systems experiencing clock drift, but on any modern Internet-connected system using NTP or on any cell phone with time synced to the network it shouldn't be a factor. The most likely problem scenario is probably a corporate network using only an internal time source that drifts.
Doing a single code as validation only makes sense to catch transcription errors since in case of problems someone could end up locked out of an account.
For the future, if you use yubikeys, yubico authenticator is a drop in replacement for google-authenticator and you can save your google-auth setup on multiple yubikeys. So, even if you lose one, you still have it on another. Then it is independent of the phone. Also the yubikeys themselves can be used as 2FA for google accounts. However, sadly, you can't set them up on firefox the last time I tried.
Happened to me too. The thing that triggered lockout was attempt to login from different country (I've changed my VPN provider). I didn't give Google my phone number so the only option left for recovery was tricky questions like "what city you've been logging in from" which I failed to answer.
Add me to the victim list. I've been running my own mail server for 13 years and have to tell everyone I'm intending to email to check their spam as that's probably where my mail will be. I send a handful of mails a week probably. Almost always goes to the bin if it goes to gmail.
Lest anyone think Gmail is less susceptible to problems because of the sheer volume of mail they handle (so any problem gets a lot of 'eyes' on it), they had an amusing and long-standing bug that meant any email containing a link to any domain name starting with "0x" automatically went to spam.
You could literally have two long standing, legitimately used accounts send an email to each other containing a link to a URL like http://0xANYTHINGHERE.com/ and it would be insta-spammed. I suspect it was a hard coded rule to avoid people using "long IP" URLs to circumvent other filters.. except there are lots of legitimate 0x domains that aren't long IPs.
It was fixed sometime in the past year but I got a lot of use out of it in talks I've given about email deliverability over the years.
Without these customisations that forum would be overrun with all sorts of spam.
However, these customisations only stop spam postings but can't stop actual registrations.
Based on the users that I see who are registering I see a great majority of these spammers love using Gmail accounts.
So while it is good that Google Gmail is trying to fix these spam issues, from where I stand Gmail users seem to be a big part of the spamming problem.
Spammers love Gmail only because they can easily create spamming e-mail accounts.
Google? Any large organization. I help a small non-profit run their site in my free time, and Google's actually been the nicest about our oldschool forwarders and Mailman lists. Despite being on no public blacklists, AT&T domains drop us at the border, with the only appeal process being via email, and I've never heard back. AOL and Yahoo? Flip a coin.
Of course, email forwarding turns out to suck, but we're just going to suck it up and move to G Suite for organizational email addresses and let folks forward from there. E-lists, OTOH, I haven't found a good integration to automate membership in the organization vs. G-Suite; perhaps it's time to just move to a forum.
A while back I'd explored trying to own my data, especially with email, and found that the efforts involved in hosting your own email server were tantamount to a full-time job.
The amount of fighting you have to do to stay on everyone's whitelists is absurd.
There's a good middlespace here, where you pay someone for email service who isn't an adtech company. Sure, you don't host it yourself, but someone who's not selling you is and you're their actual customer. (I pay FastMail, personally.)
I think you are still being sold even if you pay. Nothing really stops that. Even if they give away or even pay the bigger providers they are ultimately selling a white list. Since if they just accepted whoever gave them money with zero policing of abuse, even a warning and take down system for spam then nobody would accept their whitelist as it eventually becomes a blacklist of "the one spammers use" and their business model essentially becomes a scam.
There's some setup cost, for sure. But there need not be that much of an ongoing maintenance (for me, it's just pacman -Syu + restart of some services, perhaps 10mins a month). You can also gain some interesting features and insight as a result.
Reading this thread made me realize the magnitude of this problem. It seems very sensible that small email server administrators should unite into a single effort in order to better publicize this issue, raise awareness and shame Google into compliance.
Google seems to be suffocating the internet bit by bit on all fronts and it needs to be stopped.
Imagine a phone carrier dropping your calls before the phone even rang on the other end. This is how I feel most of the time when I email people.
Every new recipient I email, if I don't hear from them within 2 days, I have to contact them out of band to ask them to check their spam folder. The problem is usually Gmail's heavy filtering.
Can confirm, my server is also not accepted by google for years now when mailing someone with a business account. I'm not going to bother fixing Google's problems and support the behaviour, though, I'll just contact them through other means and let them know their email provider is blocking mail addressed to them.
At work we use sendgrid because of this. Have to trust a centralised third party to send out api keys. It's frustrating.
>Can confirm, my server is also not accepted by google for years now when mailing someone with a business account.
Hey, I remember you were one of the replies[1] in my previous thread that said I was exaggerating the "send emails" issue. Maybe your difficulty with Gmail's mystery filtering algorithm will warn others that depending on personal email servers for reliable outgoing email is a non-trivial endeavor.
Perhaps personal email servers are not impossible to set up but they're also not as easy to debug as some make it out to be.
- My comment in this thread was mainly about gmail for businesses, hence the contrast between "I'm not going to fix their problems and just tell the recipient their setup is broken" and the previous "it's doable". Other mail providers are usually no problem at all, and personal Google accounts are also better.
- For email to personal gmail accounts it seemed to be enough to send a few mails and get a good reputation. Since then (and since my previous comment), I think I've started ending up in spamboxes again, but the number of messages I send to Google is very low. It's not outright rejected, though, since I would notice that.
- I still think this part of my previous message is very true: "I don't think we should dissuade people from doing it, especially if the fact that more people doing it means that it'll be easier next time because it'll be slightly more common. Many of us are in tech and the field is a small subset of the population. Even if it's a small amount of servers setup by us, that could make a noticeable on those working at bigcorps who write the hostile receivers."
> Maybe your difficulty with Gmail's mystery filtering algorithm will warn others that depending on personal email servers for reliable outgoing email is a non-trivial endeavor.
I hope not. For what it's worth, I'm still at it and have no plans to stop hosting my own mail! I'm very happy not to share my data with a third party (the ideological part of it), but on a practical level, I also don't have a spam problem and I can be sure that email arrives. No message has ever not reached me due to spam filtering. It's also much faster than, say, Runbox, which we use at work and takes at least 30 seconds for any sign up email to arrive (too short to get coffee, too long not to be slightly annoyed). As another data point, my girlfriend switched from 1und1 to my mail server because it has some conveniences, without most of the downsides (since I manage it all for her). She hasn't had delivery issues so far.
As a related tidbit, I've also recently started blocking Google from my website. They're being a dick on the internet (not to me, it's more of a solidarity move, see https://lucb1e.com/!130) and like with email, I'm not dependent on Google so I am again in the rare position to say "no, I'm not going to play your game" to Google. I'd rather encourage others to do the same than to give up and treat both email and search results as a walled garden.
>I hope not. For what it's worth, I'm still at it and have no plans to stop hosting my own mail!
You misunderstand the point I'm trying to emphasize. I'm not trying to dissuade you from running your own email. There are real benefits to controlling your own email server and it's great you've configured something that works for you.
I'm saying I disagree with how some proponents communicate the drawbacks. They discount (or are blissfully unaware of) the real difficulties of running a personal email server for critical outgoing emails.
In the previous thread and this one, there are several of us with decades of email admin experience saying it is tricky to debug "sender reputation" while the opposing side insists Gmail/MSOutlook spamholes are easy to fix and not a big deal. Therefore, one of the sides misunderstands the realities. With these conflicting accounts, readers contemplating running their own email server will have to decide who is more accurately reporting the state of the email ecosystem for personal servers.
The author (Tomaž Šolc) of the article discussed in this thread had ~15 years experience administering his own email servers and tried to do all the "correct" things as an upstanding citizen of the email ecosystem and yet his emails still suddenly got rejected by Gmail. Readers will have to conclude if he got bit by forces out of his control -- or
-- he's incompetent and doesn't know what he's doing. (From my point of view, the author wasn't incompetent and his frustrations with Gmail's filter is a common reality. Maybe this time, he can update DMARC to fix Gmail rejections... for now. But eventually, a new Google AI spam algorithm will mysteriously reject his server's emails again.)
Fair points. I don't have much to add to your comment, so I'll just say that I was nodding along with every paragraph. Thanks for clarifying what you meant to emphasize, I think I understand now!
In my opinion the Gmail web interface should expose a way to generate something that is similar to a password, whose delivery is then controlled via the web portal.
This is similar to the +whatevertag trick that gmail pioneered for tagging, however that can be removed by malicious parties (spammers) via a simple regex. So Google have almost all of the infrastructure but should just add a bit more to get the rest of the way there.
What I mean in specific :
1. I want to sign up to and receive your newsletter (you[re Ted) but I don't trust you yet. so I should navigate to gmail.com, click something like "generate another inbox", leave it set it to "For now deliver this mail to my inbox", add the description "for Ted's Possibly Spammy Newsletter", and then click "generate". It should give me inbox3943578423@gmail.com - similar to a phone number but a bit longer and personalized to one recipient - and then I should give that to the recipient to use, in this case the possibly spammy newsletter. It should always be delivered to my inbox, as I've set. Once one of the spammers sells my email address (for example I start getting advance payment scams) I'll be able to disable further spam from there by sending it to the trash but also know that Ted's newsletter is the one that got compromised or sold it. You can do this today by going through the steps of registering a new gmail address and turning on forwarding, but it takes like 10 minutes to do so. it should be like 10 seconds.
This should be possible because people always have easy access to the gmail web interface. There's no reason it can't be a bit more like a social network where you confirm it from the web interface as well.
I've also run a mail server for a long time. Yahoo and Hotmail are probably the most annoying, Yahoo because you can't do anything about it and Hotmail for blocking entire IP-ranges.
I don't blame them though, preventing spam is very hard.
Spammers always have spf,dkim,dmarc,reverse dns,signing, etc meanwhile many legitimate senders has none. I'ts a hard problem to solve.
I've tried things like increasing the response time so the sender have to sit and wait, or used spam lists, but there are too many false positives. One easy solution is to make it illegal to send spam. Since my country made sending unsolicited e-mail an offense, spam decreased a lot. The second problem is hacked servers. Unless you are running a mail server, always block port 25 in the firewall. So at least the hackers wont be able to send spam on your behalf. Many ISP's already block port 25.
I had my personal domain hosted with Google years ago. One of the reasons I moved away is that emails that I was sending to other Google Apps domains were bouncing as spam when the destination address was an alias. The bounce came from Google Groups, which is what Google used for handling aliases within Apps domains.
That particular example was sent via SMTP, but I had the same problem when sending via the Gmail web interface, and it occurred sending to at least three different Google Apps domains.
After I moved my domain off Google Apps (I switched to Fastmail for a variety of reasons, but that issue was the kicker), I was able to send to those same addresses without issue. In fairness to Google, I was on the Google Apps free tier at the time, so there was nowhere to go for support.
I've always thought that, I also used to operate my own email server and I've encountered the same problem with other email providers, always the big ones, yahoo (there was a time when yahoo was big in email, at least here), microsoft and now yes, google. What is even worse is that sometimes google just drops the email messages, replies with 250 ok, but the email never gets delivered to inbox or even spam. Definetly the big 4 or 5 or whatever are taking over on all aspects of our life. Actually if I think better about it, they have already done so, there was an article here on hn about blocking google, amazon and ms network blocks, nothing really works as expected without them.
Regarding the issue with mailing lists for open source projects this isn't just an issue regarding Gmail. DKIM and DMARC cause no end of problems for mailing lists and it effects mail delivery to all major providers.
Mailman, which most open source projects use for mailing lists, have developed work arounds to address some of the issues. Unfortunately my experience is that many projects run older versions that don't have these work arounds or if running newer versions they have not been enabled. Most likely because no one has revisited the configuration since initial deployment on an older version. After all they didn't start the project to spend their time being mailing list admins.
Am I the only one here experiencing that GMail does reasonably good job in filtering out the spam, while I have to visit the Spam folder approximately once a week, just to randomly correct a mail or 2 a month? (not a Google employee)
That’s not the issue, though. The issue is that Gmail are too zealous in blocking and wind up blocking legitimate email with no way for the sender to debug what happened. From your perspective, nothing happened because how do you prove you never got an email you never knew was sent to you?
The people having problems here are non-Gmail senders sending email to Gmail recipients.
Every time I look in my spam folder, there's dozens of legitimate emails in there, and I get this dreadful feeling that over the years, thousands of legitimate mails went over the 30-days-in-spam threshold, down the memory hole forever.
If ProtonMail would standardize a client protocol that other MUAs implement (not their "bridge" kludge), I'd start encouraging people to move there.
I'm not interested in ProtonMail's encryption (and it's potentially a liability, attracting aggressive state action). I'm mainly interested in their apparent respect for the privacy of users' private communications. And also hoping that ProtonMail has a bit more reliable delivery than GMail.
In any case, rising competition lifts all performance boats, or something like that.
I've also hosted my own email for several years now, on several domains. The one thing I have that does not appear to be listed is a DMARC policy. As the comments on the blog post suggest, configuring DMARC works. I have no problem delivering to either Gmail or Outlook.
I don't think it's unreasonable to be strict regarding DMARC delivery. My MTA has a fairly strict SPF configuration - any email with an invalid spf result is rejected. This can come about because a legitimate company has misconfigured their spf records (happened twice in all the years I have hosted, discussions via postmaster@ helped them configure their dns correctly), but 99.999% of the time it is a spammer. What is worse is that rejecting email for domains without any SPF records can still result in valid email being lost, in 2019.
In this specific case, I don't think Google are "being evil". They're trying to reduce spam in the email ecosystem and they're doing it by using standards they themselves adhere to (Gmail send me reports of dmarc statistics each day google domains receive email from my box).
On the other hand, I do of course support either self hosting, or using another provider so as to ensure we do not end up with a Gmail monopoly. If I did not self host, I would find another provider like (but may not) Fastmail, Posteo etc (I would have to seriously review the options, which I haven't done).
I'm glad I'm not the only one seeing these problems. I send mail from a personal server, and recently discovered that my messages to people @gmail were frequently sent to spam or simply dropped. I briefly thought friends were ghosting me until I figured it out. I set up SPF, DKIM, and DMARC, but it seems like the only way to get reliable email these days is to use Gmail, Hotmail, or Yahoo (at least for a little while...).
As far as I can tell, Gmail isn’t actually “real email” anymore, and you shouldn’t use it for anything important. It’s verging in being a walled garden.
Does AI compares whom data to define which message is important or No?
It doesn't track any specific way on evolution of mentality when it goes to define data correlated to humans usage. Like others products on market. Therefore, if you apply this thought to AI I guess they just made with basics conglomerated informations gathering. That CAN´T move a barrier when satisfying humans usage. It is just how you confront which message is correct to say is relevant or no. AI can be freely a way companies uses their needs on market influence because they're , on a very simple association, doing by themselves, then people get a notion of arbitrariety on those machines that create discussion to the whole universe you see about robot revolution etc, but not the case. I guess they try to hard to make then appear as human to be soon accepted by costumers, its just the way it is, it is not the time to require maximum levels of perfection on how easy they're to addapt as a user
Gmail can be pretty aggressive with spam filtering, but I'm very happy with the bias toward flagging things as spam. Spam was such an infuriating problem, and lots of marketers would happily push the line if they didn't live in terror of being blacklisted by Google. If only Google could the legacy telephone networks like they run Gmail.
Flagging as spam is different. The OP is talking about rejecting it before it even gets to your inbox.
If gmail is improperly flagging a message I get as spam, I can create a filter to never send it to spam. If they're rejecting it like the OP talks about, if I want to stay with gmail I have no way to get that email.
I get a fair amount of spam from Gmail accounts. abuse@gmail.com seems defunct to me because I never got an answer to my reports of unsolicited mail. I absolutely don't want them to run the phone network I'm using.
The black box is more mysterious to me. I had a client for whom I managed a G-Suite account with a custom domain name. The domain was not on a blacklist, and things generally worked well.
One day a colleague and I discovered that he had not received some of my emails (intra-domain - me@example.com to him@example.com).
This is all within the confines of Google. Google had flagged some messages as spam, and by what determination I could not fathom. The content seemed perfectly typical.
I have had really pleasant experiences with G-Suite human support, at least in terms of the quality of interaction. But they could not answer why some intra-domain emails were being flagged as spam. I have suspicions that it would take a whole team of G engineers to maybe identify what bit of logic in their systems (incorrectly) marked some of the emails as spam.
It seems the beast (automation) is just almost not under their control anymore.
Isn't charging people to use other carriers one of the big things that got Bell broken up? Google seems to tread in a of very dangerous waters. I wonder if they're just not aware of how close they are to being shutdown entirely, simply because so far they've always withstanded most legal challenges.
There may be a workaround here: configure your own mailserver to relay outgoing messages through a service with a known good reputation. For example relaying via sparkpost or mandrill should be trivial to do. You might be able to do that under a free developer account if you don't send a lot of email.
#1 - Does anyone send test emails and measure delivery rates? As in send yourself a bunch of emails and see what happens.
USPS and its major customers and vendors do this with physical mail. They measure stuff like UAA (undeliverable as addressed). FWIW, their Inspector General estimates 4.3% of mail was UAA in 2013. Report Number: MS-AR-14-006 https://www.uspsoig.gov/document/undeliverable-addressed-mai...
#2 - What is the responsibility, liability for email relays to treat everyone equally? For comparison, a US retailer has to accept US currency, but can (sometimes) turn away problematic clients. Is there anything like that for electronic exchanges, transactions?
If people stopped using G in such high percentage, then they would most likely need to adapt back to a more open internet, instead of trying to wall it off around their services.
Reminds me of jabber.ccc.de that stopped providing new accounts because they felt they were ruining a federated system.
This is the reason I sadly don't recommend to customers that they run their own email servers, and instead outsource it to GSuite or O365. Being on the hook when emails aren't delivered for one or more of a dozen reasons that you can't control is no fun at all.
"Only when something like this happens you realize just how impossible it is to talk to someone on the modern internet without having Google somewhere in the middle." This says it all and is likely become a strategic imperative for Google along the way.
Well of course - don't trust the client is a rule for a reason. I get the spirit and how it can suck but it seems to be for a reason. Anything which blindly works for "you" can be abused by other "you"s like spammers and other bad actors.
I think the catch with AI and ML there is "reason" but it's just math or something that happens because.
If it is "reasonable" (trying to find a definition here) can only be determined by humans.... provided they care or understand it.
I always quote this one but I really like it:
“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”
― Frank Herbert, Dune
I would argue that it is clear that to some extent, Google does not care about the end results.
Like the author, I've been running my own mailserver for over a decade and am very conscientious about ensuring that no attackers use it as a spam relay.
While the vast majority of the people I exchange email with don't use GMail at all, so it can take a while before I notice any issues with it, I did happen to notice that GMail was rejecting my outgoing email a couple of weeks ago.
This week, I finally got around to trying to address the problem (it's not high priority because having GMail reject my emails isn't really a huge deal).
...and I found that it is working again without my changing anything. Weirdness abounds.
What boggles the mind is that they've a very string disincentive to fix it: I had to move away from my privately run email to Gmail for that very reason, and I'd gather I'm far from being alone.
Better that what I've been seeing, which is Google just sending emails sent from my private domain to spam, with no bounce back or notification of any kind. This is despite the fact that I send the emails FROM gmail using their 'send as' feature. Nor does it seem to matter if I'm writing to a contact whom I've exchanged many emails with in both directions for years. If I send from my @gmail address they always go through. Send the same content from my domain and it's a crapshoot.
Edit: And of course, I do have DKIM and SPF configured.
I tried to set up a Discourse discussion forum for our neighborhood this year, and got stuck on setting up an email server that wouldn’t get blocked automatically by all major providers. Email is not a healthy ecosystem. You have to pay to send from one of the big trusted providers or else you get blocked just because your email ip address is untrusted by default. Yes fighting spam is important, but it’s shocking how much email providers rely on simple ip filters and trust levels rather than AI analysis of email content.
This article didn't mention whether or not their servers are attempting delivery using TLS/StartTLS, which is a good thing to check. I think Google penalizes email delivered insecurely.
I disagree. There are reasons to switch off gmail. Not just Google eating mail but also for privacy reasons. Google knows all about your banking, eCommerce orders, your media subscriptions, health issues and many other dependencies.
A good alternative is protonmail. It is private, has a mobile app, is a free but you can also pay to support the service. I also consider protonmail much more secure than gmail.
Please list a few. I need 3 personal inboxes, 2 home business inboxes (bills, etc), and the ability to make new inboxes for stupid startup prototypes you buy a domain for, hack on for a few days with dreams of huge sales, and then forget about for a year until the domain expires. You know, normal hacker stuff.
I'm quite happy with the ease of setting up and running a Mail-in-a-Box [1] instance. After seeing this article I did an experiment and while my hosted mail sent to a @gmail.com address did not bounce it went straight to the Spam folder. At least I can add other users to my email and enjoy private conversations.
If I recall my time at a main-sleaze email sender, some ISPs will send SMTP retry requests on random incoming email to see if there is a functioning SMTP server on the other side of the originator's email. Often times, spammers won't care about the email that's being sent, so won't respond, or in cases where they are hiding the origin, won't be able to respond.
I don't know that this is purposeful in all cases. I've had issues on a mail host where sending an email to a different box on the same domain would reject the message as spam, refusing to even deliver it. With respect to google, however, the fact that they offer a service of their own that conveniently can avoid this issue is more than a little shady.
Its like the deep learning has identified a feature it can use to determine spam: is this from the top 10,000 domains? No? It's spam.
Spam from small domains might be pretty high as a category, but of course we don't want statistical judgements about categories to outweigh the merits of the individual. Maybe Google's algorithms have been watching too much Fox News.
I often see and hear about this issue a lot and I wonder how other email companies cracked it.
I use FastMail for instance and never have this issue, but I know so many people who gave up on running their own mail servers at the small enterprise level because of stuff like this I often wonder how FastMail does not have these issues but others do. Is it a headers thing I wonder?
Fastmail is big compared to the operator in this article. Fastmail is probably big enough that some Gmail engineers will know how Fastmail's SMTP chatter looks. If you don't have volume you don't get that.
I use my own SMTP server for receiving, but for sending I use the ISP's server (I think I have seem to have read somewhere that not only Google but some other services as well, will not accept mail from dynamic IP addresses). I have not had problems that I know of, so far. Please tell me of whatever mistake I may have made of such thing.
Gmail is rejecting valid content as spam and it never hits my spam folder. If I was not expecting an email as I was in this case, I would be totally oblivious. Something is up with the spam filter recently. They might have decrease the tolerance for spam and as a result a lot of legitimate traffic is being hit in the cross fire.
That's the price you're paying for a) self hosting and b) relying on decentralized protocols that don't have sensible abuse protections.
People hosting their own servers enabled wide spread abuse due to misconfigurations. Because everyone could do it and because defaults were shit for decades, stuff like open relays were common. People defaulted to the wrong ports. Almost no one bothered to offer STARTTLS/Transport Encryption. Spam would have killed mail by now if it hadn't been for major players like Google, GMX, Hotmail/Outlook/etc.
Back in the day, greylisting was commonly regarded as a best practice, leading to the impression that email is unreliable and prone to latencies.
I'm sorry it's this difficult to host mail by yourself nowadays, but I'm happy to have a spam-free inbox every day and if this is the price for that, I'm sure about 1-2 billion people are willing to pay it.
I'm quite astounded that there have been no updates to mail protocols in the last couple of years to at least mitigate the most common issues, but all I see are band-aids that are complex to setup and horrible to debug in case of issues.
I've had Google blackhole me several times. Each time without any apparent reason. It's impossible to talk to a human being at Google who is able or willing to help. I've had similar problems with MS, but at least I was able to talk to a human being and get the situation resolved.
To me this is more evidence that we definitely need public decentralized platforms that can replace Google and once we have them we need to eliminate the company. Private companies should not be allowed to monopolize platforms and control our lives.
He’s been using the same IP address for his mail server for 15 years. It’s possible his IP is on a subnet range that regularly gets blacklisted by Gmail due to other actors on the subnet sending malicious email/spam to Gmail.
Presumably, Google's smart enough to identify individual IPs in the range that don't send spam. I have a real shitty neigborhood IP wise, having my mail server on a $1 VPS. Public Outlook.com drops all my mail regardless of content for example, citing spammy IP range. Yet gmail somehow learned after a while to pass most of my messages through.
The real problem is lack of provided reasons for blocking, so people waste a lot of time trying to figure it out by guessing and trying random shit.
Does anyone here use encrypted and signed messages on a regular basis?
How does gmail handle that on either sending or receiving ends? Is delivery rate better or worse if it is fully encrypted?
Reminds me of Spamhaus. Are they still around? When I worked for an ISP we had to deal with them regularly. They'd happily hold you hostage over a single e-mail.
yes, they are, and they've been really really good. haven't seen any of the behavior you refer to. ever. now SORBS on the other hand... doesn't really exist anymore (fortunately) but spamhaus has been great.
Can you tell us a little more about your setup? I have a home email server, incoming only because of ISP restrictions, and I would love to make it into a real email server for my family and business.
I wonder if originally this technically constitutes an illegal form of interception of mail in some jurisdictions.
Perhaps there really would be a lot more spam without such filtering, but it points to the actual problem being elsewhere. Perhaps we need some kind of cheap and userriendly (uniform but decentralized) email court system, and fine / ban email accounts that misbehave?
Can someone explain in simple terms why, in 2019, anyone is running their own mail server? Unless you are the sysadmin, server guy/gal, whatever, this just sounds like signing up for a world of hurt doing it yourself.
Contrary to popular opinion, I don't think the technical aspects of operating a mail server are all that hard. Perhaps I'm underestimating the learning curve since I've been doing it for 27 years. (If you're running your own mail server, you are the sysadmin.)
The frustration comes with scenarios such as the one outlined in this blog post, where small mail server operators get bullied even though they are doing everything right. I can completely understand not wanting to operate a mail server due to this situation, or not having interest in leveling-up server administration skills.
For those of us who do have expertise in running mail servers, it's a shame we have to deal with these obstacles.
The biggest obstacle of course is dealing with deliverability problems. As most posters have commented here, getting people to accept your mail, and accepting mail from reputable sources isn't an easy problem to solve. At FastMail, a considerable amount of time is spent ensuring customers don't need to worry about deliverability problems or spammy inboxes.
To me, that's the biggest reason not to spend time running your own infrastructure for mail. Getting people to accept your mail reliably, is much harder than adjusting some postfix configuration.
Because email is one of the few edifices left of the truly open internet, and some of us aren’t ready to concede defeat of what was once a free platform to a corporate oligarchy.
It's not as difficult as you've been lead to believe. But if you have no experience with administering servers, the initial learning curve could be significant.
The upsides, sharing many of the common features of all self-hosting, are things such as pure and total control, data privacy, ultimate flexibility of configuration (e.g., infinite and unfettered aliasing), customization of interface, high-performance, etc.
It's big enough that when someone complains that a message sent wasn't received, the intended recipient will say, "I never have problems with my Gmail account. It must be you." And the sender has to switch to Gmail to reliably communicate with the outside world.
I wish this was just paranoia, but we've seen multiple discussions on HN about Google programs and policies that alter the internet in ways that only benefit Big G. It's like we're heading back to the days when people didn't know the difference between AOL and "the internet."