Both my personal domain and rsync.net are on a distinct subnet, but that subnet is smaller than a /24 and someone on a different subnet has, apparently, behaved badly.
Enter "abusix" ...
One of my engineers had an enlightening webchat with one of their engineers where we were shown the "offending" IP and it was explained that they have no ability to distinguish subnets (and no interest in doing so). So if you're not wasting an entire /24 (we only need ~10 IPs at this location) you're in danger of this misclassification.
We were also informed that our normal, business communications with paying customers should have unsubscribe notices appended to them. Which is to say, you're a paying customer of a service and we send you some kind of alert or critical announcement ... and it should have an unsubscribe link.
Those unsubscribe links should be there, for several reasons.
- The service-based economy means that entities (individuals and businesses) have numerous relationships. For the typical individual the number of password-based accounts crossed the 100 threshold years ago, at a doubling rate of every 2--3 years.
- Responsibilities can be transferred. The person who signed up for your service 5 years ago may no longer be at the company.
- List purging is a Real Thing. A few years back I'd worked for an organisation that had ... numerous relationships ... with individuals and corporations. These received regular email messages. Nominally, requested. Included amongst these was a major Wall Street financial firm whose implosion years earlier hit lead news and headlines worldwide. Despite not existing for years, there remained hundreds if not thousands of addresses being sent email on a regular ongoing basis.
- Mail can be forwarded. It's quite possible that you're sending mail to one address that is is being forwarded, manually or automatically, to others. This raises issues in unsubscribe requests, but might at the least be an opportunity to reach out to your customer to clarify the situation.
- I don't know if revisiting email contact approval on a regular basis (say once every year or two) is yet a recommended practice, but I'd strongly suggest that it be so.
Your hat may be less blisteringly white than you presume.
Everything you've said makes perfect sense - for a contact management function.
We have that. You can change contact info, set owner/technical/emergency contacts, alert thresholds, etc.
But unsubscribe means something totally different:
When I click on unsubscribe I want it to be the end of all communications. Period.
In this case, that makes no sense. Ceasing communications for all purposes implies service cancellation and, in our case, service cancellation implies a human interaction confirming data destruction.
How would we confirm data destruction for your implied cancellation if no further contact is permitted ?
You, and the blacklist operators, have become so jaded by the abuse you've suffered that you've forgotten that legitimate, paid services exist. I'm sorry.
From a comment below from the other side of the equation it sounds like the email in question WAS indeed marketing for a lifetime promotion.
People should have every right to unsubscribe themselves from that, and thus should have some sort of feedback loop attached to the email being sent (to the detriment of your bottom line I fully understand and sympathize with).
If this was indeed a marketing email, then I don't think it matters if it comes from a "legitimate, paid service" - the receiver still should be able to choose if they want to receive sales related emails wouldn't you agree?
"People should have every right to unsubscribe themselves from that, and thus should have some sort of feedback loop attached to the email being sent (to the detriment of your bottom line I fully understand and sympathize with)."
I agree.
We have a flag for such a thing and set that flag when people ask us to. They ask us in a nice email exchange between human beings. We're very responsive to this since they are our paying customers.
That's the big disconnect here: it's inconceivable to many people (including abusix, et. al) that healthy, straightforward interactions like this occur in 2022.
In their mind there are nothing but robots and newsletter subscribers forever locked in an arms race.
Did they request this marketing message through a "nice email exchange between humans"? Or did you automatically sign up this person for marketing then expect them to manually contact you?
You are wayyy to smart not too see the abusive asymmetrical theater of that scheme.
Atleast in my country, to send legitimate marketing e-mails, a user has to be given a choice to opt-out *before* the first e-mail.
So it's not enough to allow them to opt-out with an e-mail exchange or an unsubscribe link, you must allow the user to opt out when you initially gather the contact information.
If that wasn't done, it's illegal (and unethical) to send marketing e-mails, even from a paid service.
Your earlier argument about unsubscribe being effectively an informal termination only works if you properly separate that side from all promotional/marketing/new features/etc material, otherwise it seems you are avoiding the main point and purpose of the unsubscribe button
The flag for marketing etc. should be automated, I understand account-related emails like billing not having "unsubscribe" links but promotions definitely need one. Clicking that would then toggle the flag (and maybe have one in their account panel).
An unsubscribe link doesn't have to immediately cancel all service and communication. It can simply lead to account settings or even to a page explaining how to cancel the account.
I strongly disagree. There is absolutely no need to put an unsubscribe link into a transactional email.
All emails should of course contain enough information to make it clear who the message is from, why the message is being sent, and who it was sent to.
But there is no point in adding unsubscribe links to messages and notifications that are essential to the service.
I mean, what are you going to do if the user accidentally clicks "unsubscribe", and then a payment doesn't go through? Should you just cancel their account without informing them? That's absurd.
I'd be really pissed if eg. my backups were deleted because I accidentally unsubscribed from emails from a cloud service provider.
> There is absolutely no need to put an unsubscribe link into a transactional email.
Agreed. rsync alluded to it below as well.
'unsubscribe'... from what? If I just bought something from service ABC, and I get an email from ABC saying "you just bought foo from us"... what would an 'unsubscribe' even mean? "Do not ever email me about this purchase again?" "Do not ever email me about future purchases?"
What about situations where your email somehow (mistype) gets set up for someone else's account? I have 2-3 people with similar emails to my previous email address that would mistype and I'd receive their emails. These weren't spam but the companies wouldn't offer _any_ way to fix this.
My recourse is to just flag them as spam in gmail.
That's why I said the email should contain info about the sender -- there should of course be a way to contact them. Ideally you should just be able to reply to the message and tell them about the error. If there's no way to contact a company, that's a whole different problem, and not really one that would be fixed with unsubscribe links in important email messages.
> That's why I said the email should contain info about the sender -- there should of course be a way to contact them.
That implies a level of manual effort on the part of the recipient that's unreasonable. I have no relationship with these companies. They did not verify the email address before starting to send a stream of supposedly transactional messages to it. They should be happy that I'm willing to click unsubscribe when available, because the alternative is to set up a mark-as-spam filtering rule that'll hopefully tank their sender reputation.
Writing them via a contact form begging to be removed is not an option.
In 99% of the cases - there is no recourse. In one instance, I tried replying and they asked me to prove my identity as the customer to cancel the emails.
This, I've had people receiving bank alerts for an account they don't own and they can't be stopped. What these companies lack are customer-centric processes that they've thought through.
Wtf is wrong with putting contact information in the unsubscribe link, or reach out productively on request? Why would you presume somebody clicks it by accident vs. the much more likely case of it being a legitimate request? Are you afraid they really want to cancel your service? Or are you afraid you can't send spam under the guise of transactional messages? Or worse, listen to customers about how best to alert them? Truly ridiculous!
> Wtf is wrong with putting contact information in the unsubscribe link
By law, depending where you are, a unsubscribe link has to be instant. So there can not be an intermediate screen asking for confirmation or showing contact information. Well, you could show contact information, but then the unsubscribe (of in this case service critical mails, thus the service itself) had already happened.
I'm assuming you're referring to the CAN-SPAM Act [1] or equivalent in other jurisdiction [2], but nothing of the sort is implied. The requirement is to make available and process opt-out without charge and promptly (typically in so many days).
It is neither dire nor prevents clarification, nor prescribes a specific experience related to unsubscribe links. It's odd to hear the only options are between receiving messages and having service terminated. That sounds pretty user-hostile tbh.
You're expecting that a company incompetent enough to attach unverified email addresses to an account to "correctly" deal with unsubscribing from transactional email? This seems entirely futile and counter-productive to me. (Correctly in scare quotes because I can't fathom what a correct automated unsubscribe would look like in this situation.)
Emails should be confirmed before being used for ongoing communication. Simple as that. It’s easier to get right up front than it is to clutter and confuse in the cases already illustrated.
If I get continuously get emails sent to me that I did not request and I can't unsubscribe to, then it's spam. Maybe companies should make sure they're not sending emails to the wrong person, because I'm just going to keep marking it as spam when it comes my way.
Exactly. "Poisoning" the spam filter is nothing more than sending an alert back to upstream to deal with it when and where it starts to hurt business. Typically it's the only feedback mechanism taken seriously and thus a service to the community.
But...it is spam? If they don't give the tools necessary to stop the spam (unsubscribe or a link to "received this by mistake") then it's spam - intentional or not.
Yeah, I run my own email server and have the same issues with the same services as the person who wrote the piece this links to.
In my case users are sending estimates and invoices and monthly statements to their clients and while fake invoices may be a spammer thing those clients know who's sending them an invoice, and why and what for, so an 'Unsubscribe" link would be completely out of context because they are not subscribed to any email list.
I've had the same domain name for over 20 years now and none of my users have ever used my apps to send spam. And as spam and email volume go my server isn't even close to sending out a lot of email.
When I set up a new email server last year, with a new IP address, I had to go through the process of getting white listed. All the big email service providers have ways to do that. Google made it very easy. They gave you a unique string to add it to your DNS records and that's it. Microsoft is so convoluted I've still not gotten anywhere with them. Comcast and others had a few hoops and ladders but nothing that got me stuck.
Personally, while it's a bit of a PITA to setup and manage an email server, it's been worth it.
I used "Mail-in-a-Box". It's pretty easy to set one up with that. It has a built-in DNS server and that's a really great thing to have for managing several domain names and as many email addresses as you want. I've setup email accounts for family and friends as well as throwaways for my wife, who signs up for everything she sees on the internet.
I can move the IP address of my email server to the top of the list in my Mac's System Preferences for DNS and start testing new domain names and changes to the DNS immediately. I don't have to wait for those to propagate to whatever my access provider is using.
So I have 3 servers. An Email/DNS server, a database server, and a website/webapp server running on DigitalOcean's "Droplets". It's a bit of work for a small shop but it's much easier to manage once it's setup and I don't have to worry about any 3rd party service selling out or going under or changing their API to something entirely different. All of which has happened to me in the past.
We add link to accounts page to manage notifications - but we clearly say that certain notifications can be disable only by deleting the account.
But even that does not help: some people just do not understand how to manage their email: they will consider as “spam” any email they receive and they cannot act on it immediately. Like storage quota reached, cc expired, or similar. There is some strange feeling that if they ignore the email and mark it as spam that the problem will magically disappear.
We have so many cases like that: and they then contact us with questions like “why my service is not working” etc.
You guys are thinking way too literally and rigidly about this. Unsubscribe in a transactional email does not need to be an automated stop. Just have it open a support ticket and then follow up to find out why they clicked unsubscribe.
If they are an established customer, it is legal to do that in all jurisdictions, even under GDPR.
It’s much better than having a customer mark it as spam, as some people will definitely do if they can’t see how to unsubscribe.
I've also had email from the wrong person delivered to me some times. One company in particular kept sending updates for a service I had no way of using. An unsubscribe link would have been handy, though confirming email addresses before linking them to an account would also be a good idea, probably.
Those unsubscribe links should be there, for several reasons.
In some jurisdictions there is information that businesses are legally required to provide to their customers in a permanent form and email is the conventional (and potentially the only) way of satisfying that requirement.
IMHO, it is not helpful for anyone to have a system where recipients may not understand this and may treat that mail as spam, yet businesses are compelled to send it anyway.
Mail may also be simply going to the completely wrong address, for some reason, and there needs to be some way to ask that this be corrected without requiring credentials that the recipient might not have. If not an unsubscribe link, then at least a reply-to address that actually works, or some way to handle a mistake.
I had one email address that for some reason often ended up being mistakenly used for other people, and ended up enormously frustrated by the usual combinations of no unsubscribe links, no-reply senders, and notes that any changes should be made by logging in. In the worst case, the US Department of Education, for several years, sent the address private personal and financial details about a complete stranger at a university with student loans. There was no way for me to unsubscribe, and no monitored address to reply to: in their view, they were important emails for the recipient, and they were, but I was not the intended recipient at all.
I treat every unsubscribe link as though it read "click here to confirm your email address is live and being used by a human being so it will command a higher value when we resell it".
If I don't like an email I never unsub. I add it to the spam filter.
> We were also informed that our normal, business communications with paying customers should have unsubscribe notices appended to them.
You should have an unsubscribe link. You should also have your business address and identify yourself.
Even if it's not required by the letter of the law, you should add it.
As an example: Amazon automatically opted me into an "alert" when a wishlist I viewed had a new viewer. Since it's an "alert" and a "business communication" it has no unsubscribe. This is spam - this is an ad hidden as a notification.
Yes, just because I use your service doesn't mean I want to see every outage notification status update as an email. Preferably email subscription status would be granular so I can select what I want to get not what some idealized average user would want to get.
I've been a paying customer of rsync's service for more than a decade. The only mail I get is the monthly invoice, and roughly once-per-year notice of infrastructure changes that may temporarily affect availability.
Oh I have no doubts whatsoever the volume is low and the messages sent intended to be genuinely important to the vast majority of customers, rsync seems very reputable based on what I've heard over the years on HN.
It's still nice to have granular subscription though even for rare things you think 95% of users may like to hear about e.g. I've been using a similar service since 2015 and I have 0 interest in receiving their downtime or scheduled maintenance notifications as I don't care enough to take a special action for a failed sync or two in the first place so... I don't opt to receive them and I appreciate that option. I don't get the invoices emailed so I haven't had to think about it one way or the other there.
That's not how unsubscribe links are supposed to work.
Once the unsubscribe is activated -- and it's supposed to be very easy to activate -- then it's permanent. There's no "un-unsubscribe", "oops I clicked it again", "some other service glitched and clicked it for me".
Further, there's a distinction made between "commercial" and "transactional" messages in both law and etiquette. The unsubscribe link is expected in commercial messages, not transactional ones.
> Further, there's a distinction made between "commercial" and "transactional" messages in both law and etiquette. The unsubscribe link is expected in commercial messages, not transactional ones.
Most of the junk that gets through my spam filter are transactional or other "mandatory" messages intended for someone who fat fingered their email address. If those senders don't want to be marked as spam, they need to provide a way for me to make the messages stop.
Email confirmations should be standard but that's not what we're talking about here (and I'd expect that ~rsync is handling that properly).
Unsubscribing from transactional emails eventually causes the following support conversation: "Hi, uhh, rsync? Yeah, so, I'm having trouble logging in to my account and we really really need our backups, our intern just nuked a database. Yeah, it's uhh... cto@company.com. What do you mean my account's not active? ... ... Why didn't you just tell me my card expired? Well yeah, of course I unsubscribed, but I still wanted to know my account was being shut down!"
There's a scale of headaches happening here. At one end of the scale we have "nuisance", as in, "I'm getting too much email, or I have a stupid email address, or I don't know how to filter messages from reputable senders", and at the other end we have "job-ending cockup", as in, "I'm just now finding out that a critical part of our disaster recovery plan hasn't been working for a long time because somebody somewhere was inconvenienced by a notification, and I'm finding this out now because today happens to be the day we really need that disaster recovery plan".
Pushing the needle away from the nuisance end moves it closer to the disaster end.
The service is not meant to cater to the lowest common denominator. If you unsubscribe from critical notifications and get screwed over.. that is on you.
It is not fair to the rest of us to be inundated with endless spam just so some screwup can be kept from doing something stupid.
Transactional email from a backup service you deliberately signed up to isn’t spam, so congratulations you’ve got what you’re after.
Now someone will likely reply shifting the definition of what “spam” is to include Rsync’s critical service emails, and now the term spam is so wide as to be meaningless.
At that point it’s on you to manage your own spam filter if you truly feel “your critical backup service is down” is spam. I haven’t been inundated with endless spam for about a decade.
Abusix don’t know what they are talking about, and basically all services that let you manage your email notifications still send through critical “your service is about to be turned off because your card details failed” emails regardless of how many checkboxes you disable — and for good reason.
I got dogpiled on here a couple weeks ago for the temerity to suggest that "spam" is, by definition, unsolicited. Unreasonable people like this put companies in no-win situations.
You can always login and re-enable an email on the service. The service is allowed to request information needed to process the unsubscribe.
I get emails for some dude's Chevy when it needs servicing. I can't unsubscribe. I am stuck getting emails about a car I have never owned from some dealer in Pittsburg. I need an opt out that lets me communocate "hey, some dumbass fatfingered his email, stop spamming me."
Genuine question: your comment and a bunch of others make me wonder why people seem unable to filter email by sender. That used to be a pretty standard part of having an inbox. Are you using a mail client or service that doesn't have filtering built in? Do you find it difficult to set up a filter rule? Are you unfamiliar with filter rules? Do you use filters but just ideologically object to any unwanted email?
Abusix isn't their ISP, they're an email blocklist provider. Telling people what they need to do to not get blocked for being abusive is literally their job.
yea, in an ideal world, the blocklist provider would educate others in how to avoid beeing blocklisted. yet, if this course is met with success then the blocklist provider is out of business.
> We were also informed that our normal, business communications with paying customers should have unsubscribe notices appended to them. Which is to say, you're a paying customer of a service and we send you some kind of alert or critical announcement ... and it should have an unsubscribe link.
You absolutely should. The amount of junk I get because someone else signed up for something and fat fingered their email address is ridiculous. "Mandatory communication" with a company I've never dealt with gets flagged as spam.
That's a (very legitimate and important) reason to do double opt-in unilaterally for all email communications. Companies should make 100% sure that the person who signed up, and the person receiving the email, are the same person, before they associate the email with the account. Otherwise, malicious people can sign up arbitrary third parties for tons of random crap.
But it's not a good reason for adding unsubscribe links unilaterally to all email communications.
Remember, unsub links are machine-automatable; Gmail at least offers to follow any embedded unsubscribe links for you if you mark a message as spam. (Which, with hotkeys enabled, is one accidental keypress away.)
So consider the extreme case: what if the user fat-fingers an unsubscribe (without realizing) to their local electric company's e-invoices, which is what they've been relying on to prod them to log onto the site and pay the bill?
If it's clear that "bills you need to react to or your power will be shut off" shouldn't have an unsubscribe link, then clearly there's some sort of line that must be drawn somewhere.
(Note, I'm not arguing against the use of "Manage your Mail Preferences" links in these cases — the kind that act as magic sign-in links and take you directly to a page on which you can un-check a "mail me about X" checkbox. It makes sense to include those. I'm just arguing specifically against unilaterally including "Unsubscribe" links — the kind where following the link unsubscribes you with no further confirmation needed.)
> So consider the extreme case: what if the user fat-fingers an unsubscribe (without realizing) to their local electric company's e-invoices, which is what been relying on to prod them to log onto the site and pay the bill?
To name a specific example of this problem, I want Gulf Power of Florida to stop sending exactly the kind of email you speak of. Bills. Nastygrams when the person falls behind on the bills. Unwanted power saving tips. Calling the company and sending them postal mail has not helped. It all gets marked as spam these days. If they had an unsubscribe button, it wouldn't.
If the email is so damn important, they can go back to sending postal mail to the service address when someone unsubscribes.
I think you're trying to use two wrongs to make a right. If we're talking about things Gulf Power of Florida should do differently, then rather than add unsubscribe buttons to bills which is a bad idea, they should confirm people's email addresses before sending them email.
Well, I think letting people get silently charged monthly without sending them any message about it is a bad idea, even if they say they want it. Similar with not telling them about an outage to their service.
Well, then that mail is going to get legitimately marked as spam when I didn't request it, I have no business relationship with the sender, and there's no other way to make it stop. And the user is still getting silently charged because it's going to my inbox, not theirs.
Unsubscribe is the alternative to the spam button. If you don't provide it, you are asking to be marked as spam, no matter how important you think the email is.
the email w/unsub link could be forwarded also, it's often a portal to change notification settings w/o auth and leaks personal preference info - and when there is auth it's impossible to unsub when if were signed up maliciously.
it happened to me - someone charged a bunch of stuff to my cc and then registered my email at thousands of sites to bury the email receipts (it didn't work since I have simple filters for that sort of thing) but it has been impossible to unsubscribe from all the junk. livemail's bulk optout was roughly 50% effective. the dark patterns around optout are outrageous and it's worse when you have to use google translate just to find it.
a bunch of my remaining junk accounts have broken pw reset, or even after taking control there's no way to delete the account and/or the "optout of everything" option doesn't actually stop them from sending me junk regularly (ie they just dont honor their own optouts)
> So consider the extreme case: what if the user fat-fingers an unsubscribe (without realizing) to their local electric company's e-invoices, which is what they've been relying on to prod them to log onto the site and pay the bill?
I actually unsubscribed from my provider's invoices. That's because I have activated direct debit from my bank account so they're always paid, and I can view my past invoices on the website.
However you make a good point. I'd say the one thing where it doesn't make sense to have an "unsubscribe" at all is on "bill unpaid" emails.
1. Forget I had a monthly power bill for a couple of months.
2. Ignore the e-bill that gets sent to my bank bill payment service - ebills have been a thing for almost two decades. I worked on some of the early implementations.
3. Ignore the physical snail mail warnings for a couple of months.
> Otherwise, malicious people can sign up arbitrary third parties for tons of random crap.
In the early days of the internet I was a teenager in a prank war with someone so I signed them up for all kinds of spam and also free trial magazine subscriptions.
That's a different problem. They should have first sent a confirmation email, then paused all communications until it was confirmed.
But once the email is confirmed, I think it's totally fair for a company like rsync to say 'if you're a paying customer of this service, then we need to send you certain information to fulfill our obligations in the contract, if you truly don't like it cancel your account and take your business elsewhere.'
Yep. I had to file a GDPR complaint to get an airline to stop sending me "letters from the CEO" and other COVID-related reports that never mattered to me.
I never flew with them but somehow they still sent ads disguised as reassuring messages every other week.
Support constantly denied help, since I was never a customer. Only a GDPR complaint solved it.
The inability to generate IP reputation smaller than a /24 is inherent to the way internet routing works. Nothing smaller than a /24 can be publicly assigned or advertised to prevent the route table from becoming too bloated. On IPv6 the smallest advertise able block is a /48 for the same reason. Privately managed assignments in shared or further split subnets aren't publicly visible, verifiable, or accountable to anything but the organization owning and advertising the /24 (or larger).
As such the reputation score of a subnet is the reputation of the entity advertising itself as publicly controlling and maintaining that network, not the reputation of individual sub entities inside that subnet (which is known only to the controlling entity). If that entity is constantly allowing bad actors onto their block then that block is considered poor reputation.
> The inability to generate IP reputation smaller than a /24 is inherent to the way internet routing works.
Nonsense. SMTP is not internet routing, and there is no reason at all why some "reputation" system should be constrained by the same /24 limit that the global routing table ended up with.
I find it telling that these "reputation" outfits tend to serve up this sort of poor excuse to justify their businesses; whereas they're often plain old protection rackets. Like the one described in the article clearly is. They take money to solve the problem they created!
I think you're thinking domain based email reputation lists but commenting on the IP reputation conversation while also conflating what Abusix is doing in this comment chain with what UCEPROTECT is doing in the post and forming a conclusion from hand picked pieces of the conglomerate. For the Abusix portion related to this thread check out their response here as it goes into much better detail with what I originally described https://news.ycombinator.com/item?id=30227886 (note: I'm unaffiliated, I work for a generic network VAR and we don't even sell their service for email).
These domain based lists are also a thing but you need to have a pretty sizable volume to become trusted in those lists as anyone can register a domain in seconds, none of the above will have gained trust on them.
The whitelist payment option for arbitrary hosts (which Abusix doesn't offer) is a workaround to create a publicly registered identity with some stake to tie the reputation to. I mean it can be other things to, scammy rackets exist too, but you can't just claim it's a poor excuse without offering a better answer that solves the same problem.
Speaking of spammers are the ones that created the problem because people started using individual network block lists before companies started aggregating and mediating them. You're welcome to make your own community replacement solution but the problem is the replacement has to be reliable enough other mail administrators want to use it not just nice for the occasional legitimate guy that wants to run email out of his house and be trusted by default. Also it's a lot of work to mediate.
I'm the Abusix engineer in question (and actually the architect of the system in question), and you're being somewhat "economical" with what actually happened here.
Here's the actual chain of events in question:
- You recently switched ISPs and that meant you moved to a new IP block.
- Without going into massive details on how our infra works, but when we have the most serious level of listing (e.g. hitting our most secure traps as in this case), we treat the same /24 more aggressively than we normally would (because of things like snoeshoe spam that would normally spread traffic across a wide range of IPs).
We use /24s only where we cannot determine different ownership of the IPs by looking at the abuse contact registered at the RIR e.g. if the IPs are different contacts then we don't bundle them into the same bucket. In this case Hurricane Internet owns the entire range, so the /24 is used.
We do this because there is no other way to do it that isn't completely abuseable by a bad actor e.g. rDNS is completely trivial to forge and to claim multiple fake entities. If someone has a fool-proof way to do this, then I'm all ears.
Then we get to your failings in this case:
- You don't have proper, working bounce handling.
You were repeatedly sending mail to an old customer and we were rejecting 100% of the email you sent to that address - stating that you should stop sending to it in the rejection message. We always reject traffic on traps we are building so that bounce handling removes them automatically over time.
- You decided to send a marketing message to a bunch of users, including to the address described above "We (rsync.net) are experimenting with a lifetime prepayment option". This message provided no List-Unsubscribe option at all, so I could not unsubscribe it without exposing the trap to your engineer (which is my primary concern as it takes years to build traps properly).
- Your engineer said to me "we took down 600 old accounts and are reviewing our contact policies going forward" and "there are still plenty of other customers who are listed generically as bouncing so we will have work to do here".
That tells me that you knew that you were sending to old accounts and that your bounce handling was either not great or non-existent.
I take our role very seriously and I and my team go out of our way to help anyone who finds themselves listed by providing evidence and advice and we always try to find a good resolution for all legitimate senders.
That is exactly what I did here with your engineer, the problem is resolved as the specific account was removed and you know now what you were doing wrong and how to fix it so that it never happens again.
Your engineer was appreciative and I said if there are any further issues in the meantime whilst you fix things on your end, then we would help by exempting your traffic whilst you did those changes.
I can't see how we could have done anything more in this case.
On our website, we provide blog posts and videos, we take part in conferences and workshops to give advice on how to do things properly so you never have blocklisting issues.
I can easily summarise here what everyone should do to avoid issues:
- Have proper rDNS for your sending IPs that is part of your administrative domain (e.g. don't use your providers generic rDNS that contains the entire IP in the hostname). If someone visits the domain name, make sure there is a website present that has contact details as a minimum.
- Make sure your abuse@domain and postmaster@domain accounts actually go to a responsible person.
- If you send any marketing mail at all, make sure it has a working List-Unsubscribe header (preferably HTTP that allows someone to unsubscribe without having to contact you). Important note: If you use a mailto: unsubscribe, then I cannot unsubscribe a trap, even if I wanted to.
- Make sure you have working bounce handling. If you repeatedly send to an account and it either bounces or is hard rejected, then you need to stop sending to it and mark it as bad. No excuses.
- Don't send to addresses where you have not contacted them or had any interaction at all for > 1 year. If you haven't kept in touch with your customers, then that is on you.
- If you have a web form that when submitted, sends a message to an external user, then you MUST a) validate all of the input fields and disallow URLs unless the field required it and b) prevent automated submission via the use of a CAPTCHA.
- When collecting email addresses for mailing lists, always use confirm opt-in e.g. send a message containing a link that they have to click to activate the subscription. Do not send any further messages to them until this has been completed.
- Make sure you separate IP addresses being used for outbound mail from those used for outbound NAT pool. Block outbound port 25 from the NAT pools and make your firewall notify you of any port 25 activity from any hosts as this could indicate they are infected.
- When provisioning a new mail server IP, don't send more than 30 messages per minute (e.g. 0.5 messages/sec) for the first day and then increase the volume over the following week.
I'm sure some will disagree with some of these, but I guarantee that if you follow all of these, then you'll never have an issue with a blocklist like ours.
Incredibly insightful, thank you for taking the time to post this!
> - Don't send to addresses where you have not contacted them or had any interaction at all for > 1 year. If you haven't kept in touch with your customers, then that is on you.
That seems odd though. "Keeping in touch" in regular intervals is exactly the kind of thing customers may want to unsubscribe from, so a working unsubscribe function means there are customers where I cannot keep in touch. How is that on me? That means I can't send e.g. a password reset email to them if they try to log in two years later?
Apologies for not being clearer on this, I'm generally talking about marketing messages when I say "not keeping in touch".
My point here is "consent to send", e.g. I think most people would find that someone importing a years old list of email addresses and then suddenly sending marketing messages is unacceptable and constitutes spam.
If you're "doing it right", then "touching" your customers with a once a year "Hey, we haven't seen you in a while" message is absolutely fine and generally a good thing to do as it maintains consent, because they can unsubscribe if they're no longer interested (as they might have forgotten they signed-up or had an account).
If someone hasn't logged into your "service" for >1 year and now wants to do a password reset several years later, then I would say that is entirely different.
We generally try our hardest to handle this case without it causing a listing if you hit our traps with something like this.
"- Your engineer said to me "we took down 600 old accounts and are reviewing our contact policies going forward" and "there are still plenty of other customers who are listed generically as bouncing so we will have work to do here"."
(snip)
"That tells me that you knew that you were sending to old accounts and that your bounce handling was either not great or non-existent."
Yes, that is exactly right.
I have instructed everyone to be as liberal and forgiving of non-payment and failed contact as possible. These people, who were paying customers, have data stored here for safekeeping and we're not going to trash it because we haven't heard from them in 3 months or their email bounced.
I can give you hours of stories of customers who came back from military deployment, came back from depression, came back from prison, came back from financial ruin ... that contacted us, beyond all hope, to see if we still had their account. And we did.
After 21 years of doing this are there a few hundred accounts that we are giving extreme benefit of the doubt to ? There certainly are.
Bottom line: our duty of safeguarding customer data trumps this weeks fashionable spam heuristics.
> Bottom line: our duty of safeguarding customer data trumps this weeks fashionable spam heuristics.
I haven't suggested that you do anything differently to that.
Keep the data, keep the accounts, do whatever you feel is best for you and your business.
All I've said is to be smarter when it comes to sending email to old accounts that are repeatedly bouncing. I've outlined what was wrong and I worked with your engineers to resolve it.
The screenshot I got of your conversation with (Dave) was, indeed very well done and was both informative and actionable. You responded to us very quickly and with a high level of professionalism.
Further, participating here in this HN discussion is noteworthy and impressive.
FWIW, we immediately complained to he.net about the bad actor elsewhere in that /24 ... we'll see.
> The screenshot I got of your conversation with (Dave) was, indeed very well done and was both informative and actionable. You responded to us very quickly and with a high level of professionalism.
> Further, participating here in this HN discussion is noteworthy and impressive.
Thank you - much appreciated.
The quality, morals and perception of our product and how we treat people matters to me greatly which is why I'm always happy to get involved in discussions like this.
As is evidenced from other post on this topic, there are a number of competitors that don't behave in the same way and that leads people to think we are all the same and I absolutely want to challenge that perception when it comes to us.
> FWIW, we immediately complained to he.net about the bad actor elsewhere in that /24 ... we'll see.
Thanks, feel free to refer them to me and I'm happy to provide as much evidence as they need.
As a former admin of an isp & running saas apps I applaud you greatly for your approach. Some people may find these approaches are "heavy handed" but the reality this is what life is like in 2022 with email. Should we need to do this? Definitely not, but it's an arms race and each company sending email has to up their game - to both be compliant in getting email accepted, and to foster a healthy relationship with customers.
The comment about "lifetime prepayment option" is exactly something that MUST have an unsubscribe option to it. I understand everyone is trying to increase business, but this is something @rsync missed mentioning, insisting you don't need one.
One of the ways most blocklists work is by employing spam traps which can either be individual email addresses or entire domains.
It's quite a large topic, but I'll try and summarise for you.
You can't just take a spam trap and use it to block anything that hits it - that would be incredibly unfair as you might not know the history of the email address or domain and you'd generate considerable false positives if you did.
If you buy a domain or register an email address and you know for certain that it's never been used before, then this is typically called a "pristine trap" and there are then various ways to then "seed" this so it starts to receive spam and malicious traffic. This is the exception where it could be used immediately. However this process usually takes a very long time (usually years) to receive enough traffic to be useful.
Any other address or domain where you don't know the history would be called a "recycled" trap. There are various opinions on how these should be managed, but generally it's accepted that you need to reject ALL traffic on these for at least 2 years to allow genuine senders to work out they are no longer valid (which is why bounce handling is important!).
The other type of trap is a "typo" trap, which is typically an entire domain (or an email address deliberately close to another) which could easily be typo'd. I have a rule here that says we neverever use these for any blocklisting (other lists will not have the same rules!), however they can be useful to see trends and detect compromised hosts (as these typos frequently end up in compromised databases that get sold around the dark web). They can receive considerable volumes as well.
Typo traps are especially why you should always use confirmed opt-in and employ CAPTCHA on your sign-up forms (although this also helps with all traps in general).
We carefully monitor all traps to see how they are performing e.g. detections .vs. traffic from whitelisted sources and false positive reports and will immediately remove a trap if it begins to perform poorly or has been made public in any way.
It takes years to build a trap network and we have to discard all of that work should they become discovered (as they could be used maliciously), which is why a blocklist operator will never reveal the addresses to you as they are a closely guarded secret. In fact on our own network, even we don't see them - we monitor them via a hashed name! (only I and a few other select people can convert the hashes to the actual domains or addresses).
I think that covers the basics - I hope that explains it sufficiently for you
> make sure it has a working List-Unsubscribe header (preferably HTTP that allows someone to unsubscribe without having to contact you). Important note: If you use a mailto: unsubscribe, then I cannot unsubscribe a trap, even if I wanted to.
This is very interesting to me. Could you elaborate further?
- Some providers (IIRC outlook being the main one) only support mailto: and most prefer it. I have never seen advice that you "must" provide HTTP before.
- What do you mean by unsubscribe without contacting me? Isn't HTTP or SMTP just as much contacting me?
- Why can't you unsubscribe mailto: even if you wanted to? Earlier you hinted that it may have to do with revealing the trap, but while a HTTP url may not directly include the trap address it would be linked anyways. (Otherwise how would the unsubscribe work?)
Thanks for all the valuable insights. It does sound like you are taking a responsible approach here.
Sure - happy to elaborate here, re-reading my response I understand the confusion here as my choice language wasn't great ;-)
I didn't say anywhere that you MUST support HTTP unsubscribes, but I just said it was preferable (to me at least).
HTTP typically means there is some sort of form which will typically auto-fill the email address from the database (because of the HTTP GET parameters included in the link) and all the user has to do is generally click 'Unsubscribe' and the address is automatically removed from the database.
The reason I prefer this is that if I or one of my colleagues want to unsubscribe a spam trap, then we can - but only if the above is true. (Disclaimer: we only do this very occasionally, usually it's only if we've spoken to someone and are convinced they are legitimate, have fixed any issues we've identified, but are hitting traps because of a single address - if we didn't do this then we'd either have to force the sender to reconfirm their entire database by asking everyone to opt-in again, or we'd have to whitelist the sender, neither are great options - so if we ever do this, then it's done hours/days after the event so as to not tip off anyone on what the trap was).
Because spam traps have to stay secret, we only reference them by a hash value internally (and only I and a select few can convert hash to trap value to do the unsubscribe) and as the messages are received on our trap network the traps themselves are obfuscated to a know value before they are stored in our evidence system.
mailto: means we can't do this at all - a trap can never originate traffic, that's cardinal rule of them, it would be grossly unfair if they did.
> - What do you mean by unsubscribe without contacting me? Isn't HTTP or SMTP just as much contacting me?
Poor use of words from me.
What I meant is that I don't trust messages that have an unsubscribe that does not encode the recipient in it in some way e.g. HTTP method without GET parameters (as this means I would have to do considerably more work to fill out the form) or a mailto:foo@domain.com?subject=unsubscribe - both to me scream our that there is no automatic connection to a database and that someone on the receiving end has to manually remove the address - which could take weeks, or be never.
One other question. How do you manage domain based reputation vs IP reputation. I'm trying to get by running a service off of Cloud provider IPs and they may rotate from time to time as VPSs rotate. Part of me feels that once my domain has enough reputation it should be fine. It does seem like some of the big providers work like this (Google seems to trust me now from any IP) but it seems that a lot of smaller providers rely on blacklists more as they don't see as much traffic to build reputation. How do blacklists handle domain reputation?
From our perspective, we don't link IP and domains at all at the moment, they are totally independent.
Because spam traps get very different traffic to a real mail server, you can make a lot of assumptions that would be impossible to do on a real mail stream.
So generally speaking, if your domain name isn't in the top million domains (and there are still some exceptions to this - but I'm not going to list them all here) and you hit one of our pristine traps (see my post about the different trap types) with a message containing your domain, that will usually cause the domain to be listed.
Other lists and spam filters will have their own methods for this.
My belief is that Google and Microsoft only use AI for this (which is a bit of a nightmare if it gets it wrong).
FWIW, Mailgun has worked pretty well for me in the past as an alternative to handling delivery myself.
You might not be able to go this route if your customers have some expectations about how your email is handled, in which case this recommendation is here for anyone else that might read this and need another thing to try.
Both my personal domain and rsync.net are on a distinct subnet, but that subnet is smaller than a /24 and someone on a different subnet has, apparently, behaved badly.
Enter "abusix" ...
One of my engineers had an enlightening webchat with one of their engineers where we were shown the "offending" IP and it was explained that they have no ability to distinguish subnets (and no interest in doing so). So if you're not wasting an entire /24 (we only need ~10 IPs at this location) you're in danger of this misclassification.
We were also informed that our normal, business communications with paying customers should have unsubscribe notices appended to them. Which is to say, you're a paying customer of a service and we send you some kind of alert or critical announcement ... and it should have an unsubscribe link.
Unbelievable.