This is being looked into internally and I am involved. Likely won’t post an update here as it pertains to a customer account (unless customer agrees).
BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.
I've put this idea forward a number of times here on HN in regards to other big tech companies.
Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
The Ombudsman must not be part of the technology companies ordinary support processes, it must be entirely separate, and have highest level authority to demand action within the company.
To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
HN constantly has front page posts from people for whom big tech companies have support processes have failed but there is simply no other recourse unless you have "a friend in the business".
It just doesn't work to have some random Cloudflare person offer their email address as some post disaster issue resolution process on social media. Formalise it with an official Ombudsman and maybe then companies like Cloudflare might avoid HN front page bad publicity.
I had an issue at "one of the biggest tech companies" that went on for days and days in which tech support kept telling me I had set up something wrong, until eventually I emailed one of the top managers who I happen to "know" at that company - it was fixed within hours. That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.
> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
What if the cost was put onto the business instead of the consumer and the business just hired support people who are all Ombudsmans by default?
Instead of focusing on copy / pasting boilerplate scripts and answering as many tickets as possible, they should focus on the problem the customer is having by default and do everything possible to reduce the number of incoming questions by fixing bugs, making a better product, improving their docs, etc..
I personally do around the clock email support for 35,000+ people who sign up to my programming courses and support isn't bogging me down. Relative to the number of minutes I'm awake, support is one of the least business related time consuming things I do per day, but I send individually personalized in-depth answers to everyone who asks me questions -- usually within an hour or less.
Because requiring the customer to pay a cost fixes the incentives problem. A customer may have an issue that is a minor inconvenience to them, that can either stay as not resolved, is due to error, or is simply not worth resolving to them. By having them pay for fixing the issue it could greatly reduce the number of inbound cases, allowing them to go straight to people that can act on them instead of sitting in triage and getting canned responses for O(days|forever).
I agree that in theory you can accomplish the same thing by making the product foolproof, but I don't think you can accomplish that for consumer facing products, and that doing so may not be a worth while trade-off. Additionally focusing on issues that greatly impact people rather than small things that cause friction with the product may (or may not! if it causes lack of retention) be worth more.
In terms of games theory; if there's no cost to the company to ignore the customer, then that's an incentive to ignore the consumer's complaint. Especially if that company has a monopoly.
I can understand why GP suggested company pays $20.
The suggestion is the customer pays $20 so they don't try and go straight to the ombudsman with ever little issue. The company doesn't and shouldn't pay the customer under any circumstance.
In technical support, one of the largest problems you have to deal with is all of the idiots that don't know how to use the product and won't look up the documentation or learn the product on their own. These people spam the shit out of support all day for the most basic shit. Seriously, I was working support at an anti-virus company and 85% of the calls into the paid business tier support were requests for us to do installs or basic application configuration for them. I know that not everybody knows how to configure a firewall, but "where do I download the install file for x" is literally Google-able.
The idea of an ombudsman is ultimately a bad one in this case because the problem isn't with the product or the support or the documentation. The problem is people Always think their issue is special and the most capable person should help them. Our support tier was paid, so more often than not we did what they asked, but the free customers would just get routed to a sales rep because free customers are even dumber and more entitled than the paying ones.
This whole situation is farcicle to me. Yet another free customer blew a minor issue out of proportion and had to apologize when it turned out the system hadn't failed him, he was just a free loading mooch all along.
A company has no incentive to ignore the customer, all else being equal, but they do have costs in servicing the customer and costs from ignoring the customer.
Assume the company has no way of distinguishing "valid" or "high impact" cases from other cases. This means that in order for the company to handle cases they need the sum of fully treating every case to be greater than the sum of the cost of fully treating every case. This is almost certainly never going to be true, unless each of your cases is high cost to ignore (think enterprise support).
So you need to funnel down the cases. Typically this is done with low tier support that tries to suggest fixes and such. You can also offer high value added support to give customers the ability to pay for a support plan. This $20 proposal is like that but on a more ad-hoc basis.
No, I don't think so. All the major internet companies listen to their non-enterprise users. They may not provide enterprise quality support to each individual user, but user voices in aggregate are considered critically important.
not if you have a monopoly. Which is why customer support for most ISPs in the US is so abysmal.
And even if it is just a sufficiently large company, if the customer is not a large payer and the issue is non-trivial to solve, the the cost of losing that customer could very well be less than the cost of fixing the issue.
What I was addressing was specifically the notion that companies pay zero cost for ignoring a customer, or the idea that they believe the same. That's false. Now there's no question there are some customers these large companies are better off without, and there are large and small examples of companies firing customers, or ignoring them til they go away. Nothing wrong with that. All the time on this site we talk about firing unprofitable customers. If you do this judiciously it's just good business. Doing that in particular cases is not the same as doing that generally.
NearlyFreeSpeech.net does something kind of like that. It costs money to raise any support ticket (you have to have a special support subscription) but you get refunded your "support points" if it turned out to be a problem on their end.
I think it's useful and it's very much worth it to give support a salary where they are enthusiastic to be employed.
If someone uses your product or service and has an issue, the first contact they have with your company is through support.
A crappy support experience could easily be the difference between having a life time customer valued at thousands of dollars, or your customer feeling frustrated and going to another company, netting you $0. That has a butterfly effect too because having 100 happy customers who are praising your support could lead to many new sales from organic recommendations. Having a bunch of customers who felt neglected by your support could yield a situation where your company is now on the front page of HN for having bad support or worse.
For whatever reasons, bigger companies focus more on measurable metrics like "tickets closed per hour", where the emphasis is on things that aren't important because measuring customer happiness is pretty subjective and doesn't translate well to employee evaluation scores.
I know if I ever got the point where I would need to delegate support assistance to someone, you can be sure I would pay them amazingly well, at least equal to a developer's salary because I never want anyone to ever feel like they get ignored or have a low quality experience with my products.
Airbnb is a great example of both. If you call tech support and get India, hang up. If you're lucky they're useless. If you're unlucky, they're going against your direct command and contacting the drug dealers in your house about a complaint. Contrast that to reaching one of the many on-shore agents who do their jobs well. One call, one or two people max and the issue is mostly resolved.
I've got no idea what the cause is. It's probably a mix of poor management, ineffective metrics, low salaries and a work culture in India which isn't synonymous with quality. The bottom line is, even the same company can be doing support wonderfully and terribly at the same time.
Yeah AirBnB support in India was awful and I wrote the company off after dealing with them for 20 minutes. Their onshore team actually cared about the huge cockroach infestation in our unit but the Indian guy I first talked to did not give a damn. And that’s who you’re most likely to get at 4 in the afternoon on your first day.
Part of "Wear the Customer's Shoes" at Twilio (when I was there) was that each quarter every developer and product manager was expected to spend a full day doing front line support for free(-ish) tier customers and however long it took to fully resolve the tickets they got during that time. Really let you understand the type of problems people had, how to have empathy, and basically how to make a better product.
I think this happens because it's very very difficult for a manager's manager to know if the frontline employee is spending their time solving real problems or slacking off with no actual regard for the customer.
"Metrics" are an unhappy middleground for everyone.
Yeah but at a large company the ratings don't matter too much in the end because it ends up being a spreadsheet game focused on hiring for the lowest cost while maintaining a level of support that's barely 1 notch above "horrible support".
So if you end up hiring someone who produces a bunch of 1-2 star ratings, you just fire them and try again with someone else until you get someone who produces mostly 3s+.
In the end you would think this would result in better support but it never does. You just end up turning over a lot of support employees, and the user experience for the customer is still bad because you need to first get through a robot menu, then talk to an entry level support who will happily let you pour your soul out on the details of the problem for 5 minutes, and then at the end they are like "oh, I can't do that, but I can forward you to someone who can".
And now you get to be put on hold again, anticipating by coincidence you'll probably lose the connection, and if you're lucky now you get someone who is capable of understanding the problem and you get to re-tell the whole situation again.
Before you know it, with wait times included you're 20 minutes into this and you just barely got to the point where you might get help for the issue. Businesses could solve this problem, but they don't. Instead of hiring better support folks for more money, they put the burden on the customer to have to wade through a bunch of BS and essentially train their entry level support for free -- and it's worse than free too because you're paying that company money for their service and you're trading time from your life to do it.
Yeah, even russian social network (200m users) have a bunch of very smart people supporting users from password recovery to psychological help. No way big startup can't afford something like this. You just have to do so.
I like the idea of the Ombudsman, but in this case I'm not sure that it's really applicable. I disagree that the "normal tech support processes have failed".
I wanted to point out something that may have been missed by many - which is that the OP in this case is a free user. i.e. he is NOT a "customer". He is a user of free services that Cloudflare is providing. It even says so in his post that he's a free account holder and that he's not entitled to tech support, hence why he kicked up a fuss online. My company is a Cloudflare Enterprise customer - when we write in, we get a response typically within an hour by their swat team.
The idea of an "Ombudsman" in this case wouldn't be an actual ombudsman in the spirit of advocating for the customer, but would be more of a "one time support fee" kind of deal. Which I think is fair, but in this case I'm not even sure the OP would have paid it. The OP had a viable option which would have been to upgrade to a Business account and get priority support.
I run a popular platform with 1.5 million registered users and the vast majority of them are free users. We also have the same problem. Most of our support is just swamped with free users mostly asking questions about how to do things (and it's in the documentation, they just don't read or search for it, it's too simple to just email support or blast away on Twitter and @ mention us). I've even had to withdraw from Twitter and Facebook entirely just because I was everyone's "ombudsman" (even if they weren't paying customers).
If you read his blog you would know that he is very much so a customer. To quote his blog verbatim "Because Cloudflare deleted my domain registration I can't change the status from clientTransferProhibited through their dashboard so I don't think I can even leave."
He registered his domain through Cloudflare Registrar, a paid service. I'd be concerned if my domain registrar told me to head to the forums for support for tld level issues that only the registrar can solve.
Ok but I still don’t get why the op complains about it so publicly the same day (likely immediately) after submitting the ticket without even giving the company a chance to triage and resolve the issue. (Yes he apologized) He paid $10 for a .org domain, which for Cloudflare is at cost. Goodness... It’s not like Cloudflare is refusing him support, just that he’s not on a tier where someone will respond in real time. In any case, it’s resolved for him now and he got the attention of the CTO (maybe I should try some public outrage sometime instead of paying for ent support heh!)
Also I think there’s an issue with what the expectations for support is for. The Cloudflare service itself - the DNS, DDOS protection, ssl, CDN, etc are all premium services with a free tier for kicking the tires. As far as I understood in the post, the OP was on that free tier but had registered the domain on Cloudflare (which was probably the added complication).
Sidebar - AWS offers no support even if you’re a paying customer - support is an additional product you have to add on and it is a percentage of your overall usage bill, even if you don’t contact support...
>Ok but I still don’t get why the op complains about it so publicly the same day (likely immediately) after submitting the ticket without even giving the company a chance to triage and resolve the issue.
Well the blog post isn't really primarily about his specific issue, it's about the systems being severely lacking. A ticket won't resolve that issue.
This is a really great idea, but I don't think it's possible for this to not get overused for every little issue.
Once it's overused, it becomes useless.
Isn't this the point of the monetary hold? You can just raise it until the amount of entries become manageable. I'm sure higher level orgs would easily put down $10k to talk to a developer at Microsoft. For indie devs, numbers like $100 or $1000 could be manageable, as long as they can trigger the refund and close the case whenever they want.
Arguably this does block out poorer people from receiving "special" customer service, but there are not really other things people are willing to lose (or put up as collateral) for this type of service. I can't really ship Cloudflare my toaster or car until they resolve my case.
I like the setup my bank's Ombudsman has -- you must first take your issue to first level support, then escalate it with them if not resolved. If the second level of support denies you, then and only then can you reach out to the Ombudsman.
Any requests that haven't gone through the proper process get auto-rejected.
That is the logical way for things to work, but it requires first level tech support letting you escalate, which is not always the case with non-bank industries.
The current go-to move is to tweet a complaint at the company's Twitter account. This is surprisingly effective across multiple industries and actually was something my wife did that helped resolve a time-sensitive AirBnB issue.
This may be due to banks having regulatory requirements - they don’t want regulators in. Because of this, resolving issues before regulators are contacted is in everyone’s interest.
Tech companies being inundated with complaints would actually be expected. Making it easy to complain to ombudspersons will cause the complaints to dwindle naturally as tech companies stop the unethical, even criminal, behavior they engage in.
Very few people will complain about their lifetime Google ban after Google employs appropriate personnel to evaluate such cases.
Today very few people complain about Steam's refund policy, after Valve rewrote their refund policy to actually include refunds, after a judge ended their decade-long crime spree that saw an estimated 20,000 Australians robbed and an unknown quantity globally.
Here in Australia the ACCC (Australian Competition & Consumer Commission) has suggested tasking the TIO (Telecommunications Industry Ombudsman) with digital platforms.
Basically the way the TIO works like this:
1. The customer must first go through normal support until they get stuck.
2. The TIO sends the Company a nice letter requesting that they get their act together.
3. The Company then has 10 days to fix the issue or the TIO will escalate.
4. Either the problem got solved or the TIO keeps nagging until it is.
Every time the TIO has to mail the company it also charges the company for having to get involved.
level 1: $31
level 2: $260
level 3: $475
level 4: $2250
If an issue gets to the end then the TIO can direct companies to implement solutions costing up to $50,000.
This process heavily incentivises companies to actually provide functional support. I don't think your idea of charging the customer $20 to get help with support would achieve anything.
I am a tech executive in my company and my direct e-mail address is at the bottom of every Web site. Yes, this means I deal with routing all sorts of problems BUT I know instantly when there is a problem. Any problem. And I can actually help. But really it helps me with my job. It's win/win/win.
Plus, poor customer services is simply inexcusable -- you have to treat people the way you want to be treated. You're letting yourself and everyone else down otherwise. There is ethics and morals in business and they are important.
It doesn't have to be my way but there are definitely ways to do it, do it well and not break the bank.
A properly setup helpdesk negates the need for an Ombudsman. If a tech company cannot get a helpdesk escalation path correct, what makes you think they'll get an Ombudsman scenario correct?
Ombudsman services are able to resolve conflicts where the customer and business don't agree. They're appropriate for regulated businesses to draw a line in the sand as to what is a legitimate complaint by independently settling cases where the customer and business did not agree. If the Ombudsman hears 100 unresolved complaints and upholds only one then probably the ordinary customer services are working as intended. If they uphold ninety nine something is gravely wrong with your customer services and the regulator needs to step in.
This isn't so dissimilar from the method used in pathology to deliver consistency in results across multiple labs and assay methods. You identify some boundary cases. This is definitely normal, this is definitely cancer but this is borderline, repeat a few times. You replicate and get all the labs to mark your samples. Then you can identify labs (or assay techniques) that aren't reliably putting things in the right categories and demand they improve or stop.
I've worked for various technical help desks over the past 20 years and I'm saying that if a help desk is setup properly, ombudsman services become unnecessary.
A big part of this is how empowered you make your various levels. The SLAs I've always worked with, created, and worked under measured time on calls and customer satisfaction was always one of the more important measurements.
You could also think of it as a "priority lane" for support. Pay nothing, get whoever for support. Pay $x, complaint goes to an engineer. Pay $y, the product manager. Etc. Skips the middleman process for customers that are in a rush.
If a customer is willing to put up 10k to get their issue resolved, it's probably an issue worth resolving.
Some companies, this doesn't work at all. EG: Postmates. Emailed pretty much every one of their executive/management team about a literally brand damaging issue, and received zero bounces, but also zero replies. Some companies have a policy to ignore unsolicited emails, no matter how serious the issue, as to not fuel the idea that doing so will get results in the future.
This is a really nice idea. This already exists for some companies in the form of Twitter accounts. I doubt the people doing the typee-typee actually has any authority over business processes, or demand a change, but I think they at least have the business owners on their speed dial, vs normal support tickets. But having an email is far better.
Amusingly, Twitter itself could use one. My account (@acangiano) has all of its images censored under "sensitive content" even though they are 100% benign images. No amount of tweets to @TwitterSupport has done anything at all to change it. There is basically no recourse. My account is like 12 years old and has 4.5K followers, so it's not like it's a random spam account, either. ¯\_(ツ)_/¯
I find this whole infrastructure, despite my taking advantage of it, to be very flawed. There're many customers who might have alarming issues, who never get attention because they're fearful to be perceived as a "Karen" for bitching on Twitter.
Yes it's a good point that companies currently have some sort of Twitter presence trying to address bad publicity posts, often they seem to be able to get things done.
The Ombudsman role is there to get things fixed when all else has failed, and before the angry customer posts to social media.
> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
From what I read this is nothing to indicate the process failed (so far) just that the user decided to skip to the head of the line by writing a blog post internet style to get something resolved and attention. Failed is not 'I didn't get a reply or find what I needed as quickly as I think it should happen so now let me complain publicly so I get a reply'.
> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
In theory nice but first it would be a 'deposit' and also opens up a host of new issues as far as the money being paid back and how that would be done and so on.
The core of what you say is correct. I posted about this publicly with two goals in mind: getting help from someone at cloudflare and getting advice on how to avoid this sort of issue from the HN community.
From my perspective Cloudflare's process did fail. Assuming I didn't do something insanely dumb and what I think happened did happen, I would consider that a failure on its own even if the support was perfect afterwards.
Copying from elsewhere:
I address this in TFA.
Essentially I felt that this was alright because when I filed a ticket I was informed that I should expect a long wait and that they recommend that their non-business customers post publicly on their support forum for crowdsourced support because that leads to faster replies. I was unable to log into that forum, and I suspect that may be because the way they set up SSO between the forum login and their main login may have failed in Firefox (with all tracking prevention and ad blockers disabled).
I felt that if a company invites me to ask for support publicly on their forum to save on customer support costs it's reasonable to talk about the issue in another public place.
Ive run into Cloudflare admin pages that fail with ad blocking before. Test your theory in a private session, it works for me when their site "has issues."
(I think in my case it was adding google metrics from the apps page.)
I would absolutely pay for this as a service. I just ran into an issue with AWS where there was a problem with my account, I was unable to send email from a specific IP in a pool of IP's bound to an ec2 instance.
The machine / IP combination has been working for years and then one day it just stopped. No emails from ec2-abuse, nothing in my support dashboard. After troubleshooting for a while I bit the bullet and ordered business support.
After cycling through three support agents over 48 hours Amazon said that they were filtering the IP because of spam complaints but neither the Abuse department nor my support liaison was able to see the block - it has been applied at the network level and the network team did not communicate the changes to the other departments.
I told them that I would not be paying for a support plan (over $1000) to resolve a problem that I did not have any notifications or alerts or any way to troubleshoot for myself.
It required talking to two manager and almost a month before they finally agreed that the billing should be refunded. After lots of support tickets, emails, and phone calls I got the money back - but man... An ombudsman would have been amazing.
I like this idea a lot. Some people below are suggesting that the ombudsman should be the last stop in a support queue if your problem isn't resolved, and that makes sense sometimes, but other times you can't wait that long!
So, all support systems should have a triage type system with a "nurse" having a constant eye on every new case that comes into the support system. When there's an emergency, such as the one associated with this post, then it should be forwarded to the ombudsman or some other emergency team immediately.
This sounds nice in theory, but you know that tons of people would just go straight to the ombudsman, thinking they can jump the support queue or bypass established process.
The "shit filtering" workload would be tremendous..
The way this works everywhere is you go through the normal support process until you reach top level support if the situation isn't resolve you go to the ombudmen. If you go before you tried support they will direct you to support first.
Yes I understand the process, but as I said that "they will direct you to support first" work would be non-stop..
Not saying it doesn't work, just that a lot of people wouldn't follow the process, and anyone who sets this up should be prepared for a lot of triage work.
Just had to chime in here, but fifteen+ years ago, Bezo's email was ombudsman like, with him or a top level person reading it.
Now, it's just another level of very poor, very scripted support.
I'm fairly sure Amazon has taken the (perhaps wise, in a business sense) approach of not caring if a small percentage of users leave, due to support issues.
The cost of keeping customers with certain support issues, greatly outweighing supporting them.
This is why you have to hunt madly around Amazon's webpage to find contact info, why all forms of help point away from contacting a person, including their chat being bots now, until you move outside of their scripts.
Just one data point but, sending Bezos a nice hello and airing my grievances has worked for me every single time.
Obviously, Bezos may not read those emails but his aides and assistants who do have access to his inbox and act on the emails on his behalf do inherit his complete authority.
The problem is that big companies don't care about giving quality support for their products, and for the most part they get away with it. From their perspective there's no problem to solve.
Your solution basically boils down to "companies are failing at escalating support issues well, so they should escalate support issues well."
My point is precisely that to me, "jgc" IS a random person. How the heck do I know who this person is.
It shouldn't matter, and it should not be required, that someone "known and important" within an organisation decides to start doing hands on tech support in social media following a PR disaster.
If "jgc" is actually someone important within this company then maybe after fixing this issue, they can then go fiox their tech support by setting up and ombudsman and get their PR disasters off the front page of HN.
You may be right, but knowing "who's who" is very largely how general business gets done. Buying services over the internet from an anonymous black box with no support is a recent disruption.
No, normally you didn’t have to know someone in the C suite to “get business done”. That’s totally unscalable.
What’s a recent development is the complete lack of support when shit goes south. Back when you were interacting with real reps you had people that could see when stuff was obviously wrong and escalate appropriately.
For a regular generic Cloudflare customer like me, for personal use, jgc is one of the random Cloudflare person. I have started a spreadsheet with his name, email, comment link, and my copy of screenshot of comment; just in case if I need to email him anything in future.
Steve@apple.com worked amazingly for me back when I bought apple products. Both times I literally get a phone call from someone who has already talked to my local store and has his proposed solution set up for me.
But that was like 10 years ago. Not sure if it's still good.
I can confirm, in those rare instances where it’s required, similar email addresses still work. Just keep it nice and polite. I’m sure they have manually maintained spam filters...
> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
Pretty much every technology company already have "ombudsman" for their important clients.
> That "contact a friend in the business who can actually get things done" is a necessary part of a large support organisation and it simply does not exist yet in any tech company that I know of.
I can assure you that every tech company has these "friends" available for their most important clients/customers.
Whether it makes sense for companies to make these contacts available to every customer is another matter.
> Technology companies need an "ombudsman" - a contact that customers can go to when the normal tech support processes have failed.
At least in the case of telecom companies they all seem to have executive response teams when the usual channels fail. Send an email to any of their Exec team members (CEO, president, CTO, etc) and it nearly always gets assigned to a special team that solves the problem. I can confirm this out of personal experience with at least AT&T, Comcast, and T-Mobile.
Maybe it could be something that is given to someone when their ticket is closed (or maybe after the first tech response... it depends on the company/ corporate structure).
That way the ombudsman has something to work with, and would slow down the barrage that would occur by having a such a public contact point.
I'm never a fan of 'pay then get refunded' for something that's not your fault, and is entirely out of your control.
You're missing the part where some people, that actually would need this support, literally wouldn't be able to find that money because of the difference in purchasing power of the local currency.
No, He's thinking about various developing nations that are trying to have a tech boom.
Phillipines is a good example; Average yearly salary is somewhere around 12k USD, Median is 8k USD.
That means if someone is doing a tech startup there, 300$ is somewhere between a quarter and a half of some employee's pay for a -month-. And given the potential for tight margin/cash flow of startups even in the US... it would still price a lot of smaller players out of the market.
I think the only sane thing you can do is price it at the cost it takes to review it. It will still be out of reach of some people, but at least its not arbitory.
> To avoid the Ombudsman being overused, you could give it a price of say $20, which is always refunded when the case is resolved.
HN actually acts somewhat like a crowd-sourced ombudsman. People who have an issue write a description and post it to HN. If enough people find it compelling, it makes the front page. Once it makes the front page, someone in authority at the involved tech company will see it, and try their best to resolve it.
You can post updates with any relevant information. Probably goes without saying, but if the issue has to do with my billing or address please don't post specific details without asking me first.
I will link to this comment from TFA for verification. (Edit: added to the bottom. If you need more verification you have my email.)
Edit2: I see that the domain is back in my account and listed as "Pending Nameserver Update". I don't think that's because of something I did.
Even if cloudflare stopped publishing the correct MX records for a domain, wouldn't Cloudflare still be able to send email to the MX server that had been specified? Seems like email should at worst be down for everyone but Cloudflare.
If the only contact information the registrar has is the @ of the domain then that likely mean that the registrar is failing the contractual obligations that exist between the registry and registrar. While I have not read the exact contract that exist for .com registrars, I am confident enough to say that you can not do that.
The OP made it clear that he used a gmail account for contact so there was no excuse for him to have not received a message regarding the deletion of his domain records.
I think it's pretty common advice that anything domain-related needs to be a different email. I've heard that, and I've only ever admin'd my personal domain. For exactly that reason I registered for Cloudflare with a gmail account.
I can tell. Cloudflare has been a delight to use up to now and I'm grateful I'm getting this kind of support when I've only used your loss-leader services.
This is probably obvious but you're getting this kind of support because they're getting terrible PR by virtue of your post being at the top of HN. Not because he's such a great guy that cares so much about the customers. Not saying he isn't a great guy, or that anyone at cloudflare actually wanted this to be your experience.
If he’d emailed me directly I would have done the same thing I did. I emailed the head of engineering and support and asked for an explanation. I then jumped into the relevant chat room.
I do this sort of thing all the time. Sure, it’s unfortunate this is #1 on HN, but shrug. Fixing the problem and figured out what happened is important.
The more important question is what will be done to prevent this from happening again to somebody else (obviously this doesn’t need a response here, I assume an internal investigation is underway and this will be publicly communicated after its complete).
I don't think he's saying that OP should've known to email him, just that he would've done the same thing over private emails where nobody was watching, as he is doing here publicly.
To be fair this seems to happen with a bunch of tech companies, such as Stripe too. I wish more companies took customer support seriously.
I remember about 4-5 months ago I spent like 2 weeks going back and forth with Stripe's regular email support trying to understand their docs for SCA.
I kept getting a new rep who repeated the same things the previous reps were saying, which also had no bearing on what I was asking. It was basically a copy / paste from a script loop.
Then something negative about Stripe was on the HN front page and I happened to comment about a bad experience with the new SCA docs.
Within a few hours I was put in contact with a lead developer from Stripe who went as far as creating custom flow charts for my use case that wasn't covered in the docs and it was a pleasant experience, where "pleasant" felt like the person receiving the email was reading the words I wrote instead of just skimming them and pasting a boilerplate response.
But it only happened because of the HN comment. If that thread never appeared on HN, I'm not even sure I would still be using Stripe.
> was a pleasant experience, where "pleasant" felt like the person receiving the email was reading the words I wrote instead of just skimming them and pasting a boilerplate response.
With many other companies I would agree with you, but in this case it is actually just because he's running a tight ship. I've had friends who've reached out directly via email and received the same level of support.
You mention loss-leader services, but the model is no doubt setup to funnel you towards profitable services.
I switched a personal domain to them, but it won't be long before I get my employer to move their resources over to them, assuming I see this all play out well.
John, you guys seem pretty awesome about posting root cause analysis and being transparent, can we expect to see at least a comment (if not a blog post) summarizing what took place here?
Cloudflare followed up with me the day of and resolved the issue at least for me. I'm unsure if it would be appropriate for me to post the email they sent.
Maybe contact support first, figure out what happened and why before writing the blog post?
Maybe it's a cloudflare issue, maybe it's an honest mistake, maybe they are bad at customer service... either way, it's not great to shame a company before you even open a support ticket or talk to someone to find out.
Essentially I felt that this was alright because when I filed a ticket I was informed that I should expect a long wait and that they recommend that their non-business customers post publicly on their support forum for crowdsourced support because that leads to faster replies. I was unable to log into that forum, and I suspect that may be because the way they set up SSO between the forum login and their main login may have failed in Firefox (with all tracking prevention and ad blockers disabled).
I felt that if a company invites me to ask for support publicly on their forum to save on customer support costs it's reasonable to talk about the issue in another public place.
Oh, the horror of recommending a public forum. (/s, of course.)
A fair number of questions aren't unique - product questions, how to use an API, etc. Someone may have asked a similar question, in which case you'll find an answer, find it faster than it'll take to hear back from the support team, and it deflects an unnecessary (already answered) ticket. That should be a win all around.
Now if you do have a novel question or something account-specific, by all means, open a ticket. There you'll get replies from people who can look up your account and give you specific answers.
The ombudsman tip in this post doesn't make a whole lot of sense when the normal support process wasn't really given a chance before making the blog post.
When your entire site and suite of applications are offline, especially during business hours, waiting who knows how long for a support queue isn't really an option. Outages like this can kill a business or cost millions.
(op here). To be fair to Cloudflare a business with millions on the line should be paying a lot more money to Cloudflare than I, a college student with significantly less on the line, would. If I had the option to pay $1,000 right now and get everything back up instantly I wouldn't take it because it isn't worth that much to me.
I realize that... the issue isn't necessarily tethered to your paying status, and it could just as easily happen to someone with a high value business account.
1. If you write into support as an Enterprise customer, you're basically getting an automatic response.
2. Enterprise customers can dial a direct line to support and have this under investigation within minutes.
3. Enterprise customer have a dedicated account manager, and in most cases a Solutions Engineer helped get everything setup. Usually Enterprise domains get locked, so they cannot get caught by linting type services. The likelihood of a domain being removed under those conditions is very low.
4. The most common cause of this type of thing is the Name Servers no longer pointing to Cloudflare, no one noticing for a while. Cloudflare periodically checks to see if a domain is still using CF name servers, and if they aren't they get moved as the assumption is they are no longer on Cloudflare. I don't know that this is what happened here, but it's easily the most common issue on lower tier plans. Enterprise companies often have people monitoring their infra, thus they catch this before Cloudflare conducts the removal of the domain.
Source: used to work on the Cloudflare support team.
I think it is a bit harsh to say that the OP was trying to shame anyone. He did post a support ticket. When your domain is "off-line" it is literally a shit hit the fan moment, and no one likes to be given the run-around.
To combat, hey my issue is always a Sev1 ticket, one can probably institute something like, here is a red button and if you click it, we will charge 100$. If it is indeed an issue that caused you to lose 90%(say) traffic and it was our fault, we will return the money.
Microsoft used to have this policy (maybe still does) for some support options. It cost $100 to open a case, and if the case was the result of a bug in the product, you got your money back.
Imagine how you would feel though if you were in OP’s shoes, where your DNS entries disappeared and client emails lost, through no fault of your own, but support won’t even listen to you until you pay money.
I would trust Cloudflare to pay me back, and I'm already putting something on the line. If it really was my fault this is going to be very embarrassing.
Edit: In case it isn't clear from context I'm the OP so I'm pretty sure I know what I feel.
Except you can bet that a lot of cases will end up with a battle of back and forth emails arguing over whose fault it was and who should pay the $100.
The vast majority of support tickets are the customers 'fault'. They'll be things like setting up DNS records incorrectly, believing they don't need to pay a renewal with their old registrar just because they moved their DNS over to cloudflare, frustrations over cached resources, etc.
(op here) My belief is that I completely transfered everything to cloudflare, paid in advance, and didn't touch it. If this was as much my fault as any of the common tickets you listed I will deeply regret having posted this and will apologise to Cloudflare.
In the blog post he mentions that Cloudflare recommends posting your problem to their open community forum to try and get things resolved. He couldn't access their forum, so he posted it on a different one.
For me and my of my customers, having your "entire cloud deleted" is like... the #1 nightmare scenario.
So why does this capability/function even exist for active accounts at CloudFlare? It sounds like the OP fell victim to what is essentially a regular process.
Or to put it another way: No amount of explanation or assurance is ever going to make me feel comfortable with my doctor having a handgun as one of his medical instruments.
It appeared to me as though they deleted my domain registration from their system. That's mostly why I freaked out. I wasn't sure if I'd even be able to transfer my registration easily.
However, on the advice of someone from HN, I found that if I went through the flow of linking an external domain to my cloudflare account the registration details came back.
Oh wait, he was just sleeping? Ah, well... "oops".
There is exactly one reason you'd delete customer data without their active involvement: They're overdue on their bill and have ignored all warnings and communications. That's it. There are no other reasons.
Feel free to disable things if they're a security risk, but Do. Not. Delete. Customer. Data.
Ugh! I may have interpreted the comment I responded to wrong, and replied to my own strawman. I think you need the power for the user to delete records (obviously) but this could mean a bug in the system could cause the records to be deleted. But it sounds from what you are saying that they sort of "intentionally" deleted the records (not sure if a human had the intent or an algorithm?) but if intentionally deleted yes that sucks, there is no reason for that I can think of, given what they do is about handling DDOS.
Unless the mechanism exists for legal reasons, i.e. they are forced to take some sites down by the government or law suits.
Having the CTO offer up support in this way is heartening. Especially because infrastructure suppliers are getting more and more centralized, which leads to possible SPOFs even with individual company distributed environments.
The problem is that big companies don't care about giving quality support for their products, and for the most part they get away with it. From their perspective there's no problem to solve.
Your solution basically boils down to "companies are failing at escalated support issues well, so they should escalate support issues well."
Did you consult to your supervisor (or anyone with authority) to be able to bypass the support process (if there is any) like this? If so what was the response? If the response was negative, how did you convince people? After things resolve, can you kindly post how many spam or unrelated emails you receive so that it will be an example to the industry?
I'd like to put my skepticism on hold and blindly believe that your post is a reflection of pure concern and not just a PR stunt for damage control.
It's hard to understand criticism when it is not put into words.
From my perspective, a developer from the company the article is about arrives in the comment section. I ask an honest question about the communication process internal to the company. I am told this developer is the CTO and is supervised by the CEO. Then I state the fact that I would appreciate if we also knew what the CEO thinks about this situation.
I don't literally expect answers from the people I addressed and that's OK, they are probably too busy working on the subject matter. If they take time and write one, that is really admirable and I'd be thankful.
What I don't understand is why it is considered inappropriate to be curious about these topics, as judged by the amount of downvotes I received.
Maybe the tone I intended for my post got lost in translation, since I'm not a native English speaker.
It comes across as entitled, and also just strange and unnecessary. Like, why would you need or expect to hear directly from the CEO of Cloudflare when we're already hearing from the CTO? I'm not sure why your reply here got downvoted as well though, as I found it helpful that you explained you were simply interested in the CEO's position rather than actually expecting a response from him. So, just trying to illustrate how the initial comment came across to me, and perhaps others.
"I'd like to hear CEO's opinion" probably doesn't mean "I wonder what the CEO thinks about public communication by his/her colleagues inside internet forums". I'm reminded English is hard even when I think I have a basic understanding of common speech patterns (parlance?).
Correct. In fact, even the second phrasing, "I wonder what the CEO thinks..." could easily be interpreted as expecting a response. It seems you'd have to really bend over backwards to avoid that, perhaps explicitly saying something like, "Obviously I wouldn't expect him to respond here, but I'm curious whether the CEO shares this view." Or something like that.
Cloudflare is a solid company with a good reputation and has done a lot for general welfare of the internet. The staff there generally care about customers and it seems like they're trying to figure out what went wrong here, and of course preventing negative PR is always a bonus.
There's really not much of a process when you're a top officer at a company, you have the freedom and responsibility to exercise your judgement and do whatever it takes for the good of the organization.
> BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.
I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a company execs time and I am not even sure it's a good way to deploy resources at Cloudflare.
The reason is people will tend to (as a rule) do as little as they can themselves but then use as a hammer the court of public opinion to get something resolved.
You say 'ever find yourself so frustrated with Cloudflare' but you know that in itself is different for different people. What will happen is you will get people using you as a help desk and then after you don't help them as quickly as they think you should they will then follow up with a post, comment or story about how you did nothing.
Separately if someone is posting publicly about an issue (as this person is) and if you can verify that it's actually coming from the customer (I mean who says it is actually?) I don't think you need them to say it's ok to resolve online. In fact to me it's the opposite. You take the time to reach out publicly and you take what follows good or bad even calling you out (the customer yes you can do that by the way) if you think they didn't put the appropriate effort into finding an answer.
I disagree with you. It matters enormously that people like me are available. Sure there are time wasters who’ll send me email. But I don’t care. Dealing with a small number of real customers doesn’t take a lot of my time and matters.
Everyone optimizes for the worst case. They think “if I give out my email I’ll get tons of useless email”. I can assure you I get 10x the crap sent to me on LinkedIn than via direct emails from customers or others.
My little firm is nothing like the scale of CF but I do the same and have dropped my email address around a fair bit. I do get some pretty interesting missives that make it past the filters but signal to noise is very high in my inbox.
I deleted my Linkedin account well over a year ago and I still get emails from them saying my profile is being viewed. Tossers.
> I know that people will think it's great that you are doing this and I also know that you think it's good (for you) to have a feel for the issues that frustrate every day users. But I think it's not a great use of a company execs time and I am not even sure it's a good way to deploy resources at Cloudflare.
As a counter example, Jeff Bezos (whose time may be worth more than anybody else's) famously audits his email for customer complaints and occasionally derails an organization for a day or two in order to figure out what happened. He stands behind this practice and has said that he often picks out cases where the anecdotal complaint is counter to data that he's been presented, and that more often than not the anecdotes are correct and find a shortcoming in the data. IMO it also demonstrates a culture of caring and following up about anecdotes to others whose time is worth less than his own.
> As a counter example, Jeff Bezos (whose time may be worth more than anybody else's) famously audits his email for customer complaints and occasionally derails an organization for a day or two in order to figure out what happened.
Is that why Amazon is growing more and more notorious for selling fraudulent items over the years?
Not a fanboy, still downvoted your post because it's incredibly whiny and does not contribute anything. All you're doing is accuse of censorship without evidence, you say they screwed up "so many times" yet fail to show one example, and then you preemptively accuse people who would dare downvote you to be fanboys.
Sorry, but that comment is noise. Mine is too, but hopefully it helps you see things more clearly rather than let you pat yourself on the back thinking "HN is full of fanboys anyway".
When you use the phrase "un-American" please remember that you're talking to an international audience who see being American as a completely orthogonal issue to any type of behaviour. It makes you seem parochial, and slightly nationalistic.
To be clear, the parent comment is written by someone who is upset that CloudFlare terminated service to violent neo-Nazis. The fact their language is nationalistic is unsurprising.
Violent? I'm not aware of any lawsuits due to violence by the neo Nazi founder and I despise neo Nazi speech as anyone else.
You completely missed the point or maybe you just don't give a s++t about 1A rights.
The other issue is they claimed to be hands off to customers, obviously within the bounds of the law.
The sad truth is this - distasteful as DS/Anglin is, he was not running afoul of the law. Matthew Prince unilaterally decided to reverse 200+ years of 1A jurisprudence for one of its customers because he disagreed with their views.
These Matthew Prince types are the same assclowns who sue Christian bakeries for refusing to adorn wedding cakes with 2 male or 2 female figures when they exercise their 1A rights to discriminate based in faith preferences.
The height of hypocrisy displayed by Cloudflare was galaxy sized, despite it being legal.
> "the same assclowns who sue Christian bakeries for refusing to adorn wedding cakes with 2 male or 2 female figures when they exercise their 1A rights to discriminate based in faith preferences."
How is this different from refusing service to neo-Nazis?
It seems like you're arguing that the first amendment should apply if I refuse to sell a cake to a gay couple, but not if I refuse to sell DNS service to neo-Nazis.
Your argument is not philosophically consistent, and it's clear you are seeing persecution rather than philosophy. Under Conservative philosophy, it is clear that CloudFlare has a right to deny service to a customer it doesn't want. It's chosen to not provide services to violent neo-Nazis. Get over it.
Please explain to me how it's un-American for a company to choose who they want as their customers? The typical Conservative position would be that businesses have a right to choose who they wish to serve, and denying service should not be an illegal act as their prospective customers can go elsewhere to find another business to serve them, which is exactly what those disgusting hatemongers did.
Occasions are rare where I get to say "hey, I built a thing that might help here!" - so forgive me as I take this opportunity with both hands.
Whether this was a bug or a rare protective mechanisme, there will be times when your DNS provider makes a mistake and removes records. You mentioned in your post your DNS isn't hard to reproduce, but how certain are you that _all_ records are restored? How long do you have to fight DNS issues before it's OK?
I built DNS Spy [1] for this exact occasion. It monitors your DNS for any changes made, keeps a version of all DNS records (current & former) and allows you to restore/download a BIND9 zone file for your zone. You can easily import this into any commercial DNS provider or in your own BIND9/PowerDNS setup.
I would love to hear feedback on how DNS Spy could be improved when DNS disasters like these occur!
(OP here). That looks really useful. If I was running a real service I would definitely look into it. Because this is just the personal website and email of a college student I don't think I could justify the expense when using something like Uptime Robot to monitor if a single record points to a web server would probably give me close to the same reliability.
If you're a business, whether it's a SaaS or "just" a marketing website for your brick & mortar store, I think it's crucial to have back-ups. Most people think of backups as files, database dumps, previous versions, etc of their website. But the configuration data (in the form of DNS) isn't often considered.
You're tech-savvy and can restore your DNS records because you know yoru servers' IP address and your MX records, but who else could do the same?
How do you get all records for domains that don’t allow zone transfer (most don’t)? I’ve always thought it was impossible to get 100% accurate results with normal DNS queries (ex: ALL).
YOU as a customer import them. This is not a DNS spider service like https://dnsdumpster.com/ which also will not be 100% accurate in most cases, but finds a lot more for most domains.
Nice! A few years ago I whipped up something simple using dig, diff, cron and some bash scripts. It's handy to get alerted when something changes, and I've definetly caught a few unintentional changes.
Looks really useful and fulfills a very important purpose. What good is all your backups if you can't get your services back up due to missing DNS configuration.
There's a number of Cloudflare folks who are HN regulars, so hopefully you'll get some answers. Hopefully it's something they can reverse.
But as a general reminder to everyone (I think this is an unfortunately common problem from a number of companies): If this is how your company handles account issues, you're probably wrong. Whether it's automated or manual, a user should be able to access all of their own information even when you decide to no longer provide them service. And you should test and retest the ability for people who you now deny service to transfer out.
FWIW I recently evaluated a few DNS companies after Namecheap ballsed up our MX records in a similar way.
I actively looked for someone we could pay money to, so we are their customer (as opposed to being a free tier user, effectively a cost)
The winner was DNSimple[1], who do exactly 1 thing, and they do it extremely well. And they are small enough to not take themselves too seriously[2], which I really appreciate.
Oh and their normal support channel is email, and everyone in the company takes a turn. I tested out their support before signing up and quickly heard back from a competent engineer, so they passed that test too.
I am sure AWS, OCI, GCP, etc. all host scam websites with varying degrees of removal efficiency. What cases are you referring to specifically? Did they state they were not going to take these sites down or what was the context that you object to?
In this context, that sounds like an endorsement, honestly. If we're discussing providers that are willing to kill your services too easily, then saying that a provider is unwilling to cut service even to problem customers sounds like an amazing reason to use them.
You listed their good points, the other poster listed some counterpoints. The one post is no less relevant than the other in a discussion about possible DNS hosting options IMO.
Though I think the post would benefit from some citations to improve its relevance/usefulness otherwise it is little better than personal opinion/conjecture.
Unless you are specifically questioning the relevance of hosting spammers, on which case: If that is true (again, some examples would be helpful here) and you intend to host your own mail servers via their services not just the MX records pointing to other mail services, you could find yourself blocked by association at some point. False positives are a big problem in this area and can be much admin to clear up.
I use no-ip as dyndns for my home ip, so I can log in at home from outside. Recently at work my putty failed to connect, so I figured my internet line was down, it happens.
Came home, internet works fine. Everything looked just fine.
Back at work next day still can't connect. So I tried pinging, and I immediately see that the ip my home hostname resolves to is not what my ISP has. So I go to nslookup and try a DNS server I know (another local ISP), and it resolves to what I expected.
A bit of checking later I find that at work they've started using OpenDNS, and OpenDNS has blocked all of no-ip due to malware and spam.
I think they're mad that DigitalOcean's IP range shows up in their ssh logs with failed authentication. A lot of people think that it's the ISP's job to regulate all traffic on their network, judging from the comments here, DigitalOcean at one time or another has failed to do that.
I host all my personal stuff there, including something that updates their DNS via an API. They've been great to me.
We used ns1 at my last job, they were indeed great to us. We moved from self-hosting DNS because the DNS servers would randomly become unresponsive and would start returning fake records. After switching to ns1 and getting our first bill, we realized that a lot of our network equipment apparently did a DNS lookup for every log line. This resulted in an exceedingly large bill, which ns1 happily reversed (we did fix our stuff ;).
Since then I've started using CoreDNS, which does seem easier to monitor. (I don't know if it's faster or more reliable, but it has a lot of ways to figure out what's going wrong. As it turns out, DNS causes people a lot of trouble these days when they use it for service discovery, and their services discover each other hundreds of times a second. So that's why DNS servers grew APIs and observability features.)
I did the same after getting tired of NC's DNS interface. I host a few client sites with Netlify[1] anyways and moving over to their DNS (NS1) has been a breath of fresh air. It is free but they do have some paid options and the is UI dead simple which should be a requirement. Feel fairly confident I can rely on them to not muck up DNS records as this is critical to mail systems, websites, etc.
Two years ago there was a moment where I was close to working for them too so I always try to use their products where I see fit. :)
I haven't needed to talk to them much, but one time I tried to add a .ninja domain, and there backend wouldn't handle it. I emailed them to report the problem at 4:49 p.m. I got an email at 7:09 p.m. the same day (2 hours 20 minutes later later) asking me to try adding it again. [1] When a free service fixes your problem in a few hours, they get +1 gold star from me.
[1] I just checked my email to look up the actual times. This was on Mar 15, 2017.
Love the idea of this. I was about to switch providers then discovered that they are 3-4 times more expensive than my current provider for small-mid sized websites (currently on dnsmadeeasy.com)
Be wary of being part of something that is a cost center for the company instead of a profit center.
CloudFlare is selling domains at cost. That means they are not making any money from being a domain registrar, which means they will do everything to keep the cost of doing it as low possible to themselves. This means lack of customer service and use of ML dragnets for "anomalous" behavior.
.com has a price floor of $7.85. Most registrars seem to target anywhere from the $9.99 - $14.99 range for registration because, as far as I can tell, there is no real differentiation outside of price.
Sure, I could spend $lots to get a dedicated account rep from MarkMonitor or CSC but that's not really feasible for my personal site.
Are there really any registrars that hit a reasonable price point for individuals and offer service beyond bargain basement? Because if so I'm doing some transfers this weekend.
Namesilo has been a great, cheap registrar for me for many years and has always had privacy included for free.
I tried several of the lower price registrar's back in the day, and they all sucked in their own way, despite me not needed anything except the thing to just stay registered.
One or two would change the price of their domain privacy, most renew the privacy for like 3 dollars and then send you the renewal email that your domain needs to be renewed, one of them used to charge me separately like 80 cents from some weird Canadian shell company...
I actually have a domain still with probably the biggest "cheap" provider, and they now have a thing where you are supposed to keep a deposit in your account to cover automatic renewals. Just charge my damn credit card guys, please.
So I'm saying namesilo all the way. Only one that hasn't ever pulled any shenanigans on me.
I'll speak for myself and say that all my domains have been with Hover for well over a decade now, and the times I've had to deal with their customer service, they've been excellent.
In fact, I even had to call them once, and I got a human almost immediately, and that human was able to resolve my issue while I was on the phone.. I don't recall the exact issue and I'm sure it wasn't anything major, but it was still nice.
So yeah, Hover. They're nice. And I think their prices are decent?
Not true. CF claims it is $8.03. Let’s say it’s $7.85 + .18 tax or something.
You don’t deal with the registry. You deal with a registrar. Just as they can charge a markup, they can give a discount.
Registration could be free. It would mean a loss of $7.85/$8.03 per year. Maybe they make it up by selling ads on your domain. Maybe they use it as a loss leader service.
There is no floor.
Even CF’s at cost pricing loses them money. It’s not free to maintain the services.
One of the reasons I've stuck with GoDaddy is 24/7, American phone support. Their .com pricing is closer to $20 for renewal at this point, but I've called them at 2 AM before and gotten help.
From previous research, at least, most domain registrars have ticket support at best. I did move all my "less important" domains to Cloudflare for cost savings recently, but they have my most important domains.
DNS was a good idea but now there are organizations that have the power to arbitrarily take control and even remove your domain names and records. We really need to come up with a peer-to-peer solution and take back control of the naming system from these authorities.
>DNS has become frighteningly unreliable. Here are previous stories that show how it is possible to lose access to your domain for no fault of yours:
The second story you posted is about a user who forgot to renew their domain and did not wish to pay the overly-inflated fee to re-register it while it was in the grace period.
I hold no love for any registrar that jacks up rates for getting back an expired domain and agree that they should have sent a reminder email, but describing this as someone "losing their domain through no fault of their own" is, frankly, incredibly misleading.
The user:
1) forgot to renew their domain
2) had full right to recover their domain but objected to the price
3) had full right to transfer the domain out to another registrar for the original 15EUR price
and
4) eventually got back full control of the domain
I looked into self-hosting DNS and it doesn't seem like that big of a deal as long as you can ensure uptime to be honest. If you set up the two first on different hosts and possibly have #3/4 being cloud providers I think you're pretty good.
Does anyone here have experience with running their own DNS servers for their domains?
Cool, do you have any software to recommend for a casual power user who would like something a bit more lean than BIND from a configuration perspective? pdns seems nice, and there's an official terraform provider for it as well.
I've been self hosting for years. Currently using online.net secondary DNS service as my 3rd or 4th backup NS. They've lost my 10€/month box once (shitty cheap intel avoton hardware with everything soldered on I suspect) but the domain still resolved fine. I had backups and restored it in a day. You can also use a VPS image to self host DNS. Some providers offer automatic or manual snapshots. Hetzner comes to mind. They've annoyingly asked for a copy of my id card (welcome to Germany), but their services are fine.
You don't even need multiple servers (especially if both your website and mail run on the same server), it's a misconception debunked by the author of djbdns:
These are some valid arguments against third-party providers. Assuming nobody's perfect, I still see benefit in having redundancy for cases of downtime. Could just be a duplicated setup on a separate physical network (if all your DNS records point to the same network your DNS is on, I guess it's pointless with network separation, but I don't think that is very common).
The question is whether you're violating the standard or doing something unreasonable. Clearly DNS can't prevent you from using one server, just like it can't prevent you from using one network you own.
The main problem which people seem to have is that their domain name registrar decides to pull their domain. Luckily, there is ample competition in this space, my place of employment included, which should make it reasonable to pick a place which 1. doesn’t do that and 2. has reasonable real-live-person support.
Of course, if the registry (i.e. the TLD) wants your domain gone, you are out of luck whatever you do. If this is a concern then you should pick a TLD with what you consider reasonable management. There are a lot of ccTLDs and gTLDs to choose from.
Therefore, what you absolutely shouldn’t do is to pick whatever domain registrar is either cheapest or largest, and pick whatever domain name which happens to look cool and be available. Both are recipies for potential disaster.
Indeed.
I am curious to see what comes out of attempts at decentralizing this such as Handshake[0] and ENS[1]. I think I saw something similar with prominent backers come up here on HN the other week but can't recall it now. Namecoin[2] was very early on this.
I don't work for Cloudflare but I work for another large company that also manages domain names and DNS records. I don't want to risk the possibility that my comment could be interpreted by my employer as conflict the interest.
I had this exact thing happen to me as well, but wrote it off to having been compromised (fortunately I was only using Cloudflare as secondary DNS servers on a non-production account and am not using them as a registrar, so I only noticed months after the fact). I think a major reason going with someone like Cloudflare for DNS in the first place is reliability and availability and this does not speak to that.
Every related incident seems to be due to either nameservers temporarily/incidentally chanced away from CF (and CF's service not re-checking it perhaps) or the registration billing failing (which doesn't look to be the case since registration expires 2021[0]). The latest change to the domain was about a week ago[0], so if that was when it was transferred to CF, it might be the first scenario.
> Because Cloudflare deleted my domain registration I can't change the status from clientTransferProhibited through their dashboard so I don't think I can even leave.
Unless something else happened, deleting the zone from your account doesn't affect the registration. Re-adding the domain will instantly allow you to view the registration info and likely transfer away; this would only not work if the zone is banned for some reason.
> Every related incident seems to be due to either nameservers temporarily/incidentally chanced away from CF (and CF's service not re-checking it perhaps) or the registration billing failing (which doesn't look to be the case since registration expires 2021[0]).
The changes a week ago involves adding and deleting TXT and A records only. Cloudflare manages the nameservers I use as my registrar and I never changed them from the default. I just confirmed all of that in the Cloudflare audit log.
> Unless something else happened, deleting the zone from your account doesn't affect the registration. Re-adding the domain will instantly allow you to view the registration info and likely transfer away; this would only not work if the zone is banned for some reason.
i think that it would be rather unusual to update timestamps in whois (which i guessed the parent poster was referring to) based on updates to in-zone data. A transfer would be an example of something that would change information in whois and thus update the timestamp noted there for the latest update. it is sometime possible to infer the date of the latest change of in-zone data because the serial of the zone is often constructed by using a date and a counter. But that is actually just convention and not reliable. Its also unlikely the parent poster was referring to this.
Huh. I definitely didn't make any such changes within the last two weeks. Maybe the whois date got changed because of something opaque and internal to Cloudflare?
yes that is most likely what happened. A change of the nameservers with authority for your zone for example or updates of DNSEC keys would trigger that too, i think. But most commonly it probably happens when the domain gets a renewed registration period or the contact details for some person changed.
btw here are the dates referred in whois for your domain:
I believe the latest change happened afterwards when on judge2020's advice I re-added the domain to cloudflare which I think triggered a NS change back to them.
> However, I'm unable to log in to their community forum. When I click the login button I'm redirected to my dashboard, and when I then click Support on the dashboard I'm redirected back to the forum without being logged in. I suppose it's possibly an issue with Firefox blocking cookies (although I disabled tracking prevention) so it's possible this part is partly a problem on my end.
I'm into issues like this more and more, where you run into some strange behavior on a website and you wonder "How did this ever make it into production?", then you open the website in Chrome and the flows work fine. I worry that Firefox is becoming less and less viable.
This is not Firefox becoming less and less viable. This is developers caring less and less about supporting older browsers, less capable hardware and, I guess, long-term maintenance in general.
Just had a similar case today: My Mom tried to order something online on her old Android tablet - and it didn't work. She blamed the tablet for it, saying "It's just too old, it doesn't work correctly anymore! I used to be able to order stuff on this website". I had to explain to her that her tablet is still working fine, it's just the website that is broken because it's not supporting her device (or browser) anymore. Shockingly, she listed quite a few websites, which she has used for years, which have stopped working for her in the past few months and years; all of these she mentioned as evidence that the problem must be her tablet - not the websites. When I opened two of the sites she mentioned, I wasn't too surprised to find very shiny, very modern single-page applications (with service workers registered and even WebAssembly used on one of them)..
So when you are creating a modern web app, please don't just test in Chrome on your new MacBook Pro. Think about your Mom. Ask yourself: "Is this still gonna work on her crappy old device?"
Well, it's also a problem of device manufacturers dropping support for devices too quickly. There are still android 4.1 devices sold on amazon, and you really can't expect web developers to support that.
The manufacturer should be required to support it for the full lifetime of the device. Especially since your mom uses it to order stuff, which usually includes some pretty security sensitive information. I think you are putting the burden on the wrong party.
Well, I haven't analysed the exact technical reason for why submitting the order failed. But I'm pretty certain that submitting a HTML form is a solved problem in web development.. Or at least it should be. I haven't tried submitting a form with an async fetch from a web worker that communicates with a redux store implemented in WebAssembly yet (or whatever that web app is doing..).
If the order site is just submitting an HTML form in the old way with credentials stored in a cookie (also the old way) that would probably be open to trivial CSRF attacks.
If it is somehow checking for support for SameSite, Secure, CSP or any of the other mechanisms that have been implemented in the last years then it might fail. Or they might be using mechanisms that work around the problem that those three are supposed to help since they are not available in older clients, but just don't have the resources to test the random android 4.12 version that you use. I think it should have a proper error message if that is the case.
But I feel like you are pointing the finger in the wrong direction. I try to build my apps without extraneous fads, but keeping a webapp secure (in other words keeping up to date with the latest protections) does not mean "submitting a form", and it does not mean letting any old client lacking the required protections through.
It also does not mean doing "WASM compiled redux reducers in ES6 module workers authenticating over JWT to send gRPC commands to a kafka broker talking with ingressrouting over anycast and a internal service mesh with mTLS3.9 auth using curve9999.9, token binding and Wireguard to secure internal communications over a VPC-less multi-cloud k8s cluster that uses Multi-Raft, Single-Paxos to have a single, distributed, disputably non-consistent CRDT-consensus algo over blockchain RS-232".
So, yeah, I'm not for fads over usability in tech. But I'm also not for supporting insecure clients just because the manufacturer of those clients doesn't give a shit.
CSRF doesn't require any javascript to defend against. Having a nonce inside of the HTML resolves that completely. SameSite and Secure cookies also don't require any javascript, just some extra HTTP headers. I don't think security justifies this.
> Well, it's also a problem of device manufacturers dropping support for devices too quickly. There are still android 4.1 devices sold on amazon, and you really can't expect web developers to support that.
Are you kidding me? If you're looking for shiny stuff to add to your resume, yeah, you can't possibly support those! If you're an HTML5 game developer, yeah, gotta use the latest and greatest. But if you're in the business of selling shoes, why do you need anything newer than Android 4.1 in order to process the transactions?!
The latest version of Android 4.1 was released in 2012. The latest version of chrome for that seems to be around 23. A lot of android 4.1 shipped without TLS 1.1 by default. So you are stuck with SSLv3 or TLSv1. Those are just examples, there's a lot in security that has changed since 2012.
A lot of basic security was missing back then, asking anyone to enter any security-sensitive info on those devices now is like telling someone to use WinXP and IE8 for online banking. Besides that the great variety of devices that shipped on versions v4.1-v4.4 and with their different hacks/customizations it is simply not feasible to ask developers to support them years after many of them cannot be had to even test a website on, let alone debug it.
A couple of things..
- The tablet is ~5 years old. Even if it was 8 years old, I think you should still be able to order a pizza online.
- TLSv1 is far from plaintext. But in any case, the TLS version doesn't matter in this case. The site was loading properly.
- Even if the TLS version was outdated, are we now banning people from using the internet for using an older device?
- Submitting the form didn't work, the navigation was partially broken. It was probably a JS issue, but I don't know.
- The moral of the story, to me, is that, as a web developer, you should also care about those $few% that don't have all the green boxes on caniuse.com. Depending on how many users your site has, this may just be a few people or it might be millions - among them, possibly, your Mom.
This is frightening. I just started the process of moving all ~60 of my domains from Amazon Registrar + Google Cloud DNS to Cloudflare, and will definitely wait until somebody from Cloudflare chimes in here to clarify what's going on.
Are you using physical U2F keys for your Google or Amazon accounts?
Cloudflare does support standard TOTP-based 2FA like most people use for Amazon and Google. So whether or not the lack of U2F support should matter depends on whether you actually use it elsewhere anyways.
I've been planning too soon, am also now going to wait to see where this goes. DNS is obviously a critical system and I don't know if I can trust Cloudflare now. I'm not a big fish that can make noise. I'm an easy victim.
This is what I do. Domains are at Porkbun and domain registration is the ONLY thing I use them for. DNS and caching are at Cloudflare. Hosting is wherever makes sense.
Consolidating domain registration, DNS hosting, and site hosting under the same account is a terrible idea and a big risk IMO. Always ask yourself “what if this account gets banned?” All of the big tech companies have automated systems that could turn you into an outlier with no support.
How many of your sixty domains are business-critical?
Cloudflare's Domains service is new, and some of it's management tools are lacking, but I also moved most of my domains to it over the last year for cost savings. I'm thrilled with it, but I'm still keeping a few of my most critical domains with GoDaddy. (Hate them all you want, but GoDaddy hasn't screwed up my domains in well over a decade.)
You may be able to save a lot of money without risking your primary domain that you route email through.
1. Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working.
2. Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account.
3. Be able to jump ship easily, have backups of your zone, already know where you will transfer to.
> UptimeRobot and Hetrixtools are good starters with generous free tier
Are there any open source status pages/monitor programs that have build-in checks for HTTPS, DNS records (ipv4/6), arbitrary port checks, etc? I'd rather just setup a status page/alert app on a $5 minimal DO/Vultr node and self-host/support/contribute to a FOSS program than use a commercial provider.
Nagios. Or its descendant with a better configuration language, Icinga2. They're fairly easy to do a minimal install and configure in a container or on a VM.
You need to host across several nodes in different geographic locations and data centers to resist network splits. Then you need some way to slowly roll out upgrades to your monitoring platform over time.
I'm just talking about my personal infrastructure. If I host my crap in Vultr or Linode, I should be able to buy one cheap node on another provider just to run a simple status app: something with celary or sidekiq jobs to check my other stuff and intervals and generate a page with some red/yellow/green dots.
If you want email or text message alerts I would assume that's a complicated enough system you would want uptime alerts on it, and so on recursively ad infinitum.
If you can set up nagios (which one would probably consider an interesting evening challenge if you were already willing to go for your own monitoring droplet) setting up pushover or amazon sns (for sms) should be easy enough.
FWIW, a lot of cellular providers have an email gateway for delivering SMS messages. There's also paid SMS gateways, and options for providing arbitrary push notifications to smartphones.
This was a few (3) years past, but they accepted root@localhost sendmail messages just fine in most cases, and delivered alerts within a minute or two of sending. We didn't rely on this long term, but it was a "good enough" first pass.
I'd probably recommend using one of the gateways (or a more fully-featured service like Pagerduty) for more serious businesses, but for personal use (or where an outage detected the next day isn't crippling), it's remarkably useful.
I would try to set up a completely open source monitoring setup just for fun, but once I'm paying for SNS I personally would rather just pay epsilon more and buy/rent the whole system. I get that may just be personal taste. I absolutely don't trust myself to run my own highly-reliable mailserver to send status alerts.
> Setup up monitoring on your critical domains. UptimeRobot and Hetrixtools are good starters with generous free tier. You should know when your website/email/dns isn't working.
Lesson learned :)
> Don't tie your domain registration with your DNS provider. You lose everything if something goes wrong with your account.
I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money.
> Be able to jump ship easily, have backups of your zone,
Luckily I have that
> already know where you will transfer to.
Any suggestions? Ironically I recently moved from Google Domains to Cloudflare because I was worried about issues with opaque support. I've learned my lesson picking based on cost alone, but I'm a college student who can't afford something too heavy-duty.
>I don't see how that helps. How do I recover from my registrar deleting/disabling my account even if DNS is somewhere else? I think there's still only one failure point and the lesson is that I need to pay that failure point more money.
Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare.
> Your outage was a DNS outage, not a registrar outage. If you still had control of the domain you could update your name servers to another provider, import your backed up records and get the site back online without talking to CloudFlare.
I believe it was both.
If I have a registrar outage I'm hosed. If I don't have a registrar outage and do have a DNS outage I can recover with a little work. But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well?
> But in the only case I can recover my registrar was reliable, so why didn't I just have them do DNS as well?
Because they have just proved being uncapable of doing it? Because redundancy? Because you shouldn't keep all your eggs in the same basket.
I've been self hosting for at least 15 years and did not have any huge problems like the domain becoming non resolvable. I would never host my DNS on my registrar's infrastructure. It's being sloppy and lazy and it gets you embarassed.
A domain registered at a provider (but not DNS) can be down with no impact to your domain, so long as the domain is still in the TLD root servers, everything will keep going.
This happened to me with AWS somewhat recently[0], and I never found out exactly what happened. I just chalk it up to some dev made a mistake and didn't tell anyone. It's pretty alarming when things like this happen though.
I've been involved in using Route 53 to manage thousands of DNS zones, and haven't come across something like that. I'd recommend putting in a support request via the account that was affected to ensure that it gets looked at.
If you haven't already, you might consider checking the CloudTrail logs for the account in question to see if there were any API commands related to the zone.
Although not DNS related, I have had weird things happen on AWS, such as spikes of 5xx errors reported from CloudFront which was backed by ELB/EB, but the ELB is showing no errors. Even after contacting AWS support they couldn't resolve it, said they required application logs, but there is no logs because the requests never reached the application servers.
Edit: the dns record export/import functionality is hidden behind the advance search drop down for some reason.
Ignore this entire comment.
From reading the linked helpdoc, apparently your entire domain can get removed from cloudflare if your register stops reporting cloudflare's servers for the ns records.
The mere idea of having to re-enter all hundred or so of our dns records using cloudflares 1.2 second delay at every step of the way add dns record interface because namecheap bugged out for a few seconds is horrifying.
There is no way to export all of these, there is no way to import or mass add and i don't think they can lean on the api to save them here.
Dns records are data, dns records are sometimes important unbacked up customer data. Cloudflare does not offer a way for customers to back this data up, nor a way to restore or recover from a backup but it acts very callous with this data, deleting it in automated systems based on data from 3rd party providers.
I googled "cloudflare dns import" and "cloudflare dns export" and the first result both times was an apparently official support article giving step-by-step instructions on how to do so. I myself have used this function about six years ago, so this is not new or untested functionality.
Never clicked that because i have no reason to look for a dns export/import button in the advance search gui... hmm. good to know about, not sure why they put it there.
You can export a zone file and the auto deletion takes days. You’re site would be offline before the zone gets deleted. I don’t like it a ton, but it’s not even close to what you’re saying.
I only have the help article to go on and the fact that their export functionality is for some reason hidden in the advance search drop down? as i've just found out...
Anywho, the help article does not state it it "takes days", so it does not have to "take days". Otherwise known as listening to the documentation not the observed behavior. Programming 101 here.
as much as i like cloudflare (and i like them a lot), it's kind of absurd that this kind of thing can happen. a lot of red flags that, if true, would mean that their infrastructure require a lot more care (127.0.0.1 as the source of an audit event? no email when DNS records are deleted? no 1-to-1 message due to this happening?).
I had an issue with them recently where a SRV record pointing to “.” (meaning “service unavailable”) was being rewritten to the string “false”. It didn’t take them too long to fix it, but it made me wonder how they managed to push a bug like that to production without some sort of automated test catching it.
Which is fair, I'd rather be a guinea pig than look at ads in exchange for a free service. I was just surprised that the thing they broke was as well defined and testable as DNS validation.
Simple. They don't give a damn about doing what we've all been doing properly for a quarter of a century. Apparently these large companies are above owning O'Reilly books.
At the very least, this sort of lack of good process is definitely what happens when Google decides to cut you off (and another person just commented a similar experience with Amazon), but I suspect it's likely the case for a much larger number of companies and services than people realize. It's fundamental internet architecture, and often little more thought goes into account termination than what you'd do to ban someone from your mid-2000s phpBB forum.
So much business focus goes into the onboarding experience, and since you assume all of the people your service terminates are "probably bad people anyways", not a lot of thought goes into offboarding, or ideally, appeals.
Had the same thing happen to me some years ago. Had a (not so important) domain with Gandi, which pointed to the Cloudflare nameservers, and after some time, the domain was gone from the CF dashboard together with all DNS entries. The NS records were still pointing to CF and there also weren't any anomalies with renewal of the domain.
I didn't give much thought to it, as I wasn't using CF for anything in production at the time, but sad to see that it also seems to happen to other people.
Unrelated issue but sometimes Cloudflare docs/communications are not in sync with their actual system which is immensely frustrating. I was bitten a few times.
For instance, a while back I forgot to renew one of my side project domains so it briefly expired for maybe a day or two. Got this email from Cloudflare saying
> Your DNS records will be completely removed from our system in 7 days.
> ...
> Once you have completed this change, click the “Recheck Nameservers” button in your Cloudflare dashboard to ensure your domain stays active on Cloudflare.
I promptly renewed, except there's no "Recheck Nameservers" button anywhere, and the dashboard still read "Moved" for maybe a day. Eventually the problem was just gone, but the communication worried me that entire time.
Also don't forget: Cloudflare breaks many second and third world countries' Internet with their DNS captchas because they think the good guys live only in first world countries (maybe look up the word discrimination in your dictionary cloudflare) and force them to install extensions like PrivacyPass because they think "we are so big and know what is right for the world".
That's CDN captcha, not DNS. If you use Cloudflare solely as a DNS provider, your users don't see the captcha. If you route your traffic through their servers, then they do.
you're right, it's their CDN not their DNS. Nevertheless many site owners choose Cloudflare (paid or not paid) and use Cloudflare's default settings and maybe they also never check their sites from second or third world countries. Result is that the Internet is utterly broken on many Cloudflare hosted sites (and that's a lot of sites) outside of first world countries.
Stories like these scare the hell out of me. What do you do if one of the big internet corporation deletes some resource or account that is critical to your business? What happens when support isn’t responsive and you don’t have contacts in the company or your HN post doesn’t get visibility?
I get it - these are free services. You should factor that into every decision. But the risk is real even if you pay for an account. I’ve been slowly moving away from Gmail to a custom domain, but something like loosing DNS records and not being able to restore them quickly is even worse.
Back up everything that can be backed up, don’t rely on a single provider and always have a continuity plan!
Cloudflare Entrprise sales pissed me off so much the last time I dealt with them that I literally registered "cancelcloudflare.com/net/org". We didn't want to renew there $25k enterprise support for a website that got 50 hits a month. It was embarrassing (for Cloudflare) how incompetent their Enterise Sales rep was. I would rather pay to host my own cache layer then ever have to deal with them again, free or otherwise.
> Does anyone know what might have caused Cloudflare to delete my domain? Any ideas for how I could transfer my domain away from Cloudflare sooner?
I don't get the point of 'shoot first ask questions later' type approach. Obviously it would pay to get some kind of affirmative reply from Cloudflare prior to a post which everyone here with incomplete information speculates and wastes time on (like I am doing).
Also Cloudflare did not 'delete my (the) domain. It deleted the dns records. There is a difference and no I am not being pedantic either. How would 'the internet' know why this was done there could be any number of good or bad reasons.
Lastly the domain is not expired and as such the registrar is required (per ICANN) to supply an auth code so someone can transfer out. Or to allow the customer to change the primary and secondary dns to another dns provider. There is zero (legitimately) that allows cloudflare as either a dns provider or a registrar to lock the domain up pretty much (other than for a legal court order) just for some reason they might decide to do that.
> Also Cloudflare did not 'delete my (the) domain. It deleted the dns records. There is a difference and no I am not being pedantic either.
Thanks. You're absolutely right. I meant delete their record of the domain as it shows up in the UI of their dashboard.
> How would 'the internet' know why this was done there could be any number of good or bad reasons.
For many reasons luckily HN isn't 'the internet'. I've already gotten some good suggestions.
> Lastly the domain is not expired and as such the registrar is required (per ICANN) to supply an auth code so someone can transfer out. Or to allow the customer to change the primary and secondary dns to another dns provider. There is zero (legitimately) that allows cloudflare as either a dns provider or a registrar to lock the domain up pretty much (other than for a legal court order) just for some reason they might decide to do that.
I know. Again, I guess I was insufficiently specific. Cloudflare has warned me to expect long wait times before I can talk to a customer support rep. My question was if there's a way to transfer out without needing to wait on a slow support loop.
We need a neutral, third-party service that monitors other service providers. That is, checks periodically for such things as a host being reachable, DNS working, certificates working, etc., which notifies both the service provider company and the end-user when there is a discrepancy, but most importantly, acts as a source of records in the case of a dispute -- in other words, they hold a record of what the service should be and should do. That is, when the service provider creates a service record, they would cryptographically sign it, and forward it to this future third-party service. The same for deletions.
Now we can track, record, and audit that service provider's promises...
That way, the service provider can't use an all-too-easy excuse like "we can't find that record in our database -- so you must not have ever created one..."
Reminds me of Rackspace - clown IT overrpcied. They provisioned email incorrectly and swore that once we changed nameservers to them it would work. We pointed out it wouldn't using a few basic tools and asked for an escalation, they refused and we were stuck with level 1 tech support. After a few days I found someone on Twitter who esclated, they admitted they fucked it up, but the damage was done, I was frustrated and called it day - I proceeded to cancel, they then had to cheek to say "let's all be professional here"... I can deal with occasional shit support but I can't deal with escalation refusal, or endless cycles of support tennis or being redirected to irrelvant knowledgebae articles.
We've just been dealing with this for my company as well. Cloudflare has repeatedly deleted our DNS and cannot provide a reason why it happened. Last time thousands of dollars of PPC Ads were running uselessly.
Had the same or very similar issue .
My domain registrar dropped all name server settings on an 'update'/whois update requirmet change
Luckily I had an old export of DNS records so most were there but not all the settings were retained like DNS only or full proxy.
Those were not mission critical, so not that bad but I bet some emails went to the void because there was no real warning from both my domain registration service provider and cf so I found it very 'upsetting' (I was pissed on them both)
I've Cloudflare delete an entire zone before, and I could never get an answer as to what happened. They said it was deleted because the NS were changed on the domain...but they never were.
Funny that this is coming up. I just transferred over from Namecheap to Cloudflare a few days ago and had a similar issue. One of my A records (out of about 20) were missing after the transfer.
Last week, I have had an issue where a number of domains were purged from the 2nd tier registrar (Claranet) with exactly the same symptoms (domains suspended, zone-files blown away)... and Network Solutions are to blame.
An assumption of false-payment led to them suspending "300-500" accounts (mostly UK based). I am still of the opinion something far more sinister is at play... and this doesn't comfort me.
I had the same thing happen to me with the same domain - twice. It just disappeared from Cloudflare without any notice. I run multiple domains on the same CF account, yet this has only happened with that specific one, which uses a somewhat unusual TLD (.do).
same thing happened to me on GoDaddy for multiple domains when I got a call from a client that their emails stopped working. All the zones were factory reset, and no backup of the zones apparently existed at GoDaddy. I was on the call with them for hours refusing to hang up until it was resolved or they would lose the remainder of my business. After 2.5 hours of no valid reason that multiple domains when back to default DNS values and no log of access to my account for moths, I let them go.
That's when I moved the couple of handfuls of domains I had left at GoDaddy over to Hover. It's more expensive, but the Hover interface is better, and I trust Hover (Tucows) more (well, I trust GoDaddy less).
Is it really all that surprising when a big company that claims to be good but hosts phishing content in the name of free speech does whatever they want, including breaking things and not explaining why?
I don't trust Cloudflare one bit, and I think everyone should question whether their attempt to re-centralize everything is beneficial to the planet.
There are two major problems here: one, the problem itself, which is the deletion of DNS for apparently no good reason, and two, which is the bigger problem, is that it's incredibly difficult to talk to a human about what happened, so there's no assurance it won't happen again.
If people want things to be reliable, we've got to stop using companies with which we cannot communicate.
IMHO (and I know the parent post includes significant difficulties getting back out of Cloudflare), services like Cloudflare may be crucial to decentralization. I can't deal with something like my blog post being frontpaged on HN if my website is hosted in my house, unless I have a good CDN.
As a self-hosting enthusiast, something like Cloudflare is one of the best chances of having a plan that competes with "just hosting it in the cloud".
I hear you, but their DDoS services are painful to the rest of the world and to people who want or need to use Tor, and others.
I'm talking about their rather political move to re-centralize DNS by shoehorning themselves in to Firefox via DoH, for instance. Their unwillingness to be transparent makes this all the more frightening. Add to that their blatant desire to make money at the cost of doing the right thing (and I'm talking about unambiguous things - is someone going to argue that freedom of speech allows people run a phishing site of your bank?), and you've got a scenario where once they reach critical mass, they will be exercising their position to the detriment of everyone who isn't paying them, similarly to how Gmail, through doing and not communicating, say "screw you" to many small email services.
When people who don't use large providers have email issues with Gmail, lots of people have knee-jerk reactions saying that everything should move to the big providers, that people and small businesses should not host their own email, and so on. This is NOT the way the Internet should work, and we should never allow Gmail to just arbitrarily do whatever they want, then accept it as the new normal.
If you have more than a dozen megabits of outgoing bandwidth, you can easily host a blog from your home network which can handle a front paging here. Just don't expect to dynamically generate a new copy of the site for every visitor, and if your bandwidth is tight, then host your images on a static server off of your network. Cloudflare is not necessary - perhaps it's easier, but it isn't necessarily best to blindly trust a company that wants to become a monopoly.
I don't know. Sure, maybe not on your barely-broadband DSL connection, but I'm pretty sure you can run most things on a average shared webhosting, even if you're using WP. You just need to make sure that you have a working caching system in place. It's a gigantic difference even on an apache to just read & send a plain file and invoke PHP & run all the costly code. I don't believe HN should bring down any site that can essentially be cached as static HTML.
Cloudflare will only help once your server has gone down with their "Always On" thingy, if you have that enabled. They don't cache HTML by default.
There's alternatives to Cloudflare that offer affordable-but-not-free CDNs which has always felt less risky to me. I'd rather know I'm the customer instead of the product.
> I can't deal with something like my blog post being frontpaged on HN if my website is hosted in my house, unless I have a good CDN.
IPFS can work as a CDN, at least for static content that users are willing to seed. This is especially relevant to the "blog post hits frontpage on HN" case. Of course, dynamic content is not quite as easy.
This is why you need name servers from 2 different companies and dns monitoring. It doesn't matter who your provider is. Errors happen and waiting half a week to fix it is insane.
> I registered for Cloudflare with a Gmail address specifically so that I could receive notifications from them if there were issues with my email setup.
OP here. My website wasn't up when this happened because of some yak shaving, but when it is I disable DDOS protection. I was only using Cloudflare for domain registration and DNS.
I don't think I have ethical issues with DDOS protection in general, but as someone who browses using Firefox on Linux with tracking blocking I know how annoying it can get. If I don't need it why bother? Plus I generally like to minimize opaque layers in my "stack".
It's very easy to manage incoming bandwidth when you're hosting a tor onion service. The entire Tor ecosystem kind of helps to since there's a limit on the instantaneous amount of data in any circuit. Overall tor is great because I own my domain name (rather than leasing it on the whim of some corp) and it has nice DoS and bandwidth tools built in.
And if you were using cloudflare before you should be okay with some people not being able to access your site since that's the norm there.
Well, I've been operating a bunch of small ones from a home connection. So my bandwidth issues might be of less scale but certainly more acute. One simple line of Tor config allows me to limit my total bandwidth and avoid saturating my home connection due to people hitting the onion services.
I've had one that's been under a DoS for about a month now and things are going well regardless. Differentiating "real" from "bad" traffic is super hard and almost infeasible, I get what you're saying about that. But in terms of not getting knocked offline Tor is easy and really nice.
> But in terms of not getting knocked offline Tor is easy and really nice.
That’s of course only until someone who isn’t utterly incompetent tries to DoS you and you realize that you’ll need to spin up hundreds and hundreds of tor daemons (with cpu cores to match!) to tank it.
Onionbalance isn’t built in, and hardly solves attacks. There are no DDoS protection services, so you’re left scrambling to build your own infra against impossible odds.
None of the big onionland sites manage decent uptimes.
While there are a lot of reasons not to trust cloudflare, the fact that they stopped hosting nazis and pedophiles doesn't seem like a good one to open with imo
Is that a reference to 8chan? Because if so I am pretty sure that they removed controversial boards of this form a few years back - long before cloudflare banned them.
I think it is a problem when you talk about fear of ideas. You can label anyone as a Nazi today, and when you ban people, you kinda give them power.
It shows you're afraid of their ideas, and the persecution can embolden them or give them a sense of legitimacy. It can being the Streisand Effect to their cause, taking a no-name site no one knew or cared about and blowing it up into something everyone is deeply aware of.
1. Cloudflare has only banned large websites, not any "no-name sites".
2. Admittedly anecdotal, but while bans like these do increase _knowledge of_ said websites, I see no evidence they significantly increase their popularity or userbase.
>and when you ban people, you kinda give them power.
The key word being "kinda". You give the perception of power, but there are plenty of things in society we deal with by a form of "banning" (such as incarceration) in which we judge the ban to be good regardless of the power it gives people, or the ideas they represent or practice. Locking up child abusers, for instance, may give the spectre of child abuse power, and highly-publicized instances may fuel the moral panic around "strangers out to get your children", but that doesn't mean they should not be locked up.
In civil society, we are justifiably afraid of many ideas - I don't know anyone who wouldn't be afraid of a Nazi-style dictatorship, or its prospect. Fear can be a legitimate way of preventing bad things from happening. I fear dying in a house fire, thus I take certain precautions when cooking. In the same way, a demonstration of power over a person or idea may outweigh the power supposedly given to that idea by banning it.
The key is a balance; it may well be that you get an instance of the Streisand Effect, but it has to be shown that the consequences of that outweigh the very material consequences of such ideas coming to fruition in real life. For example, many people don't know who Barbara Streisand even is, and if they do, they likely don't know about the pictures of her house. The very canonical example of the Streisand Effect shows a short-lived controversy about the actual matter and ensuing attention for a few months after the incident. Then people forgot, or simply stopped caring. "The Streisand Effect" is more of a Streisand Effect than the actual incident that created it.
>I don't know anyone who wouldn't be afraid of a Nazi-style dictatorship
Do you believe that if people with ideas that you consider to be Nazi-like are allowed participate in the public debate as freely and meaningfully as everyone else (e.g. by using electronic services regularly available to everyone else), a Nazi-style dictatorship is likely to come about?
No, I don't, but I do think it increases its chances of happening - and there is historical precedent for it. For that reason, I sit somewhere between Popper's paradox of tolerance on these matters (and I go further than him), Sartre's notes on anti-semitism, and Marcuse's criticism of simple plurality as a substitute for educated and rational thought.
To be clear, I don't advocate for censorship of ideas that I simply "don't like". That's not a sufficiently rigorous standard. Ideas which advocate for targeting marginalized groups, or entire groups of people for ideas they have no control over, are fair game, in my opinion. I don't pretend to have no bias in my answer to that question. I am biased, and others have their own biases. I draw the line where I want to draw it, with no concern for pretending to derive it from first principles.
To get this straight, you are stating that you believe it is in your interest for certain other people who live in the same society as you to be prohibited from attempting to further their own interests by freely and meaningfully participating in the public political debate to the same degree that you and others in your society are able.
Do you think that's going to end well? How do you expect those people to feel about you? There is a very good reason why societies have protected the right to political speech, and it is to prevent the inevitable conflict that arises when some people in society feel that the rest of society is preventing them from attempting to further their own interests in the same capacity that other groups are able.
In the case of political speech that you consider Nazi-style, your rationale is that you believe it will make an event you consider to be unlikely become even less likely. You believe it will have a sufficiently large influence on the likelihood to make it worth bearing the consequences of telling people in your society that they cannot participate in the public political debate as freely and meaningfully as others. Why do you believe it would make that event less likely? And why do you believe it would reduce it enough to justify the risk?
Again, this is in reaction to literal neo-nazis getting deplatformed.
If your first reaction to seeing nazis get in trouble is to ask "well what if I get called a nazi?" then you might need to do some introspection my dude...
Also - deplatforming is incredibly effective at severely limiting people's reach. Anyone know what Milo Yiannopoulos is up to these days?
If you run a restaurant, you can refuse to do business with anyone you choose. If that was not the case, you would effectively be a slave; unable to choose actions for yourself and your business. Cloudflare refused to do business with people and content; that is their prerogative.
> A store cannot have blacks only and whites only bathrooms or water fountains. Bars and restaurants in some jurisdictions can allow smoking within their establishments, while in other municipalities, smoking indoors is banned for all businesses. Companies who chose to be equal opportunity employers have several criteria for which they cannot discriminate against. Laws such as the Americans with Disabilities Act mandates certain accessibility requirements in order to maintain a storefront ... Speech does not yet fall into any of these existing regularity frameworks.
So no, you're wrong. You cannot refuse to do business with anyone you choose. The Colorado cake case is a really special one, because it had to do with art. As an artist, you can refuse a commission to build a creative work if it goes against your values. The guy who ran that shop just stopped accepting custom orders, and then later got in trouble again when he refused to sell plain non-custom cupcakes to a gay couple.
>If you run a restaurant, you can refuse to do business with anyone you choose.
You will suffer legal consequences if it's determined that you refused to do business with them on the basis of certain characteristics protected by law. Property rights are not absolute. Exceptions to them can be made if people think there is good reason to do so.
The situation we are in now, where technology companies have found themselves with wide power to control the public political debate occurring among regular people merely as a consequence of successfully running some particular types of business, is not one we've really seen before. There are some very persuasive arguments for limiting their property rights, similar to how they were limited e.g. ~50 years ago by the civil rights act.
Cloudflare lost my support when they started de-platforming people for holding opinions they didn't agree with. Censorship outside of strictly legal bounds should not be tolerated from a company as powerful as Cloudflare.
At least in some cases, those people were claiming that because they hadn't been removed, Cloudflare supported them. I don't see what other option Cloudflare had at that point.
Except that Matthew Prince, Cloudflare's CEO, made that up out of thin air. There's no point where Daily Stormer said that Cloudflare supported their ideology. There's no record of this on the Internet. Can't find it, because nobody at the site ever made such claims. Stormer was kicked off of dozens of domain registrars and registries (GoDaddy, Google, Namecheap, Dreamhost, several national cctlds) in the same period -- none of them had to come up with a fake excuse like "people will think we support their ideology". Cloudflare does do infrastructure plenty of pedophile and Islamic terrorist sites, so now we can assume that they actually do support those as they aren't removing them from the service.
Cloudflare also didn't even bother telling that lie anymore when the dozens of sites they censored afterwards including 8chan were systematically barred from basic commerce.
Yes, it is censorship. The entire history of First Amendment jurisprudence was set around the idea that powerful people were not allowed to stop political and religious speech. Marsh v. Alabama is a great example: a company town owned sidewalks that they didn't want religious prosyletizers on. The courts ruled that the fact that they owned the sidewalks and roads is irrelevant. For the entire history of my country powerful people were not allowed to buy up the public square and prevent the little guy from speaking. Everyone had a right to enumerate their grievances in a free and open marketplace of ideas. This has of course changed in the age of the Internet, where a bunch of scheming Stanford grads have bought up the courts, wrested control of the key Internet infrastructure away from the public who funded its creation, and sit there and grin as they take the role of arbiter over all speech on the Internet. The wealth and power disparity between the rich and poor is at its height, and it is clear that there will be no legal or democratic solution to the concentration of power in the hands of a handful of Silicon valley billionaires.
Would you be OK with your phone provider or imternet provider to stop doing business with because you said some unsavory thing to a friend or have blog supporting the wrong candidate?
Why do you think you have a right to host with them? You don’t, you have a privilege that’s extended by them. You’re welcome to host your own thing somewhere else.
Why do successful technology companies have a right to have a proportionally large influence on the public political debate? Is it good for society to allow successful technology companies to have such a large degree of control over something so incredibly vital, merely because they were effective at running a particular type of business?
BTW If you, dear reader, ever find yourself so frustrated with Cloudflare that you feel like your only recourse is a blog post... my email is jgc@cloudflare.com and I’m happy to hear from people.