Screen readers, the programs that use synthesized speech to tell us what's on the screen, cannot read images. Good captchas usually have audio equivalents (which come with their own set of problems), but this one doesn't. If you're blind and flagged by Cloudflare for some reason, you're cut off from accessing half the internet, potentially critical banking/governmental/medical/communications/educational services. We rely on the internet way more than our sighted peers, so this is very important. This has recently happened to me on a few sites, fortunately not critical ones, but it was not a pleasant experience nonetheless. CF engineers, please fix this ASAP. I'm surprised there still isn't a huge lawsuit over this, as this is clearly violating all sorts of laws.
It might be worth filing an ADA compliance notice against cloudflare. This is seriously disturbing.
As someone that works in ed tech - anyone that supports higher education is required to support screen readers etc. If any of those sites are using cloudflare that would block them from being compliant pretty seriously.
I'm not american, otherwise I would definitely do so.
This issue is mostly unnoticeable, as long as you're not considered malicious to Cloudflare, everything works perfectly and passes accessibility audits. When something weird happens and Cloudflare flags you, you cannot access the system at all.
For those doing accessibility audits out there, this kind of issues might be worth looking into, as they're very non obvious and very critical.
Cloudflare switched to using hCaptcha a couple months ago, I think and I only just noticed that they do not offer an audio-based captcha.
However, hCaptcha integrates with Privacy Pass[0], and you can top up your tokens by solving a captcha at [1] and [2]. What you could do (and I realize this is far from ideal), is getting a sighted friend to solve a couple of captchas so you have enough tokens to last you a couple months.
There's AIRA and it can do it, if you don't care that another person will see what site you're trying to access and if you're willing to go through a super complicated techy procedure to get a free american phone number, which not everyone knows about. Also, not everyone speaks english.
> Also this ask a friend solution is like telling someone in a wheelchair to ask a friend to help them up those steps instead of just installing a ramp.
I agree, and I could’ve been clearer. The Privacy Pass workaround shouldn’t be necessary and we should demand Cloudflare do better.
(To avoid any doubt: I’m not affiliated with Cloudflare in any way, other than that I’m a customer on their free plan.)
You're responding to a rando on the internet, not the CTO of Cloudflare. Your parent is just trying to offer help to OP, and you're mad because you assume he is a Cloudflare dev? Even if he was, it's not like he has unilateral control over product decisions.
Yeah, and then... a few days later... The problem is reCaptcha, just stop using reCaptcha as Google is tracking all of us.
My personal opinion is that it's better to use hCaptcha. I wasn't aware that they don't support audio. I think a better approach would be requesting hCaptcha to implement the missing bit.
Giving less control to Google is better. Given the widespread tracking they have access to right now.
Solutions can’t be implemented instantly; a work around is needed until that ramp is installed. As much as the lack of audio solution sucks, you can’t expect half the internet to just give up on a piece of technology overnight
> you can’t expect half the internet to just give up on a piece of technology overnight
People call for this all the time when a major security vulnerability is discovered. The difference is in how the community views it.
Cloudflare should weigh harm of blocking access to the Deaf community warrant the harm of removing a captcha that doesn't support them. They should have done it when they chose a captcha that doesn't support the Deaf community.
As technology comes to mediate every avenue of life, we need to recognize that technology only has value in its positive effects on people's lives. A security vulnerability is bad because owners may lose control, property may be lost, crimes may be committed. Usability vulnerabilities can deny services essential to their users. You're totally correct that there is no magic solution, but to say that we know which imperfect solution is preferable is incorrect.
We can expect developers to think about accessibility during development. You would never see a building constructed today without a ramp or with insufficient handicapped parking. I realize that you can't change culture overnight either, but the fact is that technology like this, designed for use in large applications used by a huge portion of the population of the world, shouldn't need to be retrofitted to be accessible. It should have been baked in.
I mean, yes, in an ideal world that's true. Yet, here we are.
It's not possible for a dev to go back in time to before hcaptcha was created or when CF decided to switch to them to create said ramp, so until it's built, workarounds are the only real thing a community member can offer someone in the short term.
It's not like trying to find the Fountain of Youth or something. It's a company with billions of dollars making it impossible for certain people with disabilities to access the Internet. Let's not pretend that it just has to be this way. A world where half the Internet can't be broken by one company should be the baseline.
Nobody is saying it has to be this way. As I said, in an ideal world, accessibility happens when development happens and is not an afterthought. But we don't live in an ideal world, and it's clear here that it wasn't thought out when they did the switch, or other forces caused the switch to happen without this piece in place.
So, with that in mind, knowing where we currently are, not where we'd like to be in an ideal world, what, exactly, do you want to be done right now by members of the community?
It's not like we can all go in and change the Cloudflare code to stop using hCaptcha. The most we can all do is give alternatives and workarounds while pushing on Cloudflare and hCaptcha to support this scenario. Which is all being done in this thread already.
When things are already completely broken then it's acceptable to move fast because things are already completely broken. When things are only slightly broken you need to be careful you don't expand the breakage.
Are you sure CloudFlare isn't working on this right now? Captchas are very hard (if not impossible) to do right, and audio captchas are no exception. Check out http://uncaptcha.cs.umd.edu/ As far as I know CloudFlare might have a large team that's been working on this for a while but has been unable to create something solvable by a human but unsolvable by a bot.
I use audio captchas. Google will usually only let you do 2 or 3 before banning you and making you do image-based ones. I'm pretty sure the button is just there to make it seem accessible.
I had exactly the same issue. I am surprised how neither cloudflare nor google have been sued for making most of the web inaccessible to people with disabilities.
Shouldn't that liability be with the operators of websites that use those captchas if they're required to be blind-accessible? If they get sued, it'll apply pressure up the supply chain.
I used to feel and think this way. But recent events, and the trajectory of the internet over the last decade has changed my mind completely.
We must stop giving excuses to the massive centralization, to the enormous companies that step in and rent-seek what is supposed to be a distributed system.
The most powerful tool we could have to get a proper internet back is the simple concept of accountability. If I was a bakery and I made a cake with poison because I was specifically asked to do so by a customer, I would still be accountable for the dangerous thing I made.
Stop making excuses for these companies with basically infinite money. They are the last ones who need it. What we all need from them is accountability.
This also goes for section 230 protections. We do not need Twitters and Facebooks and Hacker Newses and other such companies that would supposedly not exist without section 230 protection. It's clear now, in retrospect, that accountability is far more important than the license to grow enormous without any responsibility for the toxicity your giant bloated corpse of a business unleashes onto the world.
Anyone making a captcha product can and should be expected to make it work for anyone. And if there is basis for lawsuit, it should be the maker of the broken thing that faces it.
Hell, sue them both. Neither entity deserves a free pass. They both had a part to play in the exclusion.
This does not really make sense, it would be perfectly fine if two products existed, a cheap inaccessible captcha and a more expensive perfectly accessible one with companies offering both of them to users.
That is not happening in what I described. A webpage could decide to offer the cheapest working captcha as a default with a "Are you using assistive technologies?" / "Do you need help?" link to a more accessible service.
I don't think that is will be common in the wild, but it is an argument against regulating captcha providers rather than websites.
If we want to regulate the captcha industry there should be a law passed specifically regulating the industry. Blindsiding the industry with judgments against them when there are no laws about it will decrease confidence in the rule of law, which will hurt the economy as businesses lose confidence in their ability to predict the future.
Rules and laws are good tools and, I understand how law works in the most superficial way but, I fail to understand normalization of ignorance of some required features with the "nobody forced us to do this, so we just didn't care".
As human beings first and programmers later, we should understand that people with disabilities exist and assistive technologies need attention.
Why, as decent human beings, don't we do something nice without the push of laws, for once?
I agree we should do nice things without the push of law. But ztjio was advocating for the push of law without following the process of creating a law, which I disagree with.
I agree. You don't sue the contractor because they built to the specification you provided.
At the same time, if Google is advertising Recaptcha as accessible and it's not really, then they need to be held accountable for that, because that has a real impact on huge swaths of the internet, and especially for sites that try to do the right thing and find out Google has screwed them.
> You don't sue the contractor because they built to the specification you provided.
In the world of licensed contracting you absolutely do. If you provide a contractor specifications that result in a code violation, they have to tell you no. If they don't tell you no then you, the non expert, are reasonable to assume what you asked for is acceptable. If they actually build the thing, they are liable for the violation of code.
The last thing we need is to hand out more excuses to evade accountability and responsibility for making the world worse.
Yeah, contractor is a bad metaphor. I started with Home Depot, but that didn't really fit either. It's more that you bought a ramp that's designed to fit over a two step rise. Was it advertised as accessible in your state/county (which will likely have specific gradient and handrail rules? If not, you should have looked. If the salesperson assured you it was, well that sucks, but I'm not sure you'll get more than your money back.
> The last thing we need is to hand out more excuses to evade accountability and responsibility for making the world worse.
I'm definitely not advocating that. I'm more advocating that you can't leave all this to someone else entirely. Even if you pay someone for a turn-key custom website and make sure it's accessible, there's no guarantee it still meets standards a year from now, just as if you pay a contractor to build a ramp for you that perfectly meets standards. Things change, and people need to pay attention to the things that matter to them to make sure they follow the law. Whether that's occasionally checking it yourself or paying someone else to do so (and thus ensuring it happens), it still needs to happen.
As sad as it is, there seems to be a CAP situation when it comes to captchas. It's accessibility, security or privacy. Choose two.
You can go the Google route and choose accessibility and security, do massive user tracking, and don't even show CAPTCHAs at all for normal users. HN users probably see a lot, because they use some anti-js or anti-tracking stuff, but normal users don't.
You can go the Cloudflare route, requiring users to solve visual challenges, sacrificing accessibility, but keeping security and privacy.
You can also implement audio CAPTCHAS, which are easy to solve for robots, get accessibility and privacy, but less security.
> You can go the Google route and choose accessibility and security
This is not the case. After just a few captchas the audio challenge will be locked off no matter what browser you use.
> and don't even show CAPTCHAs at all for normal users
You see them in firefox from the start as well as on chrome after around the 2nd or 3rd captcha, this is with the default settings + ublock origin on both browsers.
The "human detector" of the modern internet doesn't accept disabled people as sufficiently human. That's... the most dystopian thing I've heard in a while!
In the other direction, Google thinks I'm a bot if I use complex search queries (exclusions, site:, quotes, etc.) in an attempt to find what I'm looking for. Apparently it thinks I'm too intelligent to be a human.
I get that you meant that sardonically but it's worth pointing out that it would be even more unprofitable implementing ways to catch and redirect you.
Hi! I work at hCaptcha (we handle the CF captchas), and to respond specifically: audio is inaccessible for those with visual and auditory processing issues, as well as being broken by ML techniques. Not to project, but we suspect this is why Google turns it off if your browser looks at all suspicious.
While Privacy Pass is not perfect, we are working towards getting better. I'm not in the product flow for accessibility, but if you have ideas outside of audio that you believe would be able to distinguish between yourself and a robot, I'd be happy to discuss via email to ensure that we can get a solution for you and anyone who can reasonably interact with a computer.
eta: you can email me directly at work via: josiah@intuitionmachines.com
Audio captchas are considered broken completely (search for various papers over the past decade, including one from CMU), so it has limited usefulness only in a very narrow context in practice. I suspect this is a harbinger of things to come - as we close the gap on passing the Turing test, captchas are likely to get less effective, and we will need to transition to a very different solution for bot detection.
often it's easier to just use a website as api instead of using some broken xml nightmare that requires knowledge of the database tables.
If the concern is rate limiting then just rate limit the website. And if you don't like people operating websites with bots then I don't know, maybe stop making websites.
Easy proxying means rate limiting doesn’t really help all that much to defeat bots. And then if you do something like blocking or severely restricting something like Tor (the world's largest open proxy and hence, primary abusive traffic source; something which it would make sense to throw extra bot walls in front of), privacy and accessibility advocates jump down your throat.
This is a no-win situation. I’m not convinced it’s possible to have ones cake and eat it too, here. Someone upthread said “security, privacy, accessibility, pick any two”, and I have yet to see any evidence of a third option.
I think you do not understand why website owners use captcha. Website owners use captcha services, because doing so saves money for them.
Captcha will stop saving money, if the captcha becomes so ineffective that the short-term and the long-term downside (annoying users who are subjected to captcha) exceeds the cost saving ("cost" here is potentially many different things - it could be quality of service for normal users, it could be opportunity cost, it could be the cost of serving the traffic like network bandwidth or cpu or database capacity).
Yes there are subset of people with visual problems who also have auditory problems, that does not mean you should not support the ones with vision only problems.
Are there not "ML techniques" for visual captchas too ?
With more and more powerful models like GPT-3 coming along every few months, Even if the accuracy levels are less for visual over audio today how long do you think that is going to last ?
I believe they meant that audio captchas have become inaccessible to humans in general. Bots have gotten so good at audio captchas that the difficulty level has to be high enough that humans are also unable to complete them.
Visual captchas too can be quite very hard these days for the same reasons, there are many other ways to verify bot or human, many of them may not allow you to remain anonymous while doing so, pretty much any 2FA method could work for example.
I would say given the choice for a number of people the accessibility is more important than loss of privacy. At least it is choice they should have instead of shutting them off the internet.
Hi HCaptcha-er here. You don't lose all your privacy with our accessibility option. Also we always abide by DNT style protections even for accessibility. That being said our privacy pass solution is totally private and safe.
> Are there not "ML techniques" for visual captchas too?
There are.
> how long do you think that is going to last?
I don't know. So far it hasn't been difficult to recognize the garbage traffic when we take the time to look. And attackers always make it easier by releasing github source and / or announcing their results on Twitter.
Ugh. That accessibility login solution sounds awful, but at least it's available. I can definitely understand why you don't want to have an audio option, given that by all accounts they're next to useless.
ETA: two different people have tested on Chrome and Safari on iPhone 6S locally, and can't reproduce. Please take a screenshot and provide more information to: https://www.hcaptcha.com/reporting-bugs if you want to get this fixed.
Just like Google, it is also horrible for those who use more "nonstandard" configurations. I don't remember if it's CF or Google or someone else but I recall coming across sites that would block you if you have JS disabled, claiming that you're a bot, and the ban-page was the same across those sites, so it was a common "provider".
The a11y problem is super urgent for the majority of products out there. Not only it prevents blind/non hearing people of using systems. People also tend to forget that having a11y in place elevates UX on another level. It enables using apps/web with shortkeys, makes everything more readable, changes entire UX flows into simpler ones.
I have no affiliation with CloudFlare at all, but my interactions with them have all been kind, professional, and courteous.
I might recommend reaching out to the CEO, Matthew Prince, with a succinct explanation of your problem and I would be surprised if that by itself does not kick off some serious positive action!
As a founder of HCaptcha with a legally blind sister I take accessibility very seriously. We implemented this solution when we were still a tiny startup and I think our accessibility is by far the best of all the captchas. Please email support@hcaptcha.com if you are having any issues and we will find a solution. I'm sorry for the confusion and difficulty you may have suffered.
I'd love to read more about the economics of captchas - what is the "labor" of classifying images worth? What is the marginal cost per captcha of human solver farms, etc. Can anyone point me to some resources?
Seems like if this issue could easily be solved, a startup would have formed to solve it. Are there any alternatives to recaptcha and hcaptcha that are effective and also accessible?
Honest question: I've never really understood the back of the napkin math of how Cloudflare functions economically, which I feel would go a long way towards my understanding of why/how they were able to become such an integral and generally positive part of the Internet.
Did they have some crazy in to get cheap bandwidth? Did they bet big on bandwidth prices falling? Did they figure something else out that nobody saw? Do they just to a tremendous job of migrating sites from free to paid plans?
II love cloudflare. It has really helped out with several sites/projects that I have worked on and the service is top notch. I am also an investor. I tend to invest in stuff which I use a lot or trust/respect the employees. Weirdly what made me really invest is the level of geekiness on the company. I remember seeing you guys using a lava lamp wall to generate entropy and just thought "that's awesome". I just wanted to say don't change. Your lz4 implementation, aes gcm golang optimizations have directly benefitted me. Coupled with really high quality post mortem articles, articles on interesting things like compression, encryption, networking, a few random articles (the privacy focussed file system recently comes to mind, written at the same time I was making my own distributed fs) just leave me with a lot of respect for the culture there.
I was worried a bit when I saw the initial IPO that the free tier would leave (despite promises it wouldn't) but that doesn't seem to be the case. Literally the only bad thing I've seen on the site is recently you switched from recaptcha to a new one that I had a real tough time with logging in today - it was a bit glitchy on my pc. The only suggestion I thought of as well would be a simple "maintenance mode" similar to the "I'm under attack mode" which would allow those of us without super-ha to quickly toggle on something to pop up a "sorry server"/site is down form maintenance page without having to mess with our proxies/web servers.
Anyway I know this comes across as totally kissing-ass but I just wanted to say thanks to someone who actually works there. Everyone fat fingers stuff every now and again,don't sweat it.
Nothing beyond the normal "cognitive reserve" findings that high education and broad cognitive engagement makes the brain pathology take longer to reduce your functioning to a dementia state.
Sounds lame but this story really touched me. I recently visited a family member who's nearing the end and it broke my heart to see them in that condition.
I'm not sure what to take from this. But thank you for sharing it
Never heard of him but I read this article in one sitting (extremely hard thing for me) and I felt some unexplained connections with this story. A Beautiful Mind is another book which I very often go back to and go through similar emotions.
Wow, thanks for linking to the article. What an awful story. Makes you think about the wisdom of undertaking major elective surgery if it's not absolutely required in the short term.
> I remember seeing you guys using a lava lamp wall to generate entropy and just thought "that's awesome".
To be sure (and for the sake of internet rando completenessism), it does look like CF waited until the original SGI patent on the technique ran out. :)
> To be sure (and for the sake of internet rando completenessism), it does look like CF waited until the original SGI patent on the technique ran out. :)
I work at hCaptcha, we run CF's captcha. If you're having problems in the future, popping open a debugger and capturing the results can help us figure out what's going on. But also: browser, OS, site, ...?
Right now: there is an issue with Safari users on the most recent iOS and OS X, where 3rd party cookies have now been disabled by default. We're working on a solution.
If that's your issue, you can fix on your side in the short-term by not using Safari, or by enabling 3rd party cookies.
I cannot believe apple decided to roll that out in the middle of COVID. I got hit pretty hard with it and am finally rolling out a fix for my own stuff.
If you use Cloudflare workers, you could put together a maintenance page with a simple script + Cloudflare Workers KV as a toggle switch. It would cost $5/mo if you're not already using workers.
We believe our platform disrupts several large and well-established IT markets. The key markets that are addressed by our platform include VPN, internal and external firewalls, web security (including web application firewalls and content filtering), distributed denial of service (DDoS) prevention, intrusion detection and prevention, application delivery controls, content delivery networks, domain name systems, advanced threat prevention (ATP), and wide area network (WAN) technology. From our analysis based on IDC data, $31.6 billion was spent on those products in 2018, which is expected to grow to $47.1 billion in 2022, representing a compound annual growth rate of 10.5%. We also are actively developing new products to address adjacent markets including compute, storage, 5G, and Internet of Things (IoT) that are not included in the estimate of our addressable market.
That’s just general market analysis. It doesn’t say anything about what their hedge is. Or was it simply that the existing actors were that inefficient/price gauging.
LOL, I'm sure it is, but referring someone that is wondering about the "back of the napkin math" to an approximately 250 page S-1 filing is sort of ludicrous.
Congratulations with growing so big. Why aren't there dozens of lookalikes by now?
I mean CDNs that will let you override the origin's cache instructions and do a decent job of DDOS protection, and whose feature list otherwise looks a little like Cloudflare's.
One answer - It's much, much harder to build and operate a global network infrastructure that it might seem. It's also even harder to invent some sort of "killer feature" or other genuine innovation on the experience. You're likely not using Cloudflare simply for commoditized pipes alone, but for other features or designed experiences in their offering.
A second answer - there are a bunch of bottom-barrel commoditized pipe services. You likely haven't heard of them because they're so generic. They've existed before Cloudflare, and more will be created in the future https://www.citrix.com/products/citrix-intelligent-traffic-m...
CDNs are expensive to build, and often not very useful to customers until you've built out a large portion of it (actual hardware required, you can't just run it atop AWS). On top of this, much of the money is in Enterprise. So you've got to compete with Akamai, AWS, Fastly, Incapsula, Cloudflare, and several other notable ones to get any customers to speak of.
There are smaller CDNs out there. You can find them readily enough.
Depending on how you integrated it and how deeply, it may be trivial. Certainly going from one DNS-based CDN to another can be pretty easy - a Cloudflare / CloudFront swap could be quick.
Which suggests to me that CDNs are already a commodity in some ways.
Don't take this as being glib, but isn't this the entire point of funding rounds? Demonstrate growth and talk about a few approaches to creating a business model that you're investigating, which then brings investors on as they're betting you'll find one that'll succeed?
Cloudflare CEO Matthew Prince spoke to CNBC upon banning Daily Stormer: “We were worried that people would say, ‘We won’t work with you anymore,’” Cloudflare CEO Matthew Prince told CNBC.
“We had to have the conversation now because at some point we’ll be a public company. We had to prompt that discussion,” said Prince, who added “we want to be ready internally by July 2018,” for a possible stock offering.
I don't know if it's how it played out, but it's easy to see how increased scrutiny could have lead to the illegal content being highlighted and them getting dropped because it was then reported (possibly many times by many parties that wanted the site shut down for other reasons).
In that case, they still would have been taken down because of the illegal content, even if at an (extremely simplistic) surface level the timing seems to indicate otherwise.
I wonder if there are people out there who don't want internet infrastructure to be run by the government specifically because the government would protect freedom of speech "too much".
This is a bad analogy. Cloudflare is not inspecting individual packets for hate speech. They are refusing to do business with an organization that negatively affects their brand (The Daily Stormer). They should have the right to make that choice as a private entity.
"They should have the right to make that choice as a private entity."
No they should not.
And the analogy works: if you're getting filtered on the basis of your content - at the packet level or not - then it's fundamentally against Net Neutrality.
Wait until the PR team at Verizon decides they don't want to publish your content because you're too vocal about BLM. Or, they will only support you if you do support BLM, or something rubbish. And now your VPN, your server host, caching technology provider, Telco, wireless provider, Visa/Amex, Video-conf provider - it's completely absurd.
FedEx won't ship 'PlanB' because it's a 'controversial' medicine? But they will in 3 states?
USPS will ship condoms everywhere but not in Utah where the local Union forbids it?
Alaska Big Oil gets their local VPN owners to ban Greentech related sites?
Trump's buddies on the Board of AT&T get them to threaten anyone hosting 'fake news' about Trump?
California Teachers Union Pension Fund presses Cloudflare to ban all hosting of anything related to law enforcement?
And FYI nobody is acting 'morally' - they're scared executives just trying to do whatever to hush people up and continue making money - a system which hands arbitrary power to arbitrary groups. This is not what anyone wants.
For services that are inherently 'content neutral' - the content should not be allowed to be a basis of discrimination.
For Social Media it's different, as there is an inherent association between the platform and it's users, but not for Cloudflare, or AWS or Verizon, Gmail for example.
There is no end to the insanity otherwise; we need basic, smart and clear regulation.
Edit: I should add 'and that's just the US'. Imagine when a very vocal, organised group wants to ban Arabs living in what is commonly referred to as 'Palestine' from using the term 'Palestine'. Or Serbian authorities from hosting content using the term 'Kosovo' in any way that reflects its supposed 'autonomy'. Or Greek companies ganging up on Macedonia's usage of the term 'Macedonia'. Or Greens in Germany from banning pro-Nuclear energy content. There are at least a handful of Tweeters who would want those things. It gets infinitely messy, very quickly.
There is clear regulation in many countries: nazi speech gets you thrown in jail, the rest is mostly fine. See njmerious european countries like Germany that aren't some dystopia hellscape. That reflects a lot of what happens voluntarily today anyway, but it can't of course be formalized into regulatory law in the USA, because such regulation is restricted by the first amendment.
Your panic is unwarranted anyways, because many of those countries do allow free speech restrictions, yet no runaway crazy banning has happened yet. Its just a fictional slippery slope made up to protect hate speech. Words don't protect against tyrany anyways; actually fighting against tyranny does.
"Its just a fictional slippery slope made up to protect hate speech"
Ah, and there is the ugly, authoritarian, Stalinist argument right there.
"We only ban, denounce, destroy those who are GUILTY of Hate Crimes, so how could you have any worry about that?"
People are banned all day long on Facebook, Twitter, YouTube, Stripe, comments sections etc. for any and all sorts of made up reasons that have nothing to do with hate speech. And FYI you can say things that are essentially hate speech, but in a political context may not be, so it's fine i.e. "White People Are a Disease" is a common thing on Twitter. No problems there apparently.
And speaking as a former German resident, they have some real Nazis there, none of this kind of 'made up suff' like in the US.
This is a problem that needs to be addressed.
'Freedom of Expression' is the #1 Amendment to the US Constitution and the #1 thing in the relatively new Canadian constitution (#11 in EU) for a very good reason: it took thousands of years for 'every day people' to be able to speak their minds without being shot.
We can't leave this issue up to corporate CEO's, mostly trying to read the Twitter tea leaves.
I agree completely. I think democratizing speech so that it cannot be controlled by any source of power, government or financial, does have real importance. But its not the most important right, compared to life or wellbeing: instead, its important because it supports what's actually important.
But I have a message of good news: it turns out speech can be lightly restricted while leading to less authoritarianism or cronyism, as can be seen from numerous examples, by simply restricting speech in a way that is kept controlled by the specific people who are subject to it.
I don't have a lot of faith that we will resolve this, but the 'Net Neutrality' vs. 'Ban Someone I Don't Like' hypocrisy is dizzying.
The positions are ideological and naive - they are obviously contradictory - it's only that when they are framed in certain ways, often 'straw-man-ish', do people think they have clarity on them.
For example, it's easy to understand why we don't want Verizon charging customers different prices based on the nature of the content. It's easy for us to want to 'ban Nazis' from Reddit. But both of those are unrealistic ideals (i.e. straw men) because neither contemplate the broader application. It's a very slippery slope.
It's not a big deal that Nazis get banned from somewhere, and maybe it's not a big deal that Verizon wants to design services such that your profitable business costs 2x on their network - but those bounds expand very rapidly.
On the front pages of CNN and Fox right now we have a completely ridiculous war over 'Goya' canned foods because of some arbitrary comments the founder made about Trump.
This issue is only going to get much worse and more complicated over time.
We have basically no choice but to define what kinds of businesses can use what thresholds, and try to infer what those thresholds are.
> This issue is only going to get much worse and more complicated over time.
Slippery slope is a fallacy. Humans aren't required to follow the precedent of previous decisions like SCOTUS justices are. People eventually see the problems with previous choices once the consequences hit a tipping point.
We are having multiple moral panics right now. They will pass.
People will eventually get more tolerant of others making mistakes on social media; it will accelerate once more of us know someone who directly loses their job/status/etc to a moral panic. Also, there are examples of people who were targeted by moral panic mobs (eg. Colin Kaepernick) who survived the outrage and overcame it.
Once enough people get {"cancelled", fired, boycotted, excommunicated, etc}, these calls carry less and less effect over time and social media mobs lose their power.
> On the front pages of CNN and Fox right now we have a completely ridiculous war over 'Goya' canned foods because of some arbitrary comments the founder made about Trump.
Those are bad gauges of anything except outrage-for-clicks. Those two webpages rile up their core viewership to generate eyeballs and collect web-ad revenue. Trump only got to the front of the 2016 Republican primary because the rest of the field was boring and he riled up moral panics and owned most of the news cycles since.
> but those bounds expand very rapidly
Not always. And this ignores the observable fact that there is almost always a regression to the mean. There are social frictions which prevent these moral panics from burning too long.
The 'slope' has been 'slipping' for at least 10 years, and here is no indication that it's going to let up.
The opposite of 'fallacy' - it's literally happening all around you:
1) We now have major corporate support for 'social claims' with brands backing (and cashing in) on celebrities for their politics, not their skill at 'whatever', and supporting some causes surrounding a lot of public disruption. These have led to some groups trying to 'ban' others.
2) The attempted influencing of electoral outcomes via 'bad information' from outside agents, agitators, along with a plethora of information - with no guidance other than the whims of CEOs. There's all sorts of 'banning' on FB and YouTube and it's all pretty grey, often it can have major consequences.
3) CEO's brought to congressional hearings for these issues -
this is a sign that 'ban' culture is now a top concern.
4) Weekly national 'pop culture wars' over which brand has transgressed which group, promoted widely in the press, resulting in calls for bans of some products or services.
5) 'Social concern' as the primary legitimacy for populism, even among those who are supposed to be popular for their acting, music, or athletics. In the last 10 years - ever 'celeb' has embarked upon a 'personal branding campaign' to imbue themselves with moral authority. There was always a little bit of this - but now it's 'the thing'. Go and have a look at social media, try 'Kristen Bell'. Every few posts are about some kind of moral concern. Otherwise: they could get the ban. A host of globally respected individuals had to sign an piece in Harper's. This is basically 'shocking' to anyone of a certain age, because intellectuals and their ilk used to be the one's fighting for freedom of expression against the authorities - now, 'ban' culture has infiltrated their organisations and become a populist issue.
6) There are disruptions and protests around what would be normal, common, mundane events: 'The Joker' review by the Daily Beast was entitled: "Everything About the Joker was Absolutely Infuriating". The 'infuriating' parts had nothing to do with the film, but rather the supposed politics of the film. The DB and others wanted the film to not be released is it were. There are weekly arguments over who can play who in film, supposed issues of representation, wars over the composition of selection committees - often resulting in the 'cancelling' of individuals.
Twitter and social media have created the ability for agitators to move giant waves of people in an emotional, populist direction, resulting in quite a lot of effort to ban, cancel, disassociate.
It's far worse than ever before, and it's going to be worse before it gets better.
I see no pathway for it 'getting better' anytime soon, because the pathways to outrage have formed, hardened and are now part of our culture.
Individually - yes - most of us are getting sick of it - but that won't change the fact of the activity in the news, on Twitter, and journalists 'calling for the resignation' of so and so for this and that with the basic moral impetus of 'someone on Twitter said it, therefore, that's how America feels'.
CloudFlare, Facebook - your startup - does not want to deal with this, there needs to be some kind of collective clarity.
1, partly 5) Who are you to be the arbiter of what is 'supposed' to make people popular? It doesn't work that way. Can you read the mind of Kristen Bell and know she acts out of fear?
2, 3) No guidance other than the whims of CEOs... oh and congress. How does that add up?
4) On a slippery slope from yellow journalism to...
I'm sorry for the low-effort post, but something like this is quite tricky to respond to. Much like your view of the thing you're railing against, any legit points you may have drown in a sea of outrage completely lacking perspective and full of non sequiturs.
Cloudflare was purely acting out of market based fear, there wasn't a hint of moral impetus. Literally he said: "I don't want people saying they won't work with us" - which is giving into the mob.
Where is the ACLU on this?
We were all screaming for Net Neutrality just a couple years ago.
It's up to communities and governments to make decisions on content, it would actually help if the government made it illegal for CloudFlare to refuse service to someone so long as they were within certain guidelines, thereby absolving businesses of this issue.
Imagine literally the marketing and PR teams of Verizon, Facebook, Cloudflare, AWS, Google, your rando VPN provider, getting to decide if they 'think they might not like you' or not, it's just too much.
For marketplaces like AppStore, it's fine. But for other services, this is not going to work. It's not the job of your Telco or Garbage Pickup do decide if your public statements are cool/uncool enough for their Instagram.
> Cloudflare was purely acting out of market based fear, there wasn't a hint of moral impetus.
IIRC, the reason DailyStormer pissed off CloudFlare is because the users claimed (lied) about CloudFlare was somehow participating / sponsoring the site/activities. Sports teams have non-disparagement clauses; I don't see this as much different.
The only thing I'm not clear about is if it was just a rando user on DS that said the thing or some moderator/admin who can reasonably be said to represent the organization that runs the site.
Also, Macy's and Starbucks are allowed to deny their business to individuals at their discretion (subject to Equal Rights and Americans with Disabilities laws). Last I checked, being a Nazi is not a protected status (perhaps I'm wrong).
> We were all screaming for Net Neutrality just a couple years ago.
Different concept. This is about who a vendor chooses to allow as a customer. Common Carrier status is perhaps a closer comparison, but I think AT&T is allowed to drop a customer if they violate the AT&T ToS / contract.
Also, the CEO of CloudFlare went out of his way to publicize that this was a problem for the health of the internet and to start a conversation. Society didn't walk away from CloudFlare (eg. "vote with your feet/wallet") and Congress didn't choose to create any laws.
Citation for the CloudFlare / DailyStormer incident:
> The tipping point for us making this decision was that the team behind Daily Stormer made the claim that we were secretly supporters of their ideology.
This is more about enforcing ToS and maintaining reputation than "an internet infrastructure company cancels DailyStormer because of their ideology".
The complicating factor is that CloudFlare was not the only internet infra company to drop them. DS were rapidly dropped or denied accounts from other companies during this news cycle, so they were effectively kept offline because none of the large infra companies they approached wanted to deal with the issue in a "free speech over all other concerns" kind of way.
> Common Carrier status is perhaps a closer comparison, but I think AT&T is allowed to drop a customer if they violate the AT&T ToS / contract.
That would make net neutrality meaningless, surely. Just put something in the ToS that says you agree not to actually use your internet service.
> Also, the CEO of CloudFlare went out of his way to publicize that this was a problem for the health of the internet and to start a conversation. Society didn't walk away from CloudFlare (eg. "vote with your feet/wallet") and Congress didn't choose to create any laws.
Which is exactly the problem. America has given up on free speech, and CloudFlare was a prime mover in that shift.
When they're speaking for themselves? Sure. When they're acting as a common carrier? No. It's too bad regulation hasn't kept up with the realities of how important the Internet is.
> Society didn't walk away from CloudFlare (eg. "vote with your feet/wallet") and Congress didn't choose to create any laws.
Neither of those things mean you did the right thing. It’s really easy to pick on a widely unpopular minority group and not get laws passed against you or lose a noticeable amount of customers.
> Neither of those things mean you did the right thing.
I find it hard to believe the right thing would involve either CloudFlare tolerating libel (my interpretation) about them by a customer or that we should always tolerate an unmitigated amount of free speech (at least the obviously political/religious speech originally envisioned) no matter the cost to {business, society, decency, morals, etc}.
What is "the right thing" to you in this situation?
This misses the point to an impressive degree. No one here is disputing the current legality and I think everyone here understands the difference between government and private services.
The argument is that we've come to depend on privately owned backbone infrastructure in much the same way we depend on publicly owned roads. Furthermore, the operators of that infrastructure have shown themselves to be vulnerable to public outrage. Therefore, it's reasonable to ask if additional regulations might be a good idea.
A good analogy here might be to privately operated toll roads. Surely such a system shouldn't be allowed to discriminate against you based on (for example) your bumper stickers?
No, you can use any other service you want. CF has no private toll roads, in fact you can make your own 'toll' road right know. But if you have your private toll road you can forbid any bumper sticker you don't want on YOUR road.
> if you have your private toll road you can forbid any bumper sticker you don't want on YOUR road
If this existed in reality somewhere in the US (or wherever you happen to live) are you seriously saying that you would be ok with it?! Do you really not see the necessity of having neutral infrastructure?
> CF has no private toll roads
They are largely analogous. They transport your traffic from point A to point B for a fee. Yes, competitors exist. No, you can't just go build your own any more than you can simply whip up a viable Twitter or Facebook clone in a weekend. Claiming otherwise is either to be profoundly misinformed or to advance an argument in bad faith.
> > we've come to depend on privately owned backbone
> No, you can use any other service you want.
... which is also private and subject to the same pressures due to public outrage. Disputing that we have come to depend on privately owned basic infrastructure in general because we could switch to depending on a different piece of privately owned infrastructure is to again miss the point entirely. There is not (to the best of my knowledge) a government operated CDN or network backbone which you could make use of in order to avoid such concerns.
> There is not (to the best of my knowledge) a government operated CDN or network backbone which you could make use of in order to avoid such concerns.
The line between private and public gets a bit blurry when talking about edge routing and CDNs. Especially edge routing.
Then go to your Government and ask them to make such a Service, look there is just no point arguing about that, as a private firm you don't have to serve everyone, especially groups that are potentially bad for your brand.
There isn't always a line, some businesses are in the business of selling products and services and not in the business of policing content. And if some of their customers are pedophiles or the like then have the police arrest the pedophiles and leave the fry cooks and gas station attendants out of it, even if the pedophiles eat food and drive cars.
No. The reason why freedom of speech (or freedom of expression, if you're Canadian like me) is important in a government context is because the government (in theory) has a monopoly on violence. Businesses and private individuals should not be expected to uphold and enforce freedom of speech because requiring them to do so puts them at odds with themselves.
An employer expected to uphold freedom of speech must then require their employees to work with or for people who believe they are lesser people, not even people, or should be the victims of abuse, violence, and genocide. That is an untenable position from a human rights perspective. I as a private individual do not want to do business or associate with people who use freedom of speech as a platform to preach hate or ignorance, and that is my choice.
I am happy to see companies kick bad actors to the curb, whether they are fashionable nazis being deplatformed, gamergaters harassing women and minorities, and I am frustrated when I see legitimate journalists being censored by those same platforms. It's not cognitive dissonance to support stopping bad actors while rallying to support good actors, it's a recognition of the fact that our rights are open to abuse and that the current system isn't great at coping with abuse.
I certainly don't think that requiring businesses and platforms to provide guarantees around freedom of speech is a good idea, and most platforms that have attempted to or succeeded in doing so have turned into the cesspools of the internet.
> The reason why freedom of speech ... is important
This view is (IMO) far too narrow. Freedom of expression, particularly political expression, is absolutely essential to the functioning of western society as it currently exists. Personally, I also see it as an ideal to be pursued in and of itself regardless of any functional need for it.
Government regulation has a high potential for abuse for a number of reasons (the monopoly on violence is merely one) so it makes sense to take steps to constrain it in certain critical cases. Note that this doesn't imply anything about private entities; it is simply a logical consequence of a functional need or ideal in the context of our current system.
As to private entities (businesses, etc) things vary based on context. Certainly I don't have any wish (for example) to force YouTube to host pornographic content. However there may well exist cases where broader freedom of expression (either the functional need or the ideal) requires protection against private entities.
California actually has such a law - an employer is not permitted to take actions that would "influence or tend to influence" their employees political activities outside of the workplace. This can get very complicated (as you might imagine) when an employer takes an official political stance on an issue.
US telecoms are also subject to regulation of this sort (ie common carrier laws). Personally I think that infrastructure companies (Cloudflare and other CDNs, as well as those providing the physical layer) ought to be subject to something broadly similar. It would help protect usage of and access to the underlying infrastructure for everyone by shielding the relevant companies from negative public opinion (public outrage campaigns accomplish nothing if the company is legally required to serve all customers).
> ... have turned into the cesspools of the internet
Broadly, I think you are tending to conflate infrastructure providers with social networks (and similar end user sites). There are important differences (for example) between FedEx and an online retailer, even though they might both be involved in getting a physical product to you.
> Government regulation has a high potential for abuse for a number of reasons
Can you identify any government regulation that is enforceable where that enforcement isn't rooted in the monopoly on violence? Note that violence doesn't just mean capital or corporal punishment, it also depriving individuals of their possessions (for example, fines, asset forfeiture, etc), or their freedom. Each case where government can impose regulation carries the full weight of the threat of violence. If not for that threat of violence, the regulation would have no meaningful impact.
> However there may well exist cases where broader freedom of expression requires (...) protection against private entities.
I agree with you, but I don't think the answer here is to give government a mandate to dictate a policy of freedom of expression, I think the correct step is to evaluate whether a business has grown sufficiently large to constitute a threat to expression in society, and I think the model of being a monopoly is close to that. Disney is a stellar example; they don't own all of the movie studios, but their enormous size and ownership of the intellectual property rights to such a huge swath of cultural history for the last century, and the way they use those rights is growing increasingly harmful. I don't think it's right for government to say that Disney should start distributing adult films, but I certainly think it's appropriate to introduce regulations governing how a company like Disney can negotiate for, or more practically, dictate terms to movie theatres and retail outfits to carry or distribute their products.
> Broadly, I think you are tending to conflate infrastructure providers with social networks (and similar end user sites)
I do, but that is a separate discussion around the way that major platforms, cloud service providers, content studios, transit providers, and online retailers have become interconnected through acquisitions and competition that it has gotten to the point where I treat any online service as a "temporarily disenfranchised critical infrastructure". This is largely due to the idea that for many online services the end goal is to either achieve enough market share to become a competitor to another internet company, or to be acquired trying.
Suppose a guy walks into a grocery store with a shaved head and swastika tattoos. He wants to buy a bag of apples. The store owner doesn't like the cut of his jib and asks him to leave. That's fine, the guy goes to the grocery store across the street and buys the apples there.
The store owner across the street doesn't care what kind of tattoos a man has and is willing to sell apples to anybody. That's fine too, right? But then an angry mob descends on the store, filling the entrances and harassing the customers, accusing the store owner of being a complicit racist for doing business with Nazis. The store owner does some simple math and realizes that selling a bag of apples a month isn't worth this kind of trouble, so they start to turn the man away too.
Then the angry mob does the same thing to anyone who will do business with the man at all. He can't buy food, electricity, clothes, medicine, nothing, or the merchant will be the subject of a harassment campaign which will cost them more than they could get from his business.
This is clearly just as coercive as anything the government could do, if not more. Even they're not allowed to deprive you of food and medicine.
And as much as we might like the idea of making life impossible for Nazis, the target is only the current whim of the mob. Nothing guarantees that a given target is actually evil.
Or that they're paying enough attention to notice that the guy with the shaved head and swastika tattoos is a Buddhist monk and not a Nazi at all.
Which is why, when the government does it, they have an obligation to give you a fair trial and access to an attorney and respect your constitutional rights. None of which the mob gives you.
But yep, this is pretty much my thinking as well - that being ruled by the whims of the mob is hardly desirable from a societal standpoint because it ultimately results in many of the same issues that government restrictions on expression do.
We wouldn't be having this conversation if they had won the war - my family would be dead and I would never have been born. Excuse me for not protecting their right to speech (which leads to action).
Nazism isn't a point of contention anymore. We know what their rhetoric leads to (because they were allowed to rise and showed the world what the consequences are). That's why I said they lost the war. It's imho a pretty strong argument to draw the line right there, to ensure the sacrifices were not made for nothing.
Nazism is pretty popular on HN, comes with the territory when most of the board's users are well off westerners. Doesn't hurt that the mods are way more concerned with anti-capitalist rhetoric. Not surprising considering who owns the site though.
Maybe GP is referring to 1.1.1.2 and 1.1.1.3, the DNS resolvers that filter malware and malware+adult content respectively? Both are optional alternatives to the unfiltered 1.1.1.1 DNS service if so...
Why is it a precedent? Nobody was under the illusion that it isn't technically possible for them to shut down some customer they don't like.
Nor was/is there any debate about the legality of doing so,
So the only reason for continuing to work for/with violent anti-semites was that they wanted to. Until, at some point, they changed their mind.
HN has a strange infatuation with this idea of avoiding responsibility by pretending to be powerless. But it's neither morally sound nor logically or legally coherent to pretend to be bound by some principles that are entirely of one's own making.
Precedents aren't limited to the legal system. Their actions changed the expectation (ie established a new precedent) for their behavior going forward.
> So the only reason for continuing to work for/with violent anti-semites was that they wanted to.
Or entertaining broader ideals surrounding freedom of expression? Or perhaps seeing the inherent dangers of living in a society dominated by public outrage? Or something else entirely?
> Until, at some point, they changed their mind.
Due, by their own admission, to financial concerns. Concerns which wouldn't exist if they and all their competitors were legally required not to discriminate based on otherwise legal content.
> HN has a strange infatuation with this idea of avoiding responsibility by pretending to be powerless.
I don't see that anywhere in this comment section? Rather, I see people objecting to actions which are legal but which they feel should not be.
> Did they figure something else out that nobody saw?
I may not have a full scope of the history, but my own experience with DDoS protection was quite different. Whilst providers offered anti ddos protection through GRE tunnels and dedicated machines behind DDoS appliances and a heavy null route hand, Cloudflare had a simple few-click solution that worked at the web application level making things a lot easier and, also, allowing for features like caching, and thus, CDN benefit from a global network. Further, they've maximized performance on their machines, and as a result, Cloudflare is wicked fast.
Cloudflare does what it does really well and has built additional services on their global network that make a lot of a sense and provide a lot of value.
This has been my experience too, CF is just so easy to migrate to and to configure once you're there.
If AWS was able to offer a single click "DDOS protection and CDN" feature with similar pricing and features as Cloudflare then I'd consider it since most of our infrastructure is on AWS but at the moment they don't offer anything nearly as competitive. Just the Cloudfront bandwidth costs alone would dwarf our total infrastructure costs.
Indeed. I think I understand the argument claiming Cloudflare is a SPOF. However, if you're actually dealing with DDoS attacks with any kind of regularity, you're likely to have better uptime with them than without.
Cloudflare is a relatively cheap "lite CDN" for developers who want caching without putting in any work. This becomes a gateway drug for Cloudflare's more expensive plans once you outgrow the free plan. It quickly adds up; I worked for a company that wasn't even on an enterprise plan and was spending hundreds of dollars a month on Cloudflare just because they had a lot of domains.
My reservation with Cloudflare is the concept of letting a third party MitM my SSL traffic. That and it's more expensive than a cheapo CDN like Stackpath if all you really care about is CDN (and Cloudflare isn't even really a good CDN, just a quick hack to speed up small static files).
In my experience, it's extremely rare to find a CDN that doesn't expect to do TLS termination. My understanding of TLS is that it's exceptionally difficult to cache content if you cannot see into the requests.
> My understanding of TLS is that it's exceptionally difficult to cache content if you cannot see into the requests.
Not even "exceptionally difficult", but flat-out impossible. From the perspective of an observer, TLS sessions are random data. The protocol is specifically designed to defeat attempts to replay data -- a CDN is indistinguishable from an attacker in that sense.
Right, with Stackpath they do TLS but not necessarily on the primary domain. You don't have to point your nameservers if you don't want to. So you can set them up on a subdomain and use their fake SSL to serve purely static files (if the file doesn't exist on CDN at the time it's requested then their system will pull it directly from your server via a private subdomain, serve it to the client and store it for next time)
So in this way it's possible to setup CDN with shared SSL for purely static files but not the app server itself; you don't have to give the keys to the whole kingdom so to speak and it's cheaper than Cloudflare at the basic level.
Let me see if I follow. Auto-provisioned TLS (misleadingly termed "fake SSL") on the front-end for delivering static contents and caching. A private subdomain with a pinned cert not managed by the CDN to deliver static contents to the CDN. And a third subdomain for the application itself that's not going through the CDN.
I was under the impression that the same result could be achieved with Cloudflare, or indeed nearly any CDN. Was I mistaken? Though you may not actually need a secret, private subdomain for static files with all CDNs.
Again, please let me know if I've made a mistake somewhere. I'd love to learn something this morning.
You're right about that. So it might look like this
static.domain.com (CDN subdomain with auto provisioned TLS)
static-uncached.domain.com (private pass-through subdomain when CDN is missing a file)
www.domain.com (app server hosted wherever)
You're right that you could do something similar with other CDNs including Cloudflare (you can just set the www subdomain to "bypass Cloudflare" to accomplish a similar result), but I'm not aware of any way to use Cloudflare on a domain without forwarding your nameservers to them, effectively giving them complete control over the domain. At least with Stackpath I can host DNS wherever and simply point the subdomains I want at them.
Also, by the time you do the work to split static files into separate subdomains you might as well go with a dedicated CDN. One of the selling points of Cloudflare is for sites serving everything on one subdomain that they can forward to Cloudflare and get caching without any work.
Nope, but I'm sure lots of companies are in this situation, where they just barely need the features of the lowest paid plans, end up scaling across several domains and Cloudflare makes a ton of money off a vast majority of customers that will never need their more advanced features.
Plus that means hundreds of unique domains with unique content. If you have hundreds of domains all pointed back to one site with a single forwarding rule, 99/100 could be free tier accounts.
Unless you're sending all your traffic back to physical machines that you own locked into a cage in a datacenter, you are probably letting someone MITM your SSL traffic. For example if you are hosting on AWS, Amazon has access to your keys. If you are hosting on a hardware server leased from Hetzner, Hetzner has access to your keys.
When a 3rd party has access to your keys, their responsibilities to you are spelled out in your contract with them. That's true for CDNs as well as hosting companies.
There’s a difference between a VM host with the technical ability to carry out a targeted MITM attack against its customers using hardware-level access, and a provider that sells MITM as a service.
For most websites today if someone can intercept traffic somewhere close to the server they don't even need the keys, they can just fake responses to pass CA validation and issue valid certificates with their own keys and MITM like there is no encryption.
And coldboot attacks performed by a hosting provider staff of dumping memory and finding keys isn't that realistic of a threat, just like putting servers into a locked cage on someone else's property isn't much of a protection.
You don't need to use your SSL certificate for the CDN. Most websites that use CDNs only do so for static assets such as images, fonts, CSS, etc... the dynamic content comes from the origin server, encrypted with a private key that is not handed out.
If you press F12 in your browser and navigate to some major sites, you'll notice anywhere between 1-5 different CDN domain names that aren't directly related to the original host in any way.
Worth noting that cloudflare does a poor job of identifying and allowing crawlers specially Google. I have a site with 1 mil URLs , nothing big and every time you turn on cloudflare it reduces the crawler to 1k visits or less per day. If you compare your logs for visits for Google and Bing with CF turned on/off there is a huge difference.
Suffice to say, that in over 5-6 projects I have added CF to, it's never worked out in my favour.
As far as I understand it it's a freemium B2B model: If you're small you probably can get away using the free tier. Then when you get bigger you outgrow the free plan, either because you want specific features or because your bandwidth becomes too high.
That said bandwidth really isn't that expensive, at least if you're buying it at the scale that Cloudflare does. Many people seem to be used to the bandwidth prices of the large cloud hosters, which are really insane and have been marked up by a large multiplier to disincentivize people to transfer their data elsewhere for processing.
I don't really know, but I guess the biggest feature in the past was protection against DoS attacks.
Bandwidth isn't expensive compared to what most people are paying for it. Cloudflare is paying for the infrastructure as it is to handle attacks, so why not use it? As long as it doesn't affect paying customers then it's great marketing.
The NSA/CIA have a huge hidden budget. They are known to invest in startups. Why try to break crypto when you can just pay some tax payer money to large corps to get them to convince sites to mitm themselves.
Interesting perspective, but it seems like this is just an ad for easyDNS and their "Proactive Nameservers," though I couldn't imagine a better time than the misstep of a behemoth of a competitor in this space. Not to detract from the more important discussion about the internet's dependence on Cloudflare overall.
It may be just an ad for easyDNS' Proactive Nameservers [1] product but it provides a roadmap for one possible solution to this type of problem. From a quick reading of the marketing info, the solution can be summarized as "Provision, Monitor, and Fail-Over DNS Name Servers across multiple DNS-as-a-Service providers". The question I have is whether the following constraint is artificially introduced or not:
> We must be your domain registrar for this to work...
IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem. The question comes down to build vs buy and whether the solution is general enough to warrant an Open Source Software solution.
They want to be the registrar to be able to update your NS records. But ... that's not really important nor needed (So the answer to your question yes, it's likely artificial). Just use two anycast-ed IPs/domains. (Like Cloudflare.)
The magic happens at BGP level.
I considered CF as a domain registrar, but they don't allow setting the NS records. So you must use them. (They basically use sane no-nonsense domain registration as a way to gain leads for their main product. Pretty smart actually, because it's a great high-level add-on for their main product, but they just went ahead and made that the bait for everyone.)
Anyway, ideally, if you add 2 separate sets of NS servers to your NS records then you eliminated this SPoF, great. Sure, it's your job to keep them updated, and in sync (preferably, to avoid problems like half of your users landing on a different CNAME/IP/etc).
And recursive nameservers will handle the failover.
easyDNS has to be the registrar because only your registrar can change your nameserver delegation with the registry. This is, in essence, the registrar's job. To maintain your domain record and info, including nameserver delegation, with the registry.
You could do it with BGP, but it is non-trivial and you need your own ASN to do that.
But you can just add multiple DNS providers yourself. I mean you can add the namservers of both easyDNS and cloudflare. EasyDNS just automates this.
In theory they could simply create a few subsidiaries, let's call them saferDNS1,2,3 and have them build completely different redundant DNS architectures, and add then add the resulting nameservers.
That said, it'd be good to see an actual domain that uses this "proactive" feature to see what easyDNS is doing.
I'm not a DNS expert, so... It's not really that simple is it? If you have multiple nameservers I thought they get equal weight, don't they?
So if you have Cloudflare + (ex:) NS1, and you're using Cloudflare for caching, you need your NS1 records to return Cloudflare proxied IPs normally, but origin IPs under failure conditions. That's a lot of infrastructure.
It also fails completely if you're relying on Cloudflare for DDoS protection and IP obfuscation because a failure means your origin IPs get exposed. That's assuming Cloudflare DNS being down means Cloudflare proxying is down too. It might not be the case, but I think you'd have to plan for it.
Then there's also Cloudflare's detection of nameservers. I haven't tried it with more than Cloudflare's nameservers set for a domain, but if your domain doesn't actively use their nameserver they'll drop your site from their system. So, at the very least, you can't use Cloudflare as a secondary DNS provider (at least the last time I checked).
I was talking about just the DNS layer, but ... you can periodically check what IP Cloudflare would return and return that. (There's also the apex-CNAME record type, ALIAS or DNAME, I don't remember right now, but PowerDNS supports it, and you can set up the resolver to use CF's NS.)
Of course CF doesn't support anything like this, but it works well (because they use quasi fixed, static anycasted IPs for the HTTP(S and TCP?) proxying/load-balancing too), even if it's hacky as hell.
If they have any active verification of nameservers, and if they disable the proxies if they detect something bad, then ... it won't work obviously :)
But technically there's nothing amazing in being a registrar of a domain. So both CF and easyDNS are just stubborn in the name of user experience (consistency).
... all in all, working around any SPoF (reliably) will usually require exponentially more resources/engineering/care.
I meant that easyDNS should handle the BGP for its clients, without requiring their clients to use them as registrars.
There's HE.net's free DNS, and though they don't explicitly advertise as, it's anycasted. (Check via https://tools.keycdn.com/ping , try 216.66.80.18 [ns5.he.net].)
> The only requirement to use Proactive Nameservers is that we have to be your registrar, because we need to connect to the registry to update your nameserver delegation.
So I guess technically this could be achieved with an API for your domain settings.
> IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem.
The classic way of doing this is AXFR (your own DNS server is a "hidden master" and the DNS providers are the slaves).
The problem is you won't be able to have redundancy at the registrar level, but that has historically at least been less of an issue.
> Proactive Nameservers is a patent-pending system that optimizes the nameserver delegation for your mission critical domain names.
That's a huge negative and I can't believe they think it's a good marketing point. I don't want a patent encumbered, non-standard solution for critical infrastructure.
> We must be your domain registrar for this to work.
So they're updating the domain record at the registry level to facilitate failover? That's the only scenario I can think of where they _need_ to be your registrar. Assuming that's the case...
I've always seen 24-48 hours quoted as the worst case wait when updating nameservers at the registry. I've never seen an explanation of how it works, what's allowed to be cached, how long it actually takes to update, etc.. How do they do it in a way that's suitable for failover? Do they have a special SLA with registries?
How would the registries handle a deluge of nameserver updates? Imagine a Cloudflare scale failure and corresponding registry updates. Would the registry servers be able to handle it?
I'd love to see a technical explanation of how their proactive nameservers system works.
I think people are Cloudflare fans simply because so many other services suck more.
I gave Cloudflare a fair shake. But after hearing their lies way too many times, I'm calling them out for being deceptive and unscrupulous.
After being told that sites pretending to host Adobe Flash updaters and pretending to be Bank of America can't be taken down by Cloudflare because of their rights to free speech, I knew their attempts to pretend to be one of us, attempts to pretend to care, were nothing but bullshit.
They claim they don't host. If hosting DNS is not hosting, then what do you call DNS hosting? You literally CANNOT use their abuse web form to report domains which use Cloudflare just for hosting DNS. They do not handle abuse sent to their abuse email address (they simply send a form response saying to spend ten minutes filling out their crappy form that has all sorts of problems).
Of course, their web proxy services are also "not hosting", even though they're protecting all sorts of scammers.
So why should we think they're not bullshitting us when they run 1.1.1.1 and tell us they're not logging? Why should we trust them more than our ISPs by running DoH through them?
They WANT us to be dependent on them, because the more control they have, the more money they can make. It's dangerous, and they've shown they have no honor.
I've genuinely tried to correspond with them on Twitter, and they excel at not answering the question asked but instead just diverting. It's scummy, unprofessional behavior and I encourage everyone to consider whether they deserve anyone's data or business.
I think this is an important point about CloudFlare that can't be made often enough. It's been some years since I first noticed, and it seems as true today as it was then: wedging themselves into core Internet services and data flows seems to be an intentional part of CloudFlare's strategy.
There is no case given the architecture of the Internet where one company need be exposed to so many traffic flows from millions of people. The search engines got there first and we eventually stopped complaining, but this does not make it any more justifiable to copy the model.
What reason would a company have for desiring this outcome? We know Google can detect and predict flu outbreaks. Imagine what is possible when you have every click on every target web site.
There is a fair chance their data is already approaching the comprehensiveness of Google, and I'd be surprised, if not disappointed to learn they were not already working on unannounced (now or eternally) internal intelligence products based on that data. There are simply too many pockets who would be willing to pay for it.
> the company serves at least seven groups on the U.S. State Department’s list of foreign terrorist organizations, including al-Shabab, the Popular Front for the Liberation of Palestine (PFLP), al-Quds Brigades, the Kurdistan Workers’ Party, al-Aqsa Martyrs Brigade, and Hamas.
> CEP has sent letters to Cloudflare since February 13, 2017, warning about clients on the service, including Hamas, the Taliban, the PFLP, and the Nordic Resistance Movement. The latest letter, from February 15, 2019, warns of what CEP identified as three pro-ISIS propaganda websites.
All this while routinely censoring other sites. Their actions are very different than their words.
“ This question assumes the answer. A website is speech. It is not a bomb. There is no imminent danger it creates and no provider has an affirmative obligation to monitor and make determinations about the theoretically harmful nature of speech a site may contain.”
Well that would depend on what the target of the DDOS is wouldn’t it? Either way the point being we want companies out of politics. They have no business in them.
Yeah, Cloudflare shouldn't be censoring anyone. I could understand if it were requested by law enforcement but why would we want them trying to police the web?
refusing to provide your services to someone is not censoring them. did they black hole their dns entries in 1.1.1.1? did they steal their domain name? no. companies fire customers all of the time.
That is the problem with massive centralization even if it is market level and internally Cloudflare (or any other big fish) does decentralization/fail-over of their own. Many of these companies should have had fail-over to competitors at least for reliability.
The problem with near market monopolization, oligopoly, even the singularity, the fail-case is catastrophic and may even wipe out decentralized, diffused, dispersed, decoupled system solutions that can't make it due to so much relative size from the big fish that it squashes them along the way. The bigger the ship the longer it takes to turn.
This Cloudflare issue is like the recent Facebook SDK startup crashes where everyone has a single point of failure on Facebook SDK where people should be using or able to use the OpenGraph API directly as they need which is more robust to the app that uses it, it won't crash on startup.
In business it is a goal to centralize to grow, in nature and robust systems it is more differentiation and decentralization to survive. There will always be a push and pull between these two forces.
Systems and markets are like gardens. The garden must be maintained, new seeds planted and helped to grow from small to mid-sized, mid-sized plants the bulk of the garden, and then the larger plants need to be culled back when they get too big to not take the mid-sized and then all the resources from the new seeds/small plants. The problem is we have allowed the top end to take over the garden and when they fail they fail spectacularly. The bigger the scale the bigger they can fail.
The problem is customers choosing to put all their eggs in one basket.
Until relatively recently absolutely none, and now almost none of the tooling allows effective multi-cloud or hybrid cloud/private.
Basically the cloud providers work very hard to prevent the commodification of their services with special incompatible service offerings, lock-in, interdependency, deep and opaque APIs for integration, and networks of training and certification that position change as a direct threat to people's job security.
Cloud providers today are basically Microsoft in the 90s.
Much as open source challenged Microsoft, I would say that the world now needs open infrastructure tooling that positions hybrid and multi-cloud as first class infrastructure architectural cases in order to displace established cloud provider hegemony. Even then we will have to fight the hardware and real estate economies of scale available to large established cloud providers.
I wrote some observations about this space based significantly on HN community comments prior to the rise of Docker a few years ago: http://stani.sh/walter/pfcts/ ... click 'original' ... the conclusions still seem timely.
> Until relatively recently, the cloud didn't exist.
Depends:
> IBM and other mainframe providers conducted this kind of business in the following two decades [after 1961], often referred to as time-sharing, offering computing power and database storage to banks and other large organizations from their worldwide data centers. To facilitate this business model, mainframe operating systems evolved to include process control facilities, security, and user metering.
[…]
> In 1998, HP set up the Utility Computing Division in Mountain View, CA, assigning former Bell Labs computer scientists to begin work on a computing power plant, incorporating multiple utilities to form a software stack. Services such as "IP billing-on-tap" were marketed. HP introduced the Utility Data Center in 2001. Sun announced the Sun Cloud service to consumers in 2000.
[…]
> In spring 2006 3tera announced its AppLogic service and later that summer Amazon launched Amazon EC2 (Elastic Compute Cloud).
True, however at AWS at least, customers are specifically told "multi-cloud doesn't allow you to fully leverage the benefits of AWS", whatever that means.
It makes sense that cloud companies are inclined to keep customers from giving money to competitors, but they way they sell it and structure services, reserved instances, and enterprise discounts is such that you basically are putting all of your eggs in one basket.
> "multi-cloud doesn't allow you to fully leverage the benefits of AWS", whatever that means.
One of the selling points of cloud providers is managed services like SQS. If you run a multi-cloud architecture, you either can't use managed services, or have to build abstraction layers on top of them (and only use the features that exist in both cloud providers' versions of the managed service).
If you want to use a managed service that only exists on AWS, then that's obviously incompatible with a fully multi-cloud architecture.
> If you run a multi-cloud architecture, you either can't use managed services, or have to build abstraction layers on top of them (and only use the features that exist in both cloud providers' versions of the managed service).
And this is, of course, why they do everything they can do discourage it. Because if you do that, not only are you not reliant on them for availability, you can switch more of your business to the other provider(s) based on current pricing, and they do not want that big time.
I thought that was something trivially obvious stated about fully leveraging the benefits - that they don't control other's clouds and thus they can't use all of the same sorts of performance or efficency boosting tricks. (People would be way more mad if they did as it would require hacking into say Azure to gain root access.) Not a domain expert but it is my interpretation.
If you make the engineering decision to go multicloud for whatever reason those are inherent trade offs you need to be aware of. They have their own agenda of course in addition to any actual fundamental "real" in bulk efficiencies that price reflects.
Cloudflare has done really well in a short amount of time, but I'm sure Akamai is way bigger. Cloudflare is best known because they offer free/cheap tools for individuals and this post is targeted at developers. Most big enterprises I've worked with are on Akamai.
How do you do a fallback from cloudflare failing when they’re your dns provider too? Any redirection around would take too long to change wouldn’t it? They’d be back up and running before it was implemented. What’s the right approach?
I don't think cloudflare supports such a configuration.
The DNS is part of the load balancing, they serve different IPs based on location of the DNS query.
Edit: Apparently they do support a CNAME configuration if you pay for one of their business plans. That gives you the option to quickly switch away (if your TTL is low enough) but will impact performance by having to fetch the CNAME every 60 seconds.
Does cloudflare actually do geo-loadbalancing via DNS A records now? For years they only did anycast, unlike, say, Akamai, which hands out different IPs for each POP.
Actually, I'm not sure if they do any DNS geo-loadbalancing. I've seen it report different IPs from different locations at times, but that could be something else.
But I'm pretty sure they use DNS do other loadbalancing and DDoS mitigations.
For example, if a site is under attack, they can send it to different IP addresses to keep it away from other sites. Or if someone is directly targeting a cloudlflare IP with a DDoS, they can redirect all sites to other IPs and just blackhole that IP.
Where do you put your DNS servers? On-prem is going to be less reliable than cloudflare which has a 100% uptime SLA. I doubt your local ops team can compete.
> Many of these companies should have had fail-over to competitors at least for reliability.
How would you even set such a thing up? I fear that you might get a couple of collusionary companies that bail each other out and smaller providers might just be left out to dry…
Many DNS providers have the ability to do AXFR "zone transfers", so you can sync your records to a secondary provider and you would add a secondary set of nameservers for redundancy. Unfortunately Cloudflare doesn't offer this unless you pay for their Enterprise plan (they started offering it earlier this year).
I do love using CloudFlare for DNS, lots of great features and generally works well, but I wish they would support AXFR for the lower tiers. I've been working on a solution for this using the CloudFlare API, but we'll see how well it works out.
Back in the day (I haven't run my own mail server for years and years) there was a company called Secondary MX.
That's all they did. They weren't a mail host or provider, there was no UI, nothing. All they did was allow you to specify them as, well, a secondary MX, so if you were offline they'd cache your inbound email until you were back. Simple. Efficient.
I think they want to price discriminate: the costs from AXFR should be minimal, as it is an old technology, very optimized and low bandwidth.
However, cloudflare decision turns cloudflare "free" offering into a free SPOF. They should extend this offer to their free users, who could then use the secondary DNS that most hosts/domain name sellers provide for free to the IP that is proxified by cloudflare. It could even be limited to the case of proxy or cloudflare DNS failure, so that cloudflare could still price discriminate (make AXFR fail, unless cloudflare is down, like a dead man's switch)
Fwiw, you could use stack overflow DNS Control to manage your DNS records and upload them to many providers. Then the only thing you'd have to do is flip a line of code to fallback to the other DNS provider.
Amazing that this isn't "free" and folks have to resort to syncing records across DNS providers with proprietary, vendor-specific APIs. AXFR is standard. It's not just Cloudflare... AWS and Azure don't support it either.
The article posted actually talks about EasyDNS's implementation of exactly this.
> At easyDNS we experienced so much pain from this reality that we created a system to automate flipping DNS providers at the first sign of trouble.
> We call it Proactive Nameservers, and we’re the only company in the world doing it for some reason. Maybe this is because in order to provide a service like nameserver failover, it means a company has to admit to its customers the reality that their own nameservers may at some point, fail.
I imagine the implementation could look similar to how cell phones can use other carriers networks (for 911) when they don't have signal from their own carrier.
The question is: Can we do better? A natural monopoly is a good thing. It just means that the natural monopoly needs to build its own redundancy. Cloudflare total failure isn't common.
I wouldn't classify a natural monopoly as a good thing. It mostly signifies high barriers to entry for competitors and a market that's more susceptible to failure, as seen here.
I guess but there's not really a shortage of commercial CDNs. A market with lots of competitors but a clear winner says more about consumer preference and network (ha!) effects than it does about Cloudflare's moat.
I think for the sake of a diverse community and economy, monopolies should be difficult to achieve even naturally.
But for this specific example, a robust technical redundancy doesn't stop CloudFlare from going out of business. A technology company going out of business is pretty much the norm. Incumbents are a relatively new phenomenon for technology (sans some key exceptions), and I don't think CloudFlare is an incumbent. They are an accessory, and your business would probably run without them.
Cloudflare isn't a monopoly. Cloudflare doesn't even have a particularly strong moat when compared to Fastly or Akamai. Cloudflare doesn't even have any network effects.
I must disagree, and wish I had more data on how many sites host with cloudflare because it's free.
I just checked, fastly is $50 per month, and Akamai doesn't list pricing so let's assume it's more.
Cloudflare has had network affects from integration with wordpress and cpanel too I believe for some time now.
without cloudflare your site can be taken offline by any random person willing to spend $20 for ddos sellers.
The free plan is a pretty big moat imho, especially if 'half the internet can fail..' - I doubt 10% of those sites would be using cloudflare if there was a monthly fee pushed on them.
Admittedly, my cloudflare usage is only about 30 sites, so my data point is small. The few hosting and design clients I have and their budget constraints are not indicative of 'half the internet' - but I don't believe fortune 500 and SV sites are either.
If you have a "monopoly" which is this case just means you're the market leader by a wide margin, but in theory this could mean near total control of a market but the reason you have this total control is because your customers choose to buy from you over your competitors simply because you're better and or cheaper then this is a good thing. Customers are getting what they want. And the market leader in this situation will have a hard time abusing their position because any they'll have competitors nipping at their heels if they slip up.
I would personally phrase it pedantically as not that the natural monopoly is good but that it exists because they are good. A fine distinction and a baked in assumption admittedly that if they were to no longer be good they would no longer dominate. Technically there are some other variables like if an extended stay on top would atrophy any competitors or not and the time scale operated upon.
On a century scale individual murderers aren't a huge concern to society because they are either dead or infirm by then.
You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
What's funny about this outage is I'm sure many of of us (myself included) used this window to analyze large services and determine an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage! The situation has exposed just how critical Cloudflare is.
I went ahead and bought calls ahead of NET earnings next month. Cloudflare is becoming an increasingly bigger part of the internet backbone. Purely speculating here, but I wouldn't put it past AWS or another large player acquiring them soon.
> an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage!
Careful about this methodology. Some services at my org were impacted despite not being direct CloudFlare customers. They had external dependencies that used CloudFlare.
So it's much bigger proverbial 'blast-radius' lest something happen to CloudFlare? Can you elaborate a bit on that part? I'm interested in knowing more.
> I wouldn't put it past AWS or another large player acquiring them soon.
I really hope it doesn't come to this sadly. I'd be okay with DO or somebody who isn't as massive doing a merger. Maybe they can pull resources to make each other even successful and maintaining reasonable independence.
“Cloudflare apparently fat-fingered a routing update and sent all of their global traffic to a single POP, vaporizing it almost instantly.”
Made me chuckle, as it gave me the image of a large server in some massive server farm glowing red, then bursting in a massive burst of light as dozens of bearded Sysadmins run out of the building screaming.
That's not actually a contradiction. When an individual website is considered as a system, it will have multiple points where the failure of a component inhibits the system. It's possible to have both cloudflare and your database as "SPoFs" and that "single" is not meant to imply everyone only gets one.
It's absolutely true that if us-east-1 in AWS has a bad day, a significant fraction of the American digital economy will shut down. For some companies, the same is true of Azure and Google's various comparable offerings.
I read your post as skeptical. Why would you be skeptical? If you care about keeping your product up, you absolutely should have a fallback for cloudflare if you're a customer of theirs. Now, you might not care (and actually, for most folks I submit you need not care), but the folks making sure Ambulances get timely push notifications and realtime driving instructions probably care quite a bit.
This is why AWS is puts effort into blast radius reduction.
The Physalia paper is a wonderful read that explains both the justification and high-level approach to implementing said isolation [1].
Another great example is shuffle sharding [2], notably used in Route53.
The end goal of this is that aws outages would be isolated to subsets of aws customers where possible. So instead of half the internet failing it's only 10% (obviously the exact numbers depend heavily on implementation).
Doubly ironic since in doing this, they created a system where a protocol is the SPoF; namely nonsensical or false BGP advertisements can quickly kill the internet as a whole if done correctly.
it's a series of single points of failure on different levels, so, not multiple points of faiure, but much worse, single point after single point, which means the house of cards fails as often as any of them. The internet of 2020 is a monolith
On an unrelated note. Thank you so much for PortableApps.com! It was invaluable at university when studying in the library. To this day, I still use it for utilities that don't need to be installed on my desktop.
I am also grateful for PortableApps.com. Along with Scoop and Homebrew, they make using a system without root or admin privileges a really nice experience, in both Windows and Linux.
My sincere thanks to John and all the PortableApps.com contributors.
You're welcome, I'm glad it's helped and helping you! I try to keep it growing and relevant with a more synced cloud folder/work or school laptop/keep work and personal separate bent these days.
Okay here's the thing. I'm okay with people bashing other (competing) companies when they do wrong. However, I believe it is somewhat childish and uncalled for to bash another company, because of a mistake.
First of all "I use easyDNS so I didn't notice it at all tbh" is not only a childish assertion, it's borderline a falsehood. You DO NOT offer the same services, nor the same scale. (No, VOD would not work if your VOD provider used Cloudflare's offering.)
Second of all, as some have noted in other comments, you are very welcome to get just as big as them if you can offer similar (excellent) service and similar extremely competitive pricing. Otherwise, keep working on your offer and stop going for low hanging fruit like bashing the competitor for an outage when they literally might handle 1000x your traffic, and perhaps offer 20x the services your offer.
Honestly I didn't notice the domain name, and I actually thought the author was being quite understanding by saying things like "This is inevitable and unavoidable and entirely excusable. Everybody blows up, every DNS provider in existence will experience downtime. No exceptions." and that they use Cloudflare themselves.
This is obviously subjective, but to me it didn't come across as "they suck use us" but rather pointing out the inherent flaws in this quite popular SPOF and cautioning to avoid it.
I think the root cause (which, IMO, you correctly point out) is lost on many modern developers.
For whatever reason there's this modern idea that if a company A is paying money to company B for a service, that company B will handle all the 'hard stuff' for them.
The end result is we have a lot of applications/infra built with SPOFs, in some cases known, but in many, swept under the rug and abstracted away to passing the buck in case of a large failure (i.e. major AWS/Cloudflare/Azure outages).
You also see this at times when vendors pitch internal software solutions. I've been at more than one shop where a vendor's 'silver bullet' turned into a SPOF time-bomb because nobody considered this company's solution could fail. After all, the sales presentation said it had %nines%!
"Your app will go down when half the Internet goes down" is not that big of a deal to most software companies, because:
1. no one's going to blame me if my app goes down when half the Internet is also down but they are going to blame me if my custom solution to the same thing causes an outage,
2. there's no way my custom solution is going to achieve the same uptime. AWS/Cloudflare/Azure are not perfect, but whatever I roll for myself is almost certainly going to be much less perfect.
They do blame you though, most people won't be aware of the real issue, when cloudflare went down, the trending things on twitter were #spotifydown and #applemusic
Regarding 2, that’s exactly what the blog post was advocating for: Redundancy. They weren’t saying give up on Cloudflare (as they mentioned, they use it themselves).
> However, I believe it is somewhat childish and uncalled for to bash another company, because of a mistake.
The thing that they're bashing is not the mistake, which happens to everyone. They're bashing SPOF:
> EasyDNS was unaffected because while we do use Cloudflare to soak up large DDoS attacks against our nameservers, we don’t use them across all of our nameservers. I think somewhere in my book I wrote “DNS providers have a near-pathological aversion to SPOFs” (Single Point of Failures). Maybe only we do.
Well if you are going to rant, you might want to rant and quote what they ACTUALLY said, not what you conceived in you mind what they were saying
You indicate they said " "I use easyDNS so I didn't notice it at all tbh" but NO WHERE IN THAT ARTICLE was that statement
The actual quote is
"We’re familiar with Cloudflare’s DDoS service for DNS providers, because we use it ourselves. Fortunately easyDNS was not impacted by the outage (I didn’t even notice it, tbh),"
This is a MUCH different statement than you attempt to cast out as call "childish". He is stating that the services they use of CloudFlare was not impacted by the outage
I guess this is off-topic, but the stock photo they're using is cracking me up.
Guy at work... coffee cup on a tablet he's using for a coaster... except he's also drinking whiskey from a beautiful crystal glass... there's a folded paper airplane... there's just so much to unpack here, it's pretty hilarious.
> except he's also drinking whiskey from a beautiful crystal glass
And he has left the stopper off of the decanter like some sort of animal.
I think we are supposed to believe that the person here was just dicking around online, fiddling with paper, finishing his morning coffee, when all of a sudden he gets an email asking if anyone knows what's going on with the website.
So he stops what he was (not) doing, puts down his coffee, and starts poking around. At which point he realizes he needs something stronger than coffee. As he is pouring his glass he is confronted with the true horror of the situation, drops the stopper on the floor and just holds his face wondering why the Universe hates him.
I wonder if we can get JJ Abrams to option the movie rights.
Can you explain a bit more how this is a solution? Cloudflare isn't facebook. They have plenty of competitors, they don't have a massive moat, and they are almost exclusively used by businesses. Despite all of this, we should ask ourselves how we even got here. Why would companies move to DWeb, when they are already choosing to use Cloudflare instead of Fastly/Cloudfront/Akamai.
The more people that use anything other than Cloudflare, the better. I had this conversation with people back in ~2010 about Facebook and they all ignored it. They will again, but this time the consequences of centralization will be even worse.
It won't just be one single website that goes shitty with blockages and manipulation and censorship. It won't even be just the web. When Cloudflare achieves their goal of deep packet inspection at every peering and transit point it'll be the end of the internet as we knew it and the slow transition to just another cut apart "China-net (tm)".
Compared to the endless content marketing Cloudflare posts [1]? It's an ad, but they're still right. That's just good content marketing (informative, relevant, and perhaps you buy something because of it).
Whether this is an ad piece by a competitor or not, the problem with monopolies is that "the market" (if there is one) gets skewed incentives. Cloudflare has received heavy investment by FAANG (+MS) [1] before their IPO, so rather than eg Google or others with a vested interest and capability stepping up the game and invest into new IP control plane-level DDOS protection standards or similar, the situation smells more like a backdoor deal, such as an agreement to not go after a particular market segment.
Let's also not forget Cloudflare in particular have been accused to host/hide the very bad boys that make protection from DDOS necessary in the first place. Whether or not that is the case, a quasi-monopoly leaves customers with no choice.
If only DDOS attacks were taken seriously and their perpetrators punished accordingly (and maybe if the network had better ways of self-defense) instead of companies and websites having to fend for themselves (or having to resort to solutions like Cloudflare).
What I find really shocking is the abundant use of CF on piracy websites of all things. Not the serious ones of course, SciHub and library genesis are mirrored differently.
But a lot of small torrent websites and such simply won't load without JS and specifically CF code. It's pretty crazy. Luckily I don't use any of those bEcAuSe IlLeGaL but still, I find it really depressing, especially when webtorrent, IPFS etc are available, and frankly many of those pages will never have to bear a load that makes CF a requirement.
Now that Cloudflare is also a registrar, they could pretty easily implement a nameserver failover like EasyDNS. I hope this event underscores the importance of that to them.
Does anyone know a dashboard/list for past Akamai outages. Surprisingly, I haven't seen a lot of news about Akamai downtimes. I searched on Google and the last one I found in a news report is from 2011 when its customers like Facebook, Twitter were impacted.
They were a PITA to work with when I used them in the past but if they are really that good in service availability, you can have some justification for their overpriced service.
Well, I usually block it anyway, because it is not only a single point of failure, but also a single point of concentration of data, that can be used to track, spy on and profile users. As such, I do not blindly trust Cloudflare. I remain sceptical, no matter how positive their public image is. Especially I would not set my DNS to cloudflare.
I love all the comments about fail-over for DDOS/DNS protection. What is your budget? Well we are looking at around $0.00 for our maximum allowance. Ok so single point of failure it is I guess we are done here. Companies only say they care when there is a problem, the reality is that they dont.
It obviously takes more than $0.00 but not that much more. It’s a matter of adding a second DNS provider and making sure you replicate DNS records manually or with AFXR.
What’s really puzzling is if the same companies spend money on active/passive failover for application and database servers while overlooking DNS single point of failure.
Precisely. I remember when the CEO of my old company came to me and said (with respect to moving from a on-prem model to SaaS), "What's our SLA going to be?"
"Well, what do you want it to be?"
Give me a number and I'll tell you how much it will cost and how long it will take to get there.
AWS in 2012, DynDNS after that, now Cloudflare...I wrote about this a few years ago, the threat of the singularity of the Internet. What was distributed will be centralized again. http://tuxlabs.com/?p=430
Cloudflare is hell when you need to load scripts and css 403'd by them.
Here's my work around:
1) open developer tools
2) refresh page
3) move tab into a new window
4) find a blocked resource in the network tab of developer tools and open it in a new tab
5) verify humanity
6) repeat step 4 and sometimes 5 for each resource the page requires
7) move the problematic site out of the new window and close all of these tabs at once
It is a terrible experience. CF devs and publishers have no idea how inconvenient their service is. I wonder if they have ever lived in a region where every ISP is both incompetent and listed in the CBL?
I frequently skip sites that use CF. The captchas are obnoxious.
The entire concept of a webapp firewall seems a bit backwards to me. Developers should fix their insecure applications.
When I was figuring out how DNS works back 10 years ago, I was told "Why don't you just use 8.8.8.8 everywhere? Why do you need your own DNS server for anyway?" every single time when I asked a very specific configuration question. Some years later 8.8.8.8 was just replaced with 1.1.1.1 with same critical counter questions instead of helping. So, it appears to me, instead of understanding DNS and routing, people and tech-bro businesses take shortcuts by using centralized infrastructure for their systems, creating dangerous configurations. Now, Cloudflare have made a mistake, and half the Internet crumbled down. See the irony?
> Is it realistic for a small-medium sized business to have more than one DNS provider?
Yes: as the weblog post points out, you can have EasyDNS as your master with their multiple DNS servers, and then also have (e.g.) Route53 slaved to EasyDNS and have those in addition to EasyDNS in your records.
Question: is it even possible to have DDoS protection without using a provider of it which becomes a single point of failure? Or is it maybe possible to decouple this single feature from everything else that Cloudflare provides that could take out all the sites in the future from an unrelated misconfiguration?
I don't see the centralization as a positive, but I'm wondering what percentage of the websites that were taken offline see themselves as having no choice but to use Cloudflare in order to prevent themselves from being taken down anyway from malicious actors instead of by accident.
I think you could use Cloudflare as your primary DNS provider and benefit from their DDoS protection, but also specify backup DNS name servers with a different DNS provider in case Cloudflare fails.
Sure, Arbour and others sell devices to deal with ddos and many isps have clusters of these which you can use. Of course this is a service that costs money.
Starting yesterday, I have been getting Cloudflare's hCaptcha image identification quizzes on every site that uses them. Googling the issue indicates that the IP address range may have been blacklisted, if say my laptop (a Mac that doesn't visit any shady sites) or my router has been compromised and is now running a botnet or something.
It seems unlikely because my phone can access sites on wifi fine, and again my Mac doesn't appear to have any malware.
Could it be that the outage is somehow relate to Cloudflare putting these captchas up as a precautionary measure?
It's interesting to me that Cloudflare doesn't really have any competition with a similar business model. I suppose the free plan requires quite a lot of spending before the upgrades offset it.
There are other large CDNs like Akamai. They just don't compete with Cloudflare in the consumer sector. It probably doesn't matter for them because enterprise contracts are where all the money is at.
AWS, Azure and Google Cloud all offer competitive alternatives for the small business sector. Personally, I find the actual cost of CloudFlare very difficult to reason about in advance of actual use. This is in part because it's easy to miss a bit of the a la carte model you need.
Full disclosure, I work at Google on the SRE team for Cloud CDN. If you want a credible alternative for the CDN part of Cloudflare, our product is extremely fast. We were all very sad for Cloudflare and watched the whole affair closely, as we've got a few customers that use our services alongside Cloudflare's.
Depends on what you want but a very solid free CDN and DNS option is: host your site on netlify and use dns.he.net for your nameservers.
Another good DNS option is dnsimple.com or, indeed, EasyDNS. For even more redundancy, use one provider as your domain registrar and another for your nameservers (and set short TTLs for your zones so you can re-point IPs quickly if you need to).
For the other things Cloudflare offers on their free tier, I'm not sure what good alternatives exist (there must be some, I'm just not familiar with them outside of the obvious AWS alternatives).
Edit: one caveat with above advice, I have no idea if netlify use cloudflare behind the scenes...
> But if you want to use a preferred DNS provider, such as Cloudflare, who use their DNS responses to optimize your website proxy. That works best most of the time, so then you want to go with an active/passive model that will step back when things are going according to plan, and then when these periodic network cataclysms do occur (and they will), they step into the breach and update your nameservers so that you at least stay up until the crisis is over.
Copy editors are cheap and your reputation shouldn't be.
My guess is Wall Street HFT or other financial areas with very strict unscheduled downtime penalties where they are effectively incentivised to be batshit paranoid as it would take decades for any penny pinching to remotely pay off. I don't know if many of them use Cloudflare for their domains though.
There’s a lot of truth to this. Cloudflare for now has achieved IBM status in that no one will be fired for choosing Cloudflare, in spite of any issues that arise.
High availability is an insurance game, and perhaps we need to start treating it that way.
Rather than admitting that your customers need to maintain a business relationship with your competitors, you need to admit you need to maintain a business relationship with your competitors. That we need a moral equivalent of underwriting in the cloud space.
That is a fair point; SPoF depends on what level you're looking at. A RAID array removes the SPoF that is a single disk, but still leaves a SPoF in the RAID controller or CPU or power supply; a ceph cluster can withstand the loss of a whole rack but could still fall to certain software bugs. Likewise, "cloud" companies are internally redundant, right up to the point where they aren't. It depends on how you scope the question.
Maybe it would make sense to have multiple independent sections of backbone at the BGP level. Instead of having one public AS/backbone, break it down into regions at least so that it is more confederated.
Similar problems are happening with crypto - yeah it’s distributed but so many people are using Coinbase that if they go down it’s going to cause a lot of problems
Yes, but cloudflare reach is much deeper then that.
I am not a cloudflare customer but my all websites failed that day. The reason was digitalocean uses cloudflare, I use digitalocean. So apparently I depend on cloudflare.
Militaries. The internet was created so the US President could command the US military even in the case of nuclear war knocking out many of the internet's nodes.
Reparations would show an actual sense of responsibility. Firing someone would be appropriate if they were negligent. Other measures might be more appropriate. Is it enough that any time there's a Cloudflare incident, all we get are lengthy blog posts and sorries from Cloudflare?
I understand the point being made here, but what are those affected supposed to take away? Cloudflare made a mistake that caused (x millions of dollars of lost online commerce revenues, y number of missed telehealth sessions, etc.) and since we do not punish mistakes, nothing was done. Sorry everyone!
Understood. Typically, as a company, you write -
1) what went wrong
2) how did it go wrong
and
3) what you have or will put in place to prevent it from ever happening again
It's a learning process for all involved, really.
From the affected parties point of view, well, they should diversify their network a bit better. End users should hold those companies feet to the fire, not Cloudflare's.
Firing people for making mistakes is just going to foster a culture of secrecy and shame. Being so quick to fire is not how you retain talent and it isn't how you foster a healthy, blameless development culture in your workplace.
> Firing someone would be appropriate if they were negligent
Maybe, but making mistakes is far from negligence. Besides which, if a single person can accidentally break your system, at least at Cloudflare's scale, that's an organizational failure, not a personal failure.
Screen readers, the programs that use synthesized speech to tell us what's on the screen, cannot read images. Good captchas usually have audio equivalents (which come with their own set of problems), but this one doesn't. If you're blind and flagged by Cloudflare for some reason, you're cut off from accessing half the internet, potentially critical banking/governmental/medical/communications/educational services. We rely on the internet way more than our sighted peers, so this is very important. This has recently happened to me on a few sites, fortunately not critical ones, but it was not a pleasant experience nonetheless. CF engineers, please fix this ASAP. I'm surprised there still isn't a huge lawsuit over this, as this is clearly violating all sorts of laws.