I want to love CloudFlare, I really do. We currently use them, but sadly the number of times that CloudFlare has been the cause of a service interruption is somewhere around 50% mark. They are no longer in use on any critical/important end points, I just don't need PagerDuty waking us up over an issue I have no control over.
This is not a problem I expect to improve. As they start to cover all of the web, i imagine that it'll actually only get worse. Curious about how they see this progressing.
I wrote an article a while back digging into the details of an outage in a website I work with caused by CloudFlare, and while at some level the issue here was "CloudFlare didn't care enough to test their stack against popular browsers and then didn't have the in-house expertise to debug and correct issues that were later found", I frankly feel like this problem is pretty engrained into their business :/. If nothing else, the article might be interesting to some as it looks at how to even go about debugging complex interactions between web browser internals and overly clever JavaScript. http://www.saurik.com/id/14 "When 'Dumb Pipes' Get Too Smart"
I used CloudFlare for a while and it consistently caused more problems than it solved. Really hard to debug, too. Injecting broken JS into pages, serving up corrupted cached assets, etc.
Better to just pay for Amazon CloudFront or something and set it all up myself...
Those errors are usually caused by CloudFlare automatic Front-end Optimization (FEO) stuff, like trying to losslessly optimize images, add cache control, gzip, SPDY/HTTP2, etc. It used to be a primary marketing point, and while its still there (and still causing problems) they have moved away from promoting it so much.
Its best to use of them as a CDN, DDoS sink, and SSL. Their other stuff like FEO or their web app firewall stuff really isn't that impressive enough to be considered a reason for switching.
Disclaimer: I work in the web performance space, and used to work in the web security space. As a whole I'm not a fan of the "stick a magic appliance in front of a website to automatically stop [web security issues | web performance problems]" approach, whether from CloudFlare or otherwise.
I tried their enterprise tier out about 9 months ago and was entirely unimpressed. Moving a site from Akamai -> CF increased page load times dramatically. We ended up quickly switching back.
CloudFlare's complicity with ISIS, however, was what turned me off permanently. For those unaware, CloudFlare was providing proxy shielding of ISIS's propaganda websites. The CF CEO publicly refused to discontinue their service, taking an anti-censorship, pro-free-speech stance. His absolutist views didn't sit well with me and I consider their servicing of these domains to be aiding and abetting this criminal organization.
Interesting. This is the first I have heard of this and CloudFlare's reputation has just gone up my eyes because of it.
Is ISIS dangerous and deplorable? Sure. Are there ideas so dangerous that they do not deserve to be exposed to public discourse and judged on their merits? I don't think so.
I don't follow; CloudFlare is not the government, therefore freedom of expression isn't really the issue here. Who they choose to give a voice to says a lot about them.
I would oppose the government censoring ISIS; however, I would be proud of any company that refused to to business with them.
> Who they choose to give a voice to says a lot about them.
I don't think they're choosing to give anyone specific a voice. I think they're choosing not to make judgement calls.
Once you start blocking anyone, you've weakened your ability to refuse to block others. You'd see greater demands for blocking Wikileaks, censoring articles like https://en.wikipedia.org/wiki/Illegal_prime, enforcing Europe's right-to-be-forgotten, etc. on the basis of "well you were willing to block ISIS".
CloudFlare isn't the government, and so they are under no legal obligation to allow certain content. But their hands-off approach is precisely what gives them the rational standing not to censor any website any group finds distasteful. Furthermore, the threat of censorship does not only come from governments. I don't want to live in a world where a handful of corporations can decide what speech is allowed to be disseminated. Opinions like yours lead us head first to such a world.
Shouldn't you also be upset over all the other service providers including your own ISP that also doesn't block those sites? Unless it's specifically against protecting people against criminal activity, in which case we are really going down a slippery slope.
It might, but the business model Cloudflare is on makes the distinction less clear. For example with their free plans would they need to identify all clients, scan their content and judge accordingly?
For paid clients, I might be more sympathetic to the argument, but even then the (moral?) rules (for nonbusiness) are incredibly tricky to figure out and keep consistent. Selective enforcement is bad for everyone in the uncertainty it spawns.
If no ISP will host your content, then your message cannot reach anyone, and you effectively have no freedom of expression.
I wonder why you oppose government censorship, if you applaud private businesses achieving the same effect? Certainly your reasons cant be the same as the post you're replying to[0], and so you're missing their point.
[0] in particular Are there ideas so dangerous that they do not deserve to be exposed to public discourse and judged on their merits? I don't think so.
Cloudflare may be under court order by the DoD to keep ISIS on their network and protecting classified information with the free speech rationalization. Anti terrorist operations are much easier to conduct when the terrorists are using American communications infrastructure.
I actually get this. If they don't take a hardline on censorship, they would be inundated with "offended" people wanted every site on the Internet removed from their edge caches because someone in the world found them offensive. Much better to just say, "We don't censor, take it up with the website owner!"
What would you prefer their policy on censorship be? Though I was unaware of CloudFlare and this situation, I believe that one compromise can easily lead to a slippery slope.
How do you see protocols (bittorrent or ipfs, for example) where content can not be removed as long as someone has a copy - are we better off without their existence in your opinion?
ISIS, for all we know, could become a legitimate government in a context where legitimacy is clearly unclear. Why do we want CloudFlare to be involved in international politics and middle eastern wars?
Should an American business have an opinion on what Muslims, and other interested parties, ought to be viewing over the web with respect to ISIS? I would hope not. CloudFlare should not be in the business of judging whether Muslims and others are vulnerable to brainwashing from ISIS and need the protection of CloudFlare censorship, lest their fragile worldviews become corrupted.
Let the people of that region judge for themselves the future of their land.
CloudFlare is also complicit with DDoS for sale websites. People have asked them to remove them or reveal their ips to report but they have refused. Here are a few of them.
Common carrier[0] as a concept exists for good reasons, while cloudflare is not technically required to act as one, it just makes perfect sense to not have to make a ambigious content based judgement call for each of your clients.
To the extent that they are not prohibited from doing so, they would also not lose common carrier status if they did. The rules applied to ISPs under Title II -- the FCC's recent Open Internet Order -- do not prohibit blocking unlawful content, only lawful content.
On the flip side, if they were offering TLS services to these sites, they're literally man-in-the-middling encrypted comms to those sites. And in scope of US law-enforcement/intel collection.
Might be that they were asked to continue to provide services.
That's not entirely true. ISIS fuels its ground combat force with angry young people recruited via predominately through social media. The social media often consists of video sharing and much of the video is served up from sites like isdarat.tv (now apparently gone) that are/were fronted by CloudFlare. YouTube and Facebook are efficient with their censoring; ISIS needs stable hosting for these videos for maximum efficacy and CF provided them with that.
Gagging these people certainly won't calm them down or stop their communication but it has a real effect on their recruitment, which is directly driving their combat operations in the Middle East and as a second-order effect, the flood of refugees to Europe.
Citation, please. I hugely doubt censorship has clear, measurable effect on recruitment. Among other things it's got to increase resentment of the western world.
Finally, there may be extenuating circumstances of which we aren't aware. Perhaps CloudFlare is granting them use of data in exchange for not cutting cables. When you're that big your presence is indistinguishable from the internet itself.
I cannot readily cite any scholarly papers on this subject; this is from my observations as a frequent reader of /r/syriancivilwar, jihadology.net, and Iraq/Syria-related social media content. The typical pattern is for ISIS to release a video to their propaganda sites and for jihadist social media users to tweet the link.
al-Ḥayāt Media Center (ISIS's media outlet) relies on the ease of distribution via protected (proxied) channels to reach their large audiences. The reach of this content would be more limited if companies like CF wouldn't shield it.
Timeouts are the majority of the issues, when our service is running perfectly fine otherwise.
They have often been short, but sometimes long enough to cause me to sit there and flip DNS for various end points away from them until the issue is resolved.
I'm in financial services...as part of our checkout process, the customer downloads a 1 MB pdf with sensitive and unique data to them. In testing ( a test site with cloudflare set up), everything was fine. In production, apparently cloudflare cached these pdfs after some sort of traffic threshold was reached (I guess they are all very similar in size...though they are probably not exactly the same size.) Now customers are calling that they are getting other people's data! Apparently cloudflare caches files without even comparing hashes of the download first. Needless to say, we entirely shut off any and all caching after that.
I feel like any CDN or caching proxy (which might even happen at the ISP level if you aren't using HTTPS!) would have that same issue. You should serve confidential files from unique URLs or at least set the right HTTP headers to disable caching.
If CF download the file to check the hash, it wouldn't really be helpful as a caching service. That would mean CF would hit your server for each request before it served up the asset to the customer. Your server wouldn't benefit from caching. Maybe you'd get a small speed increase from their worldwide endpoints but that would reduce the advantage of having CF manage your site. One of the biggest advantages is that CF serves hits that your server wouldn't have to. That allows you to need less infrastructure.
Further, it might be better to never serve sensitive financial data via a 3rd party server. That kind of data would be best served directly to the customer's browser from your server.
Most of the times cloudflare is fine, they've gotten a lot better over the years.
However we have found random areas to temporarily go down for random periods of times. You will not be able to see these downtimes if you are only checking from 1 location.
I don't use CF-CDN but some websites that use CF gives me 50x error because it lost connection to origin servers, but when I switch to my VPN service those websites were up. Most of the time I get errors from CF's HKG point.
SSL. SSL is easy on a single server, but gets complex/expensive very fast on cloud platforms. Cloudflare have an incredibly easy solution (branded "flexible SSL") where they handle the SSL between the client and their CDN, and the CDN does un-encrypted requests to your platform.
Yes, this is a security shambles - Cloudflare is officially MITM-ing your traffic. But given the HTTPS-only movement this is a perfectly valid solution for public websites. And this announcement actually makes a big difference because if your platform is Google Cloud, then the un-encrypted portion is now over a private Cloudflare-Google interconnect.
FWIW, I've been using Cloudflare for 6 months and haven't seen any drop-outs. It's improved latency, and the only issue I've had was it mangling email addresses in transit, to prevent scrapers. It took 3 minutes to find the setting and switch it off, so I'm OK with that. Although I would prefer things that modify your HTML should be off by default.
I'm curious about the difficulties you're referring to. If you're referring to secure key distribution, check out Hashicorp's Vault project. Other than that, I can't think of any show-stoppers to deploying SSL across many commodity cloud servers.
That can be solved by using a cloud service with global POPs, spinning up nodes everywhere, and using a common core config built with Chef, Puppet, or Docker. Amazon, Digital Ocean, Vultr, Linode, Joyent/Triton, and probably a half dozen others will provide global presence with instant provisioning.
Slightly more work but you control it. Cost is probably similar.
DDOS mitigation is harder and is definitely something CloudFlare does well enough to earn some market share, but wouldn't it be nice if we had a more global solution to this problem that didn't involve third party firewalls?
Cost is definitely not similar. Building your own CDN network is not cheap and CloudFlare provides DNS + SSL + lots of locations + better routing + free bandwidth.
The servers, ssl and bandwidth costs alone would be more than their fees for any big site.
> no longer in use on any critical/important end points
What's your recommendation as an alternative. Cloudflare is tempting if you are tight on resources due to free SSl, CDN, etc. and it is super easy to setup. I'd love to try out alternatives though, just as a backup.
To offset some of the feedback here, we run close to a billion requests a month through CloudFlare with no issues at all. We use their Strict SSL setting and everything is fast and secure and we save a ton on bandwidth.
As far as CDN service goes, free SSL and bandwidth and peering + all of their datacenter locations + DNS integration gives us better latency than pretty much everyone else we've tried.
We're a small startup and serve up about 3.5M requests through CloudFlare a day.
when we were using AWS CloudFront, our costs quickly escalated to $100/day as we brought on new customers. We switched over the CloudFlare and made that $100/day cost go away.
CloudFlare's free service has been amazing and we've had no hiccups with it.
We run Quizlet behind Cloudflare, and use Google Cloud for all our server infrastructure (>150 VMs). We've been very happy on both platforms. We'll be saving around $2k/mo on bandwidth because of this deal, and we didn't have to lift a finger. Yay :)
Happy to answer any questions about either platform.
It's becoming unusual not to see sites behind CloudFlare now, pretty neat from a routing standpoint and devastating to user privacy and security on the whole. If things continue in this direction we'll have the cloudflare, and some scraps of regular internet off the side.
The more popular it becomes, the more incentive there is for other companies to enter the space and grab a chunk of the money. The barriers to entry are not insurmountable for e.g. existing CDN companies to start offering DNS, WAF and DDOS mitigation products, making them more like Cloudflare. They already have the peering/hosting relationships and engineering teams experienced at working with large volumes of traffic. I wouldn't be too worried about Cloudflare taking over the internet this early on.
Is there reason for people to enter a market with a single obvious leader though? I got the impression that making peering agreements with people on any sort of scale was a bit of a bear. It is already quite ubiquitous, a surprising number of websites use it even if it's not completely obvious, reddit.com has some sort of stealth configuration that bypasses the normal DNS setup for example.
> Is there reason for people to enter a market with a single obvious leader though?
Were there not search engines before Google? Isn't AWS the cloud computing "leader" while Google is trying to break into the space? There is no such thing as an obvious leader, only juicy prey waiting for the next hungry org to eat its lunch.
> I got the impression that making peering agreements with people on any sort of scale was a bit of a bear.
Hardly. Peering agreements are of similar difficulty level as any vendor negotiation, whether it be with a specific network or an interconnection fabric (IXP).
That's how the market works... otherwise we wouldn't have any new companies if everyone gave up before competing with the big guys. There's a lot of potential to do better against anyone.
To continue with your point, we use https://sucuri.net/ where I work because it has such a great set of tools and extension for our Wordpress blog, which kept getting hacked.
pretty neat from a routing standpoint and devastating to user privacy on the whole
I'm interested. Please expand on the privacy point. I thought the general move to CloudFlare was a good thing for privacy, as it provides an easy mechanism for getting sites onto HTTPS without having every site to worry about managing certificates.
CloudFlare pipes all traffic through an interception proxy and delegates signing of SSL certificates to their own infrastructure. A domain behind CloudFlare can be monitored (they see everything in plaintext), the content tampered with, or the origin completely changed without a single notification to the outside world anything has been altered.
If you're sitting back in your evil chair, this is the perfect vantage point through which to intercept what seems like the majority of all browsing traffic in the world. Even if you're not planning world domination, this is an otherwise unobtainable amount of user metrics you can package up and sell.
The point between cloudflare and your app doesn't need to be unencrypted, the lack of privacy is that cloudflare is a massive MITM and thus has access to all the data between you and your app.
Thus the evil guy sitting back in their chair is cloudflare or it's you for not configuring https between your app and them.
And this announcement means that if your app is on Google Cloud, you don't need to encrypt between CF and your app. It's now going down a private interconnect, and I bet that interconnect is encrypted (like all Google's internal data pipes).
Besides Cloudflare being the biggest man-in-the-middle on the internet, their DDoS mitigation offering is also questionable. If you google "clouflare bypass" [1], you get to websites that can tell you the origin IP address of a cloudflare customer's domain name. So, malicious guys could hit the real IP directly.
> If you google "clouflare bypass", you get to websites that can tell you the origin IP address of a cloudflare customer's domain name.
Those rely on a known DNS history from before CloudFlare was added to a domain. If bypass is a concern, changing the server's IP and making sure it never shows up in a public DNS record again solves things.
Yes, DNS history is one way to leak your IP. There are several other ways that the origin IP may get leaked, so you should be very careful if you use Cloudflare:
* Keep all subdomains on CloudFlare
* Don't use wildcard subdomains if you are not on Pro account
* Don't host mail or other services on the same server as your web server (email headers have origin IP)
* Never initiate an outbound connection based on user action
* Make sure that your web server and web application are patched against all known information disclosure vulnerabilities.
* Change your origin IP once configured for maximum DDoS protection on CloudFlare
If CloudFlare is compromised by an intelligence agency or forced by law enforcement and courts to cooperate, they're a large single-point-of-failure for privacy.
Less severely, they themselves know the vast majority of sites users are visiting and the content they're being served. Valuable stuff for advertisers, among others.
Indeed, CloudFlare will happily run an HTTPS front-end proxy to an origin which is using a self-signed certificate, or even to a HTTP origin. Thought that site was secure? Think again!
CloudFlare strips HTTPS protection off on their machines and views all plain text. So, you get HTTPS protection on the public Internet in exchange for CloudFlare getting to see everything.
Given what we learned from the NSA sniffing inter-datacenter communications within Google, do we really want wide scale solutions that rely on removing the SSL layer?
The operating mode described by the comment to which I was responding, that of CloudFlare performing all steps of SSL termination, is one operating mode available. CloudFlare offers at least one other operating mode in which they are not responsible for all aspects of SSL termination. In particular, they offer an operating mode in which they do not hold private keys. This is referred to as "Keyless SSL". Thus the concern voiced by the comment to which I was responding, that of CloudFlare stripping SSL, is but one available option rather than the only available option.
"Keyless SSL" is a bit of a misnomer. CloudFlare still holds the keys to encrypt and decrypt the content it is distributing and therefore sees everything (they can inject into your traffic as well). Even if they don't hold the private key, they can act in every way as if they did hold the private key.
...for a single server. HTTPS on a cloud platform is significantly more difficult and expensive.
We use Google Appengine, and although they have an SSL solution, it was an order of magnitude cheaper and easier to use Cloudflare's SSL.
I'll qualify this by saying our site is public, so we only need SSL because of the HTTPS-only crusade. If you are serving sensitive data, you do need to spend that magnitude more money and time to set up your own SSL.
That's a lot of hyperbole there. If you look at the top 500 websites very few of them are behind CloudFlare, CloudFlare's penetration seems to be SMBs mostly (since larger orgs can afford the in-house mitigations that CloudFlare offers).
Did you know reddit is? You wouldn't know it from looking at the WHOIS, I'm betting lots of places are behind it without showing the normal indicators.
Name Server: CNS1.REDDIT.COM
Name Server: CNS2.REDDIT.COM
Name Server: CNS3.REDDIT.COM
Doesn't look like it from here..
;; ANSWER SECTION:
CNS1.REDDIT.COM. 172246 IN A 173.245.58.24
it’s routed through a high-performance interconnect instead of the public Internet
The public internet does seem to be shrinking, more and more closed data silos and now huge chunks of traffic going through a 'trusted' man-in-the-middle. From an individual sites standpoint it's amazing (and I use it for most of my sites), but the bigger they become the more juicy a target they are.
Is this not some kind of peering, though ? The way I see it, Cloudflare decided to peer with Google instead of using transit or, even worse, public internet (with the overhead it has over AS-to-AS connections). I don't see how the "public internet" somehow lost anything.
I strongly advise against hosting images, particularly photos, behind CloudFlare. I don't use it myself but I'm frequently noticing CloudFlare aggressively stripping color-profiles from images. Most noticeable if the photo has an AdobeRGB profile, as many do. It makes skin tones in particular look very dull. I don't know if CloudFlare honors no-transform – I suspect they do, but most sites don't send it.
I would like to understand this. If a customer asks we do do image recompression to save space, and strip metadata (this is one of our services) but would like to know what this is about.
My experience in web-dev is that browser support for color profiles is dodgy at best. Much like alpha in PNGs. It's usually wiser to not rely on occasionally-supported features.
PNG alpha in particular was a nightmare for me. After that I knew to look out for the pain caused by occasionally-supported features in images.
PNG alpha transparency was a problem with the decade old IE6. And even then IE6 worked fine with PNG8 images that had alpha transparency. IE7 and IE8 required a 1 line style attribute to work 100% with PNG + alpha, and IE9 did away with that need altogether.
> I don't use it myself but I'm frequently noticing CloudFlare aggressively stripping color-profiles from images
They have a service that tries to perform loss-free optimization of served images. It's easy to disable, and should be disabled if you either already perform similar optimizations (optipng / pngquant, jpegtran, etc) or have images with annotations, color profiles, etc.
This makes it unsuitable for gfx outsourcing work. In my last job (gamedev) people were starting to use more and more of the profiles built-in the image formats we were exporting (we settled on .tiff, but also supported .png, and few others).
Though I remember that sometimes previews (or was it layers) saved in the .tiff were making it 10x, 20x (not kidding) times bigger, so a special "save" plugin was done to filter these out, and on regular someone wrote a script to check for such.
Side question, does anybody have any experience with their RailGun feature, is it worth it? We recently switched to CF and mostly use them as a CDN and we're quite happy, good value for the money.
I work at CloudFlare and have tried every feature on my site https://www.lfgss.com/ at some point.
Railgun is the single biggest improvement a dynamic web site can enable.
The biggest benefit is the established connection between the CloudFlare PoPs and your origin server.
The second benefit is the "compression" that is the result of each side having a shared dictionary.
But really, it's the open connection.
The things I tell my friends to use from CloudFlare:
* DNS
* Railgun
* Caching (my S3 bill is so small now)
* DDoS protection (I'm under attack!)
That's usually the order I recommend it too... Railgun is up there. After that list it tends to get more specific, about their web app and what works for them... but all of the above, just enable and use.
Thank for the information buro9, my company has started to go global and we now serve clients in Australia, Japan, locations far away from our origin server (+200ms) and I am hoping to see a big improvement there.
So does this apply to any GCE customer hosting in US? If I've got machines in us-central and I'm using CF, then I don't have to do anything? My bandwidth charge from GCE should go to about 0?
This is excellent, honestly. We recently moved a few billion requests each month behind CloudFlare and our metrics show that our median user (in terms of load time) had their assets loaded almost 40% faster. It's also worth noting that CloudFlare is the only reputable CDN that currently supports SPDY, and is (purportedly) actively working to turn on HTTP/2. Compare that to a company like Akamai that's still advertising Edge Side Includes like they're new and innovative and the year is 2004.
I'd love to see more transparency in the way CDNs decide to cache or not cache your content. For example: Cloudflare publishes crawl frequencies in their pricing table but what do they actually do with that content? Push it to all their edges? I'd doubt that. I guess it's based on website traffic, your website pricing plan, ... but it seems quite arbitrary to me.
I and others that I know get this kind of referral spam on every single domain we have with cloudflare. I know DNS records are public, but is there something cloudflare and other public DNS hosting services can do to prevent this?
Those spambots hit hard these days. However, they never ever touch your origin or your CDN at all (check your logs, nothing). They fake the log entry directly into the Analytics systems.
Google has to cleanup, but sadly, they haven't moved a bit since ages.
Most of these spammers just emit events to a large range of GA tracking ids completely blindly, think of it as the equivalent of...
for (var i=0; i<100000; i++) {
ga('create', 'UA-' + i + '-1');
ga('send', 'pageview'); // But with a fake referrer
}
They do that a bunch of times per day from a bunch of different IP addresses. Adding host filtering to your properties on GA eliminates about 80% of spam which use this technique.
It's possible you're being targeted due to being in a specific market or something, but it's unlikely to be related to Cloudflare. (At least I can't think of any reason why it would be related.)
Do you have any insight as to why those spambots would do that? The only thing I can think of is SEO for the "referral" domain, so that they boost their own domain's ranking.
This is not a problem I expect to improve. As they start to cover all of the web, i imagine that it'll actually only get worse. Curious about how they see this progressing.