I will remind HNers: is Cloudflare not the company that leaked sensitive data through cache files that were indexed by at least Google, and when the tech community were up in arms about the massive leakage of sensitive data, the CEO’s strategy was to turn up here and criticise Google for not deindexing quickly enough?
That's one of the main reasons I'm leary about them. Such a big f-up is difficult to forget. It shows that they have a move fast and break things culture which for a company that is responsible for critical infrastructure feels wrong.
In response to this incident Cloudflare has made big engineering changes, including huge work to move away from C as much as possible.
The offending parsers were rewritten in Rust (https://github.com/cloudflare/lol-html), as well as WAF, image optimization, and a few others. Nginx is being replaced with a custom cache server.
New implementations are using either the Workers platform, or are written in Rust or Golang.
I interviewed there once and they asked me what I would do if a service broke after a deployment. I said the first step was to revert to the last known good version and then investigate. Color me surprised when that was not the answer they expected.
Cloudflare's internal release tool suggests revert when monitoring detects failures during deployment, so this question doesn't describe Cloudflare's practices. There must have been something more to it, or it was a misunderstanding.
If I ever interview at Cloudflare and get this question I might answer with "call the sales team and have them fix it by selling someone an enterprise subscription paid upfront by the decade" just to see if the interviewers read Hacker News :P
Depending on the service’s criticality, the cost of rolling back versus pushing a fix, service dependencies in their environment… having to push a fix might have been the better approach.
Without more details about the environment, it is a 50/50 call.
I remember them criticising Google for not being faster at removing cached files. I don't remember them blaming Google for their screw up.
And let's be honest, if a big provider wants to offer cached versions of pages, they probably should have a way to purge those files in case there's a problem (eg: malware).
I'm not sure what you were expecting people to take away from your message, with the way that it is worded. It may not have been the intent, but the particular way you expressed your point heavily implies it.
> I'm not sure what you were expecting people to take away from your message, with the way that it is worded.
My guess would be that they were saying that Cloudflare's HN comment on the incident was to complain about google not cleaning up after the incident.
> the particular way you expressed your point heavily implies it.
You can't turn around the burden of proof this easily. Saying "the particular way you expressed it" doesn't give you license to make things up about a comment that is an inch above yours.
>the CEO’s strategy was to turn up here and criticise Google for not deindexing quickly enough
This wording implies that the CEO deflecting on HN was their strategy for responding to this problem, not that the CEO deflected on HN in addition to admitting fault elsewhere. Typically 'strategy' is used to refer everything that they planned to do, not a single action.
What you wrote is that "the CEO’s strategy was to turn up here and criticise Google for not deindexing quickly enough".
The CEO replied to someone asking about the services they were working with and complained about Google taking longer than the others. Maybe we're reading his comment in a different way, but to me there's a big gap between what he did and having a "strategy" to blame Google.
Cloudflare's CTO (jgrahamc) was on that thread too and didn't spend his time criticising Google. He wasn't hiding or saying "look over there instead!".
So I don't see any strategy from the CEO, CTO or the company to criticise Google or to ignore the fact that CF had f'ed up. Pointing out that Google was slow to remove cached pages is, in my view, a valid criticism.
Now, if you said that they had a strategy to minimise the problem, then I'd agree with you.
What's the point of a strategy to criticise Google in that case? Are they criticising Google for no reason or to move some of the blame / direct people's anger at Google for keeping secrets online? In any case, you didn't use the word "blame" and it's my fault for interpreting your comment in this specific way.
I didn't see a "strategy" to criticise Google. There was ONE reply to someone asking about the "caches" they were working with. That reply listed the "caches" and complained about Google being too slow. That's it. I think we are stretching things a bit if we look at that comment and context, and conclude that the CEO's strategy was to criticise Google.
For someone getting very aggressive for words "being put in your mouth", you're not really paraphrasing that Cloudflare CEO very fairly.
He was specificity responding to someone complaining it's still in Google's cache, by stating that "The caches other than Google were quick to clear [..] I agree it's troubling that Google is taking so long.".
By leaving out this context, and phrasing it a "strategy", is not a fair paraphrasing. These bits matter. Two things can be true at the same time: 1) Cloudflare messed up, and 2) Google is very slow to deal with this, and also messed up. Indexing all of the web comes with some responsibilities.
Please apply the same standards to yourself that you impose on others with such aggression and hostility.
The context in which Cloudflare was saying it was that a team at Google was the one to discover the issue in the first place. That's why that particular comment was taken so uncharitably by many at the time.
This was a much needed reminder. Although, it's quite difficult to find a better DDoS mitigator which is better than CF, I still wouldn't trust them for everything. Especially, since they are most likely snooping on the decrypted HTTPS connections
> Although, it's quite difficult to find a better DDoS mitigator which is better than CF, I still wouldn't trust them for everything.
Adding Challenges, TLS fingerprinting and Rate Limiting is possible on just about every major CDN platform to be honest. I guess with CF it's more "ootb" though, where you don't really have to think too much about policies - but at the same time, you can't go as granular in those policies (e.g layered) as some others.
You get what you pay for.