This was flagged to my attention and I've reviewed all the interactions between the author and our team. The site in question was using the free version of CloudFlare's service. On February 2, 2013, the site came under a substantial Layer 7 DDoS attack. While we provide basic DDoS mitigation for all customers (even those on the Free CloudFlare plan), for the mitigation of large attacks a site needs at least the Business tier of CloudFlare's service. In an effort to keep the site online, our ops team enabled I'm Under Attack Mode, which is available for Free customers and enhances DDoS protection.
The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to the site away from our network. While we encouraged the site owner to take advantage of the Enterprise tier of service given their needs and traffic levels, the site would have been brought back onto CloudFlare's network if they had upgraded to the Business tier of service ($200/mo) which included Advanced DDoS mitigation.
To be clear, CloudFlare does not bill based on traffic. However, resources are not infinite and when an attack against a Free customer begins to affect the performance of other customers we will take measures to protect the overall integrity of the CloudFlare service.
Matthew Prince, CEO, CloudFlare, @eastdakota (Twitter)
I think if the $200/mo tier was officially suggested to them, they would've jumped on that (depending on how miffed they were about the outage). However, suggesting a $200/mo plan which then gets switched to a $3000/mo plan is enough justification to just leave and be done with it.
You keep calling it an attack. This was normal (growing) traffic for the site.
From the comments on the site:
"I didn't notice any "attack" when CloudFlare began to route all traffic directly to us. It looked like normal web traffic - much of it, but no more than usual."
He said; she said. A Layer 7 attack is not necessarily something one might "notice." The very nature of such an attack is "normal" looking. I think it's impossible for us to say who is right -- OP or CloudFlare -- without substantial hard-data. Ultimately, it's not the basis of the article, and the OP is looking for a different service -- unmanaged, limited-downlink bandwidth -- than what CloudFlare is looking to provide -- managed, edge network.
Irregardless if it's an attack or not, CloudFlare's personell did not handle this well.
Yes, I know they did offer one hell of a starter/"sweet lolipop to sucker you in" pack - but that's still not what's being discussed.
It has been re-iterated many times in this thread - but CloudFlare had a sane person on the other end that was willing to open his wallet - that's something one should act on quickly.
keep in mind that the author's words are... his words. Perhaps he's not putting out the full story? How can you know which side is "right" on an issue such as this?
All we know is that the author ran on the free plan, and probably should have upgraded from the free plan when he started seeing his site getting large amounts of traffic.
In the end all is well, he got another service that served his purposes.
"While we encouraged the site owner to take advantage of the Enterprise tier of service given their needs and traffic levels, the site would have been brought back onto CloudFlare's network if they had upgraded to the Business tier of service ($200/mo) which included Advanced DDoS mitigation."
This reads like "We encouraged the site owner to pay 15 times more than they needed to."
Which is it, do they need the business tier, or the enterprise tier?
I've been a happy CloudFlare customer so far, but the lack of transparency in rules and pricing is concerning. You don't charge for bandwidth, but can disable sites at your discretion if it causes problem in your infrastructure? This sounds a lot like ISPs that offer "unlimited" bandwidth but start throttling you at some predetermined but unknown cap.
Was the customer contacted when these actions were taken? That seems to be the biggest issue I see. A simple email would have rectified a lot of the confusion here.
I don't get this. While it's cool that you don't charge based on bandwidth or TCP connections, it sounds like you basically do when that bandwidth is associated with an "attack".
Doesn't this make CloudFlare customers more vulnerable to attacks, since an attack will result in a monthly fine for the rest of the site's lifetime? (whereas a normal site just pays a one time cost)
On the one hand, I think it's insane that you hit 100TB/mo of usage on a "free" service and expected it to keep going. Maybe you figured you wouldn't rock the boat as long as it was free - I guess I can relate. But still, really???
On the other hand: CloudFlare comes off as terrifyingly incompetent here - which more or less matches with my experiences trying them out on a site serving maybe 100GB worth of traffic at most in a month. They seem to have missed dozens of great opportunities to upsell you on their paid service, and when someone finally noticed how much bandwidth you were using, they completely lost the plot.
What should have happened, IMO, is something like this: A panicked CloudFlare admin realizes your site is using 100TB/mo. Their first step is to send you a sternly worded email, explaining that for this usage level you need plan X, and if you don't upgrade within... let's say 5, maybe 7 days? They'll be throttling or limiting your service. Then you don't feel pressured to solve the problem right away, and they are trying to retain a customer that (presumably) they value.
Instead, they haphazardly change your settings behind your back (???) and then later take various steps to reduce your bandwidth usage before finally deciding that you need to pay them, without even figuring out how much money they want. Ridiculous.
To me this just says that CloudFlare is running on what may be a fundamentally unsound business model, and that by claiming their free tier doesn't charge by bandwidth (and not listing any limits) they're dramatically increasing the odds that customers will suddenly discover there are limits after all, and leave. If they were more up front about what the actual pricing structure is, it'd probably be more likely that people would start paying for the value that CloudFlare gives them.
The author stated a couple of times that they would happily pay for the service. The problem is it was free until suddenly it was $3000/month and it took two weeks with an outage in service for that duration to learn that fact.
Most reasonable people can be made ok with most things given adequate communication. Expectation management is one of the most important aspects to human relationships. As a business, what was a sales opportunity became a PR issue.
The only reason a business like CloudFlare offers a "Free plan" is to attract people to their service and then try to upgrade as many of them to paid plans.
He was happy to pay, they didn't capitalize on that.
Free DDoS protection is not a part of CloudFlare's business model -- plans [0]. A free CDN comes with caveats, like not receiving fast support, an SLA, etc... to name a few things the OP expected. There's no doubt CloudFlare lost a potential customer, but we just as well may not have all the details and shouldn't try running CloudFlare from our armchairs.
We must have different definitions of DDoS protection; what exactly are you thinking they are protecting that isn't covered instead by their Level 3, 4, and 7 attack protections? Of course, a CDN is by nature a protector of non-malicious DDoS, but the spirit of the conversation shouldn't really have to state such facts.
All CDNs offer DNS level DDOS protection by default. Not because they choose to, but because they have no other option. After all, the same IP ranges are used by all clients and this makes it impossible to pinpoint the original target. (thus no one to blame/bill)
Every CDN does it, but only CF claims it as a "feature".
They do need to break down their pricing structure in a way that people can easily buy into their service. The big issue here is the sheer amount of difference between the free plan and the lowest priced plan. In order to get people into your monthly billing cycle you must focus on getting them to take little steps. First get them to sign up. Then move them to a free plan. It doesn't matter if its profitable, just get them to overcome the barrier of taking out their credit card. Second step is to get them to upgrade their plan to a profitable one. This is much more easy because they are your customers. No need to pitch them the idea anymore. All you need to do is show them that the other plan meets their needs. You can also upgrade them for free for a limited time (say 3 months) to overcome the barrier of upgrading, and then start charging them at the upgraded pricing. This way they get to integrate into the new plan. They will feel the pain of downgrading, and will mostly decide to not do so. Third, you upsell additional services not included in plans. Use the same technique in the second step. Give them free trials, and start billing automatically once the trial is over. They can always cancel, but most won't bother or want because they have already been using the product for so long.
Realize that people will pay money in order to not lose something. This is a very strong mechanism to use to increase the profitability of your products.
I upgraded to Pro because the Free plan doesn't have the SSL option. The Pro account also gives you more page rules and the caching and response times are visibly better.
I had quite a similar experience where our site (approx 10TB / month) was taken "off cloudflare" and all traffic routed direct to our servers at peak load (we were on the Pro plan @ $10/month after many months of free). They cited network issues in the control panel and a "Layer 7 attack" in a support ticket. We quickly upgraded to a Business plan ($200 / month) and traffic was back through cloudflare within 10 minutes.
Also, similarly to OP, we are regularly (fortnightly) automatically put in "I'm Under Attack" mode without any prior warning or consent which is quite annoying as it tends to happen overnight so I am not alerted until someone checks the live site in the morning (it still returns a 200 so current checks don't pick it up)
We use it mainly for the CDN, we have estimated a similar service using CloudFront would cost over $1500 / month (750% more - I acknowledge AWS is are not the cheapest)
You acknowledge that CloudFlare's Service is offered as a platform to cache and serve web pages and websites and is not offered for other purposes, such as remote storage. Accordingly, you understand and agree to use the Service solely for the purpose of hosting and serving web pages as viewed through a web browser or other application and the Hypertext Markup Language (HTML) protocol or other equivalent technology. CloudFlare's Service is also a shared web caching service, which means a number of customers' websites are cached from the same server. To ensure that CloudFlare's Service is reliable and available for the greatest number of users, a customer's usage cannot adversely affect the performance of other customers' sites. Additionally, the purpose of CloudFlare's Service is to proxy web content, not store data. Using an account _primarily_ as an online storage space, including the storage or caching of a _disproportionate_ percentage of pictures, movies, audio files, or other non-HTML content, is prohibited. You further agree that if, _at CloudFlare's sole discretion_, you are _deemed_ to have violated this section, or if CloudFlare, _in its sole discretion_, deems it necessary due to excessive burden or potential adverse impact on CloudFlare's systems, potential adverse impact on other users, server processing power, server memory, abuse controls, _or other reasons_, CloudFlare may suspend or terminate your account without notice to or liability to you."
In other words, you can't host non-HTML, but you can if it isn't disproportionate, but if it is disproportionate, they can deem you to be a problem and cut off your service, without notice. That's not a contract at all. In legal parlance, that's an illusory contract -- when one side can modify their performance in any way at any time.
I use Cloudflare's $20 a month option and it worries me now that I might be deemed to be using a disproportionate about of space or bandwidth caching images, and then be cut off without notice.
EDIT: I love the cloudflare service and I'm not complaining. I just think their legal department needs to clarify this and the tech side of the house needs to be able to warn users when they are exceeding the bounds of what is acceptable.
My interpretation of this paragraph goes something like:
> If and when your bandwidth usage gets high enough that one of our customer service people gets pinged about upgrading you, they'll also have a look on the dashboard to see what mixture of filetypes you're serving. If it's all static content, you're in trouble.
I'm not quite sure how else that could be phrased into legalese, than what they already have there.
Based on what I have read on many HN articles, what Cloudflare offers and what you receive are two very different products.
From being shut off when you have a incoming DDoS of some arbitrary size to actually loading webpages slower than your server does vanilla, it seems the benefits of Cloudflare are mostly hype.
While I agree CloudFlare handled this poorly, there is this reply on the blog from Matthew Prince (CEO of CloudFlare):
Both enabling "I'm Under Attack Mode" or routing the traffic direct are both supposed to generate an automated message to the customer letting them know what happened. We've reviewed the logs and don't see a message having been sent. I'm investigating why that didn't happen since I agree it is not acceptable.
At 100TB/mo., pure file delivery, you'd need to be an Enterprise customer. Let me know if this works within your budget.
An interesting proposition - If we take it at face value, $3000 for 100TB works out to be $0.03 per GB. That's pretty high these days. If you are buying downmarket (which is cloudfare like traffic quality) you can get a cdn deal for maybe $0.01 on a 6 month term with these kind of levels, and somewhere around $0.005 for an xc in the us or eu no commitment. Cloudfare should be buying at substantially better rates than these (or at least, they seem to imply it - calling bandwidth free) so it's it seems they have a similar problem as many freemium models - when most of your customers aren't paying you have to really hit the ones that do.
> His conclusion of "two servers and 100 TBT for $200" seems unlikely to last long either.
A site I'm involved with ended up on Leaseweb rather than a big-name CDN for much the same reason - we have 3 x 100mbit unlimited traffic servers, and all 3 have been pushing an average of 90mbps 24/7 for the past year or so.
It's not quite as smooth and reliable as a proper CDN, but it's an order of magnitude cheaper, and they've never given us any trouble about using too much of their "unlimited" bandwidth.
Not extrapolated from a fully saturated link, no. But I didn't mean to imply that was usage billing.
I was thinking of fdc in amsterdam in the eu on a 10gb unmetered no bw sla, but I also think they offer a similar (slightly higher) deal in denver. I was thinking of somebody else for the US - in SLC - but the name escapes me. If you actually could use the name I'll find it.
I shouldn't have said xc as really they expect you to buy power and space from them. Clearly not what I'd try to run an upmarket video cdn off of, but i would be surprised if people like fdc aren't who CF buys from.
Those kinds of offers are generally unusable if the bandwidth matters. This goes back to the oversubscription model, selling the same resource to multiple customers and letting the customers jockey for use, hoping most will never use it enough to catch on.
“The FDC offer is a shared 10 Gbps and i believe in another topic was explained, that you're supposed to stay at 1 to 1.5 Gbps usage. Even a pure 10Gbps peering port will cost more then $500,- / month, so a true 10 Gbps for anywhere close to $500,- cannot be expected from any provider worldwide.”
“I always found this to be a problem with all FDC locations. So many places have such horrible speeds that no matter what speed they offer, I have a hard time making a good use of it.”
“The server is great. The single-thread transfer speed isn't as much, but it's reasonably passable most of the time, considering the server cost. But I've been seeing some severe routing problems, with 5-10% of the net simply being unroutable much of the time, as well as intermittent packet loss. Due to this, the overall fail rate of this server is about 10x that of other servers I have in NL.”
If you're running an ad supported viral image host and can fit your popular content in RAM, this kind of thing may be acceptable, though in the long run users will tend to migrate to image hosts that serve their memes quickly.
TL;DR: Not all bandwidth is equal. You get what you pay for and what your provider pays for.
Slightly OT but related to the Cloudflare business/enterprise offering: They "guarantee" 100% uptime, does anyone know details of this? Obviously there is no way to actually guarantee such a thing, but what kind of compensation do I get when they do have downtime? I can't seem to find it mentioned anywhere.
"Service Level Agreement (SLA) - 100% uptime
Industry standard SLAs often feature 99.999% uptime, also known as the five 9s. At five 9’s your website could be offline for as long as 5 minutes and 26 seconds each year. All CloudFlare Business and Enterprise plans offer guaranteed 100% uptime because we know that anything less than 100% is an impediment to your organization’s success."
Usually these guarantees mean credits toward future billing. Cloudflare's enterprise plan offers 2500% uptime guarantee, which means if they are down 10% of the time, customers are credited 2.5 months.
It sounds a lot nicer than it is, much like "we never charge for bandwidth".
It's not a directly comparable service, but serving 100TB/month over S3 would cost you more than $9,000 a month. On CloudFront, depending on object size and distribution area, it would cost more than $8,000
You have to read the fine print though. Amazon CloudFront, while expensive, is designed for delivering content globally.
Take 100TB for example (the first result in that Google Search)
d. We strive to maintain a high level of service, and a lot of customers
depend on our high standards of quality. As such, we will not provide
Services to those that are using our Services for:
vii. Using the Services for a content delivery network or content distribution
network (CDN). An authorized CDN network offered through 100TB is
accepted. Special requests to use the Services to run an unauthorized
CDN network may be approved on a case-by-case basis. Failure to comply
with this policy will result in termination of this TOS, and you will
not receive a refund of the Fees.
If you look into the TOS even more, they basically disallow anything that's bandwidth intensive.
S3 and Cloudfront are targeted at the dabbler, who will likely only be serving a GB or two per month, in which case S3 and Cloudfront would be pennies with no long term commitment. Once you move beyond that and are storing and serving more than 1 TB, you'd be better off with a monthly commitment.
@thezilch: My understanding from the article was that the poster wanted to deliver without load to his server, settled with 2 servers in NL and was not in for the CDN part.
He mentioned only one which was Amazon. It is expensive but it's doing more than serving 100tb. The S3 provides redundancy and cloud front provides edge delivery.
Getting a server somewhere that has 100tb of outgoing is something different.
None of the cheaper options I see there are comparable services. What danso was pointing out was that the service cloudfront was wanting $3000 costs $8000 elsewhere, it is not an unreasonable price. Getting maybe bandwidth from a sketchy dedicated server provider is not the same service as a content delivery network.
You get what you pay for. When I'm paying for a $16/megabit @95th on a 10 Gig port, and I get hit with a DDOS, it's my providers problem, because I'm not paying $16/megabit for DDOS traffic, so they need to stop that stuff before it gets to me (ideally without impacting my customers)
Contrawise ,if I'm paying $2-$3/megabit @95th on a 1 Gig Port, the amount of support I can expect during a DDOS is pretty minimal, so I end up having to take the hit - but my damage is limited to $3000/month so I don't really care.
Any time I see a "We don't charge for Bandwidth" service, I interpret it to mean one of (A) We'll throttle you once you exceed our unspoken limit, or (B) We'll discontinue your service. (Drop your port from 1 Gig down to 100 Megabits, or slower, traffic shape you, etc...) once you breach that limit.
There is no sustainable third option for those who provision reasonably high quality transit, and those who believe there is will one day wake up with their internet property offline, or seriously degraded.
This kind of whining and the comments on the page piss me off greatly.
A customer is a person who pays for a service. Someone who doesn't pay for a service (yet) is a lead. Not all leads are good business. 100TB of traffic does not sound like a good lead to me, not even at the $200 level.
Looking at the site, I see an IMGUR clone which was running for free off of CloudFlare's cache servers. I really don't understand the nonsensical comments on the article. WTF is wrong with people these days thinking that everything is supposed to be free? Are you all 16 and on an weekly allowance?
Commenter Matt had a very valid point that some sort of optimization of the stored (cached) files would have been a smart option for yourselves (less local storage) as well as CF (less to cache, less bandwidth). I'd recommend http://www.jpegmini.com/server (Oh wait, it's not free, now what... cry me a river)
I have a question for the Phobos peeps. Were you making money? Seems to me like you were... since you can afford the LeaseWeb servers. Instead of bitching publicly, perhaps you should have reached out to the company when you noticed your traffic levels were reaching antisocial magnitudes.
Grumpy Gramps
(who used to pay UUNET $6k+ a month in 1997 dollars for the privilege of hosting a basic database-driven e-commerce site for a luxury watch brand on a guaranteed T1 connection)
They contacted the company after being shut down. Not before, when it should have been pretty clear to them how many resources they were using. There's a simple indicator in the dashboard. "BANDWIDTH SAVED".
This line in the article "I tried to monetize it through ads some time ago, but failed. Advertising-Networks don't want us as a customer, but I'm fine with that." was not conclusive that monetization failed via other means. Not a user of the site, but at first glance some choice links to pr0n affiliate sites would do the trick.
There are a handful of contributing causes, just off the top of my head:
Once you get past the first few tiers of paid service, providers tend to reign in the bullshiting "technically true by widely misleading" descriptions and are upfront about what they provide. Properly managed expectations don't lead to as strong of emotions when a customer finds out what they're getting doesn't work for them.
Commercial users will save face by A) avoiding discussing anything that could make them look bad, ditching a vendor is a tacit admission that you went wrong when you chose their service. Additionally, if their service is built on top of the ditched service, it casts quality concerns on their product too. B) Avoiding disparaging anyone, ever, is pretty common play-nice-save-face.
Why should I give my competitor helpful advice? I got caught up in a web, I don't need to help the competition avoid my mistakes.
In a multiple person organization, the party angry about the service and the party who decided to buy/use it are different individuals and airing grievances/venting is an internal process that they don't reiterated in public.
In a free service, a failure to deliver is the only downside. If I paid for a service, I'm going to be too preoccupied beating myself up for making a stupid purchase decision and looking for an better alternative than bothering to share my findings.
These causes and more are all often balled up and interpreted as free users acting "entitled". Sure that exists, but its a classic case of reading too much into imprecise metrics when someone assumes the cause.
Furthermore, the feedback from free users isn't inherently worse, and is just as subjective as that from paying users, it just tends to be more negative. For all we know, any given product could objectively be shit and because only rubes will buy it, the free users are the only ones that can let you know.
Interesting, I have been using their services for a while now but most of my traffic (probably 99%) actually goes straight to my web servers as i'm only really interested in the free DNS hosting service, however looks like i'm breaking the ToS as well, may be time to rethink :S
Could someone explain the reply from "Matthew Prince" to me?
They turned it off due to an attack and would turn it on again if upgraded to business ($200/mo).
Does this mean turn it on while under attack would cost, and after the attack would be free again? Or it would cost the upgrade to turn it on either way?
They claim that other customers were affected, so they stopped handling requests.
The process would have been automatic if he was a paying customer. Because he wasn't, the company felt justified in not trying harder to bring their resources back up.
One thing I'm yet to puzzle out is whether CloudFlare handles apex domains. They say they do but DNSimple says in reality they just route it straight to your servers so that it's pretty useless.
CloudFlare does handle apex domains; when you turn the service on, it changes the A records to point at CloudFlare. I don't know where DNSimple is getting their information, you can see CloudFlare working on a naked domain just by trying to resolve the hostname. It'll return two (or more) CloudFlare IP addresses, not the address of your server.
4chan (and its users) probably want a fast downlink. The author touts a cheaper provider, but the service they are receiving is an order or more of magnitude slower service from my anecdotal test through wget(s) of images on both OP's and 4chan's sites. That is, a pair of dedicated and unmanaged servers is not, AT ALL, comparable to a CDN's offerings.
4chan probably also expects to have more than an email (and timely) endpoint with which to correspond with the provider of their most core site-service -- image serving.
Because moot wants 4chan to actually be online occasionally, and to load in a reasonable amount of time. Notice how terrible pr0gramm.com performs, with a tiny fraction of the traffic 4chan gets. That's why real sites can't just get phoney "unlimited bandwidth for $20/month".
Were you already using http cache of 1 year for all the images when this happened? Do you think it could have been avoided by setting it to 1 week, 1 day, or even 1 hour?
It's funny that I have to ask this to you instead of asking Cloudflare. They have really messed up on this one.
I've rented servers for high-traffic websites with no income for years. If it's something you love to run, and you can afford it, then it's just the cost of your hobby. Not everything is a business.
If he is as really rich as you suggest it seems odd to me that he would complain about Cloudflare and not have rented a 100 TB server sooner. They are not exactly difficult to find.
Whether he can tell us if he is rich or found another way to monetize adult or potentially offensive content, it is up to him to share.
$200/mo is not "rich," it is what you would spend on a hobby (think a membership to a climbing gym or buying a new gaming computer once a year). Being able to spare $3000+/mo for CloudFlare enterprise is not even close to a comparison.
Now this might just be my opinion, but considering many of the people I know in the technology field have a disposable income of around $1k/month, I don't think there's very many people that would throw away 20% of it on an image dump. If you make more than that, then you are rich by my standards.
See, for me it's not an "image dump", but a nice hobby project. Managing a community as well as the technical challenges this site presents is interesting. I can spare $200 but not $3000.
Also, I learned a great deal of JavaScript while building that page and I just now learned a bit about load balancing and setting up Varnish - something which may come handy for godknowswhat. I never imagined making my living by writing a JS game engine either.
The attack continued and began to affect the performance of other CloudFlare customers, at which point we routed traffic to the site away from our network. While we encouraged the site owner to take advantage of the Enterprise tier of service given their needs and traffic levels, the site would have been brought back onto CloudFlare's network if they had upgraded to the Business tier of service ($200/mo) which included Advanced DDoS mitigation.
To be clear, CloudFlare does not bill based on traffic. However, resources are not infinite and when an attack against a Free customer begins to affect the performance of other customers we will take measures to protect the overall integrity of the CloudFlare service.
Matthew Prince, CEO, CloudFlare, @eastdakota (Twitter)