No one likes DDOSes from China. One can plead Amazon as much as one wants. Pay or get booted, there are probably 2 engineers paid 6 figures a year by Amazon getting paged for this DDOS, someone must pay for the time they spend tuning DDOS protection instead of their primary project to make attacked website accessible for everyone else.
Source: worked for AWS, was oncall during similar attacks. Nasty things with those they tend to start around 6-7PM (guess when does working day start in China).
The two colo centers we've hosted in have always helped us with DDOS issues free of charge. Maybe that's not normal, but even a former employee telling us to GTFO looks bad on Amazon to me.
I am not telling anyone to GTFO, nor, I believe, Amazon does. It depends on a DDOS, there are a lot of smaller-scale DDOSes are just absorbed. Some are stupid filtered easy enough that no one is notified, some are serious enough. My first oncall at Amazon I got ddosed from 3 VPS machines, easy enough, a month after same attacker started to shift machines inside VPS, then a month after attacker started to spoof ips within narrow range of ips, in just a half a year (yes! they can last THAT long) attack was coming from a range of spoofed IPs with a traffic that followed no pattern except for destination they wanted to go down at many gigabits per second.
In this case - 700k QPS (gigabits of ingress) of well engineered HTTP/HTTPS DDOS traffic is not something an average colo can or even will be willing to handle at all. I'm assuming a hot-potato DDOS, when a customer comes along with a long tail of colos and providers that already booted him. All that traffic, servers and ultra expensive engineer time. Everyone wants it for free, but ALAS.
If you start talking about co-located or self hosted services, the mitigation strategies are very different.
Assuming you can find yourself a transit provider that supports BGP flowspec updates (many don't, sadly), you can do this fairly cheaply. You'd obviously want some level of support from a network tech that knew what they were doing, but it's not insurmountable.
There's a bunch of other options available too.
This sort of thing is one of the downwsides of having your infrastructure managed by someone else. If things go wrong and your provider doesn't feel incentivised enough to help you out, there's a lot less you can do about it, other than just pay whatever sum they demand.
I have no much experience with co-located services so can't really comment on that. I can't go in detail how and what mitigations are applied on AWS site also, as I feel obliged to leave as much weapons on a "good" side of ddos as possible, and knowledge is one of those.
What I remember pushing BGP flowspec updates upstream was thought about as something close to impossible though.
Interesting when your product can be spiked, and make significant increases in profit. This looks like numbers that could potentially knock a business out of business. Reminds me of old phone bills.
Also usually AWS doesn't turn attack into revenue, they push customer up the "support tier" (gold/platinum whatever they are called now) and strip the DDOS traffic from expenses as much as possible. Those tiers are quite expensive though, but are fixed support costs more or less.
My general point is: AWS is a business, and it operates as one. There are no hollywood style bad guys sitting there in cubicle dungeons on chests filled with gold thinking how to extract money, quite the contrary. It is understandable that customer cannot pay unlimited (from customers perspective) charges, but AWS pretty much incurs them, as customer being ddosed is consuming resources that would be otherwise be sold to others, or engineer time that would be put into developing new features and attracting new customers.
That simply isn't reasonable. Name one business other than maybe network providers who's revenues grow in direct proportion to incoming packets, regardless of content?
You can't disregard any business that doesn't fulfil that property as being "eventually unsustainable".
My comment was a bit more general than pure "packet". I agree thats where the disconnect between low lever service provider and customers come - providers revenues and expenses are "packets", while they don't always translate to revenue for customers.
However my note was about general "traffic", if one sells video views for example, and revenues do not grow inline with adjusted to [almost always decreasing] bandwidth costs sooner or later that will become a big problem.
I was at a product management event once and met a guy who managed a product in this space. A group of us went out for drinks after the event and he ended up explaining what he did. At some point he mentioned "Chinese hackers." Another guy in the group called him on it, wondering why he just assumed it was Chinese. He laughed and said that the near constant level of activity they see goes basically flat on Chinese New Year.
I suppose if you're not a Chinese hacker, it might pay to pretend you are by tailoring your working hours and days.
Why don't providers just set up a system that creates a country-level null route for a given destination IP? And have a UI with a checkbox for the user to do it, for any selected country. It would mitigate the issue, and once it's over, the user can un-restrict traffic / or just keep blocking if it's a non-valuable source.
I know you can do this on the server, using many different techniques. But this does not help as the traffic still reaches you (that you have to pay for).
You can also do this with Geo DNS (and get much less of a bill).
And the ISPs, datacenters, and anyone with a router can block ASIA or China allocated IP ranges. Especially if it's not the type of a flood that's designed to attack the routers (instead of the web-server).
The point of their website is to make censored content available to Chinese users.
China is attacking them to prevent Chinese people from reading the website.
Your suggestion is to make the site unavailable to China.
Do you see why it is not a solution? You are basically setting up a market for censorship-- the attack doesnt ever have to end-- depending on how much China is willing to pay to keep the website offline.
The great firewall does already block this site, and I cannot view it now without turning on my VPN. Therefore this site is pretty useless to me currently, but someone has to fight the censorship. Maybe one day it will actually succeed?
If normal citizens had access to a VPN in which to access this site from another country, it would be quite redundant to use this site then wouldn't it? Maybe I'm not understanding.
Please, don't perceive this as being rude, it's not meant to be.
Having provided IP transit at a largish network provider in a previous life, you have no idea at the complexity involved what you're asking for. It could be done, but the costs involved are non-trivial.
If you're honestly interested in the complexity involved, start reading about BGP, dynamic routing protocols, router/switch fabrics, control plane integration, autonomous systems, peering agreements, etc.
I feel it shouldn't be unreasonable to expect AWS/Cloudflare/Akamai to have policy-based routing to blackhole a lot of these source subnets. Of course it's complex, but these are some of the largest hosting providers in the world.
I've found this is a common thing to say with AWS employees. One of them insisted that Amazon's ridiculous ephemeral storage policy (immediate, permanent, and irrevocable deletion on any halt or stop event, making accidental data loss a real possibility) had to be that way because it would just take too much hardware to allow a cooldown period before the drives were wiped. There's no way I believe that. I think Amazon is just used to intimidating customers with exactly that line of reasoning: "No offense, but you have no idea how hard the cloud is", and people buy it because "the cloud" is the new hotness.
I've had RAID 6 fail. It should be extremely rare, but isn't. And at AWS's scale, It's not hard to imagine servers going offline regularly. Ephemeral storage as a policy makes sense to me in the sense that you can separate out that what's important from that which is ephemeral, and provide cheaper storage than a more HA solution like ganeti.
Why isn't the persistent data put onto an EBS volume?
If ephemeral storage is a drive local to the virtual machine's host (which I think is the case) then having a cool down period would mean holding the hardware that used to be used by you in reserve until the grace period expired.
It's possible but it's a lot of work to solve a problem that can be better solved by not relying on ephemeral storage persisting.
I mean, it's a nice idea in theory, but in practice stuff finds its way onto the ephemeral disk even if you have EBS volumes mounted, and "Sorry, we just deleted all your crap, I guess you should've had that on EBS" (which is an extra fee by the way) is not an acceptable solution to the problem.
Yes, it would mean holding the hardware in reserve for the cooldown period. I'm not talking months here, just enough time to recover from an accidental "sudo shutdown -h now" instead of "sudo shutdown -r now" (or similar). It'd be nice if Amazon sent an email warning about the condition and giving you an hour or so to go in and save your data/restart your instance. They could even make it a policy that you're charged for the time your instance is running + 1 hour to facilitate cooldown feature if they're really that worried about it; it's better than wiping data as soon as someone stops (from AWS console) or shuts down (from real console) an instance and providing absolutely no avenue for recovery, no matter how quickly you notice the mistake.
I have never had stuff accidentally find its way onto ephemeral storage. The ephemeral storage is mounted at a specific location of /mnt. Everything else on the system (OS, binaries, application code and resources) is stored on an EBS volume.
You have to specifically put something into the /mnt folder if you want it to be stored on the ephemeral storage. Any other location is safe and will persist through halts and stops.
In practice the only thing you should ever use the /mnt folder is maybe a Nginx disk cache, or as an alternative /tmp or something like that. Basically if stuff you don't want to lose is finding its way onto the ephemeral storage then you are doing something wrong.
It depends on your users. We have some users that are not super well-versed in AWS and they just see a big disk and put data there, and someone has to come back and move it to an EBS volume to make sure it's safe. /mnt is also used as a staging area for large files and the intention is always to move them to permanent storage when done, but that sometimes doesn't happen. /mnt is usually, in the non-AWS world, where the bigger, more authoritative disks, like an NFS mount to the NAS, would be mounted, so it's counter-intuitive to tell users to treat /mnt like /tmp. Even if someone is using /mnt as a temporary store because they understand EBS v. ephemeral, if they shut down from within the instance, they don't see any warning about the doom of the ephemeral data, and it may be unclear that a shutdown/system halt is the same as a "stop" in the AWS console, and they could lose the data that they had in the staging area unexpectedly.
There are plenty of plausible situations where an AWS user can find themselves with important, even just temporarily important, data on ephemeral. Whether those are the result of "correct" usage or not, it's beyond the pale to just zap that data away and tell the customer tough titties as soon as a shutdown command is issued.
I've been looking for work for a while, but I won't even respond to solicitations or job board posts that so much as mention the cloud, agile or scrum.
What you said adds pretty much no value to the conversation, I didn't mod you down, and I didn't look to see precisely what field you work in, just the same, cloud is pretty much an obfuscation for on-demand co-location and shared-hosting services.
While you may not like the trend, and some providers have better options than others, and you may require some operations over others... but very few businesses can afford to manage multi-site infrastructure that can dynamically scale. Most are probably served just fine with rented servers, traditional colocation, or don't even need more than one VPS...
That doesn't make the technology bad, and only makes you seem ignorant in your prior statement.
I'll go into it later, but for the most part I consider the cloud a really bad idea. I also regard most cloud companies as "buzzword-enabled" so as to attract investors.
My gripe with scrum and agile is not so much with the methodologies, but with companies that think they have a methodology when in reality they have a bureaucracy.
Geo DNS (to be more precise AS numbers is what is used) is something that is sometimes used as a very last resort - customers in China are customers as well, and if you drop everything from China thats effectively what the attacker wanted.
I recall only hearing about one time when packets from Chinese ISPs were completely dropped for some reason and only for short period.
I've also have an anecdotal reference that one can persuade providers that actually deliver traffic from China to do filtering on ingress on next hop routers after China, but that should be something very serious, that impacts their revenue as well and prolonged. As another commenter noted - costs for providers are very non-trivial.
In my experience DDOS is always money competition, it costs money to mount one, it costs money to defend against one. Unfortunately when one of the sides is [allegedly] a country it doesn't play very well.
the real reason is because Amazon wants to do business in China, so they absolutely cannot do something like that on their end without getting blacklisted by China's government.
So now you DDoS the captcha system. For companies not operating with massive bandwidth and computing power, you can just overwhelm their defenses. Cloudflare can get away with it, because they explicitly set out to be able to "service" those super huge number of requests.
I was working on an anti DDoS system for SIP, a UDP-based protocol. Basically the options were: 1. lockdown, just whitelist known good customers, and break many scenarios. 2. Attempt some kind of analysis, like sending out probes to determine good/bad IPs. 3. Scale the hell up. Write L7 stuff that can go at wire speed, and get lots of wires.
Needless to say, #1 is the easiest to implement, but allows you to get your pipe saturated. #2 requires compute + pipe, and #3 is the only thing that'll really work.
This matters because DDoS'ing a telecom can be very lucrative. I can say with good confidence that demonstrating DDoS capabilities are probably worth 5-6 digits in blackmail against many companies.
Greatfire is unique that they want the site to remain accessible to ordinary Chinese users while withstanding the DDoS attack (so they can't blackhole all traffic from China either).
If they put a reCAPTCHA wall in front, the GFW can simply block reCAPTCHA (easy -- it is a Google property and they block everything else from Google anyway) and no one from China can access Greatfire without a VPN. Mission accomplished.
If the goal was only to block Greatfire for non-VPN users, then they could just use the GFW for that from the start. The use of a DDoS can only imply that China wants the site offline for everyone, even VPN users.
I think Greatfire is evading the GFW by hiding their mirrored content behind innocent looking websites such that the GFW does not block it. Once the censors discovers a Greatfire node, they block it, but then Greatfire just moves on to another IP address or domain name.
With this DDoS, they are taking the different route of attacking the infrastructure of Greatfire such that they can't serve traffic from China at all. Causing massive bills and outages for Greatfire is probably a bonus, but I don't think that is their main intention.
I chuckled, because when everyone tells me "AWS is practically the internet" I can point out "The Internet is resilient at a far lower cost than Amazon".
CloudFlare is probably not a good choice. They recently blocked access to a similar service, Lantern, per the linked WSJ article.
"CloudFlare, which offers content-delivery network services, said last week it cut off Lantern’s use of the service, saying it was unauthorized. “We don’t do anything to thwart the content restrictions in China or other countries,” said Matthew Prince, chief executive of CloudFlare. “We’re a tech company and we comply with the law.”"
> "We don’t do anything to thwart the content restrictions in China or other countries," said Matthew Prince, chief executive of CloudFlare. "We’re a tech company and we comply with the law."
There's a popular idea that businesses (and people) have no responsibilities to anyone but themselves, because what they have is theirs; they built it themselves. But if you think about it a little, it's obviously false. Here's a more accurate statement:
We're a tech company whose success is completely dependent on the freedoms in our nation and many other nations around the world, and on the political and economic systems, infrastructure, and enormous wealth that blossomed from them. Without the sacrifices of blood and treasure by our predecessors of hundreds of years, and of many people today, we would not have these resources or opportunities today. There are many talented people born in many countries who, without these benefits, have no opportunity for success.
They can't sacrifice their company for every principle, every time, but there's a middle ground between that and 'we're just a tech company so we have no responsibilities'.
> There's a popular idea that businesses (and people) have no responsibilities to anyone but themselves.
It's not just a popular idea, it's why they are created as firms instead of philanthropies. There is a difference and it does matter what the expectations of the donors/investors are.
> We're a tech company whose success is completely dependent on the freedoms in our nation...
This sounds great but how is it reflected in company policies?
> They can't sacrifice their company for every principle, every time, but there's a middle ground between that 'we're just a tech company so we have no responsibilities'.
A company could easily make a statement to its investors about its moral stance on issues that it expects might harm the bottom line.
The company does have responsibility to its investors not to go rogue and burn cash just because it feels good. Most of the time the kind of corporate behavior that you praise is actually clever PR that costs the companies little.
It seems like you're saying that private enterprises should either completely divest themselves from any commitment to social responsibility (defined broadly as: "doing the right thing because it's the right thing, even when it may seem to go against the bottom line") -- or they might as well thrown in the towel and become philanthropies. Yes?
In other words -- on the blackhat-whitehat scale, it's either black- (or at least very charcoal-y grey-), or whitehat. But I just don't see modern, large companies generally acting that way -- not because they're led by altruists (they're certainly not); but because that's just not human nature (across the board). Most of us are greyhats (somewhere on the scale); and the behavior most business leadership I've either read about, or seen directly (behind closed doors) seems to fall somewhere on the greyhat scale, also.
That is: large business definitely aren't philanthropies -- but in general, most of them (even many of the traditional "bad boy" players like banks, big pharma, etc) -- aren't straight-up moral nihilists, either.
At least that's the way I observe these things. I could be wrong.
I don't disagree, but in terms of a framework for evaluating the behavior of businesses I think the following is reasonable:
Businesses should act within the law, and lawmakers and the public determine what legal safeguards are necessary. For example, if you start a restaurant you must comply with health code, fire code, etc. If you start a bank you need to keep a certain amount of risk capital, etc., etc.
One could argue that all dishes used by a restaurant should go through hospital level sterilization, or that banks should contain more risk capital than they are required to by law. Such arguments would be in the name of safety or quality.
One could similarly argue that restaurants should use at least 20% locally grown produce or that banks should lend 20% of capital to underprivileged groups. Such arguments are in the name of moral responsibility, etc., and lawmakers have actually implemented many such laws for banks.
For an investor who wishes to invest in a bank or a restaurant, there are many options. Being able to compare financials and other metrics will help the investor figure out which is the smartest investment (based on her risk appetite, etc., etc.)
Why might a restaurant decide to focus on locally grown produce or a bank decide to focus on its ethical treatment of subprime borrowers? Largely for PR/marketing reasons. If such marketing campaigns are successful, customers will flock to the bank or restaurant in question and (assuming they are still able to be profitable) make the bank or restaurant a more desirable investment.
One can pick any business and any metric that he thinks has moral significance and claim either "regulators should require x, y, or z" or that "that practice is horrible". One might be right... essentially ahead of the game morally from society's average.
The perception of moral progressiveness, like the font chosen for a brand, is one factor that helps determine a business's success. It may be the case that most of the meat we eat was raised in unconscionable conditions, or that 30% of imported electronics were assembled by modern serfs in near-slavery. The more we are aware of such things, the more likely firms are to make the most progressive choices.
Moreover, I'm not sure their argument makes sense even on its face. When they say they "comply with the law", which law do they mean? There are many thousands of lawmaking bodies. What if a small-town mayor passes a law outlawing the word "webinar"? What if China passes a law saying that DDOS protection is illegal worldwide? Or websites not properly registered with the Central Propaganda Department may not be carried by any network provider?
Cloudflare, I'm sure, will happily ignore any laws like that. The question is: why not ignore this too?
The wonderful thing about sovereignty is that a country's law's jurisdiction is whatever the country decides they it should be.
The degree to which they can practically enforce that jurisdiction becomes a game of relative power and how willing others are to constrain it, of course.
What counts as human rights is subjective. The UN says that it is a human right to receive and express opinions through any medium. Does that mean that we should hold "human rights" to be more important than revenue and forbid service providers from charging for access to information? Like the WSJ who wrote the article that's supposedly to blame here?
I'm sure whatever country you do live and pay taxes in has at some point in the recent past violated someone's human rights, or at least they have in someone's opinion.
Given that the country you live in has violated human rights to some extent, and that you could reduce your contribution to that by not earning taxable income or purchasing taxable goods, is it not also your defacto position that you value revenue over human rights?
(My apologies in advance if you've ceased paying taxes, or buying anything taxable, or if somehow no one in the world believes your country has violated human rights, or might do so in the future, or your position is that revenue > human rights)
My point is that the world is a lot more grey than you make it out to be, and that you are also in some way likely valuing revenue over some human rights abuses.
I don't agree with a lot of your other posts, but I think we're on the same page here. When I watch the youtube video where Boston Dynamics demonstrates the stability of Spot by kicking the robotic dog all I think is, "Don't kick the dog bro". It's machine intelligence descendants are going to judge us, or maybe they won't care and will kill us all of anyway.
Actually, there is some evidence that slavery was very economically efficient. 'The Half Has Never Been Told' by E Baptist lays out how enslaved labour picked much more cotton than the then free labour.
That was then, though. Now, when skilled labor is of greater importance and unskilled labor of comparatively little value, I submit that things may well be very different.
You forgot that their success is also built on the oppression of the Chinese, and low salaries etc that gives the rest of the world affordable hardware (and rare-earth minerals, metals, etc).
Not commenting on what CloudFare should or should not do, just indicating that your high horse actually has longer legs than you gave it credit for.
Replying to my own comment (I'm too late to edit it).
My comment is about that concept in general, not about Cloudflare in particular. I don't pretend to know and won't judge Cloudflare based on one sentence taken out of context. For all I know they are excellent members of the community; in fact they could do be doing good things behind the scenes without publicizing it, which might be wise if they are as exposed to China as other commenters say.
Forgive my outburst, and maybe this sentiment won't be well received given the context, but I just find it to be downright unpatriotic for a US company like CloudFlare to stand there saying things like what Matt Prince says in your quote, when someone comes under attack by an opposing nation state.
Again, I realize this place isn't exactly a bastion for this kind of sentiment, but have some thought for freedom here, CloudFlare. The US may suck at helping a lot of the time, but if you've got a group of folks trying to deliver some good ol' freedom to a country like this, and that country is trying to shut them up, maybe put out a helping hand, or at least don't shut off service.
In the case of Lantern, they were taking advantage of a bug in our system. Specifically, they were setting the SNI field (outside the encrypted packet) of a request to look like it was going to an actual CloudFlare customer (e.g., news.ycombinator.com) and then setting the host header inside the encrypted request to point to some restricted site. The bug was that we did not check that the SNI field matched the host header, which allowed Lantern to do what they were doing.
Lantern was not a customer of ours, instead they were exploiting this bug to essentially disguise traffic to look as if it was coming from one of our actual customers. One of our biggest concerns was that this would put CloudFlare's actual customers at risk of being blocked. And, beyond that, even if it weren't being used to avoid Internet restrictions, that someone could effectively impersonate the identity of a customer on our network is, per se, a flaw that we should patch. As soon as we became aware of the issue, we began matching the SNI header to the host header and, effectively, patched the bug.
We've always been very supportive of a free and open Internet. However, even if we support what someone is doing, we can't put our current customers at risk of collateral damage or keep open bugs that allow our network to be exploited.
Matthew Prince
Co-founder & CEO, CloudFlare
@eastdakota
> Lantern was not a customer of ours, instead they were exploiting this bug to essentially disguise traffic to look as if it was coming from one of our actual customers.
This makes a world of difference.
Just to confirm, does this mean that if the exact same attack had happened, but Lantern had been a CloudFlare customer, you wouldn't have shut them down?
Still curious about this quote:
“We don’t do anything to thwart the content restrictions in China or other countries,” said Matthew Prince, chief executive of CloudFlare. “We’re a tech company and we comply with the law.”"
So if Lantern were a customer, would the outcome still have been the same?
Well, if Lantern were a customer, then China could just block them like they do for any CF customer they want to block. The reason the bug was allowing people to get around the firewall was because they were pretending to access a site that wasn't blocked, but actually receiving content that was blocked.
Patriotism is not a justification for violating the law. Granted, modern politicians and civilians use patriotism to justify literally anything they want to do as long as it's in the name of the Homeland (similar to religious martyrs justifying anything they do as in the name of their God).
Usually patriotism is the last justification used by those who have nothing else to stand on, like the KKK trying to oppress African-Americans, or the Nativists trying to oppress Irish immigrants, or modern-day politicians who decry all Islamists as terrorists, or the border states trying to oppress migrant workers, etc. Each time they've exhausted all other excuses, Patriotism is the last justification for their actions. (I won't touch on Mao, Stalin, Hitler, etc because they're too tied to specific nationalist policies)
Personally, I wouldn't want to identify myself as a Patriot, because usually they're the ones standing on the wrong side of history.
> Patriotism is not a justification for violating the law.
Actually, it is. Patriotism, in being a Patriot, is a loaded word in the American (USA) context. Specifically, it is about doing what is good/right for the country and her citizens regardless of the law (i.e. British rule.) Or so says my recollection of American History. I mean... just look at the Patriots (rebels, in the british colloquialism) in the image on the wikipedia page for Patriot_(American_Revolution).
"The Oxford English Dictionary third definition of "Patriot" is "A person actively opposing enemy forces occupying his or her country; a member of a resistance movement, a freedom fighter."[1]. In this definition, if the alleged DDoSers are Chines, attempting to block the actions of a foreigner imposing influence in their own land, they are the more Patriotic? Which is why the term is utterly useless in this argument; Dare, any other.
> Usually patriotism is the last justification used by those who have nothing else to stand on[sic]
Patriotism isn't a word really, it's a neologism invented in the 18th century, probably attached to by the founders because the British hated the term. And while Patriotism's historical (and more ethical) definition might have been to defend the principles of one's country and the constitution given to the people, the modern definition is waaaaay different. At this point we should bring back the word Loyalist for the people who use Patriot to mean someone who blindly follows their government.
Not everyone desires to take part in geopolitics and become a tool of diplomacy. Some people just want to do their business and it's perfectly fine in my opinion. You can't force people to be patriotic or to feel a patriotic call.
I worked with a DDoS protection provider briefly. Suffice to say, it's quite possible that being public with identity can bring a significant chance of physical harm. Dunno about this particular case, or China, but for other people offering services to that continent-area, they had real concerns.
I got confused, are you talking about bringing freedom to the US ? :)
Kidding aside, not saying you're wrong, but companies that want to maximize profit take a too big of a risk alienating a possible big market...
LOL. Booters (that carry out DDoS attacks which are illegal in CloudFlare's own country)? No problem for CloudFlare. Trying to circumvent Chinese content restrictions? Nope, that's unauthorized!
Yep, it's hilarious. Cue Mr. Prince to come in here and give a half-assed explanation as to why it's not actually a contradiction, even though it clearly is.
That's funny, cloudflare has a project to "Protect Free Expression Online"[1].
It even states:
"Often these attacks appear politically motivated — going after, for instance, citizen journalists reporting on government corruption. The promise of the Internet is that it is a great leveler — that anyone with an idea can reach a global audience. These attacks threaten that promise."
Historically Cloudflare have been quite strong in their support for free speech. For example, they run Project Galileo to protect public-interest sites against DDOS attacks: https://www.cloudflare.com/galileo
I'm guessing in this case it's simply a case of them choosing which battles to fight. They probably don't want to commit to run an open proxy for everyone in China to access banned websites. That would likely get them banned outright in China, which, for a CDN like Cloudflare, would really hurt their core business.
Excuse my ignorance.. is it really the case that website a gets ddosed, website a gets charged by amazon for the ddos traffic... and amazon isn't inclined to mitigate the attack? Will wonders never cease......?
Actually the Prolexic product is one of the most innovative and effective one we've seen to date. DDoS attacks are not a commodity issue, you have to pay to play..Not sure how that makes Akamai horrible..
I can confirm that Akamai is a pain to deal with. Defense.Net as well. Cloudflare is what I would chose but they are siding with the Chinese gov't at this point.
Not sure what you're basing that statement on. Cloudflare very recently gave support to various parties involed in the pro-democracy movement in Hong Kong. They claimed to have weathered the largest DDoS in history in the process.
From my experience, this is confusing the issue - the OP was talking about using their DDoS filtering service, not their CDN. This is done by Prolexic - who despite being acquired by Akamai, still seems to function fairly independently.
I've no idea what the CDN team are like, but it's not really fair to say don't use product X from company Y, because product Z sucks.
We've tried a number of DDoS filtering services, and Proxleic/Akamai has been the most professional and effective so far. They're also the most expensive, by a fair bit, but you get what you pay for I guess.
Could you define buckets of cash? On the low end, one gets 30TB/month (1gbps uplink, 200mbs guaranteed) included for ~60 Euro a month, or 100TB for ~160 Euro (for a single server, outbound) at Hetzner:
Such a single server, isn't the same as a CDN of course, just curious what you consider "buckets of cash". I see cachefly asks for 400 USD for 2/TB a month, for comparison.
Agreed. Can't understand why I never see DosArrest on these lists though. They've been doing DDOS protection for a while and are good at it AFAIK. That said, I have no idea if they are interested in taking on the Chinese government.
isit possible to combine them? like, would my domain actually resolve if i change my name servers to cloudflare's, then cname it to incapsula? i'm just curious...
For the record WSJ does use Akamai already for many of it's static resources. These are mostly requests for things that can not be cached already in some way. (I'm sure the crew is looking for some additional ways now =) Source: I used to work on WSJ and now within a different division of Dow Jones.
For them, using a DDoS mitigation service might be harder than it seems. Their project, "Collateral Freedom", depends on having a set of web frontends with some special properties. Namely, 1) the provider of the said frontends must be willing to ignore takedown / access restriction requests from (Chinese, Russian, etc) authorities, and 2) the access to those frontends can't be easily blocked by the Great Firewall of China / Russia / etc without causing massive problems to some unrelated widely used services.
(full disclosure: I do devops for one of the Russian sites that is currently mirrored by Collateral Freedom)
They wouldn't just be giving in, it would take less than a day to get up and running on CloudFlare. That is probably worth avoiding $30k bill, especially for a non-profit.
As for PR, there absolutely is good will towards organizations that take on censorship. This is the #1 story on HN, so a company stepping in and saying "we got this covered, free of charge" not only shows they are good people, but also gives would be customers an idea of the quick implementation cycle for their CDN + the load it can handle.
Real world use case + helping to stop censorship = great PR
This is exactly what I was thinking. I read the blog post, then came to the comments. I completely expected something along the lines of ... "Hey, I'm the VP @ SomethingTech, we like what you're doing and will help you mitigate the attacks for free if you get in touch with us." ... to be the top comment here.
My reaction was based on the interview where they say that the 30k$ charge was harming, not killing them. You're more knowledgeable than me on CloudFlare and it may very well be a strong option.
Now in a way their $ penalty with this attack is not unlike growth. To me it sounds like they invested in their branding and will have a name as a company that can sustain damage and keep its ground - important because their goal is to be credible in bringing content in/out China.
What if they had given in? They'd be yet another victim of an attack sort of linked with China.
I do see a business sense behind what I assumed was their moral stance and I definitely stand by it.
Seems brilliant to me if you are trying to battle online censorship in China and get your message out. I agree that one of the mitigation services could benefit from some pro-bono services.
I agree with this. Silently allowing a bully/attacker do as they want is never acceptable. I find GreatFire.org's work commendable and hope they get the help they need.
If they're a non profit, then maybe this is exactly what their mission is. They can sit back and rack up a huge aws bill, then Amazon has to decide whether to collect. If they do collect, then that was what the money was for anyway. I'm guessing that giving in to bullying from the Chinese government is against their core principles here.
As xnull6guest already noted, they are most probably already financed by the US government, that's why they can be laid back: they are exactly doing what they are paid for.
Move to OVH -- they offer free DDoS protection as standard, and unlimited bandwidth. I just moved to OVH after getting DDoSed. I'm paying $109/month for a quad core 3.7Ghz Xeon, 64GB RAM, dual 2TB software RAID. It's a pretty sweet deal, and I haven't had any problems so far.
100% worth it. I have monitoring/notification set up on my infrastructure, and have had two hardware failures since becoming an OVH customer a few years ago. OVH techs in the datacenter were on the scene before I got downtime notifications. Their techs must deal with this kind of thing all night, every night, because they seem to have this process of replacing hardware down to a science. This as opposed to my other dedicated providers, who basically don't give a shit, for the same hardware at the same price. OVH is an excellent budget investment!
I think the reason they can offer it so cheap is a combination of scale, cheap electricity, and tax breaks. iWeb is also in Quebec, so I figured there must be some tax break. Lo and behold there is one heck of a tax break:
It's actually even better than that because there's also a national program that pays up to 35% of R&D salaries: http://www.cameronhuff.com/blog/ontario-tax-credits-for-soft.... There are hundreds of smaller special programs that can pay even more (and they often stack).
If you don't need the latest hardware and enterprisey features, give soyoustart.com a look. It's an OVH company on same network with same DDOS protection that uses the 'OVH' brand recycled servers. About half the cost.
And if you don't need any support or server hardware, give kimsufi a look.
I do not work for OVH, but am so impressed by their automation and value in this field I refuse to use anyone else.
Unrelated to the story at hand. Have you been recently DDoSed on OVH? I know that (at least some time ago) they just null-route/deactivate your account on spot with no notification on anything that looks like a DDoS.
This may be true. However, they are FAR more inclined to work with a customer before just flipping a switch like a lot of (for example) American hosting companies will. I have nothing but good things to say about OVH. They were helpful during a DDoS I had about a year ago throughout the incident, but I'm sure they would have nullrouted me had I not been so responsive to their support guys when things started going badly. "I'm sure" here is pure anecdote and based on zero evidence, but it's the feeling I got.
I moved from iweb because they null-routed my server during a DDoS attack that happened for no apparent reason. (My guess is that it was a user upset at some user-generated content, so they decided to DDoS my server rather than notifying me).
Been running a game server on OVH that has been repeatedly DDOS'd by random people. It may take a moment or two to kick in and cause some wonky connectivity around the times of the attacks, but their service has been able to take some attacks in the several GBps range without a problem while we have used them.
They are using Arbor in their core network, and auto mitigate attacks on customers now. I am working with Arbor systems as well as others for DDoS protections where I am working now, and I can confirm that it works quite well.
Takes about 5 mins to kick in once an attack has started, which is very good.
Yes, and the service has remained available to most users. It's just the initial flood knocked stuff out for about 2 minutes as the worker threads dropped their request queues.
No, GB is perfectly correct when referring to memory. Just because some people standardized on Gibibyte and redefined gigabyte doesn't invalidate what memory makers have been doing for ever. JEDEC still uses GB, as they should.
I was unaware they had a monopoly on language usage. A byte is not an SI unit. Base2 is vastly more defensible and natural than base10. The real issue is that everyone in networking likes round base10 numbers divided over some arbitrary cesium fluctuations. This leads to 1GB / 1Gbps not being 8 seconds, which is confusing. But in JEDEC's and others defense: "why should I have to change, he's the one that sucks."
Trick question: How many bytes are in a 1.44MB floppy? In a 700MB CD? A 4GB USB stick? And a 480GB SSD? How many bytes/s bandwidth for 10Gb Ethernet?
I can't understand how people can be so delusional to think that randomly redefining prefixes is a good idea. It was never, ever used consistently, and JEDEC just should get rid of this idiocy.
Trick question in return: I'm selling a 256GB SSD. How many bytes am I legally required to deliver?
This is why there is a need for the two different prefixes. If I am selling a 256GiB SSD and you are selling a 256GB SSD, those are now differentiable on their face.
The IEEE and/or SI are in charge of language? The same IEEE that sends car insurance offers to its members, just so we're clear. JEDEC is also a standards group, and they disagree. What now, a Wikipedia editing war to determine the victor?
Standards bodies aren't anything magical, and I don't get the slavish following they seem to get. So an RFC says something, or another group mandates something. BFD. Unless you're expecting interop to work, use standards as you see fit. They aren't an ends unto themselves.
In this case "GB", when referring to RAM is unambiguous. Only disingenuous cloud providers or petty editors would use a base 10 interpretation.
I don't think it's too good to be true. They have over 2Tbps of unused inbound capacity (or at least they did in 2013, and probably more now), and their routers mitigate attacks at the boundary of their network.
This is the correct answer. Given that this website is part of the 'civil society' sphere, it likely already gets taxpayer money from the State Department. They should go to their funders - be they public or private - and ask for help with their mission - I'm not sure it's appropriate to ask individuals for help.
But the funders won't solve this problem though, only mitigate it. (though I agree it's something they definitely should do ASAP) Putting it out there like this, seems more likely to overcome the technical difficulties of it.
This is interesting. Though Hacker News appears to not be blocked, it has been flagged as "Contradictory" on certain days. Is the Chinese government blocking certain news items?
If you click on the day in the calendar to look at the details, the "Contradictory" status is when some of their test servers work and others don't. For this site in particular, there are several servers showing a timeout and no data received. So it's possible that HN is being partially blocked.
Censorship in China isn't just an all or nothing thing. There are content filters at the city, providence, and country wide level. Separate filters for traffic leaving the country vs internal. etc.
It looks specifically like cURL's exit value and the downloaded page size varied on some requests. It would be interesting to know what the contradictory download size was, and what the curl exit value was.
All that is there when you click on the date, if you scroll rightwards. The exit values in the blocked sites are timeouts (CURLE_OPERATION_TIMEDOUT), and the broken download sizes are 0 bytes.
You should trace the attackers by tracing back. Work with your upstream providers and mailing lists (NANOG) and publicly shame these attackers. Likely, they are spoofing addresses - validate that and make sure you let the network know where the spoofed traffic is sourcing from to follow BCP38 and BCP84, defined by RFCs 2827 and 3704.
Assuming it is direct spoofed traffic and not a reflection, naming and shaming will accomplish nothing. Names of the big ISPs allowing this are not a secret.
Transit providers do not care. They make money on it, some people are using it legitimately, and they just don't care, for the most part. It's a well known problem. It might not hurt to mention it, but they know what they're doing.
First a 2.6bn request/hour DDoS and then making the front page of HN... talk about getting flooded with requests.
Hopefully making the front page will at least get them the attention of Amazon or enough donations to cover the temporarily (absurdly) high operating cost.
A number of the DDoSes from China are involuntary induced by DNS Poisoning. When users query a block dns name, they may receive an IP other than the website they want to visit. It may used to redirect traffic to make DDoS without those 'drones' even know about they're involved. Just like for a week long, whenever I try to access Facebook, I got redirected to some german IP, and receive a TLS CN mismatch error.
Even if it is true, getting an enterprise account, which is still very reasonably priced, gives you dedicated IP addresses to your site, so this shouldn't be a huge concern. Generally when under fire, Cloudflare is also happy to get you up and running and talk finer details on billing for the long term later.
The bigger problem is their stance on Latern mentioned above.
Honestly a bit confused on the downvotes, was just trying to provide additional information for people who aren't familiar with working with CF. Anyone care to enlighten me?
I hope you keep us up to date. I live in the US and I've had accounts suddenly locked out after their passwords were changed even for criticizing some Chinese issue. Several email accounts, among other sites. With the amount of money the Chinese are spending, I think it's not going to be easy in the US. Maybe you're an American and things will be easier for you, but I've learned not to trust the natives. Not even their law enforcement.
Maybe the corporations might be more stable and have more integrity, but you have to realize that China is the largest economy in the world now, and they can flick Amazon out of their market with their pinky and not even blink.
Isn't this title a little sensationalist without specifying who's under attack? I assumed it was a royal we and after clicking the link realized it was just this one site.
It's the title of the blog post that's being linked to. With that title being transplanted to HN, you need to consciously think about what the context is supposed to be, which is why HN lists the source domain.
State sponsored cyber warfare is something that happens at a scale that normal private companies and organizations are not well equipped to deal with. I wonder if there's someone you can alert in the government who can consider coordinating a counterattack or pass a backchannel note to the right people to cut it out.
Calling a simple DDoS cyber warfare is a bit hyperbolic when we know there are actual sophisticated attacks being carried out against far more important targets.
Should the Chinese government have the power to shield their citizens from information and monitor them electronically?
Should a group of people in democratic, Western countries be able to subvert the will of a world superpower with impunity?
Of the two scary worlds, I guess I'd rather choose the latter. But I don't even like having to choose.
(I doubt Amazon like being asked to choose even less, and I would be surprised if they cut you any slack. Sedition is not looked upon favorably, and abetting those perpetrating it is not either.)
I'd suggest that the ability to perform a subversion of the will of a foreign state is a necessary adaptation, preventing a nominally democratic state from sliding towards aristocratic, oligarchic, or plutocratic governance.
The ability to increase freedom in a foreign state is related to the ability to prevent a decrease of freedom in your own.
Inconvenient websites help uphold the duties of the fourth estate when mainstream media outlets have seemingly abandoned--or at least heavily de-prioritized--those duties. It is important for all governments, not just those with sketchy human rights records, to know that even the mightiest machine can be taken offline by a single wooden sandal.
For human rights, civil society and media working in difficult DDoS threat environments check out these guys, they are awesome ---> Equalit.ie https://equalit.ie/ who run a DDoS project called Deflect - which I think is free for people in those categories of people.
https://equalit.ie/portfolio/deflect/
That's not how international internet peering works though; it's entirely a set of private agreements. It's not even a given that the traffic has either Chinese IP source addresses or is actually from China.
These days, DDoSes are not just lots of GET requests, because, as you said, they're fairly easy to mitigate. These days, the most common attacks are various UDP-based attacks, like NTP reflection [1]. You send a spoofed header to a server that speaks over UDP, and they send a huge amount of traffic to the victim.
There are things like amplification attacks (DNS or NTP) where a small amount of attack traffic generates a huge amount of target traffic. Even if the traffic is symmetric (1 byte to target = 1 byte from attacker) the bad guys tend to have botnets / malware-infected systems so they don't pay the cost of the attack side.
If the attackers are just sending GETs you might be able to filter them out, but if they're sending random packets you usually need upstream help to keep them from getting to you to begin with.
There are also things like Slowloris that just use up resources vs using bandwidth, those are harder to identify but easier to deal with on a server-by-server basis.
The requests can vary. Sometimes they are simple GET requests, sometimes they're exploiting a cpu|memory|io-intensive process in the application and sometimes they can be reflected DNS attacks.
The problem is separating the legitimate requests from the bad requests. Sure I can see there are 5 Million requests to the main page of the app. But which are from poeple legitimately trying to use the application and which are from the botnet? You can't just do it by IP without running the chance you're going to cause problems for legitimate users.
There are ways to mitigate this but it requires being able to analyse current traffic and past traffic quickly, and at scale while having the expertise to set up firewalls and other filtering correctly
It would be interesting to see which IP space the bulk of the traffic is coming from. Seems like it would be trivial for the Chinese government to spoof traffic from any IP within China...
Honestly most companies I've worked for have blocked the entire Chinese IP block. Obviously Chinese hackers can use proxies but it honestly does cut out a TON of problems. The downside of course is you will never get a Chinese customer/viewer. In preparing to respond to this post I googled "China IP block" and pretty much every result was about how to configure .htaccess or iptables to block the entire country.
The more I watch the security theater, the more this all looks to be set-up. I really don't fear the chinese, as the NSA is far more competent and organized, and is much more dangerous to the world at large, not just the citizens of the United States.
anyone remember that story a few days ago where China said "Hey, we want NSA-level backdoors in hardware, too!"? can someone help me with a link?
when stuff like this happens it would be nice if we could simply refuse to route all traffic originating in the offending countries until the attack is over.
It seems a little hypocritical to me that AWS will create a service for every technology known to man, but will not create a service to help companies who rely on their infrastructure to deal with DDoS.
I would appreciate it if people who down voted my comment explained to me if it's because i'm wrong or because they don't understand what i am trying to say or just for the heck of it :).
The issue is your comment doesn't really add any value. Anyone can paste a random link saying "you should use this" but it takes effort to explain why it would be useful to them. HN comments are about fostering discussion, so say something to be discussed :-)
Thank you for your input, but 1. this is not a random link, 2. the explanation is all inside the link i posted, so why to be redundant? anyways, if i didn't care for a discussion i wouldn't post it at the first place and i wouldn't later ask why was i downvoted, thank you again for your opinion lucaspiller.
You don't really understand the distributed part of DDoS. I've had services taken down my attacks and when doing a post mortem we could see the increase in traffic, but when accounting for frequency, our office was still the main user.
Yeah, but my understanding is that it helps a lot. CloudFlare masks the actual IP address of the web site, and distributes the load through the CloudFlare platform.
It's still possible to overload that capacity, but it's a lot more than what any standalone web server can take. Furthermore, the web server itself will stay online, as it isn't actually getting hit by the flood of requests.
Sure, the target will not be affected, but CoralCDN network will, and it will not be able to handle the load of the attack. They don't have enough resources.
Source: worked for AWS, was oncall during similar attacks. Nasty things with those they tend to start around 6-7PM (guess when does working day start in China).