A Chinese here, I'm deeply ashamed by this malicious action at a national level. though DNS poisoning wasn't new for Chinese netizens but it was always poisoning A records to varies foreign military sites like DoD or some Korea sites (which also makes no sense at all). At scale of this big, those shameless bastards running state censorship machines has been outrageously attacking on random innocent site owner, this is far beyond unacceptable.
And starting from like a month ago, a lot of people on the Chinese internet started to decorate the GFW, and concludes with "for your own good" shit. I can see from those fragments of thoughts, China is not going to stop the GFW, and as Moore's Law advancing, it's going to be even worse.
Sorry for swearing, the feeling of "there's nothing anyone can do to stop this BS" just leads to greater despair.
I have seriously contemplated a special request handler for Chinese traffic that is being directed to the wrong host on our servers. Something along the lines of "Chinese government censors broke your internet" or the like.
I figure it either helps inform the citizens as to why this link isn't working or the message itself causes the GFW to ban our IP as well.
Many of us outside GFW/Cn are aware of the "decoration" (we call it Astroturfing in the US) - it's so transparent that it's often joked about on Reddit and similar sites.
>One thing I learned is that Apache can have problems figuring out which virtual host to use in some cases:
>>If no ServerName is specified, then the server attempts to deduce the hostname by performing a reverse lookup on the IP address.
ServerName is the name Apache reports in Error messages and using for redirects. It has nothing to do with the Host-Header sent by the client. If it did, a reverse lookup definitely wouldn't be of any help either.
His remedy was still ok though - just for the wrong reasons.
I'm the author of the post linked above: thanks for the clarification about ServerName. I spent a lot of time reading that documentation and got pretty confused about how it was actually being used.
Complementing it, when there is no Host HTTP header, apache displays the default virtual server for the port used. Unless you've set it, it's the first entry on your config file.
It's not a nitpick. It's the difference between making stuff up and reading the documentation. It does matter when reason about how to configure your server.
Oh man. That would be pretty cool. Somebody like CloudFlare could do that since they proxy for lots of smaller sites. Then again, HTTPS will add even more complexity.
We were the lucky targets of China's DNS poisoning on Monday the 19th. Any requests from clients in China to Edgecast's CDN was instead thrown at our public IP. After deducing what was going on, we ended up blacklisting large (class A in some cases) APNIC address blocks assigned to China.
Interestingly, it wasn't our webservers that were overwhelmed but instead the Cisco firewall that sits in front of them. 25K concurrent connections made it decidedly unhappy.
Same here. We received well over 150mbit/s, also on Monday the 19th, also from China. We also saw occassional spikes starting on January 9th. We saw the same BitTorrent /announce traffic, and lots of other random traffic.
We ended up mitigating it by moving our IP address on that host, and blocking all input on the old address.
My company's web server saw a similar unusual surge in traffic with host headers and URLs of Chinese sites for an 8 hour window a few months ago. All the IPs were from a single Chinese ISP, though the volume of the traffic was a lot lower than what OP observed. Seems like these events aren't uncommon.
I wrote a blog post about a DDoS on a website by the Chinese at http://dvps.me/ddos-attack-by-torrent. Falsely I assumed that it was torrent tracker injection, while in fact this looks like a spike in the affect of DNS poisoning in China.
This is why I really like having Varnish in front of anything Internet-facing: for something like returning a 400 for an un-approved Host header, you're going to be effectively bottlenecked by the network interface.
A Chinese here. I would suggest, if possible, use 301 to redirect the malicious requests(which are without recognized hostnames in HTTP header) to <www.gov.cn>, which is the government's face.
They (the GFW) intend to have every server owner ban China IPs so that they could claim "you guys are also banning us, what position are you in to blame us for GFW?"
Maybe China are trying a new approach to censorship. Instead of blocking things themselves, perhaps they're trying to get sysadmins to block all of China. I imagine it would save them a lot of work.
I noticed that as well. However, they seem to be friends / colleagues / associates. A web search for "Craig Hockenberry daringfireball" shows that Craig Hockenberry is (or has been) a contributor to John Gruber's show and John Gruber has recommended one of Craig's apps [1].
Thanks a lot for the post.
What suprised me the most was the fact that Google is serving 40K (now 47K in live statistic) requests per second.
That was way below what I thought.
Each Apache server without real optimization running on a typical physical machine would be able to serve 40-50 requests/second; so they only need a thousand of servers to do 50K requests/second.
Or am I missing something?
Your missing factor is "the fact that Google is serving 40K (now 47K in live statistic) requests per second" is wrong.
"The number of requests peaked out at 52 Mbps. Let’s put that number in perspective". Perspective is 52 Mbps isn't a small number, but not massive either.
We started seeing this around the 9th I believe. We were not the only people as well, based on a somewhat late ISC post[1].
We thought it was a new form of intelligent blackholing. Instead of sending traffic to IPs that could easily be blacklisted by tools to get around the firewall, the Great Firewall would start sending them to random "good" IPs for the same result. Others seem to think the same thing[2].
I had one of my clients have the same thing happen to them 3 nights in a row about 10 days ago. All of the sudden they got dumped a ton of traffic out of no where. Torrent tracker updates and what seemed like legitimate traffic routed to the wrong ip. Then after about an hour most of it stopped. During the attack and after we changed some firewall settings and clamped down our request per second and connection per ip limits on the web servers. One big change was having the web servers respond 444 to any host request that was not configured (i.e. default). So if someone came to the ip looking for say google.com they would get a 444 response as it is obviously misrouted traffic.
Then the next day the same thing happened but much more traffic. The 444 change helped some but there was just too much traffic for the web servers to handle quickly so they bogged down to a crawl. Luckily we were able to figure out through severfault and some other searches that it appears to be DNS poisoning coming from China. We ended up banning the entire country.
The third day the same thing happened but because we had blocked the traffic from China at our outside firewall the servers were unaffected. Since then we have seen some flashes of traffic being blocked but not nearly as much as before.
In all cases the increased volume of traffic only lasted about an hour. The only thing I can surmise about the length of time is DNS records only being cached for about an hour. So after that time the poisoned cached DNS records were replaced by the real resolving ip address. In the case of the article's author their servers suffered a much longer attack than we did. I am not sure if it is the Chinese government doing it or an attack for hire scheme using holes in many Chinese based DNS servers.
TLDR; Remove default server settings from your web servers and have the default server block respond 444 or 404. This may help stave off the attack until your incoming traffic takes up all available resources of the web servers. Of course you could always block all traffic based in China like we did.
Using DHT is also a particularly effective way of getting massive amounts of UDP traffic directed at you, which does not stop even after stopping the torrents. It subsides eventually, but can be quite irritating.
One possibility is to block requests coming with a different 'Host:' header than your website. Or just block the URLs like described in dvps.me/ddos-attack-by-torrent
Or put a nginx in front of the Apache, disable keepalives in the Apache server (letting nginx deal with keepalive from the clients), and reduce MaxClients so even if they're all active the server does not go into swap.
(I've done exactly that for a friend, it made a huge difference in how much traffic his box can handle).
This is something I was wondering, too. Using nginx, I can easily top 50,000 req/s (45Mbit/s) on smaller, static/cached content, using 4 cores and < 1GB RAM.
The bandwidth chart I get from my colo (where I don't own the router that measures my bandwidth usage) works the way I described. The article mentions a single server, and nothing about owning the router also. If they did, that's probably where they'd null-route the traffic they don't want.
It seemed backwards to me too, when I first saw it.
That's the only reason I can imagine they'd respond with anything else besides 127.0.0.1 or some internal web server with warnings to obey their overlords.
I live in China.
We have a department named 工信部, it blocks a site by polluting the DNS and point the domain name to random foreign IP.
I think this explains what happened here.
And starting from like a month ago, a lot of people on the Chinese internet started to decorate the GFW, and concludes with "for your own good" shit. I can see from those fragments of thoughts, China is not going to stop the GFW, and as Moore's Law advancing, it's going to be even worse.
Sorry for swearing, the feeling of "there's nothing anyone can do to stop this BS" just leads to greater despair.