And if everyone advertised /24s instead of larger aggregate networks, routers would have to store about 12 million routes. Better yet, why not just advertise a /32 for every IP address?
Most routers would crash and the internet would stop long before we ever got to 12 million routes.
Some /24 are more critical than other. There is a price to security and for sure advertising only /24 is probably not viable.
The article mentionned that some networks within the /19, in this incident, were critical. So yes, I believe that every businesses should advertise /24 for their highly critical infra. They can advertise /19, /20 and so on for the less important networks. No need to use /24 for everything.
It would have happened, and not even doing it as a mitigation would've worked, as the offending ISP would've re-advertised the same routes as google shortly after google changed it.
TransTelecom (AS 20485) in Russia, China Telecom (AS 4809) in China and MainOne (AS 37282), a small ISP in Nigeria.
I don't know what qualifies as big ISP but I am certain MainOne with a 14,000 kilometre submarine fibre optic cable may not be one to be classified as a small one.
From wikipedia:
The Main One Cable is a submarine communications cable stretching from Portugal to South Africa with landings along the route in various west African countries.
This sort of false equivalency is unhelpful. It requires a great lack of judgment to believe the U.S. is comparable to the regime overseeing Xianjing on domestic internet surveillance.
It seems naive to believe that the US isn't comparable to China when it comes to surveillance considering the actions taken and information that has come out, particulary since 9/11
edit: Why the downvotes with no response despite my comment being entirely relevant? If anyone would like to explain/counter I would be curious to hear their reasoning
I assume the "US is comparable to China" assertion rubbed a few people the wrong way. This link collection shows the gravity of the situation in the US, and that US is drifting towards totalitarianism, without having reached that stage by a long shot. The surveillance scope is quantitatively different in the two countries, given the expressed totalitarian nature of the Chinese regime.
Something I ask myself all the time: how will I be able to tell when we've reached actual totalitarianism in the West? It won't be Chinese style totalitarianism, and I'm pretty sure they won't helpfully dress up in jackboots and announce things have reached peak totalitarianism. Would we notice at all?
I think the US is very big on internet surveilance and is probably beyond what China does. You don't have to be totalitarian to do surveilance, there's plenty of other reasons to do it (though once you have the data the temptation grows to use it for totalitarian uses)
This is true, and Im not in support of wholesale surveillance with out reasonable cause and the associated court documents. That said, I don't think I would want to go to Russia or China and say, google "How to overthrow the government". I would expect someone would want to speak with me pretty quickly after I did that...
>That said, I don't think I would want to go to Russia or China and say, google "How to overthrow the government". I would expect someone would want to speak with me pretty quickly after I did that...
That doesn't mean the US isn't performing the same level of surveillance/tracking, just that they aren't publicly picking up dissenters. In reality, if they were really worried, they'd hit you with a gag order and charge you in the FISA courts so you can't go public
I suspect that in every relatively developed country nowadays executing systematic requests of the kind you mentioned would get you some kind of personal attention.
That's why they invented it, right? To make sure the political status quo does not change. That includes political systems in the US, UK, China and Russia.
These routes can spill over many different paths. There is no central authority that would be able to drop them from the whole internet. Even if one Tier 1 ISP dropped them then their routes would just flow through another one. And good luck getting all of them to agree on that unless there's some massive, persistent abuse.
It wasn’t always that way. There were people (especially in operations) that used it as a guide for certain decisions. I was disappointed when the slogan was retired.
I don't think Google has control over peering agreements. The way I understand it, ISPs/backbone providers would have to decide to drop peers from their BGP configs. Outside of calling up ISPs, I'm not sure what Google should have done differently, if anything.
edit/minor correction: I guess they'd have control over peering agreements for initial hops for Google Fiber users, but that wouldn't have resolved the issue in this case, either, unless they stopped peering with other huge providers like NTT. Even then, it'd only help their subscribers.
It wasn't even CT that was the cause this time, they just propagated what they got from another ISP. There could certainly have filters in place but that only really works for stub networks.
China has also made it their goal to bifurcate the internet in order to prop up their authoritarian state. I don't see why the rest of the world should play along with that. It's frustrating in the extreme to see big Western companies repeatedly bow to the Communist party's wishes only to get all of their IP stolen by a domestic clone with the backing of the government.
> China has also made it their goal to bifurcate the internet in order to prop up their authoritarian state. I don't see why the rest of the world should play along with that.
China doesn't need anyone to play along to prevent their population from accessing the global internet. Right now, at least the rest of the world can access Chinese websites. A blanket ban on peering with Chinese ISPs would really bifurcate the internet. I'd count that as "playing along".
To answer your question, most large companies have been subject to cyber espionage by the chinese military for several years. This accounts for the speed at which China built its manufacturing base in strategically important industries and its ability to rapidly copy products and the required tooling for their manufacture.
The Mandiant report I linked to above focused on tracking the activities of a single but prolific APT group responsible for hacks in 150 major companies worldwide for 7 years since 2006, and that they believe is actually a team of cyber espionage professionals from within the Chinese military.
Also just for fun: Anyone here drive a Range Rover Evoke? Heres China's version:
BGP is a system build based on two things primarily in this order
- shortest prefix to reach an IP ( 99.99.99.0/24 takes precedence over 99.99.0.0/16 )
- shortest number of "hops" measured by how many AS paths it takes to get somewhere ( if you have multiple ways to get to 99.99.99.0/24, the way with the last number of autonomous systems "AS" in it wins )
So the result is that you have to somewhat trust who you are peering with, and who they are peering with, etc. to the other end of the traffic.
From someone who does full understand BGP, I have a question
Somewhere, a BGP route was misconfigured to send data somewhere else.
What would happen if a BGP route was terminating at China, and the bad actor who made it happen, decided that they are not going to fix it and just leave it.
How would the rest of the BGP network deal with it?
While investigating the alarms, a network engineer at each major network will decide to stop taking routes from the Chinese network making the advertisements, and everything will sort itself out... as far as that network is concerned.
Just to clarify, it was a small Nigerian ISP that caused this, CT just propagated it to others.
It's likely that this Nigerian ISP was setting up peering with Google and misconfigured their route policy. If you do this incorrectly you can advertise prefixes you get from one peer to other peers as if you own it. (essentially you replace the full AS path (prefix metadata in BGP) with your own AS number, it makes it looks like you originated the prefix to others)
If neither the Nigerian ISP or CT refused to do anything then everyone that is their peer or customer would need to manually filter this "bad" prefix" to stop it. Customers and ISPs that only use CT would still be affected.
After glancing at the headlines, I actually did a double take. I was like "whaa? Taboola creeping into my HN"
Given the audience, it wasn't the PR managers finest hour this week. What happens in BGP land is discussed publically on the NÀNOG mailing list, and they are the friendliest crowd ever.
I've read them go out of their way to solve issues that just needed Goodwill to do in a couple of minutes, keeping the back channels open even between companies whose rivalry would dictate that they'd talk to each other only via their law firms.
This could explain why there's so little information coming out from Google as to what caused the outage. China Telekom is state-owned and thus pointing a finger at them could stir the relations with the Chinese government, regardless whether this was a bug or an intended act.
From the article, it wasn't China Telecom who made the initial misconfiguration - it was a small ISP in Nigeria. China Telecom just accepted their BGP update and rebroadcast it.
I get that they need a cool sounding title but they could have just said "BGP hijacking" and saved a click for everyone who knows what that is. We've seen this before and we'll see this again.
Does anyone know if there was any relation of this attack and the Facebook outage that happened yesterday as well? Seems weird that both FB and Google have trouble on the same day.
Ironic that the company reporting this has security problems themselves. I was trying to download their cloud research pdf via google and spammers have gotten hold of them with hundreds of thousands of fake links: https://www.google.com/search?q=site%3Athousandeyes.com+file...
"Strange snafu misroutes domestic US Internet traffic through China Telecom"
> China Telecom, the large international communications carrier with close ties to the Chinese government, misdirected big chunks of Internet traffic through a roundabout path that threatened the security and integrity of data passing between various providers’ backbones for two and a half years, a security expert said Monday. It remained unclear if the highly circuitous paths were intentional hijackings of the Internet’s Border Gateway Protocol or were caused by accidental mishandling.
No, it’s actually quite accurate. BGP hijacks are a vulnerability in the very core fabric of the internet. Just because we’ve seen a lot of BGP hijacks recently, does not in any way, shape, or form downgrade their severity.
What's particularly maddening is BGP hijacks have been happening for well over a decade. It's one of the weakest parts of the Internet infrastructure and I'm convinced something more disastrous is going to happen any time now, either by accident or malice.
If it can be done once, I guess it can be done multiple times. I don't know nearly enough about this to understand how BGP hijacks work or why they are possible. Can anyone point me to a simple explanation for a layman?
BGP is a gossip protocol. BGP roughly works like this:
As BGP nodes come online, they establish connections with "nearby" existing BGP nodes, saying what version and how frequently they'll check in, and what their AS number is (a unique identifier)
Once communications have been established, then they can start to report any network routes they know about.
"I'm AS 123456, and I am the originator for 3.0.0.0/0" (i.e. they're responsible for it), "Also, I can reach 4.0.0.0/8 with a cost of 6" (A fair number of hops away on the network).
Any neighbouring BGP peers update their routing table:
"AS 123456 is responsible for 3.0.0.0/8, and I can reach it with a cost of 1, and I can also reach 4.0.0.0/8 with a cost of 7 via AS 123456". If there's a cheaper route to a network address, no changes will happen.
Routing changes can propagate quite quickly across the internet. The routing protocol is nice and lightweight, and updates are happening with reasonable frequency, as network connections come and go.
The idea is that should damage occur to the network fabric, the network will automatically update and route around it, without need for any intervention.
It's entirely built on trust, though. You have to trust that AS 123456 is indeed actually responsible for 3.0.0.0/8.
If you get two parties indicating responsibility for a network range, it's possible to end up with routing loops etc, as things get in to a mess.
What is legitimate behaviour, though, is for, say, AS 123456 to be the originator for 3.0.0.0/8, and another AS be the originator for 3.0.1.0/24 (i.e just 254 addresses under that space). That's not an unusual situation, and it won't cause routing issues, because more specific is taken as a priority over less specific, rough analogy: "In general mail for General Electric, should be sent here, but if it's for the electronics product division, send it straight to them"
There have been different attempts to put filtering in place, provide authentication "Yes, AS 123456 is allowed to be responsible for 3.0.0.0/0" and the like, but nothing has really taken off.
Sounds like a bad actor could easily attract as much traffic as they wanted by claiming to have the shortest route to all sorts of places.
With different data snooping/data protection policies in various countries, it would also be useful if you could order your traffic to avoid certain countries.
Absolutely. ISPs, backbone providers etc can all manually filter out updates from specific ASs, and in theory habitual bad actors will have that happen to them. Filters will get put in place even as a temporary measure during incidents sometimes, depending on how responsive / capable the NOC is.
Why isn’t there a “premium” commercial Internet, similar to DOD NIPRnet, for b2b stuff and for communications to vetted infrastructure? A random business on the Internet (eyeball, not servers) cares a lot more about routes to AWS and Google than to a bunch of other eyeballs, particularly in foreign countries.
You would still want full connectivity, but when things went wrong I could lose connectivity to a Nigerian ISP without critical business risk, but losing access to Google sucks. You could largely accomplish this by prefixing the hell out of everything except the “special” connection, and ensuring DDoS and other security filtering could drop the entire normal network if needed without affecting internal or special.
Only really at colos. What scares me is businesses with key business partners, satellite offices, etc who just rely on the Internet working normally to communicate between them. If someone like L3 sold a premium network it would be worthwhile.
Yeah, you could accomplish this for hq to satellite office over private circuits, MPLS, etc. That wouldn’t be enabled by default. If a carrier made a point of on-net connections being protected by default, it would probably be easier for more businesses to adopt.
By default, Cisco and Juniper routers (and likely others too, those are just the ones I can speak of) will advertise all bgp learned routes to all bgp peers unless you specify a route map/export policy.
Leaking a route would imply you’re advertising the route to a peer that you’ve otherwise not intended to, either due to a misconfiguration or by not configuring things at all.
most likely these routers in Nigera were configured to send traffic destined to Google through China first ( a more specific /24 out of the Google prefix was configured -> China Telecom ASN ), through a route-map mistake this was advertisied out to the public internet instead of staying within the Nigerian ISP ASN.
Despite the routes terminating at China's firewall I don't think it's a good guess to say this was necessarily a Chinese attack. It seems like the interference of the Chinese firewall was simply a side-effect of the new path the network traffic was going. If you were perpetrating this attack it seems more advantageous to re-route the traffic and allow it continue working rather than discoverably interfering with the traffic. Plus if you were going to try to block the route, there's probably a better way of doing it.
China's firewall has a long history of accidental enforcement of censorship outside its jurisdiction, leaked DNS and BGP poison are the most common problem.
Sooo... paint me stupid but I find the title completely misleading and another, very, VERY annoying thing caught my eye - after reading the text, I caught myself repeating the word "thousandeyes" and it annoyed me to no end. Then I re-read the text and I found that you can kick that word out of every 2nd paragraph and the text would still have merit.
Is this yet another marketing ploy where you post something with purposely-misleading title in order to attract traffic? I don't like the fact that word "thousandeyes" got stuck in my head, nor do I like the fact that I got clickbaited. This one is going to my list of "just like every other site since 2015, don't click".
It's called content marketing. Some do it better than others but it is always trying to attract attention to a company or service by blogging about related content.
Repeatedly including your company name, especially if it’s something unique like “thousandeyes” is essentially pointless when it comes to SEO. They own the domain thousandeyes.com. Google is going to recognize the intent for searches for thousandeyes are going to be about that company.
Other keywords, yes, those are important for SEO. But repeating your unique company name, not so much.
I’ve been around the world to see some amazing/strange investments by China. Whole office buildings in Malaysia, South Africa, Costa Rica, etc that sit empty because a state-backed investor is speculating that this area will someday be more valuable.
No where was this more obvious than Malaysia which is their neighbor... but still crazy to see giant brand new office parks that look abandoned.
To your point, it doesn’t at all surprise me about Nigeria because I saw it in ZA.
China seems to have a world wide plan for the next century.
If it's not a government project, it could also be capital flight. I think it's one of the main drivers of real estate price expansion around the world. There's concern in China that things could collapse in the next 10 to 20 years. Many people seem to think that losing most of your real estate investment is better than losing ALL of your money.
