It seems like many aren't quite getting what's going on here...here's a brief overview:
Shodan.io fancies itself as a search engine where you can search for IOT things (webcams, refrigerators, etc) that are on the internet.
They have no technical issues doing this in the IPV4 space, because it's easy enough to scan every single address in the space.
This isn't practical for IPV6. So, it seems they wanted a way to identify every IOT device in the IPV6 space without having to scan all of it.
At least one approach they found was to join the ntp.org pool[1], and effectively donate server time. Since pool.ntp.org is the default NTP server listed for many linux distros (and thus, probably IOT devices), they now are getting live connections from the exact devices they want to index on their search engine.
Once you connect, they scan you back on 100 ports on so (ports unrelated to NTP) to see if you are a webcam, router, or whatever else they want to put in their index.
Pretty shady. Kind of like volunteering for a charity so that you can raid their internal mailing list for spam purposes.
Sort of, but in the global order of shadiness, this ranks pretty far below what your typical multinational corporation does to track an individual. It's pretty ingenious actually.
Well then I guess people should stop relying on their IP address being one of many in a vast space for security and implement proper authentication instead.
It's not like your service has to expose all its secrets to an unauthenticated client sending you a GET / HTTP/1.1.
That's not the irk. Rude behavior isn't suddenly polite just because I have great security. The spirit of donating server time to the ntp.org pool is not to use the data for your own self promotion.
The real-life equivalent is a pizza delivery guy who, after delivering the pizza, goes around your house, checks which doors and windows are locked, unlocked or open, what brand the locks and the burglar alarm are and then proceeds to sell this data.
I hope the NTP pool terms of service will disallow such behaviour in the future, otherwise I'm removing my server from the pool.
just spin up a separate network namespace and macvlan device to run your ntpd in. it'll get a separate v6 address via router advertisement and ntpd will be the only thing listening on that address.
just wrap ntpd in firejail[1].
ipv6-per-networked-process pretty much brings an end to portscanning.
They are taking the data from the scans and building a nice easy to use searchable signature database for script kiddie hackers. At the very least, it's impolite.
If they know what software you are running, the minute there is a new vulnerability revealed, they can break into your system. It gives you less time to patch your stuff. You think this is the stuff of fantasy? Read https://www.drupal.org/PSA-2014-003
The lesson here, I think, is that security intuitions developed in an IPv4 world with NAT everywhere may need re-examining in an IPv6 world.
If your device has a publicly routable IP address, it should probably be behind a firewall. If it's not behind a firewall, it's going to get scanned. Relying on security through address space size is stupid.
> What was most puzzling was the fact that the devices that were targeted had randomized IPv6 addresses and were not published in DNS or any public record. For all intents and purposes they were hidden safely within my lab network.
That is just wrong, and it's wrong in a very glaring, obvious way: The devices were not "hidden safely within [his] lab network", because he was not using NAT. They were connected to the internet with publicly routable IP addresses, which they were using to communicate with other hosts on the internet.
Edit: That being said, it's a great writeup, and some good technical work was done to figure out what was going on, and I enjoyed reading it. But the starting premise seemed to be "how could this be happening, my devices are hidden!" and I feel like it should have been "oh, it makes sense that someone would do this since my devices are no longer hidden".
You're quite right; people significantly overestimate the safety of NAT in my experience.
But what I was getting at is that however illusory the safety of NAT is, the author seemed to be instinctively assuming that his IPv6 devices had it, because we've spent 20 years training ourselves to assume that everything is behind NAT. It's a new world.
More like an unlisted phone number. The addresses weren't published in any deliberate way, but they were still visible on the "caller ID" when they connected to another system.
I'm not really seeing a vulnerability here. The entire IPv4 internet is scanned probably hundreds of times a day. Was anyone really counting on IPv6 addresses being longer to add security?
You're absolutely right, I agree. I think this reaction ("Oh no! My 'private' IPs are known and being scanned!") is probably one that comes from having spent a lot of time with a personal network behind the NAT of an IPv4 router, which necessarily provides a wall to internal network scans. Using IPv6, NAT is no longer necessary, and therefore it can surprise people that aren't expecting that behavior. I thoroughly enjoyed the write-up though!
A machine with a random address that makes no (direct) contact with the outside world will be a lot harder to find in a 64-bit address space then a 32. As soon as it talks to the outside world though (by talking to public NTP servers in this case) that difference is rendered moot.
Unless it's using Privacy Extensions in which case it will change it's outbound IP in a matter of hours and once again be lost in the huge address space.
Unfortunately privacy extensions are not enabled by default on most linux distributions and server operating systems.
Could be wrong, but part getting through a firewall is establishing open ports and mapping the machines behind the firewall and since you have a verified machine name this would save bad actors a significant amount of time being that they can make the assumption that the computer exists rather than timeouts to nothing. While this goes in on the hiding an IP (disable ping) is not proper security argument it is valid that you would want to stop disclosure on such a public level.
I have worried about it the past. The more a bad actor has to thrash around the better chance an IDS will trigger. As well some older (read cheaper) firewalls have been known to allow through ACK packets through. This attack is not (IMHO) intended against properly secured sites, but rather than small business with consumer grade routers or something equivalent.
Very interesting post, I've suggested to Brad that he consider probes with a decreasing ttl in the packet to see if the harvesting is happening at an interstitial node or directly on the ntp server. If you had access to a compromised Juniper router it would be straight forward to add a rule to mirror packets which were headed to the NTP pool addresses.
Oddly after being an unwitting participant in one of the NTP amplification attacks I set up my own stratum 0 NTP server based on the beaglebone black, the adafruit GPS module, and the PPS time keeper code. So all of my machines only talk internally for time, although initial installs still use what ever the distro packed into them. I brought up Debian on a Dragonboard 410c and it sets the time via an NTP call during the initial startup process. (or fails to set the time as NTP is blocked egress/ingress from the firewall)
It seems pretty clear it's the NTP servers themselves. All the servers are in the same small subnet, and 2 of them even have rDNS hostnames indicating they're owned by shodan.io.
I have been interested in using the dragonboard with linux as a small controller for some embedded things, how is that working for you?
On subject, I have always though having a master internal ntp under strict controls should be used, so good on you for that, I dont see ntp payed attention to far too often in the enterprise.
Support for the DragonBoard is much more limited than other platforms. Android seems to work best although with a stupid "giant phone" sort of design language which is silly. If they had the level of documentation that the BeagleBone (or even the RasPi has) they might be a contender but my estimate is that the platform won't go anywhere.
I've recently acquired a Nvidia Jetson TK1 development board. It is $200 USD, and basically a Linux desktop system based on an ARM processor platform, running Ubuntu 14.04. You can easily hook up mass storage and other stuff. Just 2GB of RAM though.
FWIW, I've had really good luck with running Endrun CDMA stratum 1 timeservers. You can pick them up off of ebay for a few hundred. In theory it's more stable than GPS, especially if you get one with OCXO or rubidium option.
If someone is detecting which IPv6 addresses are in use in your range from NTP requests, that implies they are all talking to the external NTP servers directly.
Surely if all your internal hosts are talking directly to the external NTP servers you are doing it wrong? My gateway box sets itself by pool.ntp.org and the internal ones set themselves by it. I thought that was how you should do things (even if it isn't a rule, it is only polite to try not overuse a public resource).
> These addresses are 128 bits in length
Only 64 are relevant here: once you make outgoing connections from any address a scanner knows there is at least one active host in that /64 and there may be more. Though of course a 64 bit address space is still impractical to scan.
> Surely if all your internal hosts are talking directly to the external NTP servers you are doing it wrong? My gateway box sets itself by pool.ntp.org and the internal ones set themselves by it. I thought that was how you should do things (even if it isn't a rule, it is only polite to try not overuse a public resource).
Depends on how many internal hosts you have, and ease of configuring them (there is a DHCP option for ntp server, but not everything will use it), and available hardware to setup ntp servers: ideally you would have each client syncing from three servers so the clients drop a broken server.
> ideally you would have each client syncing from three servers
For a small network (a home or small office network for instance) you likely have one incoming router anyway, so if that dies clock setting is the least of your worries until it comes back up, meaning setting the internal hosts by that one clock is as fine as anything else.
For a larger network you have redundant everything, including edge servers that can act as your multiple NTP sources for internal hosts rather than every internal host poking the public pool.
> For a small network (a home or small office network for instance) you likely have one incoming router anyway, so if that dies clock setting is the least of your worries until it comes back up, meaning setting the internal hosts by that one clock is as fine as anything else.
I'm not so much worried about the router dieing and losing sync with the outer world; but about the router's clock going wonky: I've seen computers where something got initialized wrong and they were 30 minutes slow after 10 hours (reboot helped in that case, but sometimes it's the oscillator is just too far off)
You are surely doing something wrong with ntp if you only have one time source configured. The only configuration that I can think of that would be worse is to have a setup with two time sources.
While it is pretty freaky, I think the point of Shodan scanning IPV6 devices that use the default pool is to detect linux-based IoTs that use the default NTP settings, which is kind of the reason why Shodan exists.
It is unlikely that Shodan is going to hack your Raspberry Pi specifically because then people will go after them. But hackers of many hat colors will use the freely available information it gathers for their own purposes, so act with this in mind.
For folks only looking for a cautionary tale, who don't worry about larger security trends and only want to keep their own machine safe, that's a fine takeaway.
For people who do care about network security, it is a very interesting post for a few reasons.
If someone is harvesting IPs from NTP queries sent to Debian infrastructure for intelligence gathering, that in itself is a big deal.
It is also a somewhat novel technique - if an attacker is placed such that they can see the first NTP queries sent by a new install, they are well placed to target that device before it is fully hardened/patched, because they likely see some of the first packets sent by that install. It is really rather clever.
It also points to the sort of correlation we need to be getting better at - orchestration isn't just for devops weenies anymore, and this sort of thing is only going to become more sophisticated.
I've been noodling about with making a tool for making this sort of analysis easier, but there are a number of problems to fix, not least of which is the sheer volume of data generated watching the wire, even on my home network (which is a bit absurd for your average home, but tiny compared to a business of any size).
Debian does use [0-3].debian.pool.ntp.org as the default NTP servers, however. Debian (and Ubuntu) also start up the NTP daemon as soon as the client is installed -- even before one has a chance to change those default servers.
The more interesting thing it's also saying is that some of the *.pool.ntp.org servers are bad apples, and results in your NTP request, or at least its source IP address being collected and results in a scan back at you.
Is there any expectation that a public address (even if it's buried in a huge IPv6 address space) is in any way secret? Once a packet leaves your network, you have to assume that the world knows your IP address. If you want to keep it secret, don't let it leave your network.
Ofcourse. But there is an expectation(not a guarantee, though) that ntp.org provides a sincere and non-shady service - which is why this blog posts, among other things, have made ntp.org started to weed out such shady servers.
Nice evidence for why site operators should be running their own NTP servers. If you run 1 server set per site which doesn't even need a strong local clock, you mitigate this information leakage.
I'd think this is something that most network designers/engineers would get. That said, I'd be a rich man if I had a nickel for every time I saw NTP misconfigured.
That doesn't really help, if you make an NTP request to a remote server, the server and all elements of the network path will know the source IP you used. You can't avoid that without also avoiding getting the response from the request.
Using a broadcast time source (CDMA/GSM, GPS/Galileo/GLONASS, WWV/WWVH/international equivilents?, ATSC broadcasts, etc), or choosing a trusted time server would be alternatives; you could also use a telephone time service and leak your phone number instead of your ip address.
I am not optimistic about this project, which seems to involve people who don't have much of a track record in secure programming trying to refactor a giant blob of C code.
Improving the security of the Internet's go-to time transfer protocol is certainly useful. However, switching to a different protocol would not mitigate this port scanning attack. According to this researcher, just sending an empty packet to an NTP server's port is sufficient to result in a port scan back.
Unless I missed something in the article, I'm not sure this is odd or malicious behavior. The NTP servers know your IPs because you sent packets to them from those IPs. So you basically told them what your big random 128-bit IPv6 address is.
And the fact that they sent packets back to you (after you sent them packets) is not surprising either. However, if you can show that a full-blown port scan occurred after you sent them packets, then that would be odd. I did not see evidence of that in the article... did I miss that?
"It takes less than five seconds for your address to be harvested and scanned. The entire scan takes less than one second and scans over 100 common TCP and UDP ports. "
OK. I'd like to see a full packet capture. And technically, the address was not 'harvested'. He gave it to them when he sent them packets. The story strikes me as very dramatic and over the top because of statements such as that.
Look in to what shodan is, its a service dedicated to portscanning the internet. With ipv4 you can easily scan every ip, with ipv6 you need some mechanism to discover addresses. This very much looks like they joined the ntp pool soley to harvest and portscan addresses.
Technically, you're trusting that collectively they are giving you the correct time, so you're not putting full trust in every single server. A rogue server with incorrect time will be ignored.
That doesn't imply I trust them in any other way though.
And even if I do (I might trust them more than a more random selection of individuals, taking them doing something for the community as a point in favour of their character) that wouldn't stretch to trusting they were not unknowingly compromised by a third party.
When you visit a web site, you trust that it will give you the data you ask, and hopefully not a lot more. You hope that it doesn't give you viruses. Sites that do are not trusted. Sites that don't are... not untrusted. :(
You also hope that visiting a web site doesn't lead to the web site running a scan on your IP address. Or doing any number of other nefarious things.
This kind of trust is at least most peoples assumptions. If it weren't, there would be no outrage or even surprise at the nefarious behaviors which abuse our trust.
No one objects to IRC servers scanning when you connect. This isn't a trust issue. If you sent packets to a free service, you have no rigts over whatever trace you left behind.
This isn't talking about incoming request for the NTP service - it is fuller scans to valid IPv6 addresses that have been harvested because hosts using those addresses talked to an NTP service that logged them.
emumeration of ipv6 hosts is likely to be the goal here, not a scan of NTP ports.
The normal ipv4 methods of "scan the entire address space" fail in an ipv6 world, so shodan needs some way of mass havresting ips for them to gather scan results for...
Shodan.io fancies itself as a search engine where you can search for IOT things (webcams, refrigerators, etc) that are on the internet.
They have no technical issues doing this in the IPV4 space, because it's easy enough to scan every single address in the space.
This isn't practical for IPV6. So, it seems they wanted a way to identify every IOT device in the IPV6 space without having to scan all of it.
At least one approach they found was to join the ntp.org pool[1], and effectively donate server time. Since pool.ntp.org is the default NTP server listed for many linux distros (and thus, probably IOT devices), they now are getting live connections from the exact devices they want to index on their search engine.
Once you connect, they scan you back on 100 ports on so (ports unrelated to NTP) to see if you are a webcam, router, or whatever else they want to put in their index.
Pretty shady. Kind of like volunteering for a charity so that you can raid their internal mailing list for spam purposes.
[1]http://www.pool.ntp.org/join.html