Sigh, those AT&T gateways are such a dumpster fire. What's sad is that it's not likely that there will be an easy way to fix this as those gateways do not have any kind of bridge mode which is a crime in and of itself. The best they've got is DMZ+ mode which fakes giving the LAN device the public IP address but it's still doing NAT behind the scenes and routing everything as normal so it doesn't work for any IP protocol other than TCP, UDP, and ICMP so no IPSEC tunnels or GRE with those modems.
The biggest crime is that AT&T uses these on their FTTH to turn ethernet from the GPON transceiver into ethernet for the home. Thankfully, in that application, you can bypass it, but it's still a tragedy. My equipment is perfectly capable of handling Ethernet, thanks (it's also perfectly capable of bridging 802.1x packets to the gateway, and keeping all the rest of the packets :)
Do you have a write-up of this anywhere? When I installed AT&T fiber I looked into this briefly, but I didn't find any success stories other than manual VLAN reassignment.
I was planning on cleaning up some hardcoded values so I could push it to some git service, but haven't gotten around to it. However, I'm not above a code dump. My IP is hard coded, but then again, I'm going to link you to my website hosted on that IP, so it is what it is. :)
http://ruka.org/~toast/steal/ (note the README which is going to tell you much of the same thing below in different words)
Here's the scoop. This is a FreeBSD program which opens up netgraph sockets against two physical ethernet interfaces and one virtual one. The physical interfaces are a) one connected to the ONT, b) one connected to the WAN port on the Residential GateWay. The virtual interface (ngeth0) is the one that you configure as your internet connection for FreeBSD (These interfaces are hardcoded in init_netgraph in steal_util.c; sorry!)
Everytime a packet comes in from the ONT, RGW, or FreeBSD, there's a decision for what to do with the packet. Mostly the RGW and FreeBSD packets go to the ONT, but through the libalias NAT engine for connection tracking. Packets inbound from the ONT are checked in the NAT tables to see where they should go... if there's no clear place to send it, then it goes to the FreeBSD. That's mostly in steal_util:loop. I haven't figured out how to setup the virtual interface in the program, so see the bottom of README for how i do that.
steal2.c is a little bit insane, it's basically setting up hot code loading. It's annoying to restart the daemon because the NAT tables are dropped, and I do a lot of Erlang work in my day job, so I really wanted to be able to hotload some things with this. So it uses dlopen and dlfunc to load steal_util.so and calls the init and loop functions. If init fails on a reload, it continues to use the old loop. There's some provision for updating the state data structure in a safe way.
Theoretically, this might work in Linux too, I think netgraph got ported there a couple of times, but I don't know if libalias did. Anyway, the main ideas could certainly be ported into whatever tool is available; namely: proxy 802.1x to the RGW, drop EAPOL logoff packets; oversend ARP packets, spoof mac address of the RGW, make pcaps of packets you don't know how to handle, etc. :)
Also of note: the ONT/RGW send packets with VLAN 0, FreeBSD strips this when reading packets via netgraph, and I haven't found a way to send packets with VLAN 0, which hasn't been a problem (yet?). If it becomes a problem, I suspect I'd need to switch to the packet apis that tcpdump uses (BPF), I think there may be a sending interface there too. And I'm also sadly moving out of AT&T fiber's reach this summer, so at that point, I won't be able to test anything else.
In this case for AT&T FTTH there are two seperate devices a ONT and AT&T's router. The ONT is a media converter that converts the GPON (fiber) to Ethernet. AT&T's router is superfluous in the scenario where the customer wants to user their own router. The only reason AT&T's router is necessary is to perform the 802.1x authentication.
Living in Central Florida, the lightning capital of the United States, I take electrical isolation very seriously. This is what is so great about fiber. Instead of having a another separate conductor running into your house (twisted pair or coax) there is just a glass insulator (fiber). So no extra isolation needed.
I think Icomera’s bus WiFi systems also use 1.1.1.1 somewhere in the hotspot process.
TBH, I’m not surprised, and this definitely isn’t the first time something like this has happened. For example, Hamachi (before being bought by LogMeIn) squatted on 5.0.0.0/8, and then (when 5.0.0.0/8 got allocated) on 25.0.0.0/8 (which was already allocated to the British MoD).
This is really nothing new, and now that I think of it, not really surprising!
Semi-related: once some hardware arrives, I should be able to finish finding a way to rip the 802.1x certs off of the 5286AC so we can use our own routers, or at least put these things into a proper bridged mode.
You can bypass these routers with an 801.1x MitM attack. You basically put it behind your Linux router and bridge only the EAPoL frames, then do DHCP from your own router.
I posted some details about this on DSLReports[1].
I have been using this method with AT&T GigaPower fiber in Austin, TX for two years and it's been totally stable and free of problems.
That said, I'd love to be able to extract the cert and not have to do this.
Please fully disclose your work! I was looking into this earlier and found the other post you mentioned. Unfortunately, they stopped following up with RE details on the 5286AC after they got a bunch of CVEs assigned. Was really disappointing.
Either way, thank you! Would you be willing to send me the URL for the firmware? I don't have time to desolder anything at the moment but would love to look at the image.
I'm sure you're pretty far down the rabbit hole of dealing with the 5268AC but the "DMZ Plus" mode does seem to bridge. I have a Ubiquiti Edgerouter on the other side with OpenVPN setup, public IP, etc. All I needed to do was enable DMZ Plus and turn off wifi.
DMZ Plus doesn't bridge - it just fakes it by essentially 1:1 NAT'ing a public IP to your router.
You are still using it's internal routing and NAT tables, which to be charitable are problematic. You also lose the ability to do ipv6 - not that AT&T's ipv6 implementation is worth even using at this point.
I just moved from Comcast to AT&T gige and the speeds are certainly great - but this CPE (and horrible v6 even if it worked) is seriously making me reconsider.
What is truly disturbing here... who wrote these firmwares? You'd think something as widely distributed as an Internet gateway for a major ISP would be written by someone who at least know some Internet basics (like which IP ranges are public and which are private). And, how do you test if this can happen. And if they made this mistake then what else is lurking in that particular piece of software?
"The WLC intercepts client DHCP discovery packets, inserts it's own IP address configured on the egress interface into the relay agent field, and unicasts the DHCP packet to the configured DHCP server."
Wait, you weren't aware that most equipment manufacturers' software teams leave every edge case and design decision to a coin toss between 'fastest to code' and 'least effort to code' and let the rest of us bitch and scream about the stupidest of egregious bs for years until that piece of gear isn't sold anymore, then do it again with the next model?
"Weren't you aware that most equipment manufacturers' managers downprioritize quality and demand that the developers to produce code that can handle the managers' incomplete solution requirements given the smallest possible budget?"
There are numerous guides/howtos floating around for configuring cisco/juniper/etc. equipment that uses 1.1.1.1,2.2.2.2 for things like loopback addresses(these are used for very different things than the lo device of 127.0.0.1), or uses such addresses for peer to peer links etc.
People just copy paste these configs without thinking, I'm not at all surprised 1.1.1.1 breaks left and right.
Comcast / xfinity was null routing 1.1.1.1/32 for residential customers up until Monday morning/afternoon. They removed the null route fairly quickly. I assume it was done to reduce the amount of garbage traffic going to 1.1.1.1 from hitting Comcast's core.
So is all this 1.1.1.1 garbage traffic now directed to Cloudflare's servers? Or will ISPs port filter everything sent to 1.1.1.1 except DNS (and DNS-over-HTTPS) requests headed for Cloudflare?
>APNIC's research group held the IP addresses 1.1.1.1 and 1.0.0.1. While the addresses were valid, so many people had entered them into various random systems that they were continuously overwhelmed by a flood of garbage traffic. APNIC wanted to study this garbage traffic but any time they'd tried to announce the IPs, the flood would overwhelm any conventional network.
>We talked to the APNIC team about how we wanted to create a privacy-first, extremely fast DNS system. They thought it was a laudable goal. We offered Cloudflare's network to receive and study the garbage traffic in exchange for being able to offer a DNS resolver on the memorable IPs. And, with that, 1.1.1.1 was born.
Cloudflare and APNIC seem to both assume the traffic is going to now hit cloudflare. Trying to block everything but DNS makes no sense at all; how can/should ISPs be keeping track of which services someone chooses to run on their IP address?
Edit: Turns out Cloudflare aren't just running DNS; they're hosting a http/s webpage with instructions on how to use their DNS too, so you've gotta hope people aren't filtering: https://1.1.1.1/
Super interesting! It’s the IP address space equivalent of a no mans land - a dead zone where stray packet radiation will cook your servers. One can only wonder what goes on under the surface..
Are people really that surprised 1.1.1.1 won't work everywhere?
I suck at networking, but I worked that job just long enough to know there is some pretty horrible stuff such as 1.1.1.1 being hard coded all over the place, even in high-end hardware.
AT&T's crime isn't really that they are stealing stuff, but rather that they aren't doing anything to fix the millions of legacy stuff with broken 1.1.1.1
Granted, the most plausible explanation is they thought: "To hell with it, it's broken, we are going to use it."
The usual non-technical word that would get picked here is "hijacking". If I get on a bus home, and some nutjob with a gun to the driver's head insists it goes to Springfield instead, the bus wasn't stolen, but it was hijacked and now I don't get where I was headed.
Hijacking can occur in several places in Internet infrastructure, but hijacking IP addresses is arguably the worst since there's almost nothing we can do about it. Cloudflare might have (perhaps even purposefully) struck the one thing you can do if you want to, which is put a popular but optional service on the address, and then let ordinary users scream blue murder because it doesn't work.
[ Other examples of hijacking: Web browsers will fetch the path /favicon.ico from your site. Why? Because Internet Explorer did that to add "favourite icons" for web sites, so now that's all you can use it for; administrator@example.com can't just be the email address for somebody who fancied the handle "administrator". Why? Because Certificate Authorities decided that if somebody receives email sent to administrator@example.com that person must be authorised to have certificates for any name in example.com. No existing rules told them this was safe, but they did it anyway, so now you have to allow for that ]
Oh, I should point out that "The Internet" (to the extent it's any distinct thing) also hijacks things. All ISO/ITU object identifiers used in Internet standards are under the OID 1.3.6.1, but, er, 1.3.6 belongs to the US Department of Defense, so how did the Internet get the nice compact 1.3.6.1?
Turns out there's just an RFC from 1988 which says "This memo assumes that DoD will allocate a node to the Internet community", it says it assumes this will be 1.3.6.1, and of course thirty years later it would be pointless to say "No, the DoD does not allocate this node to you".
To be more specific, RFC 1065 states its use of the subtree that was initially held by the U.S. National Bureau of Standards. The NBS transferred the subtree to DoD, which had not stated how it intended to manage it. Thus, the DoD might have chosen a different value for the last digit of the internet subtree, but that's all.
This isn't really hijacking, although it does seem a bit haphazard. Considering the "official" handling of the subtree transfer, I suspect that the DoD did accept the decision after the RFC was submitted as a draft, with the RFC just not updated to reflect this.
30 years later, I don't think anyone cares about OIDs.
"I suspect that the DoD did accept the decision after the RFC was submitted"
You suspect that an enormous Federal bureaucracy "accepted" something but it isn't written down anywhere? Does that sound right to you?
If you mean "accept" in the tacit sense that they can't do anything about it now, well, of course, that's why I called it "hijacking". I can't do anything about the fact I'll be home late when my bus is hijacked, but let's not pretend I've "accepted" the new destination and somehow now the nutjob with a gun has my permission to take it there.
Funny thing about OIDs, everything that needs to enumerate things and touches any of the ISO/ITU X series standards uses OIDs, yes, still in 2018 and presumably forever. For example you might have noticed that TLS 1.3 is now (probably) finished. Grep through that and you'll find it mentions all the OIDs you need to use to make it work, and introduces OID filters so that you can write key-value matches for client certificates like "I only want client certs which have the following policy OID".
OIDs are fine, there is no reason to create a new parallel system that would work the same way and presumably either duplicate the OID hierarchy or try to displace it.
This kind of makes sense, except that we are talking about hardware devices here.
The hardware is manufactured by a private party that has no obligation to follow anyone else's rules.
That party may voluntarily decide to participate in some standards body and get their device certified in which case maybe they are violating some voluntary private certification, but I don't think that is the case here.
As a consumer, unless the manufacturer specifically guaranteed compliance with some standard, then the product is not defective. You get what you specified. If you failed to specify (or verify the specification) that is your fault as a consumer.
Your imaginary standards and rules are not legally binding in the real world, which is my manufacturers ignore them, and why it is silly to call this "theft" or even "hijacking."
How can a manufacturer hijack their own device? It is their device. You the consumer chose to buy it. Unless they guaranteed you something in writing that is missing, caveat emptor.
I'd say 'abusing' is the term I'd go for - and it seems awkward for an ISP/network operator of that size to ignore the rules.
If mum and dad's home network uses 1.1.1.1 because it's "cute" or whatever, that's understandable if still wrong. But why do network providers that employ lots of people that should know better do this?
I cannot believe that it wasn't brought up, ever ("Hey, you know.. We should use 10.0.0.0/8 maybe?"). I gotta assume that someone in charge shrugged and said "Who cares?".
yes. At the very least, there are the autonomous system registration fees paid to ARIN for the prefix. CloudFlare might just be paying APNIC for the two addresses it needs, though, or in some kind of agreement. Either way, CloudFlare paid in some way to be the sole entity on the internet allowed to advertise for prefixes containing 1.0.0.1 and 1.1.1.1 over BGP to its peers.
When 1.1.1.1 launched here in Spain, it was inaccessible from several major carriers. Some had network-wide routing problems to that IP address, and some had installed CPEs that included static routes to 1.1.1.0/24 and stuff like that, probably for internal purposes.
Nothing of this strikes me as odd, let alone malicious. It's such a weird IP range, I even remember having my LAN configured as 1.0.0.0/24 at some point, because who would ever use those IP addresses?
Also reminds me of when Spanish ISPs were given IP ranges by RIPE for their customers beginning with 37.* -- those had never been used, so many network administrators had them added to their bogon list, which meant for those customers lots of web pages were inaccessible. The solution was to reboot their CPEs until they got a good ol' IP address from the ol' ranges :D
Nothing like that is an excuse. You have 10.0.0.0 for that - it's huge and you can use it for whatever you want without stepping on anyone's toes.
There are absolutely zero reasons I've seen so far (I'd be interested to hear abstruse ones? "I thought who cares" isn't one) to avoid using one of the private ranges.
10.0.0.0/8 isn't even noticeably harder to remember or type, really.
Using the 10/8 or any of the other RFC1918 had great potential to step on their customers toes. That is exactly why rightly or wrongly they used the 1.1.1.0/24 range. Hardware manufacturers generally used the range for interfaces that were local to the device and often only used on interfaces internal to the device. They knew this equipment would be deployed into environments where RFC1918 addressing would be used but they had no idea what RFC1918 address ranges, so using addressing from the RFC1918 networks meant potentially impacting their customer's data. They chose to instead use addressing which at the time they believed would not impact their customers.
APnic is not blameless here. They knew the issues with this space when it was assigned to them as a research network. For quite awhile they allowed Google to advertise the space and collect data on it's usage. I assume Google no longer was providing the infrastructure to do so and APnic saw an opportunity to have someone collect data for them for free.
Collecting data on traffic sent to this ip range is one thing but approving its use for a service available to the public knowing the accompanying issues much of the public would have accesssing it is in my opinion not responsible use of a research network.
