Instructive in the sense that for any given technical controversy on HN, there will be a couple people that have some connection to the topic and can speak with some justified confidence about it, and several hundred people who read the title and, if we're lucky, the lede graf of the story and then just write stream of consciousness extrapolations --- also, instructive to note that HN does not, ceteris paribus, do a good job of upvoting the authoritative comments over the superficial riffing.
I enjoy googling a new phrase as much as the next person... but really?
From WP: Ceteris paribus or caeteris paribus is a Latin phrase meaning "other things equal". English translations of the phrase include "all other things being equal" or "other things held constant" or "all else unchanged".
I was just gchatting my wife in Edge, and the message entry box wraps lines on the character level (breaking a single word over two lines). Something that nobody would ever want to do and nobody could possibly think is the right behavior. Google and Microsoft develop two of the only three relevant browser engines, and indeed have basically supplanted W3C in setting web standards. But a simple text-entry field on a Google site running in Microsoft's browser doesn't work properly.
So is it really surprising that AT&T handled an obscure IP address the wrong way?
I am not surprised if/that google degrades the experience of their products on other browsers. I believe this has been topic of discussion a number of times on HN.
Google also degrades the experiences on the own platform because they think they know what is right and needed.
Example: Android platform with their own Maps application will kill phone in minutes.
Maps continually will ask to enable location, which is not needed just to look at maps and search, every few seconds after selecting no previously. Only way around is by checking don't ask again. Bam, infinite loop of software asking with policy refereeing answer. Google Maps kills a phone's battery with-in minutes.
I think Google is aware their products perform better on Chrome and does not devote resources to correct this, leading the user experiences on Edge or even firefox to be similar but not equal.
I'd make the argument that the parent should not be down voted because Google, perhaps in a small way, allows degraded experiences on other platforms for one reason or another.
There was a short time when Google made Hangouts Chrome-only, but faking the User-Agent would make it load. Turned out, yep, Hangouts worked totally fine in Firefox. They just refused to let me load it and that's it.
I don't think you can do that anymore, though. But I haven't tried it in a while.
A huge problem I see with the web is that it's is missing concessions for "advanced" users. I get that we're a smaller market, but come on.
I don't know the situation, but in theory, it could work fine for you, but 1% of the functionality could have been not working properly for 1% of users. That's thousands of unhappy users that would decide that Google delivered unfinished not working product and that quality of their products sucks.
I would also say money wasted on providing support for those users but I'm still not sure if such thing exists.
Then they should have waited. Google gets zero leeway for this kind of behaviour from me. Especially considering how hard they leveraged google.com to push chrome.
Google is the new Microsoft. History repeats itself. The major difference this time is you aren’t the customer, rather more the product, but that’s just semantics in this specific case.
I tried to use Hangouts last week with the latest Firefox on a beefy machine. While it wasn’t explicitly blocked, it froze the whole browser after 30 seconds. Never got into the meeting.
I just tried using Hangouts yesterday. Did not work in FF, but oh wow it magically worked in Chrome. Also, the only Hangouts "Desktop" app still requires Chrome. Also when you use Hangouts on iOS, and you open a link, it asks if you want to use Chrome (which isn't even installed) or Safari. how about use the goddamn default browser? Nope, it's gonna ask me to use Chrome for the rest of eternity. Total shit.
I do have Chrome installed on iOS, although I don't generally want to use it. But now that I have it installed, Hangouts on iOS opens links ONLY in Chrome. It's infuriating.
Oh course the "desktop" (not really) Hangouts app also only opens links in Chrome.
Don't worry though, Hangouts will soon be discontinued and we'll have to complain about some Google chat app.
It's well known that that's simply because Safari is the only default browser option on iOS, so apps have to use workarounds if they want to let users open links in a different browser. Though even then all iOS browsers must use Apple WebKit.
On other operating systems you can set the default browser to be Firefox, Chrome, or whatever else you want, and different browser engines are allowed.
The inbox app for iOS has a “remember my choice” radio button for the Safari/Chrome choice. Somehow the app hasn’t remembered that choice for several months.
My experience differs: using Google stuff on Chrome is worse. Occasionally one or another of my Gmail tabs jumps off into space on Chrome on my Mac and pins a core.
Chrome on Wayland, with GNOME 3 in particular, pretty much always pins a core too, but not for Gmail-related reasons.
Internet Explore, Firefox, and Edge don't support the 'break-word' value for word-break in CSS.
Only Webkit based browsers support that. If you make the mistake of testing/developing primarily in Safari + Chrome, you'll never discover the error.
They might have also used 'keep-all' which isn't supported in Edge, but is supported everywhere else, and have a fallback rule of break-all leading the behavior seen.
tl;dr:
There's actually no universal way of setting word-breaks for textarea's or pseudo-textarea, so I can't imagine if that writing a rich text input field is easy considering you have no foundations to work on.
AT&T has long provided a sub-par Internet connection service.
Their wireless router and DSL modem "U-Verse" gateway device lacks a true bridge mode, has a small NAT table, and you must use it because it contains a 802.1x certificate. You can use your own router with a rather involved hack: http://blog.0xpebbles.org/Bypassing-At-t-U-verse-hardware-NA... They instructed customers (including small businesses) to get off 10.X.X.X, presumably because they were considering CG-NAT, although they have since done some IPv6 deployment.
The original AT&T faded away with declining long distance revenue which they milked to the end until cellphones and Internet access became the main market. It took tax credits meant for "video dial tone" broadband deployment and used them to build their cellular network.
Its ADSL deployment use PPPoE and ATM which can cause MTU mismatch. ATM itself is a telco mistake. European access oligopoly members wanted 32 bytes because that suited voice frame transport and they thought data transport would never sell. American counterparts wanted 64 bytes for better data delivery. They compromised on 53: 48 for payload and 5 for header. With newer "U-Verse" ASDL they've finally eliminated this layer of overhead. (But no more 3rd party modems).
It decided to deploy VDSL instead of make the true upgrade to fiber. But they also cut costs by not building enough remote terminals so many customers have slow and unreliable links. Fast VDSL speeds (e.g. 100Mbps) aren't possible if customers are a mile away, or more. How much money have they spent on truck rolls diagnosing issues? Is it less than just building a more robust network to begin with?
Since 2005 they've been complaining that customers are using too much Internet and they need to slice and dice the Internet with data caps, application specific throttling, etc. And get more tax credits and deregulation which they promise will be used for broadband investment.
I've had better service from U-Verse and, more recently, AT&T Fiber than I have from Comcast.
But really, the point this thread is making isn't about whether you approve of all of AT&T's business decisions, but rather whether their backbone engineers can properly manage a network.
I'm interested in the opinions of people who have managed default-free BGP peering before on whether AT&T does a good job, but I'm not so much interested in reading another dslreports thread about whether AT&T is a "good ISP".
My experience with AT&T was that they were overly bureaucratic but outside of those rules anything goes. For example, they didn't ask for an LOA for BGP announcements of other people's IP space. When we were turning up a new Comcast circuit we accidentally leaked Comcast's customer routes (~5000) to AT&T and they were accepted.
Good to know! Add to the set of people who's AT&T network engineering opinions are interesting "anyone who has managed a very large scale distribution layer network". :)
Eh, I'm using one of the crap Uverse gateways stuck in bridge mode connected their Gigapower service. I consistently get 900mbps up and down when connected to a GigE port. All this for $80. That's way, way above par IMHO.
As a service. The gateways are still crap. Don't use 'em for routing or wireless. They are laughably bad.
I live in a rural area and depend on ATT's unlimited hotspot plan (that's now defunct) for my internet. It uses CG NAT and provides no IPv6 addresses.
However, you throw a SIM card from your phone in the hotspot and you'll magically have an IPv6 address, even without changing the APN. It's ridiculous that I have to resort to using a VPN to bypass all the troubles being behind a CG NAT entails.
From experience I can tell you that this guy is correct. Or at least he was. I had U-Verse as my first internet service at an apartment years ago and, in addition to waiting a month for them to hook it up, the modem they sent me didn't work. The tech came out (weeks later, of course), confirmed it didn't work, and proceeded to swap it with literally every single spare he had in his van. We got lucky on the last one. While chatting he admitted they were the some of the worst equipment he's ever seen.
Some of the branch locations at work have U-Verse as their only option. I don't have to deal with their day-to-day problems but the help desk tells me they're still pretty crap.
Yeah, no. ISPs don't really employ experts any more. The experts won't work for them. They haven't done anything innovative since the heady days of the dotcom boom.
It's at least gratifying to see that the top two root comments after the dust settled were ones that got it right. Though I suppose some of the upvotes may have come in after this information was available. (Edit: I checked and it doesn't look like it.)
Why would you trust what AT&T is saying publicly? This could have very well been an experiment to "see if anyone noticed", folks noticed, so here's the default "we didn't mean it, you can trust us" explanation.
Major ISPs generally do not act in the best interest of their customers, especially when the action is allowing access to service that hurts their ability to collect more information about their users. They certainly do not deserve the benefit of the doubt.
Not everything is a conspiracy, but nor should plausible explanations be dismissed out of hand without providing evidence or arguments. AT&T has lost the right to be trusted, or even to have the benefit of the doubt, thanks to a long history of anticonsumer activities.
The reason you are getting an ICMP respose from 1.1.1.1 is that your router is responding. This is consistent with the explanation. If you still want to doubt it, clear all DNS and ARP tables, ping 1.1.1.1, and check the ARP entry for it. It will match the MAC of your routers inward facing interface.
The fact that the reverse DNS is correct doesn't mean that it's reaching the real deal. The fact that the ping time is fast enough that light could only have traveled 5 or 6 miles one way means it's almost certainly not the real deal.
I could believe they tried to sneak it through for malicious purposes and wrote it off as a mistake when they were caught. I could also believe it was a genuine mistake. Either way, I'm glad there are communities like Hacker News to make a stink when stuff like this happens and get it into the spotlight.
Assigning a 1.1.1.1 to a local modem interface? Cox: that ain't noth'n here hold my beer" and watch a 10.x traceroute go seven hops out the door and resolve somewhere on their network.
There is a big difference: 10.x.x.x is for private use. Cox is using it privately for device management. You can also use it privately and there will be no conflict (assuming you are not attempting to manage those same devices, which is probably a safe assumption 99.99% of the time.)
Those IP addresses are just being assigned to their own networking equipment. Cox using the CGNAT network for that would be just as wrong as you using it. There's absolutely nothing wrong with Cox using private address space internally. If anything, there's something wrong with routing requests to private address space out your WAN in the first place.
What Cox is doing is completely appropriate and what AT&T and Cisco are doing is squatting on someone else's IP assignments.
They need to explain why it is an accident in detail, otherwise they are just being incompetent for disrupting large number of their valued customer's internet connections, and likely wasting their valuable time. They can't just say it is unintentional, they need to convince me that it is. Technical word salad will not cut it. It has to be clear to most people.
I believe this. I’ve had at&t business connections that black hole certain IPs and ranges because they (incorrectly) use them internally. What’s worse is their own techs spend a large amount of time troubleshooting because it’s not clear to them.
I've got AT&T fiber using a BGW210-700 with firmware 1.5.11 and 1.1.1.1 and 1.0.0.1 have been and still are blocked for me. I'll be keeping an eye on it to see if it changes.
Large ISPs wanting to get into CDN revenues and DNS, just a little hiccup to start testing the waters... ISPs are mad most people use Google or Cloudflare DNS now instead of their own which facilitate their ad networks and future CDN properties.
Large ISPs want that DNS endpoint and are aiming for CDNs as well, they are gonna eventually sell it as other CDNs/DNS can't be workable due to issues like this.
Large ISPs stopped innovating for decades and are mad others kept innovating on top of the network. ISPs think they own the network and are taking back this property not by innovation, but by bribes to 'representatives' and by force.
Now that net neutrality is demolished and ISPs can sell your private data, they have a reason to ruin DNS and CDNs to eventually own that for tracking and revenue streams. We made IMMENSE mistakes in 2017 with allowing ISPs to bribe their way into removing privacy protections and removing net neutrality. We gave them reasons to dismantle systems built ON TOP of the network so they can own those areas by controlling the network. 2017 was regressive for the internet markets built on our internet utilities.
Rather than ISPs building their own companies to compete on top of the network, they want to use their network lever to barge their way in with bribes like the Kool-aid man through walls that shouldn't be breached.
ISPs are in all out war mode fighting being a utility or commodity, rather than building competitive products or innovating on top of the network, making the network better/faster/fast they want to win by force and milking it, not by building products and improvements people want.
ISPs are also gunning to help build our government firewall/filter that is for censorship and IP protection [1]. AT&T in particular has been a privileged insider to the surveillance state and filtering[2]. It is also an extension of FOSTA/SESTA censorship. Turns out, building the government filter for surveillance also gets you a super efficient ad network by peering into all private data. No way blocking Cloudflare was a mistake, it was ISPs kicking the tires of their 'innovations'.
In 2017, we allowed emboldened network provider monopolies to get more emboldened and now that they won, they want to run all ad networks and will do so in unison with the surveillance state and filtering/censorship.
Do they? I'd say most people would be troubled to find their DNS settings let alone have the confidence to change them. I'm open to correction on that, is there any data on Google /CF DNS usage?
Most people that know anything about the internet have switched away from their ISPs DNS largely due to tracking and slowness/peering.
These tools will spread and eventually everyone will, a big fear for the ISPs who want to be ad networks and profiling systems.
> is there any data on Google /CF DNS usage?
Only ISPs would know that information, Google and Cloudflare know how many people use it but not how many are connected that choose it. ISPs know who uses what DNS, who uses VPNs and common clients and other competitive advantages that should be protected. Pretty soon they will have some 'identity protection' app that provides VPN and DNS which is used in their ad network/tracking systems. Even better if large DNS providers are flaky due to their own interference. This was definitely AT&T kicking the tires to see what would happen, either that or they are incompetent and need competition. They should be broken up either way.
ISPs now that they are able to sell your private data and build ad networks by removing privacy protections, actively do not want others using alternative/competitive DNS. Along with building the government filter and surveillance state, ISPs are not in the business of providing network quality and speeds, they are milking it with data caps and bundles noone wants, they want to be the biggest/baddest ad network/profiling system ever invented, that they of course control 100% over competitors.
From the OP article:
> The blocking is affecting AT&T home Internet customers who use an AT&T gateway. Cloudflare unveiled its DNS service on April 1, and users in DSLReports forum threads almost immediately started complaining that they couldn't access it. One thread began on April 1, within hours of Cloudflare's announcement.
> Cloudflare pitches 1.1.1.1 as a privacy tool that can help deter ISPs from monitoring one's Internet usage. AT&T lobbied against broadband privacy rules last year, and the company used to charge fiber Internet customers extra for privacy. AT&T fiber customers who did not opt in to a traffic scanning system that analyzed Internet usage in order to deliver personalized ads had to pay at least $29 more per month than customers who consented to the scanning.
> AT&T ended the controversial traffic scanning program in September 2016, but it says that it still wants the "flexibility" to expand advertising-focused business models to compete against Facebook, Amazon, and Google.
By allowing ISPs to remove net neutrality and privacy protections, we've emboldened network provider monopolies to be even bigger and ultimately it will harm internet freedom and competitive business on top of the network. Events like this will happen more and more as they are empowered more and more as they pull away from being privacy protected utilities, which they are but don't want to be.
It is obvious Cloudflare has no interest in operating a real recursive DNS service. If they did, they would have provided additional backup IPs for users they knew would be impacted by these issues.
Building a production service on what even APNIC refers to as "research space" was foolish. If this was Cloudflares only line of business, they never would have taken such a risk.
We (Cloudflare and APNIC) knew exactly what we were getting in to. We knew this was the only way that we'd be able to reclaim the space to be usable. And, in about a month, we've gone from 1.1.1.1 being routable by only 92% of networks to 98.7% of networks — and climbing. Don't accept that you can't change the status quo.
This is one of the things I love about HN - someone can post an inane and inaccurate slam of a free service, and the CEO of the company providing said service responds in a clear and friendly manner.
But exactly like the commenter stated. Part of the goal is to fix IPs like this being inaccesible. This DNS is both easy to access and using what should be, a perfectly accessible IP
I’m one of the first 100 users Cloudflare had sign up when it launched. I know because you sent me a t-shirt with my domain (and the 99 others) on it.
I switched over to your new DNS service the day it launched. I’m also an AT&T customer both residential and for corporate useage. It’s been fun giving them trouble to get them to route this correctly.
Hopefully this is behind us now and the ip becomes properly routable by even larger parts of the Internet as you grow this.
One question though, is there a plan to add typo correction? It’s the only thing I miss from using OpenDNS.
Thanks for being an early adopter! I've misplaced my version of that tshirt, which is a serious bummer. Remember working with our designer to hide all the "naughty" domains in the rays of the Cloudflare sun. :-)
I doubt we'll do typo correction. We're DNS purists. DNS should answer what you ask. Doing otherwise opens many cans of worms.
Bummer on the shirt! It is a great design. Although, I'm biased since your team gave my domain a prominent placement.
That position on typo correction is a bummer. I fat finger typing domains on occasion and found it handy to have my DNS service take care of that for me. Hopefully in time you guys reconsider. I am sure common errors could be corrected and it would lead to a better browsing experience for folks using your service.
Thanks though for making such an excellent service free and fast! Love the work you do!
I think they reply with their own HTTP server IP, which then redirects you to the "correct" address.
I assume it breaks if you type a wrong https:// address, but I guess few people do that, they just type the domain and let the site redirect to https:// if needed.
1.0.0.0/8 was unassigned space until 2010 when it was assigned to APNIC. It was never "allocated for research" and the reason nobody wanted it is all the cross-talk with internal devices using something they shouldn't have.
Whois literally says 1.1.1.0/24 is a research prefix.
netnum: 1.1.1.0 - 1.1.1.255
netname: APNIC-LABS
descr: APNIC and Cloudflare DNS Resolver project
descr: Routed globally by AS13335/Cloudflare
descr: Research prefix for APNIC Labs
APNIC designated 1.0.0.0/24 and 1.1.1.0/24 out of the whole /8 to APNIC Labs for "research" because they were getting so much bogus traffic. That just happened a few years ago.
But it's a fairly low risk for both cloudflare and users.
Just about all systems I've worked on allow multiple DNS IP addresses, so just set an alternate.
Cloudflare doesn't make any money from this (aside from the advertising it's giving them), so it doesn't really harm them that it's broken due to 3rd party abuse/issues.
I think that would be counter-productive. One of the main goals is to do research on how the 1.1.1.1-like addresses work in the real world. Offering a nice and free DNS service on those addresses is almost just a side-effect, which obviously triggers real-world reactions and uproar when ISPs fail to route properly. If they had offered backup addresses, maybe ISP customers wouldn't complain as loudly, and the media / attention levels would have been lower.
Why am I today just learning that `ping` will stuff 0's into octets you omit from an address? All of the keystrokes I could have saved and minutes on the phone I could have saved doing `ping 192.168.1` instead!
It's not ping. It's a common C library routine used by quite a number of programs that does this. And this is old behaviour of that library function, that has been around for decades. Here's FreeBSD's library function from 1994, and it was fairly old then:
These corporations have zero benefit of the doubt level of trust.
A few voices to be fair pointed out 1.1.1.1 is often incorrectly used and issues with it becoming ‘live’ mostly likely caused the outage.
Not judging this response btw, I can clearly see why ATT and other big telco wouldn’t get much leeway.