This is because of an update in the systems from the Sprint side to help
merge with T Mobile. That happened from 12am-6am EST and Sprint's systems
clashed with T Mobile's. I'm glad I have off today.
For today was scheduled first migration of chunk of Sprint users to TMobile.
Probably something got "misconfigured" at mobile core that causes failures.
Not irreversible (but the DOJ rarely breaks up already merged companies due to the difficulty of "unscrambling the eggs")!
This event could create a reason to block the merger ex post, especially given last week's news that DISH may not buy Boost Mobile from T-Mobile, which was a key provision of the merger to reduce its anti-competitive effects.
Well, hopefully it doesn't cause reason to block the merger ex post. Disclosure upfront, I'm nothing more than a satisfied T-Mobile customer that switched away a couple of years ago from AT&T. No other interests in this race.
Sprint was a shambling zombie, and any public policy measure that could be undertaken to make Sprint something other than a shambling zombie, and in particular, into a viable competitor would have rewarded Sprint for well, not being competitive.
Out of the four major carriers we used to have, we had 2.5 well capitalized competitors with large customer bases and Sprint, a shambling zombie. T-Mobile put together an offer, tendered it to Sprint, and it was accepted, and out of the many ways that Sprint could have been gobbled up by one of its larger and better competitors, T-Mobile serving as daddy to Sprint's mommy was about the best outcome you could have gotten. This puts their customer base, assuming the Sprint customers stay, at relatively equal footing with Verizon and AT&T, and more customers = increased cash flow, and increased cash flow keeps them operating in the black and hopefully with room to invest.
I don't care if we have three or four or five major wireless carriers right now so long as I can be satisfied with one of them where I live and work. T-Mobile's "Uncarrier"-like lack of billing shenanigans was one of the things that put them over the edge when I decided to bite the bullet and switch carriers. Also, with StarLink and maybe other systems like it possibly coming online and into service in the next couple of years, I'm not at all convinced it's going to matter that we only have three major "cell carriers" going into the next decade. Let T-Mobile keep all its customers, all of the spectrum they acquired. They paid for it, and I expect them to put it to use on behalf of their customers.
Do we have confirmation that Verizon is actually having issues, or are people on Verizon trying to call T-Mobile customers, failing, and are blaming their own carrier, Verizon?
I have a Tmobile MVNO and I can call Verizon users but not vice versa. I can also not call any Tmobile number. My wife has Verizon and can't call me but can call others except for Tmobile #s.
> This is because of an update in the systems from the Sprint side to help merge with T Mobile. That happened from 12am-6am EST and Sprint's systems clashed with T Mobile's. I'm glad I have off today.
Or
"This is because of an update in the systems from the Sprint side to help merge with T Mobile. That happened from 12am-6am EST and Sprint's systems clashed with T Mobile's. I'm glad I have off today."
Or
This is because of an update in the systems from the Sprint side to help merge with T Mobile. That happened from 12am-6am EST and Sprint's systems clashed with T Mobile's. I'm glad I have off today.
Edit: Note to use a single initial carrot > or surrounding quotes " or surrounding asterisks
What could cause all major cell phone providers to experience outages that began at the same time in the same day?
A solar flare seems unlikely, since WiFi does not seem to be effected (for me at least).
Routing issue maybe?
One possibility: it’s all just T-Mobile, but people are bad at reporting. For instance: if a non technical person tries to call a t-mobile number from their Verizon phone, and experiences an “outage”, they may report this as a problem with their own phone rather than understanding that it was on the receiving end (and knowing to try to call other carriers).
This could also explain problems with Facebook messenger (if the T-Mobile outage is effecting data). It’s not a problem with messenger etc. it’s just that messenger is the service that some T-Mobile customers are trying to use and having it not work. So they report it as a problem with the black box they know as their phone.
That or we are getting hacked. And if that’s the case then buckle in and charge up your ham radios.
Quite possible though is a significant disruption of one of the many services like LNP which are implemented via SS7 and shared by all carriers - there is indeed a complex set of inter-dependencies between carriers that are all somewhat generically referred to as part of the SS7 system, several of which are required to successfully set up a call or deliver a text message.
All the major carriers maintain their own local LNP database. The only updates that would be distributed via a third party(NPAC) would be recently ported land lines.
What about number transfers?
Some MNP implementation rely on the original Home Subscriber Server the number was issued in or something, so there could be paths between core networks among major carriers.
The article seems ti suggests calls and texts to T-Mobile are failing so customers from other providers are likely reporting issues unaware of the source.
No, T-Mobile forces everything through Inteliquent (aka Onvoy/Vitelity) as of a few years ago.
They all likely share the same STIR/SHAKEN vendor, and Comcast is affected too (Comcast has working STIR/SHAKEN that interacts properly with T-Mobile and Verizon). Wonder if their shared STIR/SHAKEN vendor broke these calls?
How about too many IMSI catchers being deployed at the same time ? Like basically ~ all of them at the same time right ? Couldn't that cause some unforeseen disturbances ?
Other services are reporting failure to connect when trying to reach T-Mobile. I was just trying to call a coworker w/ T-Mobile, and the error message I received was "Verizon cannot complete this call." (I'm on VZ)
Could be something with 4g. My 3g flip phone is working fine (AT&T subscription plan - not prepaid). I can even receive calls from people on T-mobile who aren't able to call each other or other people (although they don't receive my incoming calls).
There is no proof of a denial of service attack. Attack maps like the one you’ve linked to sinkhole traffic from botnets to known honeypots. The only way a DDoS can be identified is if you’re sitting on the transit links themselves or you have insight into the very private networks that the carriers have. Attack maps do not have this.
Edit: Lol at all this nonsense 'Anonymous DDOS', BGP fat-fingering, NSA speculation. I got karma to burn from those butthurt at me being right about the fiber cut. Look at the timestamp and region! Look at this: https://bgp.he.net/AS393494#_graph4
Who you think AS6461 is?
EDIT2: Someone else might have been partially right about merger upgrades being a factor, perhaps removed some redundancy. Before I found the Zayo cut I saw that Crown Castle was looking weird overnight (they are #1 or #2 largest cell tower company with their own acquired fiber network, Lightower): https://map.internetintel.oracle.com/?root=traffic&country=U...
Update #2 from T-Mobile's President of Technology (3:02 PM PT):
"Teams continue to work as quickly as possible to fix the voice & messaging problems some are seeing.
Data services are now available & some calls are completing. Alternate services like WhatsApp, Signal, iMessage, Facetime etc. are available. Thanks for your patience."
This is a good case of causation and correlation being conflated. Of course T-Mobile users will report problems with YouTube even if it’s just their carrier. Any localized ISP problem will show up in outage reports for top websites because that’s what everyone uses.
That is true but I will also note that Facebook Messenger died for me across devices (even on the browser regular Facebook was fine but my messages would not load), and the email discussion I've seen largely says something similar- only Messenger failing for a lot of people around the same time a couple hours ago.
Twitter moved really fast on this one, and not in a good way. 4chan created a meme map declaring that the USA was under a DDOS cyberattack, and all of Twitter apparently immediately decided that this was the truth - including some US senators.
How do people get digitalattack map to show today? I can only see yesterday on there and some people are posting images that say the 15th.
Either way, it's being taken out of context. There are always DDoS attacks like that. The past couple of months has had constant traffic on their timeline.
I'm on T-Mobile and can confirm that I have been unable to make or receive calls for the past hour. Also, my home Comcast internet seems to be operating at dial-up speed and with insane latency. Something big is going on.
Off topic, but I had 2FA setup on my phone for various things, and had it stolen overseas. I was able to lock the phone and erase it...but I was stuck overseas without being able to access anything or reset it. There has to be a better way.
I did have some keys backed-up in the Cloud. One issue is that because I was overseas a lot of security protocols locked me out of my accounts -despite- my keys and required an SMS message first...which of course I didn't have a phone with my SIM, so it was impossible to do.
Many sites give you a recovery code of some sort that you can use to disable 2fa in the event you were to lose access to your authenticator app or texts.
There were early reports that Level 3 was having some routing issues earlier today, but it's been confirmed that they're not having any problems whatsoever (for anyone who may have been suspecting that)
I use 1Password 2FA everywhere I can and choose services specifically for that reason. I just switched cell providers today because Verizon didn’t offer it. I don’t have influence over my employers IT though.
T-Mobile user here, had to restart my device (iPhone) in order to make calls. The data network seems stable as I was able to FaceTime another person via the T-Mobile data network.
Oh interesting, I couldn't get a verification code from Chase earlier today. Just tried sending a text to myself; I can send but not receive texts. That's pretty wild, it's been like that for >4 hours for me it seems.
I tried logging in to a couple financial sites that should have sent me text codes, but they did not even try. They just accepted my name/password and went to my account page.
I tried from a Firefox private window, my work Firefox profile (which has never went to those sites), a Chrome incognito window, and a Chrome instance being run under Selenium, so make sure I had no cookies from a previous visit to allow bypassing the text code.
So now I'm a bit confused. I hope the sites aren't doing something stupid, like associating successful text code verification with the IP address, and caching that on their end.
Maybe they've temporarily turned off requiring text codes to avoid locking people out during this outage?
Considering it is impacting other non-celluar services (like Facebook), I'm guessing it is an internet backbone issue (e.g. bad BGP routing was broadcast?).
PS - This doesn't mean it is unrelated to the Spring/T-Mobile merger, they themselves could be the ones who screwed up routing.
This is because of an update in the systems from the Sprint side to help
merge with T Mobile. That happened from 12am-6am EST and Sprint's systems
clashed with T Mobile's. I'm glad I have off today.
A friend’s phone has been non-functional for hours now. With the size of the T-Mobile user base, this outage almost certainly has already had real human costs in the form of an inability to summon emergency services.
Depending what is causing the outage 911 may still function. Or it may not. Just spreading to not assume 911 is unavailable because normal calls don't work/your plan expired/whatever else is going on as 911 gets a few special technical perks that make it available more often than normal calls.
Exactly right. 911 calls are handled differently at almost every step of the process, starting with the handset and carrying all the way through to the 911 call center.
Priority access services like 911 are handled very differently. For example, I can’t place any normal outbound calls with my T-Mobile sim, but calling 311 in NYC works fine.
Same. AT&T to T-Mobile line. Dead air, insta hang up.
Signal messages appear to be unaffected so far, so one more anecdote that data works, but IDK if they were delivered to my group warning them about this via cellular data or wifi.
Called from ringcentral voip to t-mobile phone on UMA. Rang once, then dead air. Disabled wifi to shut off UMA for a second try, rang a few times then went to busy tone.
What's up with the networks that went down a few hours staggered from the rest?
Frontier Networks, Playstation Network, and Spectrum all went down early this morning; and I mean actually down. I had a Fingbox at my parents house tattle on Frontier Networks as soon as the connection got less spotty, and then the network dropped again.
The ISP/data failures are more interesting to me, to be honest, than the cell networks. I wonder if they're related. The cell networks all seem to share a lot of cascading failures, historically. It's intriguing that something is happening to affect the rest of the world.
The cynic in me says that someone just re-routed the internet through whatever new DPI-esque system is now in place, but I hope the reason for this mess is more innocent.
Come to think of it seems like telecomm companies have been elevated to the national security level of importance. I'm not sure if it has always been true, but with more and more reliance on tech that depends on these companies the danger is escalating. Does anyone know if these companies are vetted and inspected on any regular basis by government agencies the same way airlines for example are? A trivial mistake by one of these carriers can have dire ramifications for the entire nation as it seems to have been the cause of the current outage.
What might be needed is something like voice-packet across voice-network tracing/logging... sort of like, given a voice connection ID, tag packets going through it in a certain way, when routers or forwarding equipment en route see this tag, then send back an "I received the packet at <timestamp>" message, or something like that.
If something like that already exists, then T-Mobile should know exactly where the network is failing, and why.
If not, then maybe it, or something like it would be good to have...
I have learned from many years of software engineering that a working system is NOT proof that the system will continue to work into the future, that good logging/tracing of all system components is a necessity not a nicety, and that if you can't tell exactly when/where/why/how your system failed within 5 minutes of looking at log files -- you need to revamp your logging system so that that becomes possible.
If it takes T-Mobile more than 5 minutes to accurately diagnose the root cause of this by looking at log files, then that is proof (to me, as an Engineer) that their logging system ranges from suboptimal to non-existent, and that fixing that so that this never happens again (via fixing their logging system) -- should be their absolute first corporate priority -- after whatever is causing this is fixed.
Perhaps incidents like these will increase the popularity of amateur radio. I'm curious if there's any studies on the potential impacts of the US carriers begin down for a day or a week or more.
I'm curious how many other people share my ambivalence toward amateur radio? I want to be interested in it, but since encryption is forbidden I can't think of very many things I'd really want to use it for, what w/ being unable to trust the authenticity of transmissions and having no confidentiality.
I enjoy listening to stuff w/ my RTL-SDR rig. I also find the emergency communication capability that radio amateurs provide a Good Thing(tm). I just can't see what, personally, it could possibly be useful for.
Talking about what gear you each have. There is a reason that is most of the conversations on amature radio : it is the only thing that you can safely talk to strangers about.
The last that I'd explored amateur radio I thought digital signatures were still a grey area. Good to know they're not. That does increase the usefulness dramatically.
No need to study anything more than a newspaper archive. Look at the local papers every time a hurricane hits the United States, or there is another major natural disaster over the last 80 years.
Ham operators are always a communication lifeline in times of crisis.
feeling pretty good about the fact that some friends and i decided to get our licenses early into quarantine via remote online testing. highly recommend it if you're interested, some people are not aware of the fact that morse code requirements were dropped long ago and the question pool is available from a number of excellent study help sites (hamstudy.org is what I used).
My hope is for someone to complete a smartphone-based ad hoc mesh network that can use bluetooth, wifi to pass messages along, up to 6 hops or similar.
Currently, the closest thing I'm aware of for that is cwtch which is a ricochet.im rewrite
There was one out there for Android that worked great.
I wanted to build my own for emergency services and we figured out how to do some of the mesh routing but it didn't go far... WiFi Direct unfortunately only allows you to pair to one phone at a time (while bluetooth allows endless pairing outside of BLE)
this already exists in some forms [1] on top of the fact that amateur radio in general is quite capable of moving data whether by packet radio [2] (including packet radio BBS [3], things like hinternet [4], bbs over amateur radio, or plain old teletype (RTTY) as a means to operate access a console on a remote machine. there's a ton of interesting creative work being done in the space even though it isn't particularly mainstream-popular.
I have no idea what it’s called but when I was in Cuba 5 or 6 years ago they used something like this on their mobile phones. They basically had an “internet” via bluetooth but without needing to connect to the “real” internet which was very expensive and monitored by the government.
Cuba is amazing at making the most of what they have. I hadn’t heard of SNET before this, but I know they’ve kept classic cars running for decades out of necessity
I am not exaggerating in the slightest when I say that Cuba must have the highest concentration of “hackers” of any place in the world. We were on a bus going to the far west end of Cuba (where not many people travel to) and it broke down in the middle of no where and the bus driver managed to fix it with some scotch tape and a twisted up plastic shopping bag.
I’ve been to 70 countries and the Cuban people are hands down the most entrepreneurial of them all.
For definitions of dabble that don't include transmitting, its most certainly not.
One of the best ways to get ready to take the Technician test is to buy a cheap handheld and figure out how to listen in on the local repeater. Being able to apply what you're reading in books to real world stuff is a fantastic learning opportunity.
Unlikely that an attack would manifest itself in this way. I think it's an SS7 screwup since it seems that signaling is mostly what's gone pear shaped.
SS7 would not affect website uptime. All the carrier websites are having issues.
[edit] As others have pointed out, if b-number routing is failing, then their support sites would be flooded with people checking their account. So maybe the subscribers are in effect overloading the sites. That would make sense.
People are fearmongering and sharing pictures of DDoS activity out of context. There are always attacks like that going on in the background, it doesn't mean that's the cause.
Why are so many people in this thread sure that this is "just Sprint merger mishap" when i and many others have been unable to use Facebook Messenger and Instagram chat in Europe?
What does those services have to do with US cellular networks?
I came here from the DDOS thread on the HN frontpage because Messenger was down (and instagram DM's), which i haven't tried before here in Scandinavia in my 10 years of FB. Same for my colleagues.
I see at least 3 other people mentioning it in this thread, and lots on Reddit and on Twitter in connection to these articles.
That for me was at least was at little interesting when the overall theme of this new "incident" has been "something large scale going on".
Likely the same reason that a few other comments were downvoted: there wasn't an actual DDOS attack.
Additionally, your linked graph doesn't actually contain any context or resources that could be used to verify it. It is in fact an uncredited photo from [0] which was refuted in the children of a few other comments[1,2] that were available at the time of your post
That is the current suggestion: (reposted from elsewhere in this thread: https://puck.nether.net/pipermail/outages/2020-June/013124.h...
This is because of an update in the systems from the Sprint side to help merge with T Mobile. That happened from 12am-6am EST and Sprint's systems clashed with T Mobile's. I'm glad I have off today.