In case people are wondering what this is about: All links and images are not working on twitter right now. Seems like they accidentally included their own services in the api blocklist? Sending best of luck to the engineers. The whole twitter saga has given me a good chunk of valuable "lesson learned" over the last weeks and months.. so thanks for that.
For someone who has tried to avoid hearing as much as possible about this drama (and didn't notice this was about Twitter until looking at the comments), what are the condensed "lesson learned" takeaways if you don't mind elaborating?
I honestly think the engineering/ops problems are the least of their failures. They bled revenue not because of outages, but because wild policy swings and chaotic management style alienated some of their biggest customers.
If they had frozen features and left the existing policies in tact, I suspect we would have a dramatically different narrative about the layoffs. If brief interruptions like this are the worst that happens when you cut engineering to the bone, it's a good argument that is Twitter was indeed wildly overstaffed.
Instead, though, we have a company in crisis due to its mismanagement of other areas, so we're primed to view stuff like this through the lens of that broader failure.
> If brief interruptions like this are the worst that happens when you cut engineering to the bone, it's a good argument that is Twitter was indeed wildly overstaffed.
Overstaffed in order to maintain Twitter as a static service that never ships new things, sure.
I guess they've been able to ship some things that the old Twitter had already implemented and/or a/b tested. But I'm not sure those count. Meanwhile people have been paying in advance for Twitter Blue features that were promised 3 months ago.
The frequency and types of outages and failures is significantly more frequent now than pre-Elon. This isn't a surprise to anyone, given Elon's strategy for maintaining Twitter (or not maintaining it, as the case may be).
I don't interpret a lot of "drama" as you put it, but interest. Many observers here are in this field, and follow the "chaos engineering" discipline. Some of them use tools like "chaos monkey" that simulates a metaphorical monkey running through your server room turning off random things, to see how well your resilience systems cope. It's a rare and greatly interesting sight to such practitioners to get to see what happens when the monkey "disconnecting the more sensitive server racks" is a more literal one.
>What was the reason when Twitter went down pre-Elon?
To use a tennis metaphor, good players minimize unforced errors and recover quickly from forced errors.
This is a very clear unforced error that could likely have been prevented by just waiting to roll out the new feature.
To extend the tennis metaphor, it would be like Serena Williams losing a set on 50 double faults. Sure, she's lost other sets before, but it would be notable for her to lose in such a unique way, even if she still went on to win the match.
That's why people are talking about this, it's a very weird way for a site to fail and it's interesting how it happened.
He created that drama when he tried to gaslight the entire tech industry into thinking 90% of engineering jobs were redundant, you can’t expect anything else than for him to get dunked on when the completely predictable consequences of that play out, namely that the service will get more and more unreliable over time.
If it’s nothing, why is Musk on Twitter saying the whole codebase is brittle for no reason, and needs a complete rewrite? Sounds like a pretty big deal to me.
As far as the drama comment, you’re free to disengage from this thread at any time. No need to continually interject how against the drama you are. Just leave if you find it distasteful.
A 1 hour down time every infrequently is nothing, but at the same time a codebase can be brittle. Are you saying the two are mutually exclusive?
Brittle code means it's hard to change, and causes issues when changed, but other than that it's stable when running.
And I have every right to point out the drama here. People here do somewhat have the right to create drama, but it is distasteful and unprofessional and I will continue to call it out.
Glad for you to admit there is drama here though :)
- When you fire a lot of engineers, legal, security experts... things will still somehow keep working
- Also some people still want to keep working at this place (or must, because of VISA etc); It probably happens all the time, just this time everyone was watching it in full motion.
- I was sure this whole takeover was the end of twitter, but somehow with how many users twitter has, it just won't die. Goes to show the advantages of the mass.. probably the same applies to microsoft/google..
- The whole 8$ for a checkmark story.. which destroyed the whole trust in the checkmark basically immediately. The whole verification process (how that worked) - many people shared some insights on that and how much effort it was to fine-tune all of that with the scale of twitter.
- The free API just being shut down with basically zero communication to devs. and somehow getting away with it. I guess this is a reoccuring theme in tech that a small company has to be open for developers until it's big enough to turn on them.
- With the reduced engineering and the new management, it seems more errors slip into production (and incidents like the one now) - or maybe it just feels like it with all the focus on twitter right now - but anyways, you just see things breaking you normally would not expect to see..
- How people are communicating, or in this case not communicating.
I'm probably not the ideal person to write this down, there was so much stuff going on, some people probably made a whole blog-article series on this. This is just a few of the things that I'm able to remember right now.. and with everything on hn here it gives you some ideas and things to think about. Hope that Helps.
Maybe mister "genius" is working his magic by annoying as many people as he can and making Twitter visibly worse before it is finally great again! So people will finally be amazed at how "genius" he really is.
Why would you stay at Twitter? Even if you believe in the platform and the mission, this has to be one of the most stressful work environments in tech right now.
The only cases I can think of for staying:
- Unfortunate H-1Bs tied to the job. (We really need to make it easier for these folks to switch jobs and keep their visa!)
- A feeling one needs the job and can't easily get work elsewhere. Perhaps living paycheck to paycheck, haven't interviewed in awhile, imposter syndrome... (If you're worried you'd not be able to get hired, instead ask why you weren't fired.)
> Why would you stay at Twitter? Even if you believe in the platform and the mission, this has to be one of the most stressful work environments in tech right now.
I worked for a different eccentric billionaire early in my career. It was stressful, demanding, and unpredictable as he tried to micromanage everything by himself in short bursts of 5 minutes before moving on to the next thing.
The hook was an idea that it was a famous company and he was a [not quite] famous billionaire and that we were sitting on a once in a lifetime opportunity to build our resumes. He was going to reward us greatly at some future date when everything was working well again, and we’d be able to work anywhere else we wanted after this.
Didn’t really pan out for us as everyone got burned out, the billionaire’s micromanagement and constant product churn diminished the company’s reputation, and he eventually laid most people off in favor of even cheaper foreign labor.
But there'll be one or two people who get cozy with Elon and win the lottery and will write articles on how hard work always pays off (neglecting the other 99.9% of Twitter that gets fired when it all gets shipped to India).
Or we could reduce the hoops & cycle time for legal immigration. The US has an opportunity right now to cement itself as the leader in technology for the next 2-3 decades and stave off the impending population growth crisis if we could find a way to enable people who want to live here to live here.
Or we could force these tech companies flush with cash to pay market rate and watch salaries rise or let these companies leave the country if they can’t afford to do business here. These are the richest companies on the planet and they don’t need a labor subsidy, they just want one and are powerful enough to get their way.
I was in the industry before this greedy trend accelerated out of control - I know first hand how much better things were before. I’ve seen a colleague choose death over the abuse the average H1B gets subjected to. The sooner this lie ends, the better for all of us.
I'm not interested in government policy that allows companies to bring in cheap foreign labor to undercut my compensation. There are 330 million people in the us, so if companies need more engineers, they can pay more and train more.
Also, there is no population growth crisis. A decreasing population means less stress on the environment and lower housing costs. Yes, there will be economic effects, but it's hardly catastrophic. Japan is managing just fine.
This is a very xenophobic view on immigration, and arguably the status quo. Past performance does not equate to future results, but I think you can draw the conclusion that the immense diversity that exists in the US compared to "rest of world" is a competitive advantage. What would it mean to lean in?
Yes, there will be economic effects, but it's hardly
catastrophic. Japan is managing just fine.
I'm absolutely unqualified to say if they're correct, but there are a lot of economists predicting disaster for countries like Russia, Japan, and (not far behind them) China who will face this kind of demographic shift. The US is in the midst of this kind of shift as Boomers age through the system, but it is said that it will be less severe (thanks to younger immigrants) and less permanent than some other countries are facing.
Of course, you can find a lot of "experts" saying anything. Economics is an area where I haven't got the chops or the hubris to tell shit from shinola.
> government policy that allows companies to bring in cheap foreign labor to undercut my compensation
They become Americans and rise to your level of comfort and compensation. By working here in the US, they increase the wealth, tax base, and culture of our country. Immigration is a good thing. Skilled immigration even more so.
The alternative is that they stay in their home country, and that country grows a tech industry that rivals the US. Those workers will work for even cheaper than in the US, putting an even wider delta on price and creating an incredible arbitrage opportunity for talent.
Much like the automotive industry, foreign competition will drive margin out of our comfortable tech industry that has enjoyed being peerless for decades.
We're only going to see more `TikTok`s and `Spotify`s arise.
At some point, talent won't want to come to the US anymore. That should scare you.
I say all of this as someone who wants everyone to enjoy wealth and prosperity regardless of where they live. I still want opportunity and the ability to capitalize on it to be accessible to any American that would take it. And for that to continue, we should keep growing our talent pool and increasing the scope of what we can achieve together.
I see foreign competition as an inevitability, but I don't think number of engineers is a decisive factor. China and India have enough engineers already, and have for a while, but other than Tiktok, no real competition yet. Whereas Sweden, home of Spotify as you pointed out, has very few.
Even if number of engineers is decisive, we will never stop it by immigration. China + India have more than 8x the population of the US. We would never be able to deprive them of enough engineers via immigration.
Maybe folks early in their career. It's not like the cachet has totally dissolved because they did employ a lot of top notch people. Also, when they're interviewing for their next job, even if your departure was super contentious, Twitter is a great scapegoat that nobody will question.
"It says here you were terminated for breaking into a vending machine in the lobby and distributing snacks to your colleagues while dressed as Robinhood."
"Yeah it was an act of protest because management threatened to take away our lunch breaks."
Could also be that they have enough money saved up and don't actively need a job, so they're happy to ride this one out until the end and because they don't need the job they don't really get stressed or anything - if they get fired no big deal.
If they actually make it all the way to the end, maybe being the last man standing there actually does have some perks we don't yet know about?
Are you logged in? Twitter seems to be working normally (i.e. it refreshes) where I'm logged into the browser. But where I'm not, I get the error message everybody's seeing.
Definitely region specific I think. It's working mostly fine for me. Links and images are mostly fine except for the errant occasional broken one but even then it's only when opening the full image
I'm struggling to see their POV on that. The deal got them $54 for their options, which is a much better deal (probably by more than 2x) than they would have got anywhere else.
There's plenty of evidence on Twitter, from Musk firing engineers who confront him with straight technical questions, to Musk demanding ridiculous work hours as if it's a sign of strength or loyalty.
I can say from experience it's possible to be in an abusive work environment and not decide to quit (through some combination of unawareness of how bad things are, fear of having given up, not wanting to re-enter the job market, etc)
It doesn't make sense from the outside (or in hindsight), but it happens
> Did you see the photo of the employee sleeping on the floor?
I’ve done this. I had lots of fun—everyone, from the CEO down, was crunching—and became financially independent after it.
It’s usually a sign of under-resourcing. But almost every start-up experiencing unexpected growth should be ready for it. It’s unusual at Twitter’s scale, but closed to mismanagement than abuse.
> I’ve done this. I had lots of fun—everyone, from the CEO down, was crunching—and became financially independent after it.
OK, but this doesn't seem relevant to the situations Twitter's (former, in many cases) employees are in at all. There's no quick path to financial independence working at Musk's Twitter--the company is losing money faster than it was when it was public, Musk is cutting costs everywhere, and there are well-documented instances of employees being shown the door after sleeping on the floor.
In general, I'm pretty sure that in the vast majority of cases, people who sleep on the floor at the office do not end up financially independent for it.
> In general, I'm pretty sure that in the vast majority of cases, people who sleep on the floor at the office do not end up financially independent for it.
Doesn’t matter. In the context of software design it’s a well known trope and most who do it enjoy the work:
Please don’t do that. You deserve better as a worker. If a company is unable to function without infringing on the free time of their workers, that company should not exist.
If a business operates expecting this from employees, that business is being abusive. If a business got into trouble and employees are sacrificing to help it out, that's just an emergency. And it should be exceedingly rare -- if it's not, that's a poorly run business.
Certainly. I've been in a poorly run business that did this, and I quit.
But I don't mind working late every now and during crunch time, it gets paid forward, usually with time off later.
It's about your relationship with your company.
My point was from the sounds of it GP had a good relationship and had fun and no-one on HN should tell them what to do because they don't know the situation.
As far as I know there is shamefully little research on corporate behavior. Most research I hear about has to do with gender pay gap and only in jurisdictions which mandate it (I wonder why this is).
I think our best estimate of corporate behavior is to look at union contracts, and union demands. Almost universally union contracts prohibit workers from working outside of set number of daily, weekly, and monthly hours. Usually there are some clauses for extraordinary circumstances, but I’ve never worked under a union contract which had less than 11 hours mandatory rest between shifts (even in an emergency) which effectively prohibits all nighters. I’ve seen contracts that go down to 7 hours if absolutely necessary (and every hour infringing on 12 hour rest is payed with 90% penalty) but never 0.
Why do unions demand this? My best guess is that this is a common exploitation tactic by corporations, and thusly a common complaint from union members. Thus we can reasonably assume that many (most?) workers who have worked unreasonably long hours have not seen their just benefits for doing so and felt exploited.
Now it would be nice if corporations kept statistics on this, or if there ware regular surveys by governments or impartial research centers so we could get a more accurate picture, but as far as I know, as it stands, this is our best method, and it is not in favor of this practice.
> People should be allowed to work how they want to
I disagree. There is a race to the bottom and a tragedy of the commons. A company could easily exploit this by gamifying the work and hiring workers with addictive tendencies. Most national labor laws and labor contracts set a maximum number of daily and weekly working hours for a reason. Workplace safety is only one of those reasons.
Yes, and if they do, there is a risk of company exploiting that person and their co-workers for underpaid labor. One worker’s enjoyment of work does not justify this risk.
Nobody, but you must see how this can be abused to do so. A system that can be abused will be abused given the right incentives. As it stands companies are incentivized to pay as little for labor as they can get away with, and if they are allowed to “allow” workers to work excessive overtime, they will. And it will cost every other worker our free time.
If you enjoy your work, that’s great, just do your work during work hours, don’t use your free time to undervalue other workers in the industry.
I think we might be arguing about different things here. You’re saying that it worked for OP, that is fine, I don’t disagree with that, however I do disagree with this practice, because as a whole it is not good for us workers in this industry. This behavior is ripe for abuse, and companies do abuse it unless there is a collective action against it (through regulation, union contracts, etc.)
Like you said, if you work for a company which expects this from you, and you quit as a result, they can simply hire someone else that takes the abuse, and that is not a good prospects for us workers in this industry. Which is the reason I’m asking OP not to do this. Even though they had fun and were handsomely compensated, this behavior does a net disservice for the rest of us.
I had equity in the outfit, I made my first million from it, and again, I had fun. If the CEI were at home partying, it would have been a different story. But we were all there. Kidding, coding and trying to keep something we were building together alive.
[Raises hand] I spent countless weekends and evenings in my mid-20s to help the company meet a delivery deadlines (earning the company X million pounds each time) or to untangle some production fuckup (saving the company from having to “best price” trades or incur fines). Claims of overtime shushed and dismissed (“come on you already make more than $colleague…”). Promises of equity that were “delivered” in the form of some kind of ultimately worthless “units” that were granted in place of actual shares (just while the legal side is worked out, you know) and which were eventually swept under the carpet as if they never existed.
Those overnighters and the value they created mean very different things to someone who has equity vs the rest of us. As you can tell I’m a little bitter still, I realise now this amounts to wage theft and violates various protections we have in law - I just wish I had known at the time :(
> you're essentially saying (or it is being interpreted this way by many) "Do the time, it worked for me!"
I’m not saying that. I’m saying doing the time isn’t equivalent to abuse. The facts and circumstances matter, e.g. who is doing it, how often it’s required, and what everyones’ stakes are.
They make for a shitty work environment for everyone - anyone who wants to set up healthy boundaries between work and life would be let go, thus eventually reducing worker rights.
No one is in shackles. They make like $300k in compensation. It's a free country, workers can go wherever they want. Free labor market.
What if, just what if, the workers actually like working hard and making their lives better? That seems to never enter the conversation here.
Honestly, the culture on HN makes workers like me more resentful. It's 100% negative, there is absolutely 0% optimism in these threads. Not even one comment here (or elsewhere on HN).
They all make $300k and can all go wherever they want and do whatever they want? Are you certain that is the case?
> ... like working hard and making their lives better
It's true that work can be very rewarding - "Find a job you enjoy doing, and you will never have to work a day in your life" and all that. Plus who doesn't like making their lives better? I guess it is up to the reader to decide whether a job in Musk-era Twitter is one of those jobs, and whether working for the new "hardcore" Twitter would make your life better. I can only speak for myself but if I worked somewhere and I saw that after a takeover:
- half of my colleagues got fired one day
- some people occasionally get fired for on-the-spot violations like answering a CEO's question
- the company went from nearly-profitable to incredibly unprofitable (let’s even ignore how it got that way)
- the company stopped paying some bills and were kicked out of some offices and services
- the owner-and-CEO had become an extremely ... divisive figure, shall we say
- the owner-and-CEO declares that the existing product is brittle and broken and needs a ground-up "2.0" rewrite (a second system so to speak...)
... then I wouldn't see this as some wonderful opportunity, I'd see it as my workplace being turned into a plaything for a tempestuous billionaire who might not be as competent as he wants us to believe. And if I was a wealthy and financially-secure US permanent resident looking at a buoyant jobs market then I'd probably leave, take a bit of a breather then start looking for a new job at a different company. However if any one of those things didn't hold (maybe I'm an H1B visa holder, maybe I am young and don't have much savings or experience, maybe the job market kinda sucks, maybe I have am the sole earner in my family) I'd be in a much different and more depressing situation - maybe not literally "in shackles" but certainly not as free as you are suggesting.
So they don't realize now they are being abused even though they have first hand experience, but you realize they are being abused because you have what insight exactly?
In other words, what exactly do you think the people working there will "realize years later"?
Could you specifically list the abuses being done to these engineers that they are unaware of?
Usually people choose to stay in abusive situations because they don’t believe they’re being abused, or have convinced themselves they have no better option. That doesn’t change the fact the environment is still abusive.
There’s a whole swaths of science and research behind this. It’s why domestic violence can continue for years unknown to anyone, even from friends, until it’s not.
OP isn’t saying they’re being abused as a fact, but if they are — at a workplace that seems as objectively toxic as Twitter right now — many who are still there won’t likely realize they were until afterwards.
Worth noting Twitter’s status page claims it is “operational” and no events are listed under history section for past 180-days, which is obviously wrong:
Oh man, no way. If that is the case, some people thought I was full of shit when I said, "watch it, in the mass layoffs they fired that one guy who knows all about the certs." There is always that one guy and unfortunately he always gets canned.
Also weird given that Twitter is using 3rd-party service which apparently doesn’t even automatically log and post known downtime issues; service being:
"Some parts of Twitter may not be working as expected right now. We made an internal change that had some unintended consequences. We’re working on this now and will share an update when it’s fixed."
I haven't really been able to read anything on twitter for a while now as I run noscript by default and haven't deigned to whitelist twitter.
I remain astounded by how much their product has declined (even pre-Musk acquisition) from their glory days of being so light weight that they were used to organize protests when governments would clamp down on internet access.
I try to avoid the site as much as possible but I have a redirection rule to go to a Nitter instance in case I end up clicking on a link. The Nitter UI is nicer too and works fine without JS.
Looks like their redirect service for links was down, not twitter itself. I could go to my profile, but clicking links in my tweets gave me an API error. Ugh, not good.
Nitter doesn't use the official API, which seems to be what's broken, instead it talks directly to the undocumented APIs that Twitters normal web interface is built on.
I assume they do it that way to get around all the rate limiting and threat of having access revoked that the official API comes with.
Thanks - we changed the URL to that from the submitted URL https://tweetdeck.twitter.com/. (Submitted title was "Tweetdeck is now broken for normal users")
Followed swiftly by: "We apologize again for the fault in the service. Those responsible for sacking the people who have just been sacked, have been sacked."
That's an excellent question. Having given the matter some serious thought, I think what I'd suggest would be to start by making Twitter better, rather than worse.
I'm not a business genius though so perhaps I'm missing something crucial here.
Honestly I think this point is understated. Twitter (executives at least - but that's how capitalism works in America right now) basically won by offloading a dying platform onto a sucker with more money then sense. If they'd spread the joy around so that everyone who worked there got a golden parachute to glide away on before Musk walked in with a kitchen sink I think I'd be singing praises of them as folk heroes - as it is they're at least assholes that managed to pull one over on a larger asshole.
Where did this meme of them being a cash incinerator come from (other than from Musk, hoping people didn't look into the public statements)? Last time I looked, they had a few profitable quarters on the books. Looking at the last annual statement, I see a 200M loss on 5B revenue, with them sitting on 8B in assets, 2B of which is in cash. That's nothing close to a cash incinerator.
As far as I can tell, tightening up that slightly and reasonably could have turned it profitable again. The only cash incinerator I see in this picture is bull-in-a-china-shop Musk who appears to have made the problem 100x worse.
Everything changed after mister "genius" operated a leveraged buyout of the company, which essentially means that it is now riddled with debt and zero assets that have not been pledged to pay the debt.
They don’t have a quarterly earnings report because they’re a private company now, and their revenue has dropped by 40% according to an email sent from a manager. Ironically, we can only really know their cash position pre acquisition, and the only source for their being a “cash incinerator” is Elon’s word, which to any reasonable person is worth exactly zero at this point.
I only need one example but I have dozens. The only one that matters is he banned journalists from the platform after promising to make it a haven of free speech. It’s objectively one of the more hypocritical things a public figure has done in recent years.
100x was rhetorical. It's not a public company anymore, so we can't know. But if the headlines reporting a 40% drop in revenue are true, that would bring the net loss from 200M a year to 2.2B a year, which would be 10x worse (though I don't even know if you were being that specific in your critique), but that also doesn't take into account whatever he saved from chopping headcount.
> The change in question was part of a project to shut down free access to the Twitter API, Platformer can now confirm. …
> But in a sign of just how deep Elon Musk’s cuts to the company have been, only one site reliability engineer has been staffed on the project, we’re told. On Monday, the engineer made a “bad configuration change” that “basically broke the Twitter API,” according to a current employee.
This is actually funny. Just this weekend I was talking to another engineer friend of mine, and we were taking bets as to how long it would be before twitter went down for the last time.
We felt bad for the engineers that are still there, and were actually surprised it's stayed up this long, with all the knowledge that walked out the door. Basic stuff like "where are our servers actually located".
I guess the folks that left loved Twitter enough to leave behind good notes, because even the most well architected systems still need some institutional knowledge when they go down.
Just had the wild idea that Elon fires anyone who's away from their workstation long enough to be potentially interviewing elsewhere. Just crazy enough an idea to be something he'd pull.
It appears that when Twitter goes down, its engineers were indispensable but when it's up, it's because the engineers foresaw this.
Given those factors, how long it stays up must be an interesting combination of how indispensable the engineers made themselves vs how much foresight they had.
I know some of the former Twitter engineers, and they were great engineers. I'm sure they built a fairly resilient system.
But at the scale and complexity, eventually you hit an unforeseen error that requires some institutional knowledge to fix, and it's doubtful they still have that, unless the ex-engineers loved the product so much they are willing to take phone calls from the people that are left and point them in the right direction.
I don't think we landed on a time frame. I'm actually surprised it stayed up this long.
They do this every time. Last thread I saw was Twitter had an outage for only 10k people for a short amount of time and HN was having a victory lap.
I personally don't think that 100% uptime for everyone around the globe is even a worthwhile goal for Twitter. The incremental costs for that last 0.5% aren't worth the incremental benefits.
HA is most necessary for what, journalists? (Who are already antagonized under Musk)
The problem with bets with strangers is counterparty risk. The counterparty cannot prove that they will fulfill in the loss scenario, which makes it a guaranteed loss. Bets with friends don't have that problem, since the bet is usually either cash-free or for such a small value that retaining the friendship is more important.
We always build systems hoping they never need any maintenance.
The truth is, they often do. Especially when there are changes happening. I once took a week off and we went down on day 2. Other times, I'm convinced I could've gone 6 months without anything breaking.
At Twitter's scale, I'm sure there are systems where everything is automated and self-healing. But there are probably other systems that are more fragile. And then there's the eternal race against scale; eventually a system with no maintenance will get buried under the weight of its own data or the scope of its usage. That always requires some work.
Exactly right, which is why this whole saga is so irritating to watch. The commentary is heavily biased and its exhausting to filter back to neutral. I just want to know the impact of many engineers leaving
You're not going to know that impact, because most of the technical problems that Twitter as a company can have aren't even visible to us.
Sure, images and hyperlinks breaking and not paying the Slack bill and Musk having another mental break makes the front page of hacker news, but will a spam and abuse detection pipeline being silently busted for 3 months do that? A reporting backend that was providing advertisers with incorrect data on their ad spend? Some customer-relations workflow that ruined the week of a group of TAMs, and had revenue impact?
Twitter is a big ship. There are more places things can break than anyone, inside or outside it can imagine. But as an outside observer, the only signal that we can act on is 'Has the ship sunk? Or is it floating?'
Observing that binary state answers a question, but not the question of 'Is the ship working well?'
It seems really unlikely to die from a single spectacular failure, where a key bit of infrastructure dies and just cannot be brought back online for any amount of love or money.
A steady drip of annoying but individually minor issues will gradually chase people away: more spam and less relevant recommendations discourage users; bad spend data and fewer account reps make it a lower priority for advertisers. Twitter's main value prop is its network effect, and if a few people leave (or even engage less) every issue, that value will wither away....
You can never know the true impact because people will spin the narrative to say what they want. This hour-long outage occurred because Twitter laid off their engineers. Meanwhile, AWS had a major outage that lasted over half a day and took down Slack among other major internet infrastructure in December 2021[0], but we can't blame that on layoffs so we just ignore it, I guess.
There's really no correlation between layoffs and outages, I suspect.
What evidence do you expect? It's not like Twitter is going to show us a post morterm from their outage. Anyone who knows and uses twitter quite a bit can see that things have been wonky on the sites. Big users have mentioned tons of issues (their mentions being empty, twitter threads going broken, some tweets never reaching their audience not even a bit), etc, etc. It's all anecdotal of course, we can't build a case of evidence, this is a private company and all the shit internally is private. But things are getting worse, not better. Even those big initiatives announced are not actually happening, that paid and free API plan that was supposed to be released in 7 days? It's been a month and nothing. Things are not "normal" and clearly the lack of engineering talent is slowing a lot of things down (both new development and infrastructure resilience).
Assuming that you are correct that things are breaking more at Twitter (which I'm not sure is accurate to begin with), how do you know that is due to "lack of engineering talent" and not due to higher pressure to deliver features?
As anyone who has operated a software system knows, the #1 reason things break in production is new code, and from a third-party observer with no inside knowledge it certainly seems like more changes are happening at Twitter than ever. This is orthogonal to layoffs, though.
All still present (positive and negative) on Twitter, plus the benefit to open-minded people of hearing opinions from all sides, which has improved post-Elon.
I used to run a bot that broadcasted my town's current alternate side parking rules (for snow removal operations) but after twitter got weird the town decided to make a text messaging service for it.
I still use it to follow certain artists but if that subcommunity moves I'll follow them elsewhere.
Local status updates and the occasional tidbit of interest from people whose thoughts I like to read but who twitterblog instead of using an actual blog.
That second one, poking fun at the BBC for saying that Twitter "can't protect users from trolling," was really the BBC shooting itself in the foot with a laughable headline burying the lede of the article:
> Twitter insiders: We can't protect users from trolling under Musk
> Twitter insiders have told the BBC that the company is no longer able to protect users from trolling, state-co-ordinated disinformation and child sexual exploitation, following lay-offs and changes under owner Elon Musk. [1]
For some reason, the headline decided to focus on "trolling" over an increase in child sexual assault and state-co-ordinated disinformation...
Honestly the "insider" even talking about "protecting" people from trolling and state co-ordinated misinformation(?) in the same sentence as protecting them from csam makes it very hard to believe it's something genuine.
Sounds like the people who say all the petty things they dislike about someone on the internet and then finishes with "he's a pedo" to add gravitas.(yes elon does fall into this category)
“I would expect to start seeing significant public-facing problems with the technology within six months,” he says. “And I feel like that’s a generous estimate.”
A certain percentage of engineers have to keep the lights on. It's usually around 10%-40%, depending on staffing levels and engineering quality.
But there's one more gotcha, a company can't just keep 10% - 40% and expect the service to still run, because it's rotating and oncalls can always reach out to other people on the team.
Twitter not only fired indiscriminately, with perverse pattern (more capable to find another job leave first), and didn't even keep the minimum to keep the lights on.
From a software engineering perspective, it's doomed to fail, maybe it can retract to a smaller footprint (abandoning features/stacks) and still work, but that would also be pretty hard. That, and I don't think anyone not desperate will apply for a job there in the foreseeable future.
> If these systems ever go down, its link rot at scale
Only if one's mental model is that t.co links were ever permanent.
They're privately-owned obfuscator links. Unless someone in the wild was building a mirror mapping, they were always doomed and never to be trusted; they knocked the "U" right out of the "URL."
That's what I'm saying. We must reframe our mental models so that everything on Twitter is thought of as temporary. None of it is, or really was ever, stable.
It's even more-so true for t.co URLs though because instead of just 1 point of failure there are 2 points of failure. The original URL may work for another decade but if you don't know what the URL is behind the t.co URL and Twitter is down or whatever, then you're just out of luck.
I don’t think it necessarily changes the reliability of the URL. A server or dns change is similarly problematic.
The only issues I have with shorteners is that they obfuscate what you’re accessing (so if it changes, it’s harder to know) and they’re a privacy issue
Things fail slower then faster than people expect.
Lots of people were waiting with baited breath for Twitter to fail the first week Musk bought it, but most of the failure conditions had been automated away...
Instead the reliability loss creep happened more slowly. The people that understood the edge conditions in the system were fired. Then new changes need implemented and the old, now not understood systems, were ripped out and new pieces put in. And lo and behold they have tons of failure conditions and edge cases that can knock out the system.
Even if your programming team wrote immaculate documentation on all these edge conditions, it can take an immense amount of time to both read them, and then fully understand them, and when you have a micromanager breathing down your neck for changes 10 minutes ago this is what happens.
I don't spend enough time on Reddit for that to really resonate, but the badges and icons flying all over the place and talk of Twitter Gold (or whatever they're calling it) do feel very Reddit-esque these days.
I’m in what is likely the top 1% of Reddit users by volume (I have a problem, send help), and I have absolutely no clue what you’re talking about.
Further, as a dev of 15 years, I’m not even sure what you mean by “crash” here. Are you saying that Reddit serves exclusively 500s or timeouts for a sustained period of time on a weekly basis? Or that it serves more than zero 500s at some point during a given week? Or that you, personally, encounter an issue of any kind over a week?
So many ways to describe “crashes” that yeah, by some definitions it probably does, but “can’t use the service” such as what Twitter is experiencing, I don’t think that’s accurate.
One of the triggers used to be that if a thread got too many comments it would bring down the website. So sports communities had to start splitting up their game threads during massive events so Reddit wouldn't break.
It definitely used to happen more often, but it was doing it for me yesterday coincidentally.
> And when it's not crashed it's slow as cold shit.
And stale. When they changed algos in ~2016(?) to try and stop certain subs I swear it got stale and feels like it updates on a cron job once a day anymore.
It's kind of funny that another thread (https://news.ycombinator.com/item?id=35043330) is in top 5 Google results for "Your current API plan does not include access to this endpoint" search term, and is flooded with newly created accounts asking for support.
> half the time stuff isn't loading at all or is broken in obvious, overt ways
I use the app for a couple hours a day (according to my iPhone weekly reports), and have barely noticed any degradation since "the takeover". (Notwithstanding the obvious outage happening currently).
There's been a few minor glitches for sure, but then what large service is immune? Facebook, Exchange, AWS, etc. have all had notable outages this past year, and the Apple Services I regularly use (Apple Music, Siri, etc.) "fail" more often for me in daily use.
Maybe you're trying to do something more advanced than me, or are using a less reliable platform somehow, but generally I find all the "Twitter is so dead" talk being confirmation bias from folks that really want it to fail in light of recent changes.
I've never seen a large company whose entire existence is a single app perform so poorly. It was down for me a whole day a week or so ago. Bugs are pretty rampant, they're small, but didn't happen to me previously. It's just been gradual degradation for months now. Idk how you aren't noticing.
> Maybe you're trying to do something more advanced than me, or are using a less reliable platform somehow, but generally I find all the "Twitter is so dead" talk being confirmation bias from folks that really want it to fail in light of recent changes.
In my ideal world, Twitter - or something similar - succeeds. I don't care much about the politics of the ownership because that's generally going to be an iffy subject with anything coming from the valley.
But to regularly use it and not see things breaking here and there every day (not even including the times when the API is borked or it's only recommending Elon Tweets in your feed or a person's timeline won't load no matter what, not being able to send a tweet, etc.) ... I have to imagine you're very lucky. There was about 11 hours when people had exceeded their daily tweet quota by trying to send a single tweet. You don't remember stuff like that? It happens multiple times a week in the last few months.
I'm certainly not doing anything advanced. I'm mostly a casual user.
I can corroborate. Videos struggle to load, timeline doesn't refresh unless you pull down multiple times, my current position gets lost multiple times forcing me to scroll down and then scroll back up, tweets showing up multiple times, many more ads than before and so on
It's really been kind of miserable to use the app and I only open it once or twice a day. It had issues before but never all at once like this
Elon's own account is that he's "experimenting" with taking production services offline to cut cost and simplify their code base. Essentially a literal Chaos Monkey. With this in mind, service degradations aren't surprising.
It not just the layoffs it is also the major changes Elon is forcing through at high speed .
Hiring for twitter is going to be quite expensive, nobody is taking this up without huge payoff and probably upfront given Elon pre disposition not to honor severance etc
As someone that I think the previous twit sphere would accuse of being right wing adjacent, I’m shocked at the dumpster fire that twitter has become from a content standpoint.
Like.. wokeness was annoying but the trash heap of craziness it is today is 5 steps backward. My feed is nothing but conspiratorial nonsense.
If twitter was “tilted left” before, today it seems to have capsized rightward.
Right, it turns out that bringing back a bunch of previously banned trolls will chase more centrist people away, leaving only the trolls and those who like to argue with them. Not sure if there have been algorithm tweaks as well (aside from the obnoxious removal of the sort by latest button Elon swore he'd keep), my feed is the same. Combined with the stupid "Trending" tweaks and it's a worse version of Reddit.
I honestly can't believe that not only did he welcome back actual Nazis that were previously banned, but he sold them checkmarks that let them have outsized influence.
The Twitter suicide is the most crazy thing I’ve seen in my career. Rich and bored billionaire bought a “socially valuable service” just to drop its revenue by half and destroy it completely
I still don't understand why every fifth post in my feed is an Elon tweet, I don't even follow him. I could block him but I find it interesting to see how much of his content is pushed toward me who hasn't asked for it.
Maybe better to block him. I did so because I was a bit sick of seeing his tweets - some of them are borderline offensive, some are him cosying up with far-right but the majority are just a bit try-hard, unfunny and ultimately sad.
I'm not a delicate flower when it comes to offense so I'd just roll my eyes when the first two kinds of tweets popped up. But seeing a grown man repeatedly trying and failing so hard at being a poster is a bit much to take.
Shameless plug: I created a browser extension to help transition to Mastodon[0]. If you don't yet feel like you can leave twitter.com, but want to explore alternatives it's a great way to get started. Essentially it injects Mastodon posts into your Twitter timeline, so you can retain your existing Twitter following while getting exposed to Mastodon.
That will happen the same time email "does something about that". It's an open standard. If you join a server that doesn't federate with servers you don't like, or you just don't follow accounts that post things you don't like, you won't see things you don't like.
My question for you is... what the heck server are you on, and what accounts are you following??? I've never seen a single thing you've mentioned on Mastadon.
Maybe just don't follow people who post that stuff/turn off boosts/block furry instances/etc?
This isn't like Twitter, where it fills your feed with shit it thinks is relevant. On Mastodon, you have pretty much total control over what shows up in your feed.
One of the criticisms against the old twitter engineering team was that they failed to set up proper dev and testing environments. A leaker exposed all of this and it was later confirmed by Musk in a Twitter Spaces call.
On an unrelated point, Californian employees in the main layoff were employed till January 4 for legal reasons. Many of them were demanding their contracted 2 months' severance to follow that. Now that 2 months have just passed, does anyone know if they received this pay?
No... he got the class action thrown out because of an arbitration clause... so each individual case must go through arbitration, and those are all still pending (when I checked a week or two ago).
There were many different hypotheses on the outcome of Elon firing 80%+ of Twitter employees:
1. The site would crash and burn the next day.
2. Nothing would change. All those engineers were anyways just sitting around.
3. There would be no immediate impact (servers can run by themselves after all), but the site would slowly degrade over time as institutional knowledge around maintenance, upkeep and all the various system quirks was gone.
Yes, but my recollection is that the whale fail was usually due to huge growth or unprecedented traffic peaks. I don't think Twitter is growing at a fast rate any longer.
Rails was never meant to be run on such a massive load. It was the perfect framework to get twitter take over the market but once that goal was reached a rewrite was unavoidable with a tech stack that can sustain such a load.
"Record high" in an established business like Twitter isn't quite the same thing as a traffic spike in the growth phase of a startup (i.e. Twitter in 2011).
Record engagement these days is most likely measured in single digit percentage points over average engagement (at most maybe 10-20%), where when you're early and your traffic is spiking, it can be an order of magnitude more traffic than what you're used to seeing. The absolute numbers are larger yes, but from the perspective of "what can my infrastructure / architecture handle", relative changes are far more relevant.
> Musk claims often that Twitter engagement numbers are at record highs. I don't believe these are lies.
You're talking about a man who has more than once been fined by the SEC for making false statements about public securities. What makes you think that he'd be a more reliable narrator for a private company for which he has no external accountability and a large personal financial interest?
> At the very least per tweet it’s possible to see really view patterns, which to my knowledge what’s possible before.
Those numbers have been proven to be incorrect, by people who are posting from locked accounts with 0 followers racking up large nonzero view counts within a few minutes.
Some part of my brain immediately replied to this with the theory "what if those view counts are real and Twitter's just kind of ignoring whether an account is locked now" and damn that is a hell of a possibility to ponder.
There's another plausible theory that the just implemented it by counting database reads. So of course a locked test account will have more reads every time you look.
> There's another plausible theory that the just implemented it by counting database reads. So of course a locked test account will have more reads every time you look.
It'll have one, maybe two more reads every time you look. Not dozens. One person posted an example that had literally over a hundred alleged "views" within a minute, despite not refreshing their timeline after posting the tweet.
The fail whale was very common during a time of sustained growth when Twitter did not have the infrastructure it has today, but it all but disappeared in recent time.
The fact that things start to fail after that infrastructure has been put on place and running find for a while is noteworthy.
The fail whale was also a very specific case - I have no idea what was going on in the backend, but from a user perspective it was all "twitter's not available". Now we have inconsistent behaviour and countless small bugs that detract from the overall experience.
I feel like I'm living in an alternate reality. People seem to be acting as if Twitter was good prior to 2022 until Musk ruined it, but my experience has always been that Twitter is awful, both from an engineering and content viewpoint.
Maybe it's because I use an Android, but I've always found the experience terrible. I get notifications, but when I click them and they don't take me anywhere; the back button is completely broken; mentions randomly don't produce any alerts; it resets me to the algorithmic timeline every few days; replies often fail to load; the spam problem is completely out of control and most adverts are irrelevant crypto-spam; search never finds what I wanted; it's constantly putting tweets on my timeline from people I don't follow; and the whole app seems optimised to produce as much hatred and outrage as possible. Nowhere on the planet is as toxic as a Twitter thread, and yet the media and political sphere collectively decided that reporting on and catering to Twitter spats is the most important aspect of their career.
Frankly, if Musk destroys Twitter, it might be the best thing he could do for society. I may be misremembering, but I thought that prior to Musk buying Twitter, everyone was decrying it as toxic and lots of people were suggesting the government should step in to reduce its influence. Now it seems to be imploding by itself, yet everyone's upset.
4. Something in between? In short term we'll see some issues but overall nothing important will change and these downtimes and small issues are a justifiable cost for letting such massive cut in the payroll.
lol, I guess every tech company that has major KPIs for availability is wrong and people are totally fine with apps being down constantly, and new users will totally come back after trying to signup and failing. It's a tech company, a few thousand employees supporting perhaps hundreds of millions of users is pretty standard economies of scale.
What keeps Twitter up as a major social network is not %99.99 uptime. It's strong network effects in play.
What will sink Twitter is not %99.8 uptime. It's those network effects fading away.
For example, if people stopped using MySpace, Facebook, etc it was not because of their availability.
Again, I'm not saying this is cool. Im just saying if that extra %0.2 downtime (in short term) costs you 80% of your work force costs, a business manager may consider that acceptable.
>these downtimes and small issues are a justifiable cost for letting such massive cut in the payroll.
Well we're not even a year in and these "small issues" seem to be repeatedly occurring. Let's get a little further out with some more stability before we call them justifiable lol
I think 3 is happening but the thing is that can this HC-twitter build knowledge and code back so that site functions stay about same before users leave is the real battle.
Also #4 they are shipping features much more than before. Shipping more always will likely lead to more errors happening. IMO it's a combo of #3 and #4
4. The institutional knowledge is not gone, in the event of any massive catastrophe employees who previously worked at Twitter can return as “consultants” and charge very high rates to fix issues as they occur. It’s pay as you go.
> 4. The institutional knowledge is not gone, in the event of any massive catastrophe employees who previously worked at Twitter can return as “consultants” and charge very high rates to fix issues as they occur. It’s pay as you go.
What you're describing is the loss of institutional knowledge with a fantasy tacked on the side.
1) There's no reason to expect those employees would come back, even at rates far higher than Musk is likely willing to pay. Everything indicates Musk has destroyed a lot of goodwill with his shambolic layoffs, and getting people to come back is one of the situations where you need goodwill.
You're basically talking about gig-work on a shitty boss's terms. Only desperate people would play that game, and the people Musk would need probably aren't the desperate ones.
2) Any employees that come back are going to be rusty and lack the institutional of what happened after they left, so their effectiveness at immediate fire-fighting will be greatly reduced.
3) If there's a "massive catastrophe" at Twitter, do you think they even know who to call back? I wouldn't be surprised if they've lost a lot of the institutional knowledge about who knew what, at this point.
> return as “consultants” and charge very high rates to fix issues as they occur.
I'm sure there are some people who would do that, but those opportunities become very expensive-to-impossible when you burn bridges the way certain new management has. People hold personal value on things like respect and principle that can make them behave "irrationally" in naive analyses like this.
Massive catastrophes are averted by people with institutional knowledge correctly adding to the hardware/codebase over time, with new features that integrate well with old one, knowing how to scale hardware correctly, patches being applied, etc.
Not by bringing people in during a fire and pointing into the smoke and saying "fix that!"
Most of those people will presumably have other jobs now, which would tend to preclude this. You do _not_ want to be in the position of begging employees you fired to come back as consultants; it's not at all as easy as you seem to think.
Interesting to me that when Twitter was ascendant the same predictions of doom were not made so frequently when the fail whale was a regular occurrence.
It's possible Twitter didn't need to be spending all the money they were to be maintaining five nines or whatever, given the shift in product focus (humans and not a general/universal message bus - bots can't @-mention anymore, etc).
Given that they are unprofitable, if I were a shareholder I would be upset if I did not see experiments being run to see how much cost can be cut and where. I doubt people will abandon the platform en masse over a few minutes of downtime here and there. They will lose 100% of users and be down forever if they don't stop the bleeding, however.
> Given that they are unprofitable, if I were a shareholder I would be upset if I did not see experiments being run to see how much cost can be cut and where.
Twitter will never get to $44bn market cap again (much less a valuation with a decent IRR) by cutting costs alone. They need to grow way more than they need to cut costs.
If any of those experiments lead to less growth (which I’d argue is true of a broken Tweetdeck), then they are wholly not worth it.
If I were a former Twitter shareholder, I'd be extremely grateful to the board for selling the whole thing for 44 Bn. No way tje company would have reached this evaluation without Musk. And Musk knew it, he just signed himself a contract he couldn't het out of.
So, shareholders got paid the pre-downturn price after the downturn? At a time big tech went down double 10+%? Sounds like an incredibly good deal to me.
> Interesting to me that when Twitter was ascendant the same predictions of doom were not made so frequently
... wait, they absolutely were, though? I remember at the time people saying that Twitter was doomed for this very reason, by analogy to Friendster, an early MySpace-ish/Facebook-ish thing which was never able to stay up.
It was a pretty common belief at the time that unreliable Twitter would be replaced by Pownce/Google Buzz/something entirely different.
We're just pretending it doesn't now take the better part of 10 seconds for twitter.com to load now? That response times have gotten perceivably, measurably worse over the last few months? Or that these glitches aren't now a weekly occurrence?
I don't use it anymore but the odd time a friend will send me a link to it, and it is horrifically slow. I wonder if anyone will write about this time at twitter in a few years. Would be a case study in how to decimate a websites popularity and what not to do to avoid that.
"Meet the new Boss, same as the old Boss" -- and that's a slam on Elon who promised a ton of things like open sourcing the algorithm, etc. All I get now is a bunch of right-wing weirdo crap in my feed that I didn't subscribe to or ask for.
And this thing he was complaining about back in May, he completely f'd it up post-purchase:
This is one of several different glitches and downtime seen this year.
A revenue collapse of 40% with a surge in fake verified profiles does not make Twitter better than ever. It is crazy to see a sudden sharp 40% revenue decline being described as better than ever.
While I've no trouble believing revenue dropped, where are we getting this 40 percent number and surge in fake verified profiles data from? With Twitter now in private ownership, Musk doesn't need to disclose real data and to best of my knowledge hasn't really.
I blocked him and I don't see his posts any more. What did you try to do?
I'm the opposite of an Elon fan, but so many people don't seem to know how to actually use social media. Who/what you follow and who you block can completely change your experience.
Social media growth is all about recommendations and algorithms, people actually curating their network a la 2010 facebook is dead, so your idea of "how to use social media" is also aligned with completely not knowing how to make a growing social media platform where almost >99% of users are consumers and automatically are recommended things and curate their experience implicitly. People do not want to manually do anything for the most part.
The general issue of forcing recommendations and the general population not doing anything manually anymore is separate from the fact that as an individual you have lots of tools to curate your feeds on twitter/facebook/etc.
It's also just really silly to have conversations like:
You're missing the point, recommender algorithms are supposed to show people what they want. Even if what you said is true, the experience is absolutely doomed, because its a shitty UX
I thought I scrubbed things by hitting "I am not interested in Elon Musk". Prompted by your tweet I checked what it says, and the offending post on my timeline has an option "I am not interested in Business People".
So that's probably how it got through. I still stand by my grossly hyperbolic diatribe about how twitter is definitely getting shittier, though.
> How is it better than ever? I told the damn thing I have no interest in seeing ol' Muskrat's posts, yet there it is on my timeline: [0]
I bet that's by design. His posts weren't getting as much engagement as he thought they should, so his tweets now get an artificial boost to "fix" the algorithm.
Just top 3 Mastodon instances(mastodon.social, pawoo.net, mstdn.jp) collectively has > 2mil users, same level as Twitter in 2008. Misskey, an ActivityPub/Mastodon compatible server software, now has >10K users. Nostr isn't much, Bluesky is still invite only and had just went past 2k registered, but those are all available options for direct substitutes to Twitter in case anyone still in need of a legacy twitterform social media.
> Just top 3 Mastodon instances(mastodon.social, pawoo.net, mstdn.jp) collectively has > 2mil users, same level as Twitter in 2008.
You really think 'pawoo.net' and 'mstdn.jp' are fabulous advertisements for Mastodon of over 2M users on those instances, unless you think CP and loli culture there is a great thing since it is totally illegal in hundreds of countries?
Secondly, both pawoo.net and mstdn.jp instances are owned by a single company called Mask Network [0], which essentially means it is not community owned and is under central control of a company that can do whatever it pleases, even if the users on the platform dislike it.
Finally you missed out another large Mastodon instance after mastodon.social which is braag.et which promotes CP and illegal content. So it seems that the instances with the larger amount of users also allows CP and illegal garbage on it.
It's quite hilarious to believe that Mastodon with these examples remotely has a chance against Twitter as an alternative.
> You really think 'pawoo.net' and 'mstdn.jp' are fabulous advertisements for Mastodon of over 2M users on those instances
Yes.
My understanding at this point is that, firstly, instances that do not federate with these, and those cheap “CP” accusations, has no relevance in this discussion, because nothing comes out of those.
Secondly, it seems completely plausible to me at this time, that a central Mastodon instance will become the core of a Western European social media landscape, in the way this wartime Russian Federation maintains VKontakte, or that Communist China maintains Sina Weibo. And Misskey could be another such for another region, each contributing to slow segregation of the Internet at religious-social-political interfaces.
Such violent responses that you have expressed against the names of .jp and Pawoo is plausibly an indication of those developments.
> My understanding at this point is that, firstly, instances that do not federate with these, and those cheap “CP” accusations, has no relevance in this discussion, because nothing comes out of those.
It is totally relevant since not only you just mentioned both of those instances, but they also have over a million users there which CP and loli culture is pervasive on those instances and hiding the problem under the rug only makes Mastodon a far worse place, given that the majority of those instances is full of that CP content which is completely illegal in lots of countries and is hardly appealing to normal users.
> Secondly, it seems completely plausible to me at this time, that a central Mastodon instance will become the core of a Western European social media landscape...
Right, eventually defeating the whole point of federation with more re-centralization and co-ordinated instance-level banning and you advocating a centralized government controlled social networks being much better. What could possibly go wrong? /s
> Such violent responses that you have expressed against the names of .jp and Pawoo is plausibly an indication of those developments.
What "violent responses"? The basic facts about Mastodon may indeed hurt, but you advertising those two instances that have over a million users using it for loli and illegal CP content doesn't put Mastodon in a good light and makes it completely unsuitable as a 'direct substitute' to Twitter.
> given that the majority of those instances is full of that CP content which is completely illegal in lots of countries and is hardly appealing to normal users.
I say they can totally beep beep beep about that. I don't advocate violence and/or harm, but that's the Internet. If that isn't appealing to normal users, the Web 2.5.x and social media in their entirety are not for normal users, which it probably is a correct observation given recent reports of mental health issues skyrocketing. But I doubt Web3 or later will be any "better".
>> Such violent responses
> What "violent responses"?
.jp, Pawoo, Misskey at this moment are kids running around. I would think something very similar can be said about *chan, though personally they creep me out. Either way, it's not a horde of groomers conspiring over Tor. When a bunch of smart people actively tries to reframe and crush kids running around as "dark web illegal CP sharing activities left unsolved in un-enlightened countries" that's a violent response.
With the continuous growth of streaming and video social medias and looking at what are trending there, I doubt the adoption of so-called "completely illegal in lots of countries and is hardly appealing to normal users" content will even slow down. It will completely normalize, and if it did not in the EU, the EU is going to isolate itself leading to further fragmentation of the 'net scape.
> and you advocating a centralized government controlled social networks being much better. What could possibly go wrong? /s
Oh, forgot this part. I think it's within possibility that more municipal level governments start hosting ActivityPub servers, irrelevant to whether it's idealistically ideal, in case there will be significant numbers of entities that can't migrate to Insta but actually needed non-Twitter-Twitter.
> If that isn't appealing to normal users, the Web 2.5.x and social media in their entirety are not for normal users, which it probably is a correct observation given recent reports of mental health issues skyrocketing. But I doubt Web3 or later will be any "better".
Who mentioned or is talking about Web3 here?
> .jp, Pawoo, Misskey at this moment are kids running around. I would think something very similar can be said about *chan, though personally they creep me out. Either way, it's not a horde of groomers conspiring over Tor. When a bunch of smart people actively tries to reframe and crush kids running around as "dark web illegal CP sharing activities left unsolved in un-enlightened countries" that's a violent response.
It takes me ONE second to link to open evidence of CP and loli content on the TWO original instances YOU promoted. If you're going to mention another instance; I might as well mention 'baarag.net' again. Since that is an equally large Mastodon instance in the millions and is one that you know that is not 'kid' friendly which the content hosted there is unsuitable and even illegal in the majority parts of the world and you know that.
> With the continuous growth of streaming and video social medias and looking at what are trending there, I doubt the adoption of so-called "completely illegal in lots of countries and is hardly appealing to normal users" content will even slow down. It will completely normalize, and if it did not in the EU, the EU is going to isolate itself leading to further fragmentation of the 'net scape.
By allowing loli and CP content for all normal users? You do know that there is a reason why the majority of countries that have CP laws have made it illegal?
> Oh, forgot this part. I think it's within possibility that more municipal level governments start hosting ActivityPub servers, irrelevant to whether it's idealistically ideal, in case there will be significant numbers of entities that can't migrate to Insta but actually needed non-Twitter-Twitter.
Governments go where the majority of normal user voter base is and that is on centralized social networks like Facebook, Twitter, Instagram and TikTok. They don't follow where techies, furries or loli and CP fans are hiding.
You keep repeating that in multiple threads. I've been on Mastodon for 5 years now and i've yet to see any. I'm not saying there isn't anything, but "flooded" it isn't.
They're repeating it because it is the truth you don't want to hear or fix and that's the problem.
It has been 7 years since Mastodon has been around and it is no better than where it was before since little to no-one has bothered to stick around or even know where to sign up. Twitter had beyond 10 million+ daily active users in 7 years, yet with Mastodon it only has spikes of users that don't stick around for years.
Like Facebook, it looks like the Twitter network effect has never been more strong; even when it goes down.
I've seen two or three bursts of the porn popping through, mostly communities moving on to Mastodon from twitter that get pretty immediately defederated / blocked.
For the former, the instance you're thinking of is on _all_ the blocklists; if you're seeing it, you're on an instance that basically doesn't do blocking at all.
For furry stuff... I mean, you realise Twitter is full of it too?
Twitter was definitely better at keeping the weird porn siloed off the front pages, but this is a weird thing to twig on given the massive number of commissioned "art" collectives drawing naked fox people on twitter?
It does appear that the undocumented api that nitter uses still works and is unblocked by this change as I've been able to access twitter through nitter.
No, it was also working with the self-hosted nitter instance I was using which should have almost nothing cached. Double checked with a new tweet posted in this thread.
I'd never used tweetdeck prior to now. Yet, it just now loaded fine for me. Once it loaded, I clicked a button that said something like, "setup some fields", and then tweets were displayed. What's not working? I'm just a free user of twitter.
It feels like we're being deliberately herded into Mastodon.
Obviously, that's paranoid thinking. But the results are the same in many cases.
A market for lemons developing. A reinforcement of the strategy for others. And more stressful things in our lives.
If people are resentful, I understand. I hope they can find peaceful communities and safety wherever they go. I'm sure there will be political or technical reforms to accomplish that.
I hope people remember in the end that there were beautiful and good people in different spaces on the Internet, including Twitter.
I was literally adding a 2FA authenticator app to my Twitter accounts that aren't Twitter Blue accounts when this happened. I thought it was something I did.
Good to know that I can rely on H/N to tell me what's really going on... and it's not "just me".
Twitter went down constantly in its early years and did fine; as long as errors get fixed, I don't see why that dynamic would spell doom for Twitter now. There are a lot of people for whom the idea that Twitter can get along without most of its former headcount is a bitter pill to swallow, but that doesn't make it untrue.
One reason the site was stable pre-Musk is that no one made any changes to it. For better or worse, it's now owned by someone who actually uses the product and has ideas for how to develop it.
Yeap. Most people commenting here are just bitter and are looking for any excuse to confirm their biases.
From a business perspective does it matter that Twitter is down for a moment? It's not a plane or medical machine. It's not blocking anyone from doing anything useful. Everyone are just going to check & post when it is up again.
It also doesn't mean that Twitter employees have a bad work experience like most here assume. Maybe they've got a permission to iterate fast, even if it risks bringing things down, and everyone are having a jolly good time, skipping on the boring chores ensuring everything will go smoothly usually takes.
Seems to be affecting web view but not mobile clients. Wonder if this has anything to do with them not paying their AWS bill as reported the other day.
Wow, I have never seen the Twitter front-page before.
What a colossal piece of shit. Everything there is information-free and radicalized. The only maybe useful thing is the result of some game, that I assume it's soccer. (But British soccer, why would I care about British soccer? The rest of the page is localized into Brazil.)
Even before the current downtime a string was data-mined from the TweetDeck beta which referred to itself as "a feature of Twitter Blue" so the days of being able to use TD for free are probably numbered anyway.
Does anyone know of time-series data with Twitter incidents/up/downtime? Would be interesting to do an analysis of the service level and change point analysis wrt the firing.
This happened right as I was using Selenium to log in to Twitter for a project. Was wondering what the hell I was doing wrong until I saw this and realized it wasn't just me.
Not that I particularly like Elon but people are now making a big deal out of every twitter outage as if twitter didn't spend down half of the time before Elon bought it.
I've become accustomed to the fact that it's beyond clear that pre-Musk engineers pushed changes to the servers live without testing them first, and didn't even bother doing an incremental roll-out either. Either they had no engineering culture, or that culture was gutted long before he arrived, because Twitter had problems with frightening regularity.
Every speed bump and hiccup of Twitter's outages since the acquisition is now magnified by the new standard expectation of 1000% uptime by the schadenfreude brigade.
Before it is all '#hugops' and 'best wishes to the engineers at Twitter' when it had downtime before the acquisition. But as you can see, they all love to hate it because a person called Elon R. Musk owns the blue bird site.
Just like when everyone here thought that Facebook, Instagram, WhatsApp all collapsed and went down for hours it was declared as the 'end of times' the 'death of Meta', etc, with it's network effect still unaffected by that.
The same thing is happening for Twitter as even when it goes down, it's network effect of 220M+ users is still sitting there waiting for it to get back up.
The former you said is in no way implied within this post or its link.
The latter is a kind of anecdotal evidence that does have no value in discussions like this.
The linked post is from the official Twitter account stating that things are broken and it is for many people.
If you don't see it this could be due to many conditions on why it might seem working on your device/client/account.
This does not lead to any logical conclusion that these information might be wrong.
At Twitter's scale with the number of API users they had, I can't imagine how they'd filter the signal out of the noise.
This is very much a "someone would have had to know we're using our own API to avoid this disaster" scenario, which is an institutional knowledge problem. The kind of problem that gets hard to solve when XX% of your workforce suddenly left to seek new opportunities.
Ah, good point. Twitter should have been the largest caller.
That 467 error code suggests to me something else is going on and Twitter is emitting the wrong error message (467 isn't a standard HTTP, so I'm wondering if something lower-level emitted a 467 error, it was misinterpreted as an HTTP status, and the error string chooser logic is giving that API access error message due to fall-through).
This isn't "you can't run Twitter with half of its 2021 headcount".
This is "Elon Musk is deliberately mandating bad software engineering practices". I will decline to speculate whether that is out of malice or incompetence.
twitter is only important if your stuck in the bubble of twitter people. Tons of people dont use twitter, let alone even write on there. to get even like 1000 replies is still an insignificant amount, even though it feels like a lot. there are 300million people in just the US. thinking twitter replies are meaningful or represent enough people to have discourse is disconnected from the real world.
why wouldnt you just use their mailing list? ive tried this, to keep up todate with and theres so much uncontrollable noise anything important gets lost.
Because email has been so thoroughly decommoditized by Big Tech in the name of "fighting spam" that trying to receive email successfully feels like a trip to the roulette wheel?
This is more disconnected from the real world than Twitter, because you're focusing solely on a US-centric viewpoint. There are plenty of other places in the world where Twitter has been a core piece of communication.
you want me to just take your word for that? or are you just going to point to arab spring or something? twitter was only more popular and used in japan then the us. but use is overall down. facebook seemed to have penetrated a lot more then twitter around the world
instagram seems to have much better reach then twitter could. throw away updates are popular with stories. i see more people putting effort into instagram then twitter. some dont even have twitter accounts
This is the sad thing. I don't even use Twitter, except when a site is down, I go to the company's twitter to see if they're saying anything about it. Can't do that this time, I guess.
Yes, it is a bummer. Twitter was far from perfect, but it was the only large social media platform I felt took its role as a publisher at least kind-of seriously and worked towards clamping down on misinformation and hate speech. Musk spending truly unimaginable quantity of resources in order to turn the only kinda-sorta-semi-decent social network into yet another hands-off also-ran is pretty devastating. Here's hoping for its swift fall so something better can fill the gap.
Y'all are silly and insufferable. Twitter had a temporary outage that was quickly rectified and you act like it’s some massive catastrophe directly caused by his highness Elon. Like you’ve never pushed a breaking change before…
I won't argue that firing all the engineers so swiftly wasn't a mistake (I mean I _could_, but don't want to dig myself any deeper than necessary in a single post). What I will say though, is that this gives me hints as to the quality of engineering that went on at Twitter.
I feel like some stellar Rube Goldberging went into solving problems that could have used simpler, if not _less CV boosting_, solutions. I've been at places like this. The engineers have fun building convoluted "solutions" for the sake of saying they worked on cool-tech-x. And surprise, it needs endless maintenance.
