* Apply security patches - there's thousands of dependencies.
* Manage hardware/cloud resources according to volume of data you need to handle.
* React to the changes in operating systems/browsers you run on.
* Fix bugs - they are there, because simply there's not enough time, money, and need to write "perfect" software. Competition will not do that and will beat you to the market.
There's no internet connected software, that can "just run", because world around is changing and you must catch up.
There are also integrations with partners, new regulatory requirements, changes to support ownership's new priorities (and to undo the old priorities).
I don't work at Twitter but these things happen at every business.
This is all needed to continue to iterate on software. It’s not as needed for the ossified state Twitter has been in for many years, as the parent pointed out. Acknowledge there is _some_ work needed, maybe refreshing certs (though software systems I have built have always been set up to do that automatically), apply security patches, keep an eye on dashboards. But this is a job for a skeleton team, not 7500 engineers or whatever. It’s honestly kind of sad it took that many people to basically “keep Twitter up” all these years.
I believe that 7500 is total headcount, not just engineers, but yeah software development can be terribly inefficient, when all problems are solved by throwing money at them, like VC funded startups like to do.
We'll see what happens now, when there seems to be a drought.
To be clear - I don't argue that it's impossible to run Twitter with smaller crew. Hard to say without a look at the internals, but if I had to bet - yeah, probably there are inefficiences. But OP was coming from "you can leave it and it will just run by itself" angle, which is very much different thing.
What your saying is true for almost every web app. I don't see any reason to justify Twitter's complexity that requires 1000 engineers.
Scale is such a weak argument since their entire product paralelizez very easily.
I don't think you understand the fanout/fanin problems their product has.
Twitter's raison d'être is allowing folks with millions of follows to see their posts in near real time, then monetize it with ads. That means analytics, billing, and more. From pretty basic requirements you can create a huge amount of work, especially when you're managing your own infrastructure.
I'm not saying that it's easy, just that most work could be done by much fewer people. I'm not talking out of my ass, I have been working on scalable systems for more than a decade.
I'm reminded of when, out of sheer curiosity, I once skimmed the proof for Fermat's last theorem by Andrew Willes. It was a monstrous work of which I could understand very little, but that clearly couldn't have been made by a team of a hundred mathematicians working together.
It's not that it isn't possible for a single person to do a monstrous amount of work, but that it takes an organization to do a breadth of things and maintain it over time.
I also don't disagree that some problems are self-inflicted. Large organizations breed large organization problems. But large organizations usually grow from seemingly simple requirements and compliance efforts.
My team certainly got a lot more done before we required tons of process, safety checks, data storage requirements, etc., where we now spend a non-trivial amount of HC as overhead and avoid building other things entirely.
> I'm not talking out of my ass, I have been working on scalable systems for more than a decade.
Ditto, friend. I currently work in an organization that builds a billion user product with far fewer than Twitter does. However, we also depend on a battalion of internal services, and other shared resources (HR, recruiting, PR, finance, ...). Twitter maintains their own datacenters, are among the few users of Aurora and Mesos at this point, and purpose-builds a lot of their infra for requirements we probably don't have a great grasp on.
Twitter, like many products, is like an iceberg: the majority of the complexity is underneath.
Well, I think we're pretty much on the same page regarding where the complexity comes from.
For Twitter though I'm thinking though that they should have put a lot more effort in simplifying the systems/processes/data they store. This might even mean pushing to remove some business requirements that a PM might have introduced just for a promotion to reduce software complexity.
If you don't reward people for refactoring and removing complexity, I don't think you can just blame it all on organic growth.
If you're gonna condescend like in that last line, your spelling ought to be impeccable.
Security patches... so 10 full-time engineers ought to take care of that?
"Thousands of dependencies" was a mistake. The ideal is minimal complexity.
Any browser or OS decides to commit a breaking change (rare), you bring in contractors.
Scaling up and down (and out and in) is handled for you automatically by cloud services now without anyone lifting a finger. "Fells" weird to be explaining that on HN.
Bottom line, you don't need thousands of engineers to keep Twitter running for "the next few days to week or two (at most)."
When has the world NOT been changing? The leadership of Twitter changed also.
twitter is mostly run in their own datacenters, and does not have autoscaling. so capacity overall is more or less fixed, though how much capacity is given to each service seems to be human adjusted.
the lack of autoscaling at twitter, imo, is worrisome as it sort of hints that the system might not be as mature in the self-healing department as compared to other large tech companies. time will tell.
you are right though, you don't need thousand of engineers to keep twitter running a few weeks. you also don't need a driver for a car driving a straight highway (until it bends). how many engineers are needed for maintenance? who knows. I don't think we have answers, we just have what elon decided.
disclaimer: facts based on twitter threads/online reading, not a twitter employee or anyone with first-hand inside knowledge.
By what I’ve heard from people who worked there they do have autoscaling but it’s services running on things like Mesos clusters instead of someone else’s data centers. That can lead to some real cost savings but requires people to run, and most of them just left the building.
* Apply security patches - there's thousands of dependencies.
* Manage hardware/cloud resources according to volume of data you need to handle.
* React to the changes in operating systems/browsers you run on.
* Fix bugs - they are there, because simply there's not enough time, money, and need to write "perfect" software. Competition will not do that and will beat you to the market.
There's no internet connected software, that can "just run", because world around is changing and you must catch up.
Fells a bit weird to explain that on HN :-|