I can't even fathom how it's possible to use $5B/yr to run Twitter.
So, I co-architected the Opera Mini infrastructure. It peaked at a similar number of users (250-300M monthly active users). Sure, Twitter is much more DB-intensive, but transcoding web pages is pretty CPU intensive too, and typically we transcoded every single web page for them. Opera Mini was their only browser.
Twitter is spending $5B/300M =~ $17/user per year
I believe that from public sources, it's now possible to deduce that we spent less than a 1/100th of that per user/year, almost a decade ago.
Since we didn't have crazy money, we optimized things at every step. Or, well, mostly avoided doing stupid stuff.
That re-implementation of solved things in engineer's favorite language/framework does not happen for free. Its pretty clear from outside that company like Twitter is jerked around by its engineers who have far too much autonomy to implement things their way instead of high perf C/C++ that any company at Twitter's scale must use.
Twitter engineers endlessly tweeting about having JVM perf team, presumably for their low performance Scala codebase.
Here [1] is an example where supposedly Twitter's supposedly Silicon valley calibre engineers couldn't write a half decent log management system and end up using vendor which otherwise only boring enterprises would use.
It reminds me of ignoramuses at Yahoo would claim that Google's frugality is useless and they can do better by just buying from Sun/IBM/HP and so on.
That Splunk post is a painful read. In my world, Splunk is well-known for being an overly expensive service aimed at sucking money out of enterprise/pre-software-style companies.
According to their financial disclosures for 2021:
On income of ~$5.08B, they spent
- $1.8B as "cost of revenue" (costs incurred to run their business),
- $1.25B on "research and development",
- $1.2B on "Sales and Marketing", and
- ~$1.3B together on "general and administrative" (overhead) and "litigation settlement".
Then there are a bunch of small monies related to interest expense and income, etc etc.
They're spending huge amounts of money and could be profitable if they really wanted. I can't imagine what Twitter is doing with 1.25B in research. Elon could make Twitter profitable simply by cutting their research department.
Well presumably that 1.25B is mostly engineering salaries since its "research and development" (for tax purposes where they probably get huge write offs).
Anyway, they need that org to get their 1.8B in "cost of revenue" down, which is presumably mostly the cost of massive server farms to store what are mostly text mesages. Although these days with all the machine learning/etc to sell ads they probably "needs" all that hardware to run their models and can't just optimize it down to a higher perf system painting web pages.
If the salaries are towards specific operational roles they'd be listed under "cost of revenue". If the salaries were for general duties not specific to an operational role they'd be under "general and administrative".
Research and development needs to be for research and development. It could be for engineering salaries but those salaries would be towards research and development rather than operational duties.
General engineering usually falls under the "development" category. The details may be state/country dependent. That said, even in stricter jurisdictions I know for sure that companies tend to group their engineering resources in such a way that they aren't justifying individual engineers (except maybe WRT timesheet filing) rather a bunch of people working on a "project" which meets the definition all get lumped together for purposes of R&D.
Random google search, talking about how broad this can be:
I'm reminded of the reddit articles a few years back when they were talking about moving to AWS and having to batch database requests to maintain their database perf. Apparently, at the time they were literally sending tens/hundreds of thousands of database queries for each page load request for a logged in user because each comment was collecting all its up/down votes and adding them up rather than just having a per comment karma attached to the comment.
This is what happens when you hire a whole bunch of recent grads that frankly have no idea how to write code, and think they are the smartest people on the planet when it comes to distributed systems.
It's always premature to optimize until your company is failing, because if you aren't failing yet it means it's worked so far. You should always wait until your company is falling apart to do a full rewrite of your core product.
People. Salaries are expensive, especially SV salaries. Then you need well paid management for the extra heads, and group manager for them. IIRC (feel free to correct me, I'm not going to dig it out again) they spent >$1B on R&D itself... which is pretty much just couple hundred of engineers who, (judging from the service changes recently) mostly did nothing.
