My experience is, # of engineers is directly related to # of product people, which is in turn related to quantity (not quality) of ideas that the founders/leaders are desperate to try out.
Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers.
WhatsApp success is not about what WhatsApp did, it's about everything they didn't do. No desktop, no browser (until after reaaaaaaallly long), no account creation (your phone number is enough, thanks), no hidden ad engine, no hundreds of stupid features that product people stuff down in the name of experimentation.
There are some leaders who want anything and everything: Throw 100 darts in the wall. Leave what sticks (don't touch or nurture it). Go back to throwing 100 more. The other guy does that? We want that too. Mobile? Yes. Desktop? Yes. Browser? Yes. Different affiliate channels? Yes. Custom features for B2B partners? Yes. Merchandizing? Yes. Build a mini Ad engine? Yes. Multi user profile management systems? Yes. Yes, yes, yes, every requirement is a go.
The crucial difference is: What happens when you have a restricted pool of people, but insist on doing 100s of things? Some leaders simply treat this as an "org structure" problem, not a problem of focus. They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step – just so that more quantity of features can be seen on paper. This is how most startups are, except for those rare few which don't "talk" about focusing, but actually focus.
It’s comical that all of the replies to what you’ve written claim you’re wrong, because you’re actually 100% right. And this is even more true when you remember that the company had fewer than 20 people for a long time, and that there was only one real goal, that seems to be forgotten: they needed to clone BlackBerry Messenger and get it working on every platform possible. That was it. That’s all they needed to do, and that’s really all that they did.
I tried to convince the CEO of Research In Motion, who I worked for, that WhatsApp would end up destroying his business. He thought I was insane for thinking something so small and simple could have such a big effect on his big, important company. Yeah, well, whatever you say, man.
Also can't overlook that for many SMS was ludicrously overpriced, but the early basic smartphones (palm and symbian) often had wifi capability; WhatsApp provided secure, ludicrously cheap instant messaging across all meaningful handsets.
And that's all it had to do; the telecoms weren't interested in making SMS cheap or secure, RIM didn't want BBM on anything but their devices, ICQ/AIM/MSN/etc were similarly unavailable.
TBH, I was incredibly disappointed when I discovered that BB shared the same encryption key for all users unless you were on their enterprise system. While I’m just one user, I also discouraged others from buying into them too.
Well, FWIW, the first people I knew using WhatsApp were my friends who were drug dealers or chinese nationals, and both lauded the encryption features.
> I tried to convince the CEO of Research In Motion, who I worked for, that WhatsApp would end up destroying his business. He thought I was insane for thinking something so small and simple could have such a big effect on his big, important company. Yeah, well, whatever you say, man.
Well, he had a point. What killed BB was the iPhone, not that their chat app was going to lose out to the competition. They simply had a product that was inferior in every single important way.
BB and BBM are network effect. iPhone alone is not enough to disrupt the social-media aspect of BBM (which was very sticky at that time in quite a few countries).
WhatsApp + Android actually killed BB+BBM in those countries and not the expensive iPhone
Mike might have taken longer to get it than Jim but he eventually understood the existential threat they faced. He realized this at a time they could have possibly changed course but was unable to do so because the board did not have faith in such a drastic change. The plan to cannibalize device sales today vs promise of future software revenue did not sound so promising for the board, who I imagine were interested only quarterly profits and their stock price.
Mike wasted a ton of my time, and squandered the opportunity to use the brain trust of Facebook insiders, including the guy who designed Facebook, who I assembled to jump ship and build a new future on top of BBM.
A few months later he sent me a message letting me know his private jet had just landed in Palo Alto... Did he want me to be proud of the fact that he seemed to be selling out all of RIM’s IP to Zuckerberg? I never got what that was about.
Both of Mike and Jim were in over their heads. And too arrogant to let people younger and smarter operate the ship. A tale as old as time itself. I’ll write about this in detail at some point, I promise.
> I tried to convince the CEO of Research In Motion, who I worked for, that WhatsApp would end up destroying his business. He thought I was insane for thinking something so small and simple could have such a big effect on his big, important company. Yeah, well, whatever you say, man.
I remember selling every BlackBerry stock I had when I got my hands on their touchscreen phone (the Storm was it?).
3 years behind the original iPhone, while Apple had just launched the 3G with the App Store. Guess which phone (and stock) I purchased then?
Was there really no-one at BlackBerry following what was going on in the valley at that time?
There's a number of stories about the insane level of hubris at RIM. Before the iPhone smartphones were seen as "business" phones. RIM reportedly felt like hot shit because they had managed to just not fuck up while Palm, Microsoft, and Nokia tripped over themselves to try to make a BlackBerry But Better.
RIM was so focused on the "business" market they really couldn't conceive of a real consumer device. Their upper management apparently looked at the billions of feature phones sold (IIRC Motorola had sold a quarter billion RAZRs by then) and said "nah let's double down on the order of magnitude smaller business market". They dipped their toe in feature phones with the Pearl but it was a joke compared to the original iPhone with its original software that didn't even support MMS.
By starting with more consumer friendly features (Web, media player, phone features) Apple had an extremely competent baseline of functionality. As soon as they added workable Exchange integration (iOS 3 or 4?) the iPhone (including older models) instantly became workable business devices. RIM had far more ground to make up for going from business to consumer than Apple had going consumer to business.
Thanks for sharing your insights. With all kindness, this sounds like post-UnionPacific. The neo robber barons have won. To go on with the metaphor, not sure the future holds a car to fulfill point-to-point transportation demand, as our digital information needs are vastly oversupplied.
I’m having one of those “why can’t there be 36 hours in a day” moments in life, but I’ve got to publish more about this very soon, so, it’s on the way.
When you combine the story of WhatsApp’s small team / high impact development with the remote work shift of the past few years, it seems that the next cycle (regardless of when/what it is) will be the first that isn’t centered on SF/SV/California. It’s interesting to think of how that’ll affect the psychology of the area.
This is why, after being both an engineer and a product manager (and VP of Product), and now a CTO of a seed stage startup, I’m fairly convinced PdMs are unnecessary. At least for the the type of company I’d like to build (WhatsApp as a primary example of that).
Off loading the founders vision to someone else is a failure of both communication and org structure (at the seed-SB stage). People may not think hiring their first PdM is offloading their vision, but it is. That person is going to come up with their own ideas, validate their own assumptions, and push for product updates that directly serve their own self interests. To think that this won’t happen is naive.
What I’ve found works the best is hiring competent engineers, giving them adequate business context, a decent design spec, and letting them figure it out.
This is always a controversial opinion, at least in some circles, but: I've long said that the PM role was created solely because companies can't find enough good engineering talent.
That isn't to assert that PMs don't do fantastic, necessary work. More-so that there's a more ideal reality where the engineers should be doing that work; the outcomes, in my experience, are better in orgs where engineering is not just influential in, but responsible for, customers, business outcomes, product planning, etc.
This isn't always possible. Most orgs don't resemble Whatsapp; they resemble Google. A hundred product divisions building a thousand different things. Its already difficult to hire engineering talent to fill an org like this, let alone the additional number you'd have to hire in order to account for the fact that a double-digit percentage of an engineer's role should be PM-like stuff.
So, throw away idealism; outsource that part of the engineer role to another specialty which is easier to hire for: PM. Its reasonable.
But the issue has arisen in that industry has forgotten than this was a necessary sacrifice; its not desirable. We've now got university and bootcamp programs dedicated to training for PM roles. Many companies have entrenched the idea that every team/org needs a PM. Tons of people are now adding financial momentum to this idea that their skillset is valuable and necessary; the ship can't be turned so easily.
At the end of the day, this is a disservice to many engineers, as their people, product, and business skills fall out of practice and they're further relegated into code-monkey positions. Its actually a disservice to PMs as well; they could have been trained in engineering and unlocked far more professional mobility. In the worst cases, it leads to PMs resembling the Office Space "what is it, exactly, you do here" guy ("I bring the specs from the client, to the engineers!" "Then, why don't the engineers just talk to the clients?" "Well, they're not people people!" mhm)
> Most orgs don't resemble Whatsapp; they resemble Google.
When they were a startup, Google didn't resemble Google, either.
For large organizations, I'm a fan of the "internal startup" management structure. This just means each product team should be largely self contained, but has the resources of the larger organization to fall back on.
So the leader of each product team can act like a mini-founder, defining and driving the product without being tightly coupled to any other product teams release cycle.
I can attest Product Managers were the downfall of one startup I worked at. The company had dozens of millions in funding and managed to piss it all away with ill-founded escapades driven by a handful of Product Managers. If anything, the Product Managers seemed to be managing their careers, not the product. They babied engineers until most of the engineers stopped caring about the product and just gave up working hard at all. All the brilliant ideas were trite insights you could read on two year old Gartner reports -- "release a cloud version".
CEO and CTO beware, do not let Product Managers happen to your company.
On the flip side, my old PM had an incredibly deep understanding of both the product and the customers buying it (an area in which us SWEs were lacking), in addition to a semi-hands-on understanding of the engineering process too.
He wasn't a professional software engineer, but he took the time to actually learn how to program through some side projects and through this gained a real understanding of what would be trivial for us to implement and what wouldn't.
To top it all off, he had good rapport with the CEO and was adept at managing upwards.
The product wouldn't be half as good as it was if not for him.
Rather than tarring an entire profession with the same brush, I think a much fairer "warning" to founders is to make sure the people you're hiring are competent and that the job they are doing is something that actually adds value to the organisation. PMs, when deployed appropriately, can be worth their weight in gold.
What you describe is the ideal PM. Someone who understands the business and customers and some of the technology. There are two kinds of PMs that are good. The one you describe who understands the business and the former developer who has been with a company many years and has made transition to PM role. However in a lot of companies this is not always the case.
Ditto. As an engineer, it feels like product managers often feel a need to keep cranking out new features as a way to justify their seat at the table. I would love to work in an engineering culture which found a way around that problem. If that means the job falls on me instead, so be it.
> If that means the job falls on me instead, so be it.
Precisely this. I don’t know when “engineer” turned into “order taker”. There’s no reason to have a Product Manager define and engineers work. Good engineers are more than capable of defining their own work.
> There’s no reason to have a Product Manager define and engineers work
The PM isn't supposed to define features in isolation, but crystalize them into consensus between engineers and stakeholders the PM is accountable to.
If your PM isn't your peer, the org is setup wrong. Tech leadership should be helping define this, but people don't know what they don't know and so things end up with PM's becoming de-facto "team leads".
I've always called this Feature Cram, and it's standard procedure at basically every company I've ever worked, small and large, B2B or B2C. Nobody is willing to say "Our software is a success and it's done. Let's fix the major bugs, put it into maintenance mode and start a new, separate, great idea!" Nope, instead it's always "OK the MVP is going well. Let's jam ten other unrelated things into it and see what takes off and what are the dogs (and be stuck with the dog features as technical debt forever). Then, when none of them reach the success of the original actually solid idea, they hire moar people and jam 40 more feature ideas in. Eventually, the original great thing that brought the company success is so hard to find and use because of the weight of the other crap, that a new upstart that Just Does One Thing Well comes in and eats their lunch.
Then, at that new upstart, someone asks "What else can we bolt on to this thing to make it really grow??"
This may very well be unpopular, but this is what happens when you pay someone in measures tied directly to share price, or even more directly in shares; they have every incentive to do whatever it takes to juice the share price up. VC money has even more extreme growth expectations.
There's absolutely no good reason why the same team can't just make another great focused product under the same company umbrella instead of ruining the original
Yes, although there is a balance - lest we forget that you couldn’t use WhatsApp on your computer unless your phone was turned on until a month ago, and there is still no iPad app.
I think WhatsApp is in the place where it is so ubiquitous in some social circles that it faces less competitive pressures than other apps and can afford some of these pretty large gaps.
> They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step
This is the key mistake. Having teams of 2-3 with a manager each is way too many managers. If you have a clear focus, small teams are great because they can work autonomously, with little management and good results. You can't hire your way out of coordination hell.
I was at WhatsApp between 2011 and 2019; for the most part early on, we had essentially the CEO who does a mix of product management and engineering management and engineering, an actively engineering server manager (who also did some client work), and a client engineering manager. Android team grew enough to have an engineering manager and we also ended up with one product manager. Post aquisition, staff grew and so did management.
That's all nice and well, but some of your examples are not things that save effort -- using phone numbers is surely better for network effects, but not in any way simpler then having regular account registration, unless you just completely ignore security.
WA apps are not better than the competition. Basic stuff like a way to disable automatic downloading of media for groups you are in is still missing. No desktop app. The web frontend is still crap. Outsourcing account registration to mobile phone companies is ridiculous (same as Signal) -- in most of the world, it makes it pretty much impossible to preserve your privacy while using WA. Endless nagging about plain text cloud backups makes WA E2E encryption completely worthless, since even if you opt-out, 99% of your contacts will have them enabled.
The impressive part, considering the size of their team, was their backend stuff and scale.
Discord, with similar team size, was able to build something that succeeded in displacing IRC. (Their Android app was laughably bad for the longest time as well.)
In my experience it displaced Mumble and Teamspeak only for casual gaming. If your group is strict enough about muting/kicking people who refuse to fix their mic settings, Mumble or Teamspeak is generally better -- better quality, local volume adjustments, less resource hungry, open mic not being the default, etc. Think tryhard esport players, or just people with old hardware (probably 90% of CSGO players )).
For my group this wasn't about mic settings but mumble being bad out of the box. Maybe because we didn't pay for the servers and used a free something, maybe because of bad luck, I don't know, but every experience we had with mumble was terrible. For teamspeak, while the voice part is great, the message part is (or was at least in ~2016) lacking compared to discord. We were coming from skype/twitter which had better support for """rich messages""".
WhatsApp was originally a super simple product. Send and receive messages. It was fundamentally a solved problem. It was all about the fact you could send free messages in a world where sms cost 10 cents a message. It wasn't that their messages were faster, their messages were anything other than free.
They had a year fee of $1. Which I never paid yet I used their service anyways. The reason they were able to scale so large with so few devs is simple, their product was super simple. 50 tech people to build a messaging system is actually a lot.
> The reason they were able to scale so large with so few devs is simple, their product was super simple. 50 tech people to build a messaging system is actually a lot.
Tell that to Twitter. Whatsapp staying at 50 instead of growing to 5000 is impressive.
You used to be able to use it as a way to create a free self-managing SMS distribution list. Pretty cool, before I had phone with data capabilities.
That was even a use case I understood. Now I feel too old to understand how to use twitter. But that can't be it. Much older people seem to have no problem.
Why didn't MSN become what WhatsApp is today? I always wondered how they could screw this up. Smartphones are perfect for chat apps, that's basically what they are supposed to be used for, and MSN / ICQ already covered a gigantic portion of the instant messaging market. Did they just assume people would stay on desktop for chatting forever?
Good question. I remember Microsoft had an app around 2010, but it just failed to gain critical momentum, maybe because people were already moving en masse to Facebook and WhatsApp itself.
The app was only ok-ish. There were some alternative apps in the AppStore. But people just preferred talking trough WhatsApp and Facebook Messenger (which was already two years old at this point). I guess people gravitated to WhatsApp because it was less cluttered and snappier, and to Facebook because it also had a proper social network in it, but I have no idea. Two years later Microsoft would kill MSN and integrate it with Skype.
In my recollection it wasn‘t about saving 10 cents in the beginning, but about saving 1-2$ per message, since one of the first use cases it enabled was free international texting.
That was a HUGE benefit.
All the first evangelists I knew were in long distance relationships ;)
This is a great point, but I'd add that the problem isn't experimentation. Neither is it having a lot of ideas. It's that they aren't using good experiments to kill off most of the ideas.
I think of it in terms of divergence and convergence. Divergent thinking is great. If you think of an army on the march, leaders will be thinking about possible paths and sending out scouts. But when the scouts (experiments) come back with reports (results) they don't just send some soldiers along every path that looks promising. They winnow the options down, converging their thinking.
Bad leaders don't have the discipline for that. And I think a lot of them pick up that habit from sales and/or seeking investment. "What will it do? What won't it do!" That's great for manipulating the feelings of the person you're currently talking to so they'll sign the deal. But it's terrible for running an effective, focused product development organization.
I know I've seen companies die like that. And it's probably way more than we know, because as WhatsApp demonstrates, focusing hard on what's working is a great way to succeed well enough that people study and talk about you later.
There was an article about when do startup brings in their first "product manager". Before bringing in "product people", the founders are the product people. I see it as the inflection point when the founder start delegating the product making to others. This is when the organization started scaling.
Doing one thing good is a thing in the past. Venture capital has pushed basically all start-ups into 100-feature shops
Uh yeah, but it was only possible to have such a simple product because they reached mass adoption. Without network effects, you've got to differentiate and thus add complexity to gain users.
But then how did they reach mass adoption in the first place? It’s not like they were a Swiss Army knife and before and stripped features once they got big.
As dleslie and that_guy_iain have mentioned, it was expensive to send an SMS text back then, especially in Europe when your friends have different country codes. So WhatsApp was a free alternative that tapped the unmet need for millions.
And signing up with only a phone number seems like a nobrainer nowadays, but the mindset of the time was still email + password. Sure, Microsoft could have launched a competitor, but they would most certainly require you to use an MSN account, thus adding friction to adoption.
This comment flies in the face of actual history. While your point about Microsoft may be right (who knows with how they’ve ruined their messaging platforms?) it certainly doesn’t hold true for iMessage which came out in roughly the same timeframe as WhatsApp (2011 vs 2009).
iMessage still has yet to come to other platforms, and at least for a few years after the iPhone's launch, many people viewed it as a luxury product and were unlikely to spend the money to get one compared to cheaper phones (which WhatsApp supported).
Except for that $19 billion exit. Companies don't need to be turning a profit anymore -- they just need to prove they can scale, and someone else will figure out how to monetize it after a buyout. Hell, you don't even need to be a startup aiming for a buyout anymore to get away with being financially incompetent. Reddit is about to file their IPO at a valuation of $10 billion, even though they've been losing money for nearly two decades and are noticeably (and unsuccessfully) scrambling to find any presentable evidence that they're not complete buffoons.
Not true. They were charging $1/year from users in the richer markets. They could easily reach hundreds of millions per year in revenue if they followed this model.
It didn't go from $0 to $1. It was always "$0 to most, $1 for those that really didn't care about paying because they knew that anything else would have much higher real costs"
Hell, If WhatsApp ever got to say "Facebook wants to buy us, but we are also considering changing to a pay-what-you-want model to avoid selling out", I'd easily give them $10/year without batting an eye.
> What happens when you have a restricted pool of people, but insist on doing 100s of things? Some leaders simply treat
this as an "org structure" problem, not a problem of focus. They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step
> Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers.
Part 1: By definition: more engineers = more ideas.
Part 2: Not so much.
On the one hand, mediocre engineers are sometimes sequestered into small teams where they can't interfere with the important projects that all the smart people and other resources are being thrown at. They just sit there and churn out crappy ideas and crappy non-vital tools. Like daily indicators.
On the other hand, I can see that as more of a tautology: ideas come from one person and are grown by a team. So be definition a quality idea comes from one person, and turns into a team.
I think there is something else that I find hard to explain, but it stems from the same vein.
The best way I can call it is the impedence mismatch between what business people imagine and envision, and what the real products, their code, their systems are actually doing and are actually designed to do.
It creates a kind of cost explosion in the velocity needed and it is often solved by throwing more developers at it instead of rethinking the actual design of the products so that they can better grow to support additional use cases and scale.
You can't totally call it a hack, but it kind of is. A system was designed in a way that doesn't scale to the user demand and to the additional features or the new abstractions that it wants to grow into. Instead of rethinking holistically about how that system should be re-designed to now encompass all of this new stuff, and then put the work into changing it from its current model to that new one. What people do is they break up the current system in two, so now you have two service where you had one before. And now they hire a new team of engineers, manager, PM, etc. to start working on this new half. This repeats again where each of those two half are split in half again, hiring a new team, now there are four teams and four service, etc.
As that happens, you can no longer redesign the way the systems fundamentally work, so each team accrues more and more complicated workarounds on their individual pieces and all new feature becomes cross team projects where you now need someone to coordinate the work between all teams for each little project.
Once you start down this path, there is no way out, the only solution becomes throwing even more teams at it, breaking things down again and again into smaller pieces with a whole team behind it as each piece becomes more and more complex to all work around the inherent holistic limitations that were never reconsidered.
It's a vicious cycle, the piece cannot grow in it's current state because it is too complex, break it up, have two teams tackle each half, now they can take the complexity until they themselves add more stuff to it which makes them too complex once more, break it again, add another team, rinse and repeat.
The issue is often that complexity is accidental, due to the design in place and the tech dept accrued till now, as opposed to being inherent to the domain problem. But instead of addressing the accidental complexity, it is simply broken down in two so we can throw more people behind taming it.
As an end user I think WhatsApp is a terrible product for some the reasons you mentioned (tied to phone number, no desktop client) but I cannot deny that it was the smart thing to do from a business perspective.
Getting together with your peers to discuss a problem is entirely different than being called into a meeting with someone who can only use a iPad to evaluate progress on things they cant understand while several other people sit idle waiting for their turn.
There is no need to turn this into an argument by suggesting that somehow WhatsApp banned its engineers from getting together. That never happens, ever. We are obviously discussing scheduled meetings and scrums.
This confuses the mistakes of a mature company. Different dynamics.
For example Google had thousands of engineers back when they had only 3 products: search, auction, and AdWords. I remember Travelocity having over a thousand developers and not being able to redesign a webpage.
Its deeper than a product problem. Its an execution problem. It’s the ability to do what works with great criticality opposed to what’s expected, standard, normal, or comforting.
I get scared every time that senior leadership pushes a new member on the team, because I know that overall quality will drop; their view is completely related to how much things they want to done on every cycle, and they don't care about quality (until it blows up in their face)
The WABetaInfo account on Twitter said one is finally coming. And I think it seems likely considering we don't need the phone to be on while using Whatsapp Web.
>WhatsApp success is not about what WhatsApp did, it's about everything they didn't do.
I wrote something similar [1] about its success not long ago on HN,
>WhatApp grew because it was the only IM that worked on Smartphone. Thanks to Erlang. Yes. ICQ didn't bother. Microsoft refuse to accept Smartphone is the future of computing hence there never was MSN on Smartphone. Other contender tried but none of them managed to Scale. I still remember all my colleagues and friends used to download instant messenger on our phone to test them out, creating a group and basically DDoS each other until the system or network crash. No IM was able to handle the traffic load, not only WhatsApp did it many times better than their competitors, they also had almost all platform support, from Symbian to Blackberry.
There is another part of the story that dont get much appreciation, is how they managed 2 million connection per box on a what is now a relatively slow server. The latest M1 Max would have been a lot faster. I often wonder if they could have pushed to 10 Million connection per box with same latency under current CPU and Memory. ( Not saying it is a good idea. )
I'm honestly not that surprised about the size. In my experience, efficiency and the number of engineers have not necessarily correlated.
For example, I used to work for a company with about 500 employees (relatively large by German standards) and now work for a competing company in the same industry with less than 150 employees. The core engineering team only consists of a dozen engineers.
The output is noticeably larger. The number of new features and especially the stability is significantly better. It is noticeable that due to the small team, each project has one or two "heroes". They are responsible for most of the commits and know the system inside out.
These "heroes" have been working on the same or similar systems for over 30 years in some cases and have experienced a lot in their careers. As such, they can bring experience and advice that is just out of reach for others.
Given the choice, I would always prefer a small team over a large team.
Having worked in a range of companies, there is definitely an inverse correlation between company size and speed, especially for individual teams. At 500 people you're well into enterprise level organisation, where a lot of processes are put in place to minimize the amount of damage a single person can do to the organisation. That also puts a speed limit on those efficient and knowledgeable contributors.
Some companies spend a lot of time & energy trying to implement operating models that try to overcome this but actually end up putting in place more decision processes & oversight. Sometimes the real solution is to reduce control and let the efficient teams lead the way. But then what would all that management do to fill their days...
I think this is more correlated to maturity than size. It's just that size is highly correlated to maturity.
I've been in large companies where I was the admin/owner of their AWS account. And I've worked at companies with 200 people where I have to submit tickets to get anything created in AWS. The major difference was how mature the product was.
My current employer has a nice balanced approach to this: everything is a code review. I may need a gatekeeper to approve my change, but I can always submit it.
It’s also safer this way. Not even the proper owners should routinely be on production shells or tools with live write access. They also go through peer review with each other. Once the infrastructure is in place for that, it makes sense to open up.
I work on a suite of products over 50 years old that have been incumbent for about that long and still have a submit tickets to get just about anything done.
Heroes are a red flag for me, as a manager, trying to build a resilient team. Which is not to say that you can't have a range of experience levels. But what happens when those heroes are unavailable? Maybe they're out on vacation, or medical leave, or they eventually retire or leave the organization because they're tired of propping it up? Heroes tend to be a single point of failure.
You hire more "heroes". Pay what they're worth. Make them happy.
Or you just pay an order of magnitude on hosting, performance consults, bandwidth, etc. etc. etc. And STILL won't EVER get to the same result, because you can't always pay (I'd say almost never) your way into a stable and performant solution. Look at all the others.. Nothing works as well as whatsapp.
Good luck hiring 500 engineers who have no idea what happens throughout the whole stack.
I'd rather pay what you save on cloud providers and snowflakes to them. (not even taking into account the result of these "heroes".. the business value / dominance whatsapp has because of this)
You can hire 1000 people who can't draw, or maybe just a little bit, or you can pay an artist and get the result you'll otherwise never get.
> You hire more "heroes". Pay what they're worth. Make them happy.
i dont think its as easy as you make it sound. these "heroes" generally are as productive because they dont sync and already have a clear idea of what they want to achieve/how to implement what they wish.
if you add a second person like that to the same team you run a massive risk of having two competing ideas which creates a lot of friction.
At my company, “heroes” are given their own projects with them given full creative control over every detail, including the people they work with. I think I’ve read somewhere on HN to never let any two person do the same thing, to avoid the conflicts you mentioned.
Benefit of this is that these heroes can each break new grounds, and their BKM shared amongst themselves making the team extremely productive.
The difference between hero or primadonna can be environmental. When there are multiple experts for a department/tech stack, you get heroes. If the bus number is one, a hero risks converting to a primadonna.
I find if you hire heroes with different skill sets you can avoid this dilemma. Or don't hire two Supermen. Hire Batman and Superman instead. But if a hero converts to a primadonna then they were never a hero. On the other hand if you're skimping a hero of the equipment and resources they told you they needed from day one, you're the a***.
I agree. I'd add that having multiple experts in competition with each other (due to bad culture, bad communication or even just bad personality) also has the potential of turning heroes into primadonnas.
In my experience, insufferable mid-level management can bring out the worst in the best people, and then accuse them of being primadonnas because they are absolutely incapable of any self-reflection themselves.
I work as a contractor, thus I have read a lots of code from various different types of projects and the conclusion is always the same, if they had spent more money on skill to begin with I wouldn’t be needed.
My first interaction with this management philosophy was when our partner AOL replaced experienced SRE serving our project with 3 far less experienced people for the same price. What used to take an hour now literally would take weeks on the bright side for the manager he had way more reports now.
I tried to clarify in my top comment that I wasn't saying we should avoid having experienced people on a team, but based on your response, maybe a little more clarification is needed.
When I think about a "hero" in terms of team dynamics, it's a person who is consistently relied upon to save everybody else's butt, often by doing things that are flashy but not particularly healthy, such as working long into the night to meet an unrealistic deadline. When you have this sort of Superman figure who's willing to swoop in and save the day, the problem isn't so much their level of experience but the unhealthy level of dependency that's placed on their shoulders.
I'm all for having highly skilled, highly experienced engineers who are productive themselves and can further increase productivity by helping unblock others on their team. And I agree with you that replacing them with some number of juniors without the same institutional knowledge can be disastrous. But if your team becomes so dependent on heroics that they can't stay afloat otherwise, then when that hero gets hit by a bus or just quits to take a position elsewhere, you're screwed.
If you have good trust and communication within a team, the scenarios you describe are surmountable.
Every strength is a weakness, and every weakness is a strength. IME, heroes are no more a red flag than pretending that good engineers are interchangeable. It depends on the context.
The expected outcomes of these teams are different, and that’s OK. If you’re a very small company and don’t have a couple heroes, you won’t build anything important. If you’re a very big company, the heroes that built it left years ago, and you need resiliency more than new heroes (unless the business is going sideways and needs saving).
Many/most teams don't have heroes at all, they only have noobs :) I.e. most people stay on a team for a year or three and change jobs afterwards - there's no chance of accumulating knowledge. Nobody is sure how things work and why they are the way they are.
I believe many managers are fine with this, because the team is producing something (i.e. "velocity" is higher than zero), but are oblivious to the fact that team could be way more productive if the knowledge was actually retained in people's heads and not just lousy attempts at documentation.
I don't know if they are oblivious to it - but it might not be possible to pay people enough to get them to stay due to pay bands set by HR etc.
OP mentions that he is in Germany - there it might be more possible as SWE doesn't pay as well as it does in the USA and there are fewer companies so it might be feasible to pay a lot more than the competition, whereas in the USA that's unlikely to be viable outside of companies like Netflix etc.
You're right up to a point, but I've seen far worse red flags. Like companies that can never acquire these heroes in the first place because no one competent would work there that long.
As somebody else mentioned, these companies end up regularly throwing millions at consultant projects that always fail. In other words, they pay a hefty premium for shitty temp "heroes" that give them less than employee heroes would have given them.
You see this a lot in the traditional finance sector, where managers don't appreciate tech workers and relentlessly fuck themselves over trying to save a dime.
Yeah, somewhat counterintuitively treating tech as a cost center will inevitably lead to wasted funds. But hey, that just makes incumbents die faster and give way to organizations treating tech as an investment. Which are the ones you want to work for as a tech person anyway.
Being replaceable tends to make work less satisfying. All the places I've worked at that followed your advice had the most churn and the least productivity/ROI on engineering $ spent.
Being replaceable tends to make work less satisfying.
In my experience the opposite is true. My personal goal on any project is to make myself replaceable. There's nothing I find more tedious than having to work on the same thing for years because no one else can take over from me.
You’re talking about something else, where you are an key employee building a valuable system that doesn’t need you. The other person is talking about a company hiring you to fit into a system that never needed you.
> My personal goal on any project is to make myself replaceable.
So you admit that you aren't replaceable? If the company mandated that you should always be replaceable then you wouldn't need that goal, it would always be fulfilled. And working in a way that makes you always replaceable isn't fun.
Edit: What we learned from your comment is that making yourself replaceable is fun, but being replaceable isn't fun, since as you say you work to become replaceable so you can start working on something new rather than to stay replaceable forever.
I would hate to be stuck on the same product or set of features, but I very much enjoy getting to a point where I know all of the systems, how they relate, and roughly where all the functionality lies and who's worked on what. It takes a lot of time to develop that experience and it's so valuable, and it seems so strange that companies don't want to select for this and we're dominated by a culture of job hopping every 2-3 years because it's the only way to maintain appropriate pay. Then companies wonder why everything is over budget and never on time
I think like most things there is a balance. The sweet spot may be I can go hard on the project but if I feel like I need some time off - there is the ability to step back and have the work continue.
> Heroes are a red flag for me, as a manager, trying to build a resilient team
Heroes are your most powerful asset, but you have to use them responsibly. The best thing you can do when handed a 10x unicorn developer is to try to document 100% of the things they say & do, and also make it a requirement that the hero mentor others some % of the week. E.g. For 1-2 hours every Friday you force them to hold a "no stupid questions" session.
5 heros on vacation for 7 days, and coming back and finishing the work in next 5 days is more comfortable position than 50 clueless engineers claiming work will be over in 4 days.
I understand your point, but what managers fail to understand is that someone usually "carries", i.e. is actually responsible for the success of the product. I have even seen it be a manager...once.
> But what happens when those heroes are unavailable? Maybe they're out on vacation, or medical leave, or they eventually retire or leave the organization because they're tired of propping it up?
I don't know why people insist on using examples like this instead of the more obvious one: What if they are hit by a bus?
Any of the other examples have trivial solutions in the event of an emergency: call them and maybe offer them some money. If they are very stubborn then go to their home and beat them with a wrench. But you cannot get information from a dead person no matter how much money or how big a wrench you have, and people die suddenly every single day. That's the worst case you need to be making contingencies for.
Managers prefer a team reliably working at 0.1x speed rather than have it work at 1x speed 95% of the time. Yes, sometimes people leave and you will have less velocity for a while, but people pick it up and you go back to high velocity. To fix that you ensure the team always works at slow velocity, that way it wont get slowed down since it was already performing as if you lacked those heroes from the start.
This is definitely a potential risk. However, you have this risk with small teams (less than 10 developers) even without hero roles. The smaller the number of developers relative to the complexity of the product, the higher the probability of a single point of failure.
However, you can reduce the risk through good documentation, good engineering practices and so on. As long as you are aware that the team is partly made up of heroes, you can prepare accordingly. So as a manager, I can try to ensure that a few heroes keep productivity high while making sure that the rest of the team understands the basics of the system through things like code review and the like. In the worst case, a teammate can then at least get up to speed quickly.
Having a "hero" at a 5 person startup can mean life when death was inevitable. Having that at a 500 person org likely means another team is left cleaning up crap.
It's really not hard to be a "10x engineer" if you disregard tech-debt and error-handling. I was part of a very productive, complementary duo where my partner churned out a lot of code that only catered to the happy path, and I'd clean it up/"productize" it.
I actually enjoy that sort of work, but didn't receive as many accolades as the guy "churning out features quickly", even though he'd break the build and block the entire company at least once a week before I paired up with him.
Once I saw how the sausage is made, I'm a lot more sceptical of the "10x" label, either there's an invisible support system, or the code base had a short half-life. Any other scenario is a set-up for disaster.
The worst team is the one with a single point of failure. All hell breaks loose if they're unavailable. This makes me wonder why managers have teams like this?
There is one type of team which is even worse than one with a single point of failure: those teams without anyone at all capable of fixing the system. Given that alternative, having a single point of failure is better than having nothing at all. It would of course be even better to have multiple capable employees, but you cannot always have the team you want with the hiring budget you have.
Not all managers have a primary concern of "trying to build a resilient team".
Sometimes, the #1 goal is building a high-performance and/or high-quality product. Lack of team resilience, (temporary) downtime, risk of project failure due to an insufficient bus factor...they're all not good things, but they can be compromised in the pursuit of that goal.
If you want high resilience as well as high performance and also high quality, well, you should go work at NASA, and hope that the Senate works out their budget issues...
As long as the core system is built on solid engineering, this definitely makes sense.
From my experience in a deadline-constrained environment, things end up being slapped together until they "just work". Adding new features gets more and more difficult and if you're still constrained by deadlines, then the solution is to hire more, which is effectively a brute-force solution to meet the deadlines. Output per employee gets smaller, but total output should grow by some marginal amount.
Absolutely. This is also when bigger and more pervasive bugs enter the conversation. Decisions that kicked the can 2 years ago are now a big problem and no one team can fix it.
I worked at a place that made a group of loosely-related websites. Basically different spins/themes on the same products. There were 4 of us on the development team. There were 10 websites. Some new, some in maintenance mode.
I quit after a few years, and about 3-4 years later they went on a hiring spree. Tripled the number of developers. Refreshed some products, discontinued some, and some went into maintenance mode. They still had around the same 10 websites, but 12 developers.
The products didn't get better. The products didn't get made any faster. They didn't have any less bugs. My wife actually works there now, and when she tells me about the issues they have I laugh a little bit.
It's all the same problems we had back then. Invites don't work. Teams doesn't work. Uploading profile pictures has issues. Entering data doesn't work. Registration is broken.
I had a somewhat similar experience with a large tech company. Grew from <100 to 600 in a few years, but in the meantime they weren't even able to update the homepage. Not even the content changed, in almost three years. Everything was convoluted and over-engineered. The whole thing was rewritten from scratch twice, and when I left there were plans for a third rewrite from scratch. COVID destroyed the company: there were zero customers. But the website still struggled to stay online. That's when I decided to leave.
I looked at it this way. The thing most engineers, hero's or not, want to work on is new projects and to build products customers enjoy. The last thing they want to work on is maintenance. In my last personally owned company, small as we were, the engineering team and I documented how things worked as part of what we did every day. We found that by having the inner workings documented so that anyone in the team could effect a repair or update a piece of code freed us to work on new products and services. Just having the system continually documented freed our minds and kept us mostly happy (everyone has a challenge from time to time) and productive.
The documentation, not that it really matters, was done in a MoinMoin wiki. It was written by engineers, for engineers. We didn't stop marketing or sales or anyone else from reading it, but the target audience was the engineering and operation staff thus no one had to over-explain the vocabulary of our architecture.
In my experience this is the hardest part. Can't count how many times my team started documenting and then slowly, for one or the other reason, the documentation became outdated.
The cycle is: Meetings for communication -> becomes too cumbersome and time consuming -> implement policy to document things -> everyone start documenting (for a month or two) -> docs are not up-to-date because people forget/leave etc.. -> go back to meetings
In my experience it's a culture thing. When I learned to love Jira/Confluence was when the product team conducted meetings directly from Jira and during task discussion almost always asked us if tasks and documentation were done.
It could have been really annoying but the product lead was extremely empathetic and that could be felt in any task discussion, and his attitude made me want to do it instead of making it feel like busy work like it has in other companies.
I must be weird. I love maintenance. The best bugs come out of that. And it's hard to tell if a system was well designed except in hindsight; maintenance programming is the best place to be for that.
You’re not alone. I may not love “maintenance”; but, I do love resolving performance issues and shoring and standardizing existing stuff a lot. I think I most like a balance.
In my experience, documentation is the first thing to go when the workload increases. There are always tasks that get labelled as "important, but not urgent" that get pushed out; and documentation that is already known to everyone immediately involved is easy to push out beyond the product release date. Of course, by then other priorities are moved up, and documentation never recovers.
The only way I've seen documentation get the priority it needs is if management mandates it as part of the release package on which developers' performance evaluations are based.
They not only used FreeBSD for their servers, their CEO at the time Jan Koum donated $1 million dollars to FreeBSD foundation too. Talk about giving back!
The only thing they failed at was making money. I really wished they had. They knew what Whatsapp meant to the world. Look at WA now.
WhatsApp was successful only because they had no intention of ever making money. Simple lightweight app, no tracking, no ads, no upsell. Yet they were funded by VCs who would want a return on their investment at some point.
But if they started charging even 1$/user/year, most of the 450M users might not be on the platform. There are other free platforms offered with similar functionality. Security/Privacy is not a concern for a lot of that 450M users.
The founders hearts were in the right place. Brian Acton created Signal Foundation and Jan Koum like I said also have done good things.
They also tried the paid version for a while if you remember. And they were also dead against ads on the app. Unless I missed something or if I am wrong somewhere (At which point I would love to be corrected), Selling was the plan all along doesn't seem like a right observation.
>The founders hearts were in the right place. Brian Acton created Signal Foundation and Jan Koum like I said also have done good things.
I'm not saying they are bad people, and selling is not a bad thing either. I'm just pointing out that selling was the plan all along.
The WhatsApp story that gets told regularly (like in this instance) is a story of how such few employees and such small infrastructure is enough to build a massive product. You know what that translates to? Low opex. If they wanted to, WhatsApp would've been quite profitable if they wanted but instead they chose to sell to the highest bidder.
Again, that's not a bad thing, but also, I'm sure there were PLENTY of people that were interested in buying a billion user platform with 99% user retention. Why did they sold to Facebook? Only they know, but back then Facebook already had a tarnished reputation, so they definitely didn't do it "because of their mission and values".
This is a good point of view and this is entirely possible. But from an employee POV, FB was touted the best place to work. Things like that would've been a reason to select.
There is another chance that FB was seen as the lesser evil compared to Google then, who was also in the bidding for Whatsapp IIRC. My younger self would've chosen FB instead of Google for sure.
Not to mention, people wouldn't have guessed how horrible FB would turn out to be.
Edit: Funnily enough, after I wrote this comment, It became clear that money was definitely a factor.
It was free for the first year iirc, and then a dollar a year. A dollar a year is way more than needed to cover the cost of hosting, and most would spend that to avoid paying for SMS.
They ended up with a billion or two users. $1-2 billion with the subscription is good for a company of a few dozen employees. They could have branched into all sorts of value-add stuff in the future if they wanted to, all without tracking and such even. A simple payments or shopping interface a la Instagram Stores could have done the trick.
I think one thing to consider is that WhatsApp is probably by nature a scalable application. It is in the end, simply a client app that sends data to other client apps. So by nature, you scale in parallel very well, there are no 1-all communications. So unlike something like facebook where the data all needs to go to a central server and be polled by an unknown (but potentially arbitrary large) number of clients, in this case you have a huge number of point to point communications. It also means your server application can be largely stateless.
I'm not saying that it's an easy problem to solve, but it is a much easier problem to solve than a lot of other applications.
To add to this, not only is it a simpler problem, but they stood on the shoulders of giants that already solved it. They used ejabberd, a full-featured xmpp server written in Erlang. Imagine half your job already done for you. Granted, they heavily customized it by this point, but they did a great job picking the right off the shelf solution to begin with.
Why do you assume that WhatsApp uses point-to-point connections?
My assumption is that most devices cannot talk to each other directly and require central servers.
I could be wrong but I don't think he was saying "point-to-point communication" in the technical aspect of the term (i.e. peer-to-peer). Rather, just making the point that they're not storing tons of data on their server that has to be polled on demand by an unknown amount of clients forever. Even if there's a server between the users, once they receive a message and pass it onto whoever necessary, their job is done and they don't need to worry about that data any more.
> It is in the end, simply a client app that sends data to other client apps
That's not the case. Consider the "one tick" messages - they must be on a server somewhere. One tick means they've left your phone but aren't yet on (all) the recipients' phones
Servers proxy the message yes but once all recipients (maybe a dozen clients in the worst cases) receive the message, the server can drop the data once and for all. No need to optimize for harder problems like search, caching, or recommender systems.
"The simpler product makes it much easier to maintain and scale."
This is good news for the developers, the product owners and the customers.
I still can't understand why today people choose a (Javascript) stack of build systems, ton of dependencies, and all kinds of exotic tech that is the latest and most hyped. As a developer you need to support this in the future. It might be nice to build today but it will be a nightmare later.
I have seen days go to waste because of Docker misconfigurations, problems with dependency versions, outdated dependencies not available anymore, bugs deep down in an unknown dependencies, and so on. Personally I want to build systems. I don't want to spend my day debugging all kinds of weird problems.
The (old) WhatsApp teams sounds like a great workplace.
One is that the future is unknown. Your goal might be to keep the product simple and scale to 1bn users, but you don't really know if it will play out like this.
Maybe you plateau at 1m users with the initial idea and pivot the product somehow. Now the app does dating, social media or specialised communications for highway construction teams.
Tradeoffs are easier to make in hindsight. Whatsapp is a good demonstration of what you can gain with good trade offs. Do less but well. Whatsapp did trade some things off though. Their web version came late, is feature poor and a little clunky. Same for a lot of features, compared to similar apps ATT. IDK what exactly can be traced to the stack, but their decisions around user identity are similar. Phone numbers only. One device at a time. They gained simplicity, but lost options.
It's easy to see the benefits in hindsight. Real time is harder. Play that game again and it might turn out different. Most apps that set out to have 1bn never get close. At this point, scalability doesn't matter and tradeoffs made for maximum scalability don't seem as wise.
I think the Unix philosophy should apply here. The fact that WhatsApp did one thing well and it had a good business model (was it $1 per year?) is the important bit.
When you say, “Now the app does dating…” I think the right move is to scrap the project because that’s a colossal fuckup. Unless you have Microsoft cash to have multiple colossal fuckups in a row, don’t add another dating app to your messaging app. Ergo, less is more.
WhatsApp probably started with passion and a solid vision. The Zuckerberg gave them an offer they couldn’t refuse. Anyone will take 19 billion for a basket full of Indian users.
Another thing WhatsApp did well is they targeted ALL phones. They didn’t abandon their users like the fang-bangers (sorry been watching True Blood— FAANG is an annoying acronym so they are hereby fang-bangers).
>Another thing WhatsApp did well is they targeted ALL phones. They didn’t abandon their users like the fang-bangers (sorry been watching True Blood— FAANG is an annoying acronym so they are hereby fang-bangers).
That was clearly the killer "feature." Whatsapp is synonymous with communication on the developing world because of this. I remember when I was introduced to it fairly early on in Brazil and someone claiming that yeah, anyone no matter the OS could get on it. I couldn't believe it, to be honest, it felt like what iMessage was starting to look like... but for everyone? I can't imagine what it would've been like to support so many things, but clearly it was a lot of sweat that paid off extremely handsomely.
I'm not saying that philosophy is bad, just that reality is complicated.
"Scrap the project and move on" works in some contexts, not others. The way startups/products actually work, often, is evolutionary. If your texting idea didn't work, but you see a chance to pivot into something... are you really going to just fire everyone and tell investors "sorry?"
That said, the "one thing well" philosophy really does have big engineering advantages. You can't have everything. I'm just raising the "retrospectives" warning.
In any case, the "$1 per year" was never a real business model. They never even got around to actually charging it... because anything that limits the usership of a messaging app will sink it. It's the opposite of "support everything" strategy that made them successful.
Yeah I agree with the idea of pivoting. I suppose to clarify:
* Pivoting would leverage existing technology built in the process of initial concept. Which to me is the equivalent of scrapping the initial idea (while salvaging the generally-useful IP/technology).
* Adding a bunch of tangential features to a product to increase revenue is a colossal fuckup scenario (maybe the language is a bit over dramatic).
For instance, Google is great at search; gmail is cool; docs was innovative (albeit limited); and then… https://killedbygoogle.com/
Unfortunately, after seeing this time and time again it’s tough for me to get behind mainstream tech. I loved the old Microsoft/Nokia phones; and the Zune. You can tell a lot or love went into the design/engineering but then projects just get axed by corporate interests.
Meanwhile, you can by a mechanical device or appliance from 1950s and it’ll still work just fine.
Again, I don't disagree with you... just think reality makes it messy.
Pivoting via "scrap & salvage" is pretty tough for a startup. The tech behind whatsapp was evidently good, but the IP behind a messaging app is probably not enough to give you an edge. Users are.
Made up scenario: whatsapp loses the SMS replacement game. They have millions of users, but not a billion. Meanwhile, they find that a subset of users like to use whatsapp for dating (or customer support, etc. doesn't matter). They pivot to focus on those customers, and evolve into something else.
This might be a (drama noted) colossal fuckup scenario in an engineering sense. A tractor that you are now converting into a ship.
Evolving is definitely a worse way of engineering than starting with the intention of designing a ship, with neatly defined tonnage, speed and size requirements. Instead, it takes a miracle to implement basic ship features like floating.
Evolving is how a lot of actual software gets invented. Spreadsheets were intended for accountants. They weren't meant to be used as a database, incident report generators, a casual programming environment, or a HR tool. It became those things by evolving.
It happened that way because inventing UIs is hard, and evolving into them happened to work. It's still true that "colossal fuckup scenarios" arise because of this approach. Excel programmer spent decades making excel better at things it's architecture wasn't good at. It's ugly and messy, but life is sometimes ugly and messy.
Flexibility is valuable. Knowing the spec in advance is valuable. Very valuable. They're in conflict with each other to some extent
I think it was 79 euro cents for a lifetime at first (this thread brought back memories of my installing WhatsApp on my phone while I was asleep), then it became 89 euro cents per year (except for the users who had been grandfathered in), and then came the Facebook acquisition and they made it free for everyone.
Well, WhatsApp had to support exotic cases like BlackBerry, Symbian OS (Nokia), and Windows Phone. I will say, dealing with JS stacks, seems easy in comparison to mobile fragmentation from back then.
I don't see how this is related. Server-side you write and expose an API. Then you have 1-3 developers for each platform that write code that consumes that API. Those developers don't need to know anything about the internals of the server.
I wish that WhatsApp could make a public API. I'm very thankful that Facebook is integrating WhatsApp into their services, because that gives me hope that a true web interface will be developed.
For me, it takes between 10 hours (worst case) to 5 minutes (best case) to open WhatsApp.
My iPhone 4S is too old, so opening WhatsApp gives a message "This version of WhatsApp ha..., Update WhatsApp, Your update will be free of cha...". Clicking Update WhatsApp opens the App Store, and clicking Update results in another error message: "This application requires iOS 10.0 or later."
So I only use WhatsApp through BlueStacks Android emulator on my personal laptop. That's what leads to the 10 hour worst-case response time: a full working day, including commute. The 5 minute best-case is because BlueStacks takes a very long time to open, and runs my laptop fans really loud.
This is not only a pain point with WhatsApp, it's also a problem with WeChat, KakaoTalk, LINE, even 9gag (for posting videos).
Meanwhile, email continues to work just fine on my decade-old phone, mbasic.facebook.com is particularly speedy, iMessage still works, and Hacker News is great :)
Hey, if you do prefer that, I too have a broken phone, but the new multi-device beta let's you to have whatsapp in your computer with a powered-off phone. That works for me.
But this speaks to good architecture decisions, right?
If they tried to make a shared framework that ran the same codebase everywhere and then port it onto different platforms (cough facebook) then you don't get these nice isolated issues, for example.
Yes, if you don't write a native app for every platform the result will be rubbish. That's not a "good architecture decision", that is common sense. If they don't do it that way it's to save money, because as we all know Facebook is strapped for cash. (Talking about Facebook and Instagram here, because as far as I know WhatsApp is completely native on every platform)
This then also means a well defined server API without undocumented behaviour that individual implementations depend on. I stand by my assertion that the good scaling and separable builds is a sign of good architecture.
But this would be client side only, right? These issues would also only occur in the corresponding app repository, when building a separate native app for all platforms. The backend for all of these apps would be the same, not the frontend.
Sure. And so do I. Specifically, I don't want to rebuild systems that have already been built for me which I can use as dependencies. And I don't want to spend all my time troubleshooting bugs, either - I'll take a well-maintained repository on GitHub with thousands of users bug-testing it for me over some janky thing some guy on another team threw together a few quarters ago before he quit.
Simple isn't always better. Reusable patterns are. WhatsApp followed pretty standard erlang design patterns.
Rails and Django to some extent force a standard pattern, they're not really low dependency systems(although they are compared to most node projects). But that pattern to some extent makes sure that you can hand over the project to the next dev and he'll be able to make sense of it.
Somehow node.js has become what PHP used to be.
Whatever happened to interoperability? How are there hundreds of queuing system that use redis and rabbitmq under the hood, but you can only process things in python, javascript or ruby? The data structures are considered private.
So if you want to process your code in python you have to use the python message queue, if you want to process it in javascript you have to find a node mq. How does that make sense?
Want to do business intelligence on your javascript based system now? Gotta write javascript code. Welcome to debugging the same kind of memory leak and processing issues every other queuing system has had to go through.
I agree, but this assumes you actually know what you are building. This isn't usually the case for a startup.
I'm sure whatsapp wanted a million users, but they didn't originally know that would happen. If whatsapp later turned out to be about serving fewer users with more features, this becomes a story about premature optimization.
Do we really think Javascript is exotic? Hasn't it proved itself over many years now?
That doesn't mean it can't have issues like anything else. And because of the massive scale of the language you'll see a lot of shit. It's a language and ecosystem like anything else - it's what you do with it that matters.
I wouldn't say JavaScript is exotic, it's more to do with what is being done with it.
Traditionally it was used to enhance HTML pages with some DOM manipulation but it's now being used for a million other things too.
So historically, if your JavaScript code broke your page would still render as it was just html but now there are all sorts of build processes and long chains that use JavaScript to create the html in the first place... I'd consider that the exotic bit.
Traditional JS running in the browser manipulating DOM elements is very much a foundational aspect of the web and won't deteriorate with age (with the exception of perhaps deprecated functions in the far-off future) but all these js libraries and tools and build processes with massive dependency-chains are what's being referred to.
It's been used as a general purpose programming language for over a decade. It is a 'boring technology' nowadays and hardly exotic.
Of course, you could decide to do something exotic with it (edge rendering, data programming, whatever) but that's not a problem with the language but people wanting to stretch themselves...
JavaScript has matured massively over the past 10 years. I'm not a Node developer myself, but JS + TS is a very palatable combination to me. I'd pick it over many other languages.
10 years ago I would not have touched JS unless I was forced.
I'd be interested to know which ones, because I can't think of one advantage that Javascript has - aside from the ubiquity of browsers. If we're comparing any other aspect of it though, I just don't know what advantage it has (seriously).
Functions are first class. Async i/o by default. Familiar syntax (I'm sure Haskell or Clojure are better languages, but they sure take some time to get used to. You can be fairly productive in js in very little time). There are packages for anything you can think of, no need to reinvent the wheel most of the times.
And I even like it's single threaded nature. Being able to keep a process waiting without needing to spin a new thread with all that implies is very convenient.
Edit: Original commenter had already mentioned typescript.
So what's the deal with TS? I think on top of adding much needed type safety to Javascript, TS is also one of the best type systems you can ask for.
Firstly, thank you for answering. I think most people would be too defensive (devs are a touchy lot:) to bother or just dismiss my sincere question. I'm going to disagree though (quite gently, I hope:)
> Functions are first class.
Always a good thing to have in a language but I struggle to think of a language that doesn't have this now. Java has it, C#, Python… if a language is being developed and didn't have it then now it does, right? Perhaps C doesn't - I checked, functions may be passed as arguments but does not allw nested functions, but it has callbacks so a type of closure is possible… hard to tell. Still, I'd say not using C was an advantage!:)
> Async i/o by default
I'm not sure which languages this gives an advantage over?
> Familiar syntax
I don't think this is an advantage, most languages are much of a muchness, and there are some serious footguns that still lay around in JS. Is `==` familiar? No, that's a footgun. Automatic semicolon insertion (ASI) rules? No, they're another footgun. Are fat arrow operators familiar? Doubtful. The new backticks? They're the other way round to what I'm used to.
Certainly though, it can be picked up quickly enough.
> There are packages for anything you can think of
This is definitely an advantage over some newer languages, but over those in major use, it's not. Still, the way packages are installed and handled is… eye-opening. That wheel has been reinvented several times, and it still looks wonky.
> I even like it's single threaded nature
Sometimes that's a good thing, true, but it's not an advantage to have it set that way all the time.
> TS is also one of the best type systems you can ask for
I haven't used it, tbh, but I would say that an optional type system (it's optional in that you don't have to use TS, I don't know if it's optional once you start using TS, that would be better) is a definite advantage over some languages. It's not baked in though so that's half a point.
I just don't see it. I think the driving force of the move to Javascript everywhere was because devs were tired of learning at least three languages (JS, something, SQL) and not being very good in all three (usually poor at SQL) and thinking they'd get more control by kicking out the DB guy who'd stop their bad ideas (because they didn't know SQL) and they could become an expert in one language to rule them all. Companies love it because of fungibility and more new devs tend to start with Javascript than anything else.
Perhaps I've just got used to keeping several languages in my head?
> I think the driving force of the move to Javascript
> everywhere was because devs were tired of learning at
> least three languages [...] Companies love it because
> of fungibility [...]
>
> Perhaps I've just got used to keeping several languages in my head?
Another possibility that doesn't involve you being smarter than everybody in the room is that many people that were capable of learning several languages saw that the majority of companies wouldn't care for this and instead would prefer to hire for a fungible language. They specialised in this language and then used their free time to learn other skills so they could become more valuable in the marketplace.
You wrote a lot of words about how others were "tired of learning" and "not being very good" and "kicking out the DB guy who'd stop their bad ideas" and then finished it off by congratulating yourself on keeping several languages in your head. I can be rude but you should think about how this would be perceived by others.
No, you should try to keep your temper, you could’ve provided your objections without resorting to rudeness. As such, I feel that I’ll simply ignore your point of view.
Somehow I suspect this isn't the first time you've ignored other people's points of view and felt righteous about it. But, the way you act, must give the people around you a lot of entertainment so I'd certainly not want you to change!
Mate, there are published guidelines for the site. If you don't want to follow them, that is up to you. People aren't even supposed to be snarky but they are, sometimes I am - we're all human and can be a bit spiky at times - but outright rudeness isn't defensible. Of course I'm going to ignore you because of it.
> But, the way you act, must give the people around you a lot of entertainment so I'd certainly not want you to change!
I wanted to point out to you that there are other reasons for people choosing different choices to you which don't involve them being "tired of learning" and "not being very good".
You described yourself as being "used to keeping several languages in [your] head" as if this was an achievement that others couldn't muster.
I still feel that your derision towards others and self-congratulatory tone were deserving of rudeness and snark.
Anyway, I apologise for being rude since it was hurtful. I thought your comment was bad and let you know this by directly accusing you of exactly what you were doing, instead of finding some non-confrontational way of saying it.
Typescript has a pretty cool type system. A mixture of dynamic and static typing. You can be as dynamic as with js (or python), or stricter than Java (you can state a reference cannot be null for instance), or anything in between (and have dynamic parts and static parts in the same program).
I haven't used Typescript but I've generally heard very good things about it. I didn't know you can mix and match, that might give me a bit more of kick to include it in a project, thanks.
Whether or not it's an advantage of Javascript… I'm not sure. I mean, if someone wrote a transpiler for Java that overlayed stricter typing, would that really be an advantage of Java? Not sure, but of course, worth considering when choosing a language for a project.
> I still can't understand why today people choose a (Javascript) stack of build systems, ton of dependencies, and all kinds of exotic tech that is the latest and most hyped.
So the same cannot be said for "newer" languages like Rust, Go, and other's whose ecosystems and paradigms aren't completely fleshed out yet?
This comment reeks of someone who is looking from the outside in when it comes to building stuff in JS. Most comments I see criticizing JS stacks are so superficial and demeaning that's its obvious the commenters have little to no experience working in JS.
Why do you compare it to Rust though? JS is as old as Java, however its ecosystem seems to be much less mature (at least from outside). I haven't touched Java for 15 years, but I am fairly confident that people still use Apache Ant as build system. Every time I try to peak into JS world, it appears similar to tiktok trends in terms of how quickly people move from one thing to another.
It's even worse from the inside. The last 2 companies I have worked at have had significant amounts of JS code (even the majority of code) and it's inevitably became unmaintainable mess through a combination of lack of solid frameworks to instill structure and trying to apply it outside of its domain of the browser.
The last 3-5 years have convinced me that JS isn't appropriate outside of a browser almost ever. I am sure if you think hard enough you can think of cases where it's superior to some other lang but in general it's a very poor choice for almost anything that isn't DOM manipulation.
Worst was definitely attempting to diagnose an off-heap memory leak due to C extensions. Naturally the JS folk gave up and dumped the problem on my lap so I proceeded to do my usual "C guy" stuff and was amazed at just how bad stuff like node-gyp and friends are and just how fragile everything is. I found the leak and patched it and all was "ok" again but just peeking inside those layers makes you deeply uncomfortable with the runtime in production.
The rest of the problems can probably be attributed to lower quality developers but point remains. Things like lack of structure leading to insane architectures, pushing for microservices without understanding the tradeoffs because they didn't want to work on "legacy" JS code that was built with last years hipster tech rather than this years, etc.
These problems are endemic to to culture and ecosystem which IMO are inseparable from a language/tool in practice despite what we want to believe in theory.
I don't refuse to work with JS but I definitely make my concerns abundantly clear and I generally don't hold back with "I told you so" when it inevitably bites people in the ass.
> I am fairly confident that people still use Apache Ant as build system
As in: there are still (older) projects around that haven't (yet) migrated away from Ant? Sure.
As in: Ant is still the go-to build system or at least still commonly used? No, not at all. When I create a new Java project in IntelliJ for example, I can choose between Maven and Gradle as the build system. Ant isn't even offered as an option.
JS is as old as java, but they're both talking about the build process of JS.
in the beginning, JS wasn't really transcoded/compiled. The first time i personally found out about that was sometime after 2010 with coffeescript, not sure if there were preceeding examples.
and your claim that people move around is really false. React has been the defacto standard for a pretty long time at this point.
yes, other UI libraries/frameworks exist, but reacts marketshare has been extremely dominant since it displaced jquery/ember etc
Express isn't a framework though, maybe a "micro-framework". The nearest competitor to real frameworks like ASP.NET or Spring is Nest.js and it's definitely not got the same longevity as those stalwarts.
JavaScript is from 1996, it's 25 years old, and apps built with it are used by billions. With regards to the backend, it has a vibrant full stack ecosystem, it's performant, simple, and easy to hire talent for.
It's okay not to enjoy JavaScript (I'd always prefer to go with Dart or Rust myself), it's okay thinking that other languages (and ecosystems) are better equipped to solve some problems, but calling it unproven and exotic is very surprising to me.
Is the implication that FreeBSD is less mature now, intended? I'm assuming not. The BSDs have grown in popularity significantly in the last couple years.
No, but there are signs of distress, IMO. I think their decline in usage has resulted in a decline in port maintainers, which is a bigger deal now that systemd is the default on Linux for so many things.
In the 32 bit days it wasn’t hard to get most Linux binaries to run, now it’s virtually impossible for an end user to do on their own.
Pretty much nothing important depends on systemd, for the simple reason that it’s proprietary and non-portable. Thus, even if there is some code that depends on it, it’s optional.
Also, Linux binary compatibility - the capability to run unmodified Linux binaries - works much better now than ever before.
Seems redundant to have a Express backend + REST API, why can't you just build your REST API on the Express backend?
And also, React is complex, hardly anyone seems to know the internals. The API interface might be easy to learn, but "simple" is not something that you should label React with.
I'm not sure I understand why people build websites like that. My theory would be that usually at some point you have to go lower in the stack (learn how Linux/Node/V8 work or stuff like that). You can avoid going lower in the stack, but the complexity has to go somewhere, and that's what we see here.
"Personally I want to build systems. I don't want to spend my day debugging all kinds of weird problems."
Don't we all, I guess we all crave that snap, click, build -feeling Lego gave us. But in my experience building a system means debugging all kinds of weird problems. At least, when I start something, I usually for a long time feel like I'm just hopping from weird problem to weird problem. But, TBH, those problems don't seem weird anymore as experience grows.
I still can't understand why today people choose a (Javascript) stack of build systems, ton of dependencies, and all kinds of exotic tech that is the latest and most hyped. As a developer you need to support this in the future. It might be nice to build today but it will be a nightmare later.
You know that JavaScript developers don't see JavaScript as weird or exotic, right?
Honest question, is specializing in a JS stack bad in terms of career prospects? I've been using express + react/vuejs professionally for a couple of years and I wonder if it'll be in demand in the next 10-15 years.
You’re fine — don’t listen to JS haters on Hacker News. They pine for the simpler days of sites that primarily use HTML and CSS, with maybe a little bit of JS sprinkled in. But consumers are increasingly demanding an app-like experience from the web, which requires JS frameworks.
As for Express, JS on the backend will be popular as long as JS on the frontend is popular. JS backends have their flaws, but there’s a lot of value in using the same language in the browser and server.
> But consumers are increasingly demanding an app-like experience from the web
This seems tendentious. Which average user was looking for these changes? Can you point to a site that shifted to an "app-like experience" and became successful because of it?
I think it's a "general trend" rather than a website by website trend. I think that 20 years ago most of the discussion online took place with things that looked like websites (forums, mailing lists). These days, most of it takes place in places that look like apps (Twitter, Facebook).
This is a strange way to phrase the question. Sites that had to shift to app-like experiences are hard to find, because nowadays pretty much every web app is an app-like experience from the beginning and is created with a JS framework. The shift happened years ago.
Barring informational sites like blogs and news publications, it’s actually more challenging to come up with new web products that are NOT app-like in nature, and that do not use any kind of JS framework. Craigslist may be one of the few big ones and even it is losing market share to FB marketplace, which is an app-like experience.
If I start with 1990s ebay, does it become app-like when I add the ability to zoom images without a pageload? When I add a WYSIWYG listing editor? When I let people drag and drop images into their listings? When I add JS infinite scrolling to search results? When I add AJAX search autocomplete?
Or do I have to go as far as Google Docs, re-implementing copy/paste functions, taking over the mouse wheel, and adding my own text highlighting and zoom implementations?
My personal "line" is when links won't work without js and urls aren't written in the location bar. It makes a site quite useless without js. I know that progressive enhancement is supposed to be a thing but I've yet to see it outside of tutorials (browsers not properly supporting PWAs could be part of that, but I doubt it).
Google Maps was successful from the start, and in my view, has become worse over time. Docs started off "app-like", though I haven't used it in such a long time it may be different now.
JS is everywhere though. You even have parasitic languages that compile to JS (ClojureScript, TypeScript, etc). You got Node and also Deno for back-end stuff with support for multithreading. Concequently, JS is among the most popular, and most used languages.
I do admit the ecosystem needs to find a way to have more stability, because NPM outdated package tree nightmare is pretty bad, but I guess that's what you get with an industry that moves so fast, so in that case it's up to you as a developer to not use too many dependencies.
Anyway this is all to say I don't think JS is going anywhere for the next 10 years.
Yes, but you'll find that a lot of work becomes maintenance of older systems, and less new and shiny things.
I do think the 'craze' of new frameworks popping up left and right quieted down some years ago, and things have roughly settled on React, Vue, maybe Angular (which lost a lot of its shine after the Angular 2 announcement fiasco), etc.
For a new application, I picked React + Go because I'm confident that ten years down the line it will still be maintainable - although, Go moreso than the front-end, which doesn't feel nearly as solid and stable.
But there are great things happening. The wider community has solutions for everything. (For example here's the "reactive forms are not strongly typed" issue [0] that showcases both the good and the bad. The need for this feature has clearly emerged in 2016. People stepped up and a PR was created, but ... basically no signal from the Angular team. Of course using a wrapper was an easy workaround with a distinctly sour taste in the mouth. Then finally something happened and an Angular team member now seems to be working on it in "full steam ahead" mode.)
I recently had to maintain a large React + NestJS application and I was seriously considering organizing a terrorist cell to go back in time and ...
Same stack as what I chose for my personal projects. This is just my opinion, JavaScript on the server was a horrible idea. There are far better languages to reach for when writing API’s and other server related stuff. I’m curious how you are handling authorization/authentication (assuming you are developing api’s)?
No, it's great. But keep up with the "ecosystem". Learn a bit of frontend, backend and testing too. (Then keeping it fresh won't be a problem.) And we shall see where it goes in ~5-10-... years.
These issues that you described can happen with any language, ask any programmer or system engineer with long enough experience they’ll have stories with weird problems.
Regardless which language you pick, there will always be these kinds of issues.
> These issues that you described can happen with any language
While you're not wrong, I do strongly feel it depends on the language and architecture chosen. I'm a Go developer these days, and there's a big mindset to e.g. avoid dependencies, and for those dependencies to avoid including even more dependencies, keeping things fairly lightweight and with a low 'attack surface'. I mean I personally wouldn't mind a stricter type system and native enum support and some other things, but for now, I enjoy how basic it is, whilst avoiding the footguns that C/C++ brings.
See whilst obviously the JS ecosystem is quite volatile and can be trend-based, I think there are benefits - the language breeds innovation. Innovation doesn't always make the correct choices, and some tech-conservativism is important in many types of companies. But if our ancestors had just been lazy and just stuck with what was known and good, we'd never even had computers in the first place.
This article does very little to explain how they did that. It basically states the type of tech they used (Erlang, FreeBSD and SoftLayer), and something about not trying to over engineer stuff. The title also makes it sound like 500 engineers would make it easier to scale up WhatsApp, which does not make too much sense either.
Thanks, it does seem like choosing a language that was made for doing things in parallel to begin with was a good idea. To be honest, I've always felt that programming in functional languages creates less bugs, but perhaps that's just my imagination.
The technology stack isn't even that relevant. There have been a few different articles about WhatsApp and their 1B users with only 50 engineers. The key in previous articles was that WhatsApp hired really smart people.
Having extremely talented engineers and a rather small scope is what allowed them to grow til 1B users, with a minimum staff. Yet people seem to focus on the technology, because that's more easily replicated, and easier to accept, in my opinion.
I think the technology stack they choose is highly relevant to their success.
The choice of Erlang was crucial: it's basically built for communications and has all pesky issues like version updating, resilience to failure and scalability already taken care of by nature of its architecture and framework.
Of course, these 50 engineers were smart, most in that specialised space are, but I strongly doubt that they could have achieved the reliability and scalability of what made the success of WhatsApp with a PHP or Ruby back-end without more complexity, more ressouces and more hardware (like what FB and Twitter had to go through).
Well true, but my point is that you're not going to be able to scale as successfully as WhatsApp, just by using the same technology stack, if your problem is different.
You're right that Erlang was/is the right choice for WhatsApp, and it was most likely picked as the language of choice, because of the smart people working there. It's the same with FreeBSD, a failing startup isn't going to be able to layoff 50% of its engineers just by switching from Linux.
Erlang was picked for a particular reason. WhatsApp uses a prebuilt open-source chat solution, an XMPP server written in Erlang, ejabberd. This thing was first released in 2003 (old). Already a smart move to get started.
I’m pretty sure they hired Erlang developers to dig into ejabberd internals and optimize certain things. They didn’t just decide to become an Erlang shop out of the blue.
It wasn’t Erlang that was the initial right choice, it was using xmpp and ejabberd that was the root reason. Erlang just happened to be a consequence of that.
I will contest your attribution of ‘smart’ with Erlang. These types of correlations are generally bias fitting. It justifies ‘smart’ being correlated with any and all niche languages, eg ‘so and so likes Haskell, so they must be smart’. No good.
It’s better to attribute ‘smart’ with the pragmatic decision they made to simply use a pre-existing chat server solution that already has the capability to scale. Harder to assess this as smart since there’s no ‘signaling’ here, you have to objectively assess if it was the right tech (which it seems like it was). Way less vanity in this assessment as opposed to what I already pointed out, how your Haskell or Rust devs must be particularly smarter, as opposed to say PHP or JavaScript devs who are considered dumber. I don’t buy it, I need to see more than just your affiliations.
So, I reject your initial post contending ‘The technology stack isn't even that relevant.’ It was precisely the tech decisions that mattered, and the right people to make such decisions. Chicken and egg scenario, I’ll concede that.
In any case, one does not simply pick a old chat server written in Erlang out of the blue - this decision was critical. How many over-funded tech teams would try to do this from scratch in Go? Plenty, and that whole team would easily be full of ‘smart’ people.
> I’m pretty sure they hired Erlang developers to dig into ejabberd internals and optimize certain things. They didn’t just decide to become an Erlang shop out of the blue.
Eh, I was there since October 2011, and we didn't hire any people who knew Erlang until much later. All of the early server engineers (including me) learned it on the job. By the time I left in 2019, I think we hired two people to the server team with previous Erlang experience; it's hard to find people with it, and while it might have been nice, it's not important.
It's possible we had some consulting possibly before I was hired, but I don't remember seeing any evidence of that; OTOH, I do remember setting up and working with a FreeBSD consultant and Moxie when he was consulting on end to end. From my understanding, when things started bottlenecking, Rick Reed was hired to fix bottlenecks; which he had been doing at Yahoo! for many years and had worked with Jan and Brian there.
FWIW; WhatsApp the service started as just a text status, built on PHP and MySQL, but people were using it to chat, so the founders went looking for a chat server to use rather than building one from scratch. I don't know the decision process, but ejabberd was then powering Facebook chat at the time. (Of course, Facebook abandoned Erlang, they said because they couldn't find people with Erlang experience to hire)
Anyway, by the time I got there, I was told that the chat server had been mostly refactored over time and while a lot of names remained the same, and some of the basic architecture was the same, it wasn't ejabberd anymore. Mostly I worked on things that weren't chatd, and I don't think I've seen ejabberd code, so I can't verify, but it seems likely, as we customized the protocol, auth, offline messaging, contacts, session handling, etc.
Erlang is a tremendously right fit for a chat server, and hot code loading is almost necessary when you have hundreds of thousands or millions of connections per chat machine and want to push small changes. Of course, changes to BEAM itself, or the OS kernel take restarts, so you still need to do those from time to time.
> Yet people seem to focus on the technology, because that's more easily replicated, and easier to accept, in my opinion.
I think you are correct. It also works as some sort of advertisement. Others feel like they made the right tech choices, if they choose the same tech as WhatsApp or Spotify. Completely forgetting that they probably need to be somewhat proficient in Erlang to get the benefits from the language etc.
I guess that "hire extremely good developers" is less sexy than just choose Rust/Erlang etc...
It's not just less sexy, it leads to frustrated readers/viewers of your article/blog post/conference talk etc. If someone asked how to build a better system for their company and the answer they got was "first spend 5-10 years becoming a better programmer", they will be less happy than if you sell them on the idea that adopting Rust/Erlang/etc will magically make the hard task easier.
But if you choose technology for its intrinsic merits you attract engineers with taste, who have the drive and luxury to care about their work on a different level.
There are plenty more people who can't be arsed to or don't have the time or simply don't find it valuable enough to adopt "niche" technology, despite technical merits.
I don't think it has anything to do with smarts, but rather with priorities and human nature. Most people follow the mainstream because they favor stability and convention, those who break out in different directions favor autonomy and freedom.
From my experience it's always a good idea to choose a language that people would have to learn by themselves, i.e. none of the school languages (Java, C, C++, C#, Python, Javascript etc).
In that way you only attract people who do sit down and learn new languages by themselves in their spare time, which filters out people with less interest in programming at least.
Absolutely. A relatively small, highly efficient team that invests in their knowledge, skill and processes is the cause of correct technology choices and not the other way around.
The article isn't really "an article" but it's an email newsletter I send out. I'm quite surprised to see it on the frontpage of HN lol.
So what you're reading is one of my emails.
It's meant to be more of a summary with additional links to sources where people can read more if they're interested (emailing people long-form articles doesn't work well as a content format from what I've seen - even ignoring the length limit that email has).
I'll add in some more links for additional reading (like the high scalability post) to give some more detail for people interested.
OK, I guess any post on this site is dissected a bit too much - looking for insight that was sometimes never even meant to be there. It is usually a good thing, as we expect anything making it to the front page of HN to contain some profound knowledge. Sometimes a popular title referring to an interesting topic can gain a lot of up-votes as well, and then the criticism can be harsh :-)
> The title also makes it sound like 500 engineers would make it easier to scale up WhatsApp, which does not make too much sense either.
It's something I think we have all seen a lot of times - that by the time a company is serving 1 billion users it has quickly expanded out it's engineering team hugely, and because of that additional abstraction is required and the complexity/LOC skyrockets.
"this team of 7 engineers is responsible for formatting the text content of a chat, including alignment and sizes of emojis and gifs"
"this team of 4 engineers is responsible for formatting the date of a message"
"this team of 7 engineers is responsible for the overall formatting of a chat"
"this team of 5 engineers is responsible for the formatting of the non-chat pars of the application, settings, profile page"
"this team of 4 ux persons is responsible for aligning the non-technical parts of formatting and the user experience over all parts of the application"
"this crossfunctional team of 5 is responsible for creating a framework to let the configuration of formatting be disconnected from the actual implementation of the formatting"
"this team of 7 QA engineers will aid in manual verification of changes and bugfixes but will also automate test cases for formatting in the entire application"
"this supporting team of 4 will develop automation tools and enable the formatting teams to collaborate in a high speed agile context"
> it has quickly expanded out it's engineering team hugely
But makes little sense (as a developer/engineer myself) to think that growth in users, requires a ton of new developers/engineers. We are not tattoo artists, our job can scale indefinitely if set up properly, i.e. all you should need to scale further (to 1BN or 7BN) should be enough money to buy more hardware; which WhatsApp/Facebook clearly has.
You're assuming that their use case & technology stack & application scale cleanly horizontally and they're not spending all their time fighting fires.
True in this instance, but far from a universal truth.
I think that makes sense if the application doesn’t need to also grow much functionally during the expansion, but I believe the more common pattern is more users = more functionality to manage those users, support teams, global legal compliance and accommodate more niche phone systems = more engineers.
There was a joke, or maybe anecdote, going around the Israeli high tech industry when Google purchased Waze
When it came time to get an overview of the codebase being purchased. The Google Maps car navigation iOS team had over 200 engineers. The Waze iOS team had 2
> The Google Maps car navigation iOS team had over 200 engineers.
I don't really believe that. I worked at Google and the frontend teams were usually just a couple of engineers per target client, I doubt any projects had 200 entire engineers just for a single target client. Not even big projects like gmail has nearly that many.
Edit: If I were to guess Google maps would have 200 engineers in total, or at least 200 figure would be the entire team of frontend engineers for every target client, and then many more working in other roles. 1 engineer per million users is a pretty good rule of thumb for Google products, and a large majority of those will work on backend systems.
off topic but I recently started working with an Israeli tech company and I'm amazed by the culture they have, what about Israel produces such high levels of entrepreneurship?
Some of these points are a bit suspicious, and I bet don't tell "the whole story" the way they're intended. For instance, the statement about not investing in automation unless absolutely necessary or the point about hot-swapping Erlang code.
Would be good to actually hear from one of these engineers for a discussion on the paint points of working on WhatsApp back then.
while WhatsApp is the largest I am aware of (it's one of the leading platforms Worldwide anyway), there are quite a few, sometimes in the form of Elixir:
Why is this being downvoted? Whatsapp is notoriously "just" a highly-optimized fork of ejabberd [0]. ejabberd is written in Erlang/OTP which enables quasi-horizontal scaling and amazing failure recovery, along with incredible uptime.
What's different about Whatsapp compared to a traditional XMPP experience is:
- it uses a single centralized server, without federating
- it uses phone numbers as identifiers instead of nicknames
- it has a tightly integrated (closed-source) client
So you can not only "build your own WhatsApp", but also federate with others doing the same, which is pretty cool and prevents a single actor from abusing the whole network.
Doesn't anybody remember how literally Whatsapp's founder wrote on one of the ejabberd mailing lists with something like "I just installed the server, please help me configure authentication" or the like?
Retrospectively that has always colored my perception of how relevant technical aspects are for a successful startup (i.e. not much). Marketing (and viral, user-capturing & monopolist practices) are (sadly) much more important.
> how relevant technical aspects are for a successful startup (i.e. not much)
That's why i'm very interested in human and political aspects of free software. Successful FLOSS projects usually have a strong community backing them, and strong connections between developers and the community so that the technology isn't too disconnected from practical needs and UX concerns.
I think non-profits and cooperatives building upon free-software solutions are a better alternative model. For example, framasoft.org (french NGO for libre culture/software) has been instrumental in developing new solutions like Peertube, and in kickstarting a new wave of hosting coops (chatons.org federation).
In the IM world specifically, i'm very interested about Snikket.im project. It aims to build a complete client/server distribution of XMPP software tailored for specific user expectations.
If you look at https://www.infoq.com/presentations/whatsapp-scalability/ though it gives a very different picture. Sure, the founding team may have tried to prototype without knowing full technical details, but they knew enough to choose software that could fundamentally scale towards the future, and they rapidly brought on fantastically talented developers with a hunger for optimization. You can't choose the right technical people without technical chops; that being said, in many cases the right technical people are those who know how and when (and when not) to stand on the shoulders of giants!
It doesn't look like "they knew enough to choose software that could fundamentally scale". It looks more like they choose the first option that was available to them. Apple also choose XMPP for Facetime as did a million other companies that serve similar scales of simultaneous clients.
They may have hired people later on (though this entire article is kind of showing that no, they actually didn't hire that much technical people). But by the time they even hired the first Erlang engineer they had already won over the European market. And for the record it was not precisely due to the quality of either their Android or iOS android apps -- the J2ME and Symbian clients had quite a following and I would even bet were more used.
Original Whatsapp was a disaster and this did not prevent them from gaining marketshare. You could basically login as anyone just by having their phone number (which maps to their JID) and defeating their XMPP obfuscation layer (some XML encoding, which was easily done due to the easily RE Java clients).
It was only later on (2011ish?) that they started getting somewhat serious, which coincided with them, already swimming in users (and money?), starting to get more evil (with a more hardcore stance against 3rd party clients).
Not at all, I have a bunch of friends on my private server and getting someone an XMPP client and account takes less than a minute. No phone number or email required.
Sure, everyone out there isn't going to be on my server - but for the close friends I care about - it's been flawless.
I decided to delete all proprietary messaging apps and use XMPP exclusively in Janurary this year. It took some time to explain but now most of my friends and family can now be reached via XMPP.
The many comments alluding to the technical challenge of building a chat app with 50 engineers being relatively easy are forgetting that scaling said chat app to 1 billion users is absolutely not easy and a function of more than just the engineering quality and technical challenge. There's MUCH more to a successful product than engineering...
This site is getting quite repetitive. I think the balance of News to things of interest has gotten quite out of whack. This article is not news, and is also in recent memory so not an interesting re-visit. Is HN average age decreasing? The content character has changed in recent years. Something's changed/changing.
I thought I was the only one who felt this way. Almost every day I see the same "style" of articles that isn't news or that interesting. One thing I've noticed is almost every day I see an article explaining why remote work is better and/or how tech interviews are broken, both of which are popular opinions a lot of people on this site share. Maybe it's just me though
I would expect there to be certain scaling. Specially when you move to more severs and more redundancy. Cloud solves part, but still not everything. If single person can support 10k or 100k users. You might still need more than that one to move to million or hundred million.
Also some features like language support and such get more complicated when you want to widen userbase from just English speaking ones.
I asked if there was a correlation here, using an if statement as an answer doesn't answer it haha. You're now asking the same question I was, just in-line
Whatsapp used to charge 1 eur or usd per year per user.
Assuming a 20% vat, that’s $800M in yearly revenue. If you spend $500k per engineer per year, you’re still left with $775M to handle infrastructure and all other expenses.
Whatsapp would have been sustainable without fb. That’s the regret.
> If you don't sell to us, we'll build our own and offer it for free
Weren't they already doing that? Facebook Messenger officially launched in 2011 (essentially an upgrade to Facebook's built-in chat functionality which existed before that) and was free, yet Whatsapp continued to grow.
Exactly. Facebook bought WhatsApp’s users like cattle on auction. WhatsApp guys just needed the right price offer. Now with pseudo tech posts like the current they are trying to mask it’s public image behind some ancient tech innovation from it’s past…
WA Management was misled by Facebook about what FB wanted to do with WA. I agree that you'd have to pretty naive to believe FB but it's not like WA founders sold solely because of money.
WhatsApp founder Brian Acton left 850 million USD on the table when he left Facebook prematurely in protest of Facebook's lies about the vision they had for WhatsApp after its acquisition. After that he donated 50 million USD to the Signal foundation.
Technically, but GP is allocating over 95% of post-tax revenue to everything other than engineer salaries, which is probably extremely generous.
WhatsApp had 200M users in February 2013[1], which was a year before they were acquired by Facebook. Perhaps they wouldn't have gotten to 1B users without Facebook, but I'm sure they still would've grown in the last ~8 years, and fewer users likely results in lower costs. My guess is they would still be doing very well.
Even before the sale, most of the customers are non paying. They are offering the service for free in countries with largest userbase(India and Brazil I think). I really doubt it had reached 50 million paying at any point in time.
They could also easily have had different pricing based on location. Requiring EU and US users to pay €5 per year, while have perhaps just €0,25 in Asia, and free in Africa.
If you can't get someone to pay €5/$5 per year for a service they use every single day, then there's something really wrong in how we as users think about the service we use every day.
> If you can't get someone to pay €5/$5 per year for a service they use every single day, then there's something really wrong in how we as users think about the service we use every day.
Which is absolutely the case with most people and recurring payments, even with one time payments. If Signal costed money (even 1-2$), I could not have convinced even 10% of the people I got over to Signal - recurring or one time.
I've used Whatsapp for a few years when they had this pricing model. Use 1 year for free and pay after. I've never paid anything and nobody who I know needed to pay to keep using WhatsApp.
Whatsapp had 700m+ monthly users in early 2015 [1] before they scrapped their subscription fee in early 2016 [2]. Whatsapp was free to use in some jurisdictions for the first year, so we can't assume there was $700m in revenue, but there was certainly enough to pay for 50 engineers you would think.
Whatsapp only had a subscription fee for android users, and I don't know anybody who ever paid for it. They would keep giving you more and more free time; free subscription time would never run out. I imagine it ran out for some users, but I don't know what the criteria was. Perhaps for users in certain countries? Or for users with lots of iphone friends?
On the other hand, iphone users had to pay for the app upfront to download it off the app store.
Whatsapp is built on a massive, mature, global system specifically designed for sending messages. It's sad that it needs to exist and even more sad that it has to "make money" for someone. Aren't we already paying for our internet connexion/phone bill? Why do we need to be used by Whatsapp just to utilise it for its most basic purpose?
Does it? Dumbphones from 30+ years ago can still send SMS messages just fine with zero software updates. Maybe once a decade or so that could have been enhanced with longer messages, pictures etc. But overall the messaging problem really should have been solved by now.
Yes, it does. SMS aren't encrypted. And messaging is closer to being solved than ever before. If you want to use a "dumbphone" with SMS you can do that.
intuitively I would agree, but the point of my comment was that intuition might be a poor guide if we want to understand what kind of team and software architectures "succeed".
if you rewind some years microservices was also the word of the day (and probably there were some iconic projects picked to justify...)
Messaging is a weird app to focus on mau/dau. There have been many messaging protocols which scale to hundreds of millions of users, and there is prior art of large not particularly valuable chat systems like aim, gchat etc.
No one is going to use a consumer messaging system which charges money or shows ads. Which brings to mind substantial questions on the business model others are using the messaging platform for e.g. data collection.
What is amazing to me is that they did scale to 400 million monthly active users with less than 50 engineers... In 2014.
That is: on hardware from seven years ago and older. Which is an eternity in tech. Since then we've seen incredible new hardware come out (EPYC 2 in 2018 for example).
One can wonder: up to how many MAU could a service scale today, on today's hardware, with a small team? And what about tomorrow's hardware...
Seven years can be an eternity in tech, but is it an eternity counting from 2021? I mean, AWS launched in 2006, Netflix expanded to Europe in 2012, Docker had their initial release in 2013 and so on, so none of these major events are that "new" anymore. Obviously things have changed (a lot) during the past seven years, but have they changed as much as they did between 2007 and 2014?
What I'm saying is that for the general case of "scaling with few engineers", it seems to me that the tools available matter a lot more than the specifics of what hardware is available at the time. Launching an international service "from scratch" with a small team and a modest budget just wouldn't have been possible 20 years ago, but not because CPUs were too slow.
Ultimately, how many engineers a company has is not that relevant when it comes to scaling.
A small team of product-oriented engineers with a devops mentality can easily develop, maintain, and continuously deploy a considerably complex application.
A similarly complex project in the hands of a larger sales driven company often resembles a spaghetti ball, and is held together with rotting shell scripts and unproductive manual processes.
This just makes more clear that we didn't try enough creating an open-source instant messaging app that respects our privacy as it should be. It's probably much more difficult to do the same thing as WhatsApp did because of the dependency we created around it but I think we just need the right initiative to do so.
It is hard to get the general public to care about privacy when they post pictures of every meal and share every thought and emotion they have with no filter.
It’s hard to get the general public to care about inferior things pushed on them by patronising people sneering at them. How is a photo of some food important to privacy enough to be the first thing you teach for? Seems more like punching down at people you see as inferior because photographing meals is “so lower class, right guys?”.
Because complexity kills. Something that has yet to be learned by the new generation of developers. While disciples of FAANG create a gazillion microservices that they spend most of their time debugging in production ("observability"), smaller teams just get shit done.
Since "monolith" has become a dirty word, it would be instructive to remember that some of the major companies out there are still light on distributed systems, which we all knew even back in the olden days of the internet were an absolute killer of efficiency and productivity. Yes, "microservices" is just another word for it - believe it or not, there was life before Node and Docker.
Whatsapp, Dropbox, Instagram, StackOverflow - all existed or still exist as one easy-to-maintain application. But go ahead, "be like Google".
It crossed my mind while considering this that a figure like this doesn’t mean the same as it would have pre-cloud infrastructure.
My team is fairly experienced but not so much in infrastructure. Even so, we build and deploy on Google cloud with fairly excellent scalability and ease. Ten years or longer ago I suspect the story would be different - the infrastructure and scale constraints could have pinned us at times. We would have needed to hire more to support that effort.
And what I’m getting to is that yes, they had 50 engineers in their pay roll, but many more supporting various SaaS products and infrastructure they use and pay for.
Not to say this isn’t impressive still, I think it’s a great thing and worth aspiring to. I love small teams.
> And what I’m getting to is that yes, they had 50 engineers in their pay roll, but many more supporting various SaaS products and infrastructure they use and pay for.
I was at WhatsApp from 2011 to 2019.
Hardware was outsourced to SoftLayer, of course. I think they had a sizable operation, but if you move that in house, you've got a team of 4ish for 24x7 colo work (we were only in one location for an embarassing amount of time), plus a networking person, and maybe a manager. We certainly got better service with SoftLayer's larger team including ease of getting more servers, but we would have had better visibility in house, so it's a tradeoff. Softlayer's staff was probably smaller than the combined staff if all of their major customers in-sourced, anyway.
Other than that, we used multiple providers for SMS and voice verification; some of whom have a lot of staff. You need more staff for this, IMHO, because in addition to 24x7 coverage, you also need to arrange connectivity with carriers in all timezones and all languages. But some of the companies we used seemed pretty small, so I dunno.
We were using Zendesk in our customer service system after growing out of Gmail and before in-sourcing it after aquisition, and Dyn for DNS (which doesn't take much staff to insource).
What other SaaS do you think we used? Do you want to count the OS app stores and push services? Maybe payments while that lasted? G Suite for corporate email when forwarding to personal mail became too silly?
We self-hosted our code repository and bug tracker.
I hope I didn't come across as though I was criticizing at all. I meant to point out that having 50 engineers build something today means something different than it would have a decade ago, although in retrospect my age/career are getting away from me and 15-20 years ago would have been a much better example.
I should emphasize that the accomplishment is incredible, regardless of how much we outsource things today. I know you didn't outsource crucial talent or the ability to deliver a good core product; you can't do that as far as I know.
I apologize if it seemed like I was putting down the accomplishment. I'm only meaning to reflect on how much more technical and infrastructure work happens outside of our teams these days than it has in the past. Perhaps though it isn't how much _more_ happens, but how _different_ what happens is. Building these products has changed a lot in some ways.
Edit: Also thanks for overview, that was a cool cursory look at how you operated.
I mean, honestly the way we built WhatsApp 10 years ago is pretty similar to how we would have built it 20 years ago... except that we'd probably have had to move towards dedicated colo space, instead of 'managed hosting' bare metal. I think we may have ended up going that way if we didn't get acquired and move into FB datacenters.
SMS aggregators were certainly around in 2000, just as well as in 2010 and 2020 ... although they've got fancier webpages now.
I mean, you can outsource a bunch of stuff, but we just didn't use very much. You can't outsource the core product and deliver reliability, and we didn't have much that isn't the core product. SMS verification was only reliable (ish) because we used several providers with real time statistics guiding the choice. (Of course, some of the providers use the other providers, so it's very messy)
I remember reading they worked very hard to make their app available on many, many types of phones. Which is weird because they regularly announce they drop support for certain phone OS.
I was there from 2011 to 2019. We had pretty clear numerical guidelines for continuing support of a platform or OS. If the numbers are close, things like upgradability or difficulty of support make a difference: if all the devices running OS 1.1 can actually be upgraded to 1.2 and 1.2 fixes important things, then dropping support for OS 1.1 can happen with more users still there.
If builds need to be signed by the OS maker and they won't sign any more builds, user count doesn't matter (sorry Nokia S40 and iPhone 3G).
We did make some longer lasting builds for platforms we were ending support for, but there needs to be an end. Older clients don't support newer features and we didn't want to have a network where that was the status quo. I don't know what the current expirations are, but mandatory updates a couple times a year helps keep the feature support similar.
WhatsApp has to be one of those most easily scalable projects to work on. You have the freedom to divide chatrooms across different databases/shards. Same goes for notifications, where users can be divided since they're all isolated. The hardest part is just coordinating all the moving pieces in a way that can grow well, and that mostly just involves avoiding pitfalls in your design that might bottleneck.
Not the same efficiency but it goes to show how certain domains and technology stacks can scale exceedingly well. Plus in DDG case I believe they outsource a lot of their tech to 3rd parties.
I could scale to billions of users as a one-man engineer, but I would need a marketing team. :D What I am trying to say is that I could make a WhatsApp-equivalent relatively quickly, alone, yes, but that is definitely not enough to gain billions of users.
The users are all handled through a common set of code, I don't see why its remarkable that it takes 50 engineers to keep things going. Customer support, I can see that eating manpower, but not keeping the code running on a few racks of servers.
It is silly to focus on number of engineers rather than on total cost. Any company should keep hiring more engineers until the additional salary expense exceeds the decreased operating cost/increased revenue that engineer generates
User count is fairly meaningless. How much was that in daily active / weekly active users? How many of those were bot / spammer accounts being “seasoned” to then sell to abusers?
This is misleading. Likely, those 50 engineers were standing on the shoulders of the giants, i.e. those open source maintainers who build wonderful value for everyone out there.
I've honestly never understood how faang companies can have such incredible numbers of devs when the products basically move at a snails pace for years now
I thought what allowed them to get to that many users was that they were on so many platforms and that they used a large team of remote engineers in Europe to do that.
The client is another story, but for the server they forked ejabberd which is a reliable, established XMPP server. So they probably had "fast-cheap-secure" on the server-side, because 6 years had been spent ironing out ejabberd bugs before the first version of Whatsapp came out.
My understanding is that vulns in Whatsapp so far were client-side, but i'm interested if i missed some on the server side.
I bet these engineers got only a tiny tiny fraction of the billions of revenue.
It always fascinates me how in the engineers' mind it reconciles that their work brings corporations they work for billions and yet they have to rent a tiny apartment for a substantial portion of their salary and otherwise lead a pretty average life. Then going to lavish offices witnessing how the money they could make use of is wasted on vanity.
That's the founding principle of capitalism: private property.
People living someplace don't own it, and people working some field don't own it. Some "owner" owns the land and means of production and extracts value from people doing the actual work.
Yes, it's a deeply broken system. A slightly better system revolves around self-organized workers coop where everyone gets equal pay and there are no shares to hold (or everyone owns the same amount). An even better system abolishes money and private property so that people can live meaningful/useful lives without worrying about imaginary numbers ruining their entire existence.
Higher quantity of ideas == More engineers. Higher quality of ideas == Fewer engineers.
WhatsApp success is not about what WhatsApp did, it's about everything they didn't do. No desktop, no browser (until after reaaaaaaallly long), no account creation (your phone number is enough, thanks), no hidden ad engine, no hundreds of stupid features that product people stuff down in the name of experimentation.
There are some leaders who want anything and everything: Throw 100 darts in the wall. Leave what sticks (don't touch or nurture it). Go back to throwing 100 more. The other guy does that? We want that too. Mobile? Yes. Desktop? Yes. Browser? Yes. Different affiliate channels? Yes. Custom features for B2B partners? Yes. Merchandizing? Yes. Build a mini Ad engine? Yes. Multi user profile management systems? Yes. Yes, yes, yes, every requirement is a go.
The crucial difference is: What happens when you have a restricted pool of people, but insist on doing 100s of things? Some leaders simply treat this as an "org structure" problem, not a problem of focus. They slice and dice the org until the 30 people become split into teams of 2-3 with a manager for each, running into coordination hell on each step – just so that more quantity of features can be seen on paper. This is how most startups are, except for those rare few which don't "talk" about focusing, but actually focus.