I work at LinkedIn now. Chris’s role and the podcast describes ember and front end web dev. I think the LoC and build he’s referring to could be voyager-web. Our monolith flagship web app. And, there’s more systems at LinkedIn with MM lines of code and long builds example is the mid tiers, the offline data stacks, the metrics systems, KafkaKafkaKafka.
Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good.
I worked at LinkedIn in infra. Their internal tools are a nightmare. The true definition of Jugaad.
The entire company has zero concept of testing. No QA at all. Engineers push out half baked initiatives to add to their promotion packet then move on to the next thing.
I trouble shot so many issues just in day to day usage of the internal tooling like for some reason, engineers just trying to do their jobs, are QA.
Seeing you use 'Jugaad' make me unreasonably happy.
I feel like this sort of flinging shit over the wall mentality is very much becoming de-facto nowadays. Very often I have been required to just 'get shit done quick and dirty' over spending time to come up with a permanent solution.
Quick and dirty is never the short term fix it is meant to be. It always ends up being left in place until it inevitably breaks down the line.
LinkedIn thinks so. Their tech stack is so over engineered and spaghetti that they spent a hundred million and years trying to move into Azure and were beaten by how bloated and unmovable the tech stack is.
Theyre still using python 2 and centOS in tons of systems.
Yea, give everything 3x estimate because everything that can break will break along the way.
Fresh clone of master won’t build, the local build command is broken, gradle, remote build, GitHub, staging is an inside joke, prod host os upgrades, dependencies bumped in repo, http dependency’s network route changed, etc. etc. etc.
In my first SWE job, I was perplexed, I never felt much impact for solving complex bugs, even if they had side effects like reducing latency for everyone in our infra. Anyways, I realized it's like you say - you get rewarded for pushing features and shipping stuff. This incentive can easily become a 'tragedy of the commons' situation.
My biggest pet peeve with this is that a lot of people see these values as immutable, and because building/testing/running every single push takes too long, we should not do that.
As opposed to making builds faster, or build infra faster/cheaper.
There are mostly only a few ways to make builds faster.
* Ship less code (Very hard to get a large org to do)
* Rebuild the same things less often. Requires using build tooling that supports this org wide, Still hard to do but not as hard as shipping less code.
* Build more pieces in parallel. Also requires using build tooling that supports it org wide as well as structuring the code in such a way that it is amenable to parallel building.
These are all expensive investments that will pay off huge if you do them but can be quite difficult to justify until you reach a certain size.
I mean both. Smaller pieces but also less of those pieces. It is my experience that many systems code size is less purely a function of what they need to do than a complex set of layered components and systems in the name of ease and fear of NIH syndrome.
Sometimes doing the same thing with simpler components and fewer components is just better.
The top 1/3rd of Eng don’t see it as immutable. But the stack can consist of 10 other teams tools, and you’re not rewarded for the amount of time you’re spending to fix other people’s stuff. So it goes on
Just for another perspective, I've worked at a few companies including LinkedIn as a backend developer and I think LinkedIn is probably around 70-80th percentile in code quality.
There was significant emphasis on code quality at least on the team I was on, and an ever improving culture.
I did work on one voyager task though and I remember it being a nightmare
They're trying to be a site that makes a lot of money from ads. To do that they need to get people to spend a lot of time there. To do that they need an infinitely scrolling feed that has addictive characteristics. What else are people going to spend a lot of time on there? Scrolling through their contacts?
Oh, you meant what are they trying to be that's helpful to the user? Doesn't matter.
I wonder, is "being facebook" is that an organic goal LinkedIn came up with, or did they pick that up from their users?
I suspect the user base has largely driven it. From the beginning it seemed that regardless of the stated purpose users wanted to use LinkedIn as a "different facebook". Personally I hate that, but a lot of people have been doing it for many years.
In what way? Users/customers do not generally drive development. Indirect measurements of users do, such as measuring "engagement".
At best, what likely happened was A/B testing showed "what users want", which rarely and only by chance intersected with what users actually want, and instead showed over and over that socials media patterns (light and dark) hijacking attention drives engagement.
If people didn't want to use twitter, they wouldn't and it would be gone. But users do use it, and even the folks who tell me they're upset ABOUT Twitter, most choose to be there.
Whatever the reason they make that choice, at some point that's on them.
Nah, basically every other social type thing got screwed by FB convincing everyone that MAU's (i.e. the metric FB looked best on) was the right metric, when it clearly wasn't.
Like, LinkedIn only needs MAUs who are trying to sell something or looking for a job, they 100% don't need people to log in every day (as their ads business is only a small proportion of revenue).
Some users love it, the type who want to sell their every thought online to anyone who can tolerate them.
Other users (almost everyone I know) absolutely loathe it. It is hands-down the worst service I have an account with, but it's also practically required to get a job if you don't have lucky personal connections. I was hoping TFA was actually about leaving as a user, that it might be inspiration for me to free myself.
They made loads of product changes to get people to engage in that way. As an example, showing who viewed your profile in the email vs needing to log in.
Part of that might be the "reward for spamming" - any "social media" site is always amazing at the beginning, because only the die-hards know about it and the spammers have no return on investment.
LinkedIn has slightly different style of spammer, but it's basically the end game for anything.
It's because LinkedIn is old in terms of web-age. Really old. They, along with thousands of other companies, had the need for a reactive web front-end when JavaScript had not yet matured to where it is now.
this plus lack of performance eval/culture/strategy to grow responsibly and enable future success
its a great learning experience though despite what people say about the inability to learn the new hot hot tech. the nuance of software development is the decisions that other people make, that is inescapable and is a skill worth developing. i'm not buying the "just do a startup" because i think its a cop out.
In my experience, a good platform team will increase velocity a bit while a bad platform team will tank it. Perpetual migrations and re-writes, "simplified" platforms that fail to meet future developer needs, half-abandoned NIH systems as people move teams, ownership moats if product teams try to touch platform code, etc, etc. So if a company has been burned a couple times they may decide to just not bother.
Preface: I’m not in front end or flagship teams so I’m saying hearsay.
The initiative Chris refers to is still alive but likely hasn’t made any meaningful progress. These things tend to have a lot of fan fair and then suddenly leave the front conscious of the company as we get a new GDPR/DMA fire to put out. at that point it will be dead. Or, it will stay alive for 2+ years as slowly but surely 50% of clients migrate to it and see even poorer performance.
There was a blog here a few months ago about migration from restli to grpc. It’s still going on, and nearly every app i see is still restli.
Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good.
Anyways, AMA