I worked at a certain FAANG company at a time where the promo process rewarded "solving complex problems". The more complex of a problem you could solve the higher your level (and your pay, your status, etc).
Naturally, people were incentivized to find complex problems so they could solve them. Not only that, I think a lot of tech stacks at other companies evolved by copying this company's ideas, so even smaller companies with even less need for complex solutions ended up having them as well.
I’ve seen teams destroy themselves by solving complex problems that didn’t need to be solved. Instead of solving the real problems, they imagined a problem that was more interesting and complex, and tried to solve that.
in my defense I can think of at least four times I needed to prototype a complex solution to understand why it was the wrong solution and then discard it
+100. A lot of us love to over-engineer things because that's more fun to solve and looks good on a resume / evaluation.
Kubernetes, (micro-)services, queues, nosql, lambda and other aws service are examples that I've seen in the wild. These can absolutely be valuable and cost-effective, if the problem needs that complexity.
I love engineers that focus on the user, problem and business. They are soo valuable, especially for startups and smaller companies. Engineers that love solving complex problems without thinking much about the user are still awesome, if they can operate in an organisation where that critical thinking is done by others (product people or other engineers)
I once wasted several months of schedule and payroll wrestling to improve a DNN model - which resulted in that model getting much much better thanks to tons
of creative and interesting work.
We swapped the learned model for a perfect a oracle and realized the actual problem
was elsewhere in the system and was easily fixed. None of those model enhancements were needed.
Not the person you asked, but working on remote sensing and perception I have seen (not once) the teams turning from improving use of available sensor data to things like cutting edge messaging technologies to support real-time data delivery from thousands of sensors (when they have three sensors today and might get another one next year and their current simple streaming will work just fine for up to 20).
While those are interesting problems they have nothing to do with their current perception challenges (that have to be solved for a successful deployment), steal resources from the team and make life harder for a few remaining people actually working to solve current challenges, as they now need to switch to a more complex messaging and rewrite some of their existing utilities.
These weren't fatal to the project or team, but did contribute extra work for net negative value (slower & more maintenance, with no relevant extra features):
- Using Kubernetes to host an application when Docker Compose would do (specialist application with <10 concurrent users).
- Using DynamoDB to store a few kB of temporary data in a processing pipeline, despite having several existing PostgreSQL instances already and no DynamoDB ones.
I worked on a team for a short while that decided to use Cassandra to store a few hundred thousand accounts, despite the data changing only slowly and being an exact match for a relational database. It was one of the worst choices I've ever seen in my career, and the only reason I could find was that people at the time wanted to learn Cassandra.
Sounds somewhat familiar. I worked at a place that replaced a basic LAMP stack with microservices written in Perl and MongoDB as a backing store, for the sole purpose of raising complexity.
And left Mongo on the default settings for the time (speed of return over reliability of saving data), so they ended up with a reporting replica which was MySQL.
Don't think I've seen anything before or since which was architected so exactly backwards.
Think my company burned a year on a solution using Cassandra. I think the problem was 1/3 Cassandra and 2/3rds the people that would pick Cassandra amd Java over PostgreSQL and C# or go.
Years ago, my team at AWS considered writing our own declarative language to configure EC2 instances which would replace some old functionality within CloudFormation (cfn-init) until I pointed out you could achieve the same thing with our existing features (State Manager Ansible playbooks)
I worked at a place where they finally ditched their Hadoop project after a couple of years and who knows how much money. Only to implement a big Kubernetes cluster instead, where they four years later have burned through three different commercial Kubernetes distributions.
No, it’s not. Identifying which problems are worth solving (regardless of their complexity), and solving those is worth promotion.
Lots of engineers have a fascination with solving complex problems purely for the sake of solving them. When you make that your criteria for promotion you end up have to do many rounds of layoffs in harder times because you’re staffed to the gills with people solving complex problems that deliver little to no actual value.
Reality disagrees with your worldview. There is only one thing that's "worth a promotion" and that's convincing the person able to greenlight it that you deserve it.
This very rarely corresponds with the ability to deliver good software in medium to large enterprises and entirely with your ability to market yourself. Which is easier if you're able to show them a super complex problem you've solved, because it sounds impressive to a mba...
It is very difficult for management to over time not change the rules on promotion enough that it tends towards a random walk where your level is approximately proportional to your length of service. Also,in at least the places I have worked, the enormous emphasis on individual performance both detracts from looking at group performance, i.e. actual productivity, and sucks up so much time. Copying from what execs have heard about Google they have promo boards, overly defined level descriptions, endless work of writing and rewriting promo packets, tactical discussions on which VPs need to be impressed independently of the merits or scope of a change, etc. There is a clear incentive to help people in a sister org over helping your team accomplish its goals.
The whole system seems destined to be replaced by something more useful.
You’re assuming what results in a promotion is the same as what’s worth a promotion. Nepotism makes it really easy to convince someone to promote you, but it doesn’t mean someone else doing the same thing would get a promotion.
the practice among those with power or influence of favouring relatives, friends, or associates, especially by giving them jobs.
how is that related to someone finding a complicated but irrelevant problem to solve to show how competent they're to the mba thats in the position to greenlight the promotion...?
Well, an MBA that wants to promote their relative might need that "complicated but irrelevant problem" solved as a kind of "cover your ass" paper trail.
> Reality disagrees with your worldview. There is only one thing that's "worth a promotion" and that's convincing the person able to greenlight it that you deserve it.
It seems pretty clear to me that the person you’re replying to was talking in the sense of how things should be, not describing the way things are. Also, “worth promotion” is not their wording. You’re nitpicking the wrong thing.
> Wouldn't the easy problems have been solved as they are by definition, easy?
No, not at all. “Easy” does not mean “obvious”. I once replaced a call to scp with a call to rsync in a script I didn’t originally write. The result was that an operation which used to take more than an hour and was thus run occasionally now took seconds and was executed every time. Both speed and reliability were increased with one simple and easy line change.
While the bottleneck had been recognised earlier, the original authors had learned to live with it. They weren’t aware of how it could have been improved so drastically with so little effort. That’s another thing: easy is relative.
Either way, you’re conflating easy (the opposite of difficult) with simple (the opposite of complex, which is the word used by the person you’re replying to). They are not the same thing. Easy mathematical formulas can make complex fractal shapes and complex programs can do simple things.
Similarly I sped up a python script by 3 orders of magnitude by running a linter on it; the script intended to cache a value for reuse, but neglected to actually reuse the value, resulting in an unused-variable warning in the linter.
> Wouldn't the easy problems have been solved as they are by definition, easy?
Only in an efficient market. And real world markets often have vast inefficiencies. The reasons they're inefficient don't always generalize, either. Companies can behave in certain ways because of personalities (2024 Twitter vs 2018 Twitter), embedded institutional norms (2024 Boeing vs 2020 Boeing vs 1980 Boeing), changing conditions (any 2024 startup vs any 2019 startup's finances), or a million others that don't necessarily spell doom for trying a different approach.
One that's relevant a lot in the real world is market share. Large organizations very frequently get away without doing easy good things for long periods of time, which they can do because network effects and platform lock-in are powerful defenses against disruption. A lot of startups get founded on the idea "large incumbents are not doing this easy good thing, so let's do that and beat them". The fact that this ever works is a sign that incumbents must be leaving a _lot_ on the table, or their lock-in would never be overcome. The very existence of the startup scene is proof of the frequent inefficiency of markets in the short-to-medium-term (or at least of investors' belief in such).
Even in cases where the incentives _are_ aligned and the market _is_ efficient, the world is often in non-equilibrium states. I like to think of incentive gradients as something akin to a (very complex) differential equation, and consider what _simple_ DEs can teach us about them. Consider, say, Newton's law of cooling: dT/dt = -k(T-T_e). Some calc 101 will tell you that solutions to this equation trend (exponentially! so not even slowly!) to a constant stable equilibrium T = T_e. But if you try to use that analysis on a fresh batch of french fries, you're going to get burned, because it turns out T(0) is very relevant to predicting T(1 minute) for realistic values of k.
There are lot of problems that are not identified as “easy” before someone tries to solve them.
Hence, “identifying” step is often required to find the low hanging fruits.
One catefory for these types of issues is “Complex root cause analysis, easy fix”
So there might be a “medium severity” issue - something that is not business-lethal - and that appears hairy enough that everybody just avoids it. Then someone spends some time digging into it and finds it’s some arbitrarily small change that fixes it.
These things happen more often than not.
The trick here is to look at the problem long and hard enough and realize a small change will fix it - instead of doing something completely tangential and complex to circumvent the problem.
These problems fall into the class of “anyone could have fixed it” but only the person who fixed it actually did the hard work of figuring out the root cause.
Do engineers at FAANG choose what problems they want to solve? At places I worked, it was more of management task to choose problems to solve. Developers just worked on tasks they were given.
If all you do as a developer is just work on tasks given to you then you aren't going to rise very high in the ranks in any tech-forward company, FAANG or not. Sure you can make a decent career out churning tickets given to you by someone else, but you'll run into a glass ceiling pretty quickly. The tasks don't just appear out of thin air.
A key part of being a senior+ engineer anywhere is the initiative and autonomy to identify problems, quantify costs and benefits, and work with management and stakeholders to come up with solutions that balance cost, complexity, schedule, and quality.
Much of my work as a staff engineer is working with engineering, product, and business leaders to collectively identify what our biggest challenges are and how we should go about attacking them from the engineering side given all our resources and constraints. And then spearhead the implementation to get projects going the right direction and overseeing more junior team members.
The actual coding is the "easy" part that comes after first defining the problems and specifying solutions and wrangling the competing needs of the business. Engineering should have a seat at that table since we're on the hook to build (and maintain) it, after all.
I agree with you up until the cliche "code is the easy part".
People say this at the same time that they complain that people with 10 years of experience can't even solve FizzBuzz.
The amount of badly architectured, buggy code I've seen in a couple of decades working with software tells me that "code is the easy part" is an absolute lie.
Writing good code that's well tested and mostly just works, while being efficient, is such an extremely difficult task that I am not sure many developers in the world can do it. So, no, code is not the easy part, it's only the part where you can probably escape accountability (as no one will be able to tell) unless you have good devs in your team that can properly code review (which is also very rare).
I mean, code is both the hard and the easy part. I can write the code to get good code, or I can spend more effort on finding the right problems to solve and let others write shitty code.
It hurts to let people do that, but ultimately shitty code still works, while having solved the wrong problem means that all the work goes down the drain.
¿Por qué no los dos? - find the right problems to solve and write good code (or insist on good code being written) to fix them? Because bad code (unless it's throwaway code) will quickly become a problem all of itself...
Solving problems that move the needle for the company in terms of customer experience, financials, or things like that are worth promotion. Sometimes these are complex problems. Sometimes not.
Software isn't an art project; it's a practical tool to solve practical problems.
Some of my more impactful changes haven't been all that complex; it was just a matter of knowing where and when to spend my time, based on working with our customer team, sales team, and directly talking to some customers.
One of my more praised changes was adding a link to our support backend that saved our customer team minutes of manual work every time. This was literally just adding a permalink: 3 minutes of work. Simple and low-complexity? Yes. But also impactful and moved the needle, and no one else had done it in the previous years.
> Software isn't an art project; it's a practical tool to solve practical problems.
Yet me and my colleagues have seen code that resembles abstract art far too often - it's interesting to look at, but mind-bogglingly complex. Getting rid of that is a delicate balance act, as you risk stepping on the shoes of the people who wrote the code.
I do see writing software like writing a book.
I write it with the future reader in mind. It needs to be meant to be read.
I picture my colleagues or young engineers 20 years from now combing through my programs, learning the problem domain, learning the tradeoffs we made at the time and why it made sense.
>One of my more praised changes was adding a link to our support backend that saved our customer team minutes of manual work every time.
I used to support a ton of random apps written as domino databases, and often would make someone's day by doing small changes like that. Things like adding one new column that saved them tons of time in cross referencing data. It was super fulfilling to have someone mention that they spend literal weeks of the year doing some task and I'd be able to totally eliminate the need for that task altogether.
Sometimes the promo committee can't tell the difference, i.e. was the problem complex to warrant the complex solution. They just see the effort put into solving it.
At the Austrian Institute of Advanced studies there was a professor that got real good at stochastic differential equations and made a career out of applying them to anything and all in economics. One big advantage was that almost no other economist can actually follow it. Those few that can are obviously super invested in the technique
Stochastic differential equations is fundamental knowledge to do option pricing (black-scholes) among other things. I would expect an average guy educated in that field to be able to understand
In functional analysis, it is somewhat of a sport to attach indices to all four corners of a symbol. I remember overhearing a discussion between two big players in the field, and they obviously did not share the same starting point for reading the indices out loud. For sure, I didn't understand what they were talking about, but neither did they.
Similarly, when the promo process rewards impact and influence on other engineers you get all the smart ambitious people making overengineered platforms, frameworks, and standards while no one competent or engaged is minding the actual end-user products.
This is a really tough problem to solve. If the company has amazing leaders, the leaders can correctly judge the depth of the problem and its impact, therefore rewarding the problem solver properly. Unfortunately promotion is really a tool for most leaders to advance their agenda. In a way Google invented this "solving complex problems" just to counter balance the misaligned incentives of the leaders, but to no avail.
When the value generation is so far removed from the individual, the system will get skewed towards private taking over something more globally beneficial (globally as in for the entire company).
Given the high bar that FAANG has set for filtering in self centered innovators, I bet some employees simply invented problems for solutions they already had at hand.
Naturally, people were incentivized to find complex problems so they could solve them. Not only that, I think a lot of tech stacks at other companies evolved by copying this company's ideas, so even smaller companies with even less need for complex solutions ended up having them as well.