True! But it might also grind the company's ability to release features to a halt creating a stagnant stock price and an underwhelming IPO because they chose to avoid the hard work of migrating, refactoring, and simplifying, or setting a culture of maintainable code preferring more easily marketable (and probably more fun and less failure prone) feature work that also looks much better on a resume.
I'm not trying to say that you can't incur technical debt if you pay it off. What I am trying to say is that a culture of technical debt stagnates growth.
I would point to twitter as an example of the point I was trying to make, so I don't think it's a very good counter example:
Do you really think it’s because of the software or because of the product? I bet no one has said that if only Twitters software was better it would gain more customers and advertisers.
All indications are that Twitters architecture is now top notch. There are entire sections of “Designing Data Intensive Applications” (the Bible for system design interviews) about Twitters current architecture.
I think as systems get more complex and bogged down, it's harder to innovate. As innovation is stifled by build problems, dependency problems, political problems, operational problems, the amount of code you have to read to do anything problems, etc, I think engineers start to say "could I be more productive, and therefore theoretically get more reward, elsewhere?"
I think as soon as the innovators start to jump ship, growth momentum is lost and it's nearly impossible to regain that momentum. Who does interviews? Who sells candidates? What is the marketing? What kind of sense of morale will potential hires perceive?
If a large refactoring project did take place, refactoring projects are generally very high risk and low reward. That is the type of project that most senior devs will avoid like the plague. It is not fun engineering. It is drudgery and toil.
So a history of poor software results in required cleanup/developer pain, which results in high value devs leaving for greener pastures, which results in a perception of high inertia non-growth, which then makes it harder to sell people on the prospect of "high growth" stocks which results in the tanking of average dev quality (and morale for that matter), which then makes it even harder to hire innovative devs or leadership, which ultimately results in stagnation.
Senior leadership, both managerial and technical, will play hot potato until the costs of not addressing the real, highly unpleasant, problems is too high and by then its too late. Things continue to get worse between the start of the effort to fix problems and the actual fix of the problems and a lot of people will choose not to suffer that.
I didn't work at twitter, but I watched this process happen. To me, there is a clear relationship between systemically unaddressed and festering technical debt and ability to innovate with direct personnel consequences.
> I bet no one has said that if only Twitters software was better it would gain more customers and advertisers.
But whats the cost of setting up a new service? Of creating a new interface? Of testing a new feature? Of creating a new payment system, etc. All of these are software problems. How efficient would an organization be if every team had their own repo? If every team had their own monitoring systems? If every team implemented their own release process? If every team re-implemented the same behavior over and over again because higher up leadership couldn't allocate resources to problems that every team has, what effect does that have? That inefficiency is innovation opportunity cost.
Imagine you acquire a company you think can provide an innovation to your own. What is the time/resource cost of integration and what effect do you think dev environment would have on success of the acquisition?
I think technical debt clearly has compounding interest. I think companies get into a position where the debt becomes debilitating particularly through pathological short term thinking. At some point the compounding technical debt exceeds growth and it's over.
If Twitter has a sane architecture and all indications is that it does now, creating a new feature shouldn’t require back end refactoring. You have core shared services and you build another service on top of it. I know how at least one of the newer services work at AWS and how many existing back end existing services it depends on and that if anyone outside of AWS had the resources, they could build it themselves by calling publicly available APIs.
I seriously doubt Twitter is one large monolithic app
And let’s not pretend that Google for instance is any better at product development. It’s failed at everything it’s done not related to advertising - ie Stadia. It’s one success hides a lot of failures.
> Originally, Twitter ran their public APIs through a single Ruby-on-Rails application (“Monorail”), which had grown into one of the largest Rails codebases in the world. And so it was increasingly difficult to update. By 2014, Twitter went the route of microservices, migrating the API service to a set of 14 microservices, running on an internal Java Virtual Machine (JVM)-based framework (“Maccaws”).
> “While the microservices approach enabled increased development speeds at first it also resulted in a scattered and disjointed Twitter API,” Cosenza said. Independent teams designed and built endpoints for their specific use cases with little coordination. This led to fragmentation and, inevitably, a slowing of developer productivity.
Microservices don't make a lot of sense to me unless you can print money or have a public infra offering. It points to an engineering org that doesn't know what they are doing. Turning O(N) code complexity (virtual complexity) into O(N) operational complexity ("physical" complexity) points to an org that doesn't understand where their complexity is coming from or how to minimize it.
> creating a new feature shouldn’t require back end refactoring.
The argument I was making is that it doesn't matter if it gets better. Once momentum is stopped by poor architecture, inertia is too high to get moving again.
> I know how at least one of the newer services work at AWS and how many existing back end existing services it depends on and that if anyone outside of AWS had the resources, they could build it themselves by calling publicly available APIs.
Yes they product-ized their services which is absolutely brilliant. I think the fact that amazon can print money plays a very key roll in why they won while other companies lost. They scaled early in the game rather than scaling too late and reaped incredible rewards for doing so.
> And let’s not pretend that Google for instance is any better at product development.
Google is famous for developers "just shuffling protobufs between services." While it's generally said derogatorily, that means that developers are generally not in the weeds and minutiae of what they are trying to accomplish, but are actually working on the higher level concept they are trying to achieve.
> It’s failed at everything it’s done not related to advertising
I don't agree that failure is the rule. Cutting lackluster offerings rather than pouring investment in them shows at the very least a sane short term strategy with the weakness of potential harm for long term consumer trust. That's a reasonable trade off to make with good arguments for both sides.
> Microservices don't make a lot of sense to me unless you can print money or have a public infra offering. It points to an engineering org that doesn't know what they are doing
A little company I’m sure you have heard of may disagree. This was way before AWS was thing and was an edict for Amazon Retail
> Yes they product-ized their services which is absolutely brilliant. I think the fact that amazon can print money plays a very key roll in why they won while other companies lost. They scaled early in the game rather than scaling too late and reaped incredible rewards for doing so.
That’s not the point. The point is that new internal initiative don’t start from a blank slate at any large tech company. There is already infrastructure in place to build on top of even when that underlying infrastructure isn’t publicly available.
- Android: it came out in the Oracle trial that Google only made $27 Billion in total during the first six years. Google gives Apple 18 billion a year to be the default search engine on iOS. Apple makes more money from Google in mobile than Google makes from Android
- Pixel: depending on who you believe, Google sells 1.2 million phones a quarter. If it were a separate company, it would be an abysmal failure
That’s about the same number Apple sells in 2 days in a down quarter.
YouTube: while YouTube contributes 10% to Google’s revenue, most think it’s still not profitable once you take into account bandwidth, storage, and revenue share with creators.
Project Zero is not a profitable product - nor was it meant to be.
> This was way before AWS was thing and was an edict for Amazon Retail
Yes, you are telling me the point I was making. Product-izing your microservices is a justification for microservices.
> 5. All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
That is the property that makes microservices make sense. Without that property, technical leadership is setting themselves up for failure.
> Your information about Twitters current architecture is way out of date.
The only point I was making about twitter architecture, of which I know far too little about, was that as you stated previously "Every indication is that the code base was horrible" leads to loss of momentum and inability to innovate, so it's not surprising their growth stagnated for, what I believe, are primarily technical reasons. It doesn't matter if they got better if they went through a period which stopped momentum.
> That’s not the point. The point is that new internal initiative starts from a blank slate at any large tech company. There is already infrastructure in place to build on top of even when that underlying infrastructure isn’t publicly available.
The point I'm attempting to make is that if underlying infrastructure is poor, that's not a fertile ground to grow new product so innovation becomes stifled because resistance to new functionality is too high. If maintenance costs are too high resources must be spent maintaining rather than innovating. I've seen developers stumble around for a week because they put `30` instead of `10` in a config file. I've watched an entire team waste months of effort because production was not advanced enough for what they wanted to do. I've watched an entire engineering organization lose hours every day to a poor dev environment. Time was spent fighting infrastructure, not creating features, so it's not surprising when new products and features weren't created and growth didn't meet expectations.
> google stuff...
We have different definitions for failure. You said google wasn't able to develop products. Google is clearly able to develop new products/services, maybe not monetarily successful ones, but google as as a company has a disproportionate effect on my life and how my life works past showing me some ads.
> Yes, you are telling me the point I was making. Product-izing your microservices is a justification for microservices.
This was not about Amazon selling its micro services. This was only about AWS Retail developers internally.
> Every indication is that the code base was horrible" leads to loss of momentum and inability to innovate, so it's not surprising their growth stagnated for, what I believe, are primarily technical reasons
This was in the early days. They moved away from the Ruby monolith years ago.
> We have different definitions for failure. You said google wasn't able to develop products. Google is clearly able to develop new products/services, maybe not monetarily successful ones
For a for profit company, the only definition of success is does it make a profit.
Edit: I change my mind slightly. We are talking across each other. From a technical standpoint and being able to get products to market, Google is great. From a business development standpoint/program management standpoint, Google sucks. Google’s saving grace is that it has one great revenue stream.
Maybe if Twitter had better product development capabilities - not computer development, business development, they may have found something that sticks.
I also just don’t think Twitter is as compelling of a product for most people as a Facebook and an Instagram
