I remember talking to a former software exec at Amazon and he actually told me a personal philosophy that I thought was pretty profound at the time.
To set the stage this was when web services where coming into fashion and he was a big proponent of them, he said that he likes to focus teams around business domains (e.g accounts receivable, road logistics, etc.) then let them control their destiny. He said that he liked a service based architecture because it allowed him to build Amazon scale software, a piece at a time and more importantly the nugget that I really took away from the conversation was that you cant stop bad software from being written, but you can isolate bad software from the rest of the system which allows you to measure all kinds of things that give you an overall metric on the quality of that software. Things like time to defect resolution, uptime as well as true business metrics given that they are aligned with a vertical (i.e did the software empower the core business goal). If the numbers are lacking the software is bad plain and simple. Isolation makes it easy to replace.
I don't think enough people who architect large systems give thought to how do we protect the overall system from bad code which is a certainty to happen on large projects because they necessitate large teams. As developers and architects we tend to focus on elegant designs, but the overall support of an application thru it's life is anything but elegant.
Generally products built by large teams are large products. There are extra challenges associated with coordinating large groups of people, and there are extra challenges associated with developing big, complicated software products with lots of different subsystems that have to work together.
Hopefully no one is using a large team to build a small product. If so, yeah, don't do that. But you're going to get into different problems building large products with small teams.
Ultimately there's no silver bullet, the best you can do is maintain awareness of the tradeoffs, identify and deal with problems as they arise, and otherwise do the best you can to maximize the likelihood of success and minimize failure.
Sometimes people feel like they've found a silver bullet just by identifying the problems, in which the silver bullet is "well just don't do that then". In contrast, I feel like everything that's nontrivial in life consists of choosing from a set of compromises and tradeoffs. There's a natural tendency to feel the pain of the compromises and assume that if only we weren't making those compromises, everything would be great, without taking into account the tradeoffs and compromises that alternate approaches would create.
There's no 'silver bullet' - but surely a 'large product' does not need to start off as a 'large product'.
A 'small team' making an MVP, and then moving on incrementally from there, adding more people to do peripheral features as the project progress might seem like a good way to overcome some inherent 'monolithic' risks - as well as keep costs down at the start.
It takes an open mind from management to grasp that building complex software is not like building a really big building. You just can't predict outcomes, so, you have to build a good team that can forge a path, and have some 'kill-it-don't-kill-it' guidelines, and expect that many will fail.
Or sometimes your prototype or proof-of-concept simply gets deployed to production by executive fiat because they don't understand that it's put together with bubble gum and baling wire, and over the next 3-5 years you are never given the resources to do anything but (a) keep it running somehow, and (b) add more features that got promised to customers in order to land them and that need to be live ASAP.
What's that...? No, I've never experienced that situation myself, why do you ask?
> What's that...? No, I've never experienced that situation myself, why do you ask?
Another way to look at it is to say this provided you with a job and an entire company with a revenue basis, probably for years.
I mean on the other side of the fence there are plenty of stories of software developers being allowed to "properly design", no technical debt, then after 1 year of zero result they come with a product, which doesn't even start (the tests run and succeed, though). Then the third or so feature request that comes in ... needs a rewrite.
The key really is making sure a large project is not a monolithic project. Break it down, break it down, break it down. The same logic that works for implementation is also valuable on the product side.
Scaling engineering is much easier if the project is designed with that scale in mind.
Agree 100%. So many people make lip service to SCALE at my company, but out of 20+ teams we are the only team running real scale tests against real defined limits with real (obfuscated) customer data.
I am reminded by something I read by a person who studies complex systems. He said that every complex system that works well started with a simple system that worked well, and then gradually added new features over time.
Gall's Law is a rule of thumb for systems design from Gall's book Systemantics: How Systems Really Work and How They Fail. It states:
A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system. – John Gall
I agree with the article mostly, but one issue I've always had with product owners is the lack of willingness to lay the proper foundations for technical products to succeed. That can be anything from training and allowing developers to do a proof of concept before they decide on the technology. The importance of automated testing,code reviews, refactoring, etc.
Usually there's some sort of engineering manager / lead engineer who's in charge of deciding how the product is built from a technical point of view. He's the main collaborator of the product manager/owner and they need to get along well.
I agree with and have experienced this situation. I've also experienced it more often with small teams (3 seems to be critical mass).
Having a product manager is key, even for internal products. You don't even have to call them a product manager. That person can be a senior engineer who knows what the system has to do to be useful in the company's ecosystem. Someone has to have the final say when there are decisions to be made about the direction/execution of development and feature choices.
Also, there must be a manager above the "product manager" who has the guts to kill the project or remove a poorly performing product manager. This seems to be a big factor in most of the projects I've seen go south.
TMMM contains truly insightful and universally applicable knowledge and insight into building large software systems.
It also contains truly insightful knowledge regarding building operative systems at IBM in the seventies.
It's one of my favourite books on software development. However, it requires some insight and preexisting knowledge from the reader in order to tell what's what.
The absence of a clear product definition and strategy is hardly unique to large teams, and in fact will manifest itself more fatally in small teams in my experience.
I may be suffering from a bout of biased sampling but it certainly seems that the problem of actually defining a product is getting substantially worse these last few years and in the absence of real product strategies there is instead a proliferation of cargo cult product thinking.
in my company they first start a project by coming up with a budget, then they hire many people for that budget, then they have a long series of meetings figuring out what the thing should really do. So you have the pressure to get started with a ton of developers while still nobody has a clear vision. These people produce something but usually it's not what you really need.
I would much prefer to have a small team of very good people figuring out things, develop a clear strategy and then see where more people are needed. I think the main evil is to ramp up team size before the team is ready.
Most developers in small teams fall into one or more of the traps of freedom that hurts technically and makes their lives tougher than necessary. With enterprise start-ups you need to be just as agile and fleet-footed as all the b2c start-ups when it comes to backends to survive. Depending upon how much overlap of features across customers your sales pipeline is bringing in, you may share only 10% of code between different customers (basically glorified outsourced software development while pretending to be a product company) and you will need to architect your infrastructure with it too. Getting a feel for where debt exists, fixing it when appropriate, and going at start-up speeds without getting too sloppy is a challenge most b2c developers are familiar with. Hockey stick growth exists as a huge spike as a single customer gets signed, and now you need to scale your software 100x past what you tested within weeks while oftentimes not being able to use modern tools like cloud infrastructure services because your customers forbid it legally. Now add in much tougher (and worse - specific) infrastructure and security requirements across the board and it's no wonder b2b software tends to suck.
Hiring developers early on with prior experience in the business domain is critical for company success for this reason. Every developer knows how developers work, so tools by developers for developers tend to work fairly often. Most developers probably don't know anything about how your customers work though and will have difficulty prioritizing what's the most fluid parts and what's not regardless of how brilliant and fast they may be.
It's tough to tell whether haphazard selling or haphazard technical design has a bigger impact upon codebases. Add in realities like team turnover and even technical design may matter little compared to how well a codebase is documented for the health of the codebase. I can stare at thousands of unit tests but without some history and organizational know-how it's likely for a successor to repeat the same mistakes by trying to mimic existing patterns.
If you want to get more done, figure out how to shrink the size of your team. The quality will go up, too, since ownership is more keenly felt the fewer ways you slice it.
Projects that require large teams are larger, and thus more complicated and difficult. More difficult projects fail more often.
Artificially shrinking the size of the team without reducing the scope and complexity of the project will rarely improve its chances (unless the team size was only an effect of beaurocracy, and you manage to only kick off the people who aren't contributing).
Right, so the key is giving the product owner the authority to decide what the product will be, what features will be included, which features will be excluded, and prioritization of those features.
What products are we taking about? Why does the author simply assume the fact that a large team exists is the reason why the project fails?
He doesn't seem to touch on the fact that many times large teams exist because a project is large and complex, which makes it easier for projects to fail.
They needn't be but large teams are usually bureaucratic and bogged down with politics and empire building. Even when they are successful in creating something it's usually down to a small number of individuals working against the tide.
That will mean that most small teams will fail, as they will be composed from average developers. While a few teams will succeed as they have the "extraordinary", whatever that means, developers.
But the reality is that big teams have problems independently of the capacity of the companies to attract talent. So I don't think that "average ability" is the source of the problem.
As someone else posted, The Mythical Man Month is a good read on this area. And communication overhead is an unsolvable problem as the people involved intro a project grows.
>> That will mean that most small teams will fail, as they will be composed from average developers.
No, it means if you have a large team you will have more average developers - you could have a small team of 3-4 hotshot developers and they turnout a fantastic product - but add 10 normies to the team and it just drags them all down.
If you take 10,000 developers and distribute them among 1,000 teams, the expected average ability of each team is the same as if you distributed them among 10 teams. What changes is the variance--the averages among the 10 teams will be more tightly clustered than among the 1,000 teams.
That is a different argument and may or may not be relevant. Really, read Mythical Man Month, the problem with large teams is that their size leads to internal communications overheads, politics ...
I've been at two places where they were hiring like crazy because of huge rounds of funding. Even though they were hiring like crazy, they weren't hiring the right people.
I had an argument where I was explaining that engineering would be more efficient if we hired a researcher - at least just one to start with. Managements answer was to hire 10 more engineers to solve the underlying problem.
If you are working on a large product that is the future of the company and involves several silos, and do not have an executive sponsor who is responsible for making sure it gets done, you are doing it wrong.
This is a "yes but" kind of post ( i feel ). Executing a large product can often require several project managers reporting to a product owner.
It's not product vs project.
This article simply talks about one reason that could cause a large product to fail. In my experience, an internal management focused approach is fine, but can still lead to an organisation that is good at successfully failing.
The focus on the team is interesting, specifically the "hold them accountable for clearly defined KPIs" point. What about the product owner and manager? What happens to them when the product is delivered feature complete and on schedule but still fails?
As developers we have peer reviews, in my experience, product managers rarely have people review their work. Meaning there are very few people that can push the product manager to improve his/her strategy/tactics/approach. I find it's usually a senior manager or C level that has the most influence over a product manager. The flaw in this is that the person reviewing the work is not an expert practitioner thus cannot provide concrete advice.
The author mixes in methods from LEAN and agile with statements such as:
- "Start with the smallest scope you can deliver that provides value to your users."
- "Let them choose their own tools and processes. And then support their decisions."
However, given that is well tread ground, I think the author's main point is to address the issue in large teams whereby "no one is fully committed".
The key piece of advice is "Pick a group of people who are willing and able to shake up the status quo in your organization." Nevertheless, I would say that the act of doing this naturally turns your project team from a large one into a small one.
The advice is not unique and has definitely been tried, with myself witnessing a product manager that effectively created a startup within a startup. One should not discount the manifestation of this phenomenon as a visible sign of organisational disfunction.
In addition to the implied comment that people not on a team were not good enough to meet certain criteria, you also end up with a clique situation as described by a former Valver [1].
All this comes back to the point that no one is fully committed and a toxic environment begins to form. Unfortunately the answer to getting commitment from people is even higher level and ethereal, best described as ensuring the company has mission and purpose. Similar to Dan Pink's focus on mastery, autonomy, and purpose - but executed at a cohesive organisational level.
I would summarise what's needed as real leadership. Meaning someone that can rally everyone around a product and the company. Once you have defeated apathy, commitment can be channeled into extra-ordinary outcomes. What a lot of managers don't seem to understand, or be willing to practice, is that getting and building commitment often means building meaningful professional and personal relationships with the people you work with, which is at odds with the often mercenary nature of businesses.
To set the stage this was when web services where coming into fashion and he was a big proponent of them, he said that he likes to focus teams around business domains (e.g accounts receivable, road logistics, etc.) then let them control their destiny. He said that he liked a service based architecture because it allowed him to build Amazon scale software, a piece at a time and more importantly the nugget that I really took away from the conversation was that you cant stop bad software from being written, but you can isolate bad software from the rest of the system which allows you to measure all kinds of things that give you an overall metric on the quality of that software. Things like time to defect resolution, uptime as well as true business metrics given that they are aligned with a vertical (i.e did the software empower the core business goal). If the numbers are lacking the software is bad plain and simple. Isolation makes it easy to replace.
I don't think enough people who architect large systems give thought to how do we protect the overall system from bad code which is a certainty to happen on large projects because they necessitate large teams. As developers and architects we tend to focus on elegant designs, but the overall support of an application thru it's life is anything but elegant.