Slightly OT, but I enjoyed the discussion about (physical building) construction in the book "the Checklist Manifesto". In The Mythical Man Month, Brooks advocates the use of something like the "master builder" model for software. It turns out that in actual (physical building) construction, the master builder model is no longer used, because the construction process has become much too complex to be understood by a single person.
Instead, in construction, they use a system of checks to ensure that different experts consult one another so that every decision is reviewed by a relevant expert.
I suspect that the "chief architect" approach that Brooks advocates may have become obsolete as well since the Mythical Man Month was written. Perhaps software developers could learn something from the newer methods that replaced the "master builder" model in construction.
I still remember the jolt I felt in 1958 when I first heard a friend talk about building a program, as opposed to writing one. In a flash he broadened my whole view of the software process. The metaphor shift was powerful, and accurate. Today we understand how like other building processes the construction of software is, and we freely use other elements of the metaphor, such as specifications, assembly of components, and scaffolding.
The building metaphor has outlived its usefulness. It is time to change again. If, as I believe, the conceptual structures we construct today are too complicated to be specified accurately in advance, and too complex to be built faultlessly, then we must take a radically different approach.
Let us turn nature and study complexity in living things, instead of just the dead works of man. Here we find constructs whose complexities thrill us with awe. The brain alone is intricate beyond mapping, powerful beyond imitation, rich in diversity, self-protecting, and selfrenewing. The secret is that it is grown, not built.
That quote is beautiful but it reminds me a lot of tales of hubris, although I do not believe in any god(s), I always felt some truth in that mythical sin.
Can we grow designed software? Or, can we design truly growing software? I guess it depends on the definition of growing you use, "scalable" ? "alive" ?
Refactor, refactor, refactor. Growing things adapt to changing circumstances. Software must adapt to changing requirements. Refactoring is the only way.
This is so important. We all know beautifully "designed" API's or libraries.
The secret is that the "elegant" API design you see is usually the n-th iteration.
Growing code is much like growing trees. The tree grows itself, what it requires is that it is pruned and helped to reshape in a fashion that will allow it to stay alive for a long time and bear fruit.
You start planting trees, maybe it's in an empty field, or maybe it's in an old forest. At some point it takes root and multiplies. People help plant your forest in unexpected places and it expands. At some point you try pruning and controlling the trees. And at some point a forest fire destroys it making way for a new forest to grow.
We can - but if we keep the analogy of grown things we can only grow stuff that has already been grown before and make only tiny changes. It's worth remembering that evolution has neither purpose nor direction. Even "adapting" is an aimless process, the changes are predictable only for very small adaptations. Major changes may lead to anything, including a "restart" (extinction event followed by something different).
So the model has its limitations for human projects where we don't want to have quite that propensity for surprising outcomes.
See, I've always seen the statement that evolution is intentionless to be a bit arrogant. Consciousness, as we are just starting to learn, is a property of emergent networks. Further, we are just starting to discover that phenotypical expression is so much more complex than 1/2mom + 1/2dad = baby. Evolution is calculating something, even if it's inscrutable to us.
By designing systems in layers of interacting components, we can grow more and more complex systems. The internals of different components are irrelevant so long as the external interface is consistent.
We evolve them by improving the interfaces (see APIs deprecating functions/methods/messages) and by refactoring/rewriting the internals or implementation.
>Perhaps software developers could learn something from the newer methods that replaced the "master builder" model in construction
I don't know. I get the feeling that large construction projects suffer from many of the same issues that affect software projects.
And every time something goes wrong enough to end up in the media, the reason is invevitably that the various contractors/experts didn't coordinate properly, that no one really feels responsible, that major design flaws were overlooked early on and now there's no way to undo them without going even further over budget. (Yes, I am aware of the selection bias)
Big projects are hard and their hardness increases disproportionately with size. If a project's goals are such that there can be no "master builder", then things become a lot riskier and they go wrong more often.
So I think what we actually want is a bazaar of projects where each project is architected and overseen by a master builder.
Like with the endless state vs market debates, it's always a question of balance, not one of either or.
What you're advocating for seems like a SOA (or micro-services) approach?
I agree with your reasoning. It resonates with what for example Richard Brandson says about companies: split them up when you reach ~50 employees.
By splitting large projects into smaller sub-projects, we've now created the need for a higher level master builder to oversee how the sub-projects integrate, haven't we?
>What you're advocating for seems like a SOA (or micro-services) approach?
No, I think SOA and micro-services are technical considerations that have relatively little bearing on division of labour or organisational/project structure. You can statically link a library created by someone you don't even know, or you could have one huge dysfunctional in-house team split up a monolith into fifty different micro-services without creating any new ownership rules at all.
So I think splitting up projects in smaller subj-projects alone is not good enough if all it does is create a deeper organisational hierarchy based on exactly the same command and control principles as before. As you say, it just creates new planning issues on the next level.
We need manageable units of planning, and we need to know when things become too complex for planning and require some principle of self organisation. The difficulty is that self organising systems are not easily steered towards a single goal.
It's interesting that you quote Branson, because the most important feature of a separate company is that it is a unit of economic responsibility and ownership. It usually serves more than one customer, unlike a micro-service that is a fragment of some in-house monolith. And it sets its own goals, mostly independently.
I have posted multiple times and in multiple venues that I think large software projects need to adopt a proper systems engineering process. Rather than a "master builder" you have a "lead systems engineer" whose job is to manage the complexity of the project. It goes beyond just making sure the experts and team leads communicate. It includes deciding how functionality will be divided between the functional boxes of the system, how those boxes will interact, and managing changes to the plan so everyone affected will understand what is going on and have input into how things change.
Traditional multidisciplined engineering domains (aerospace, automotive, marine) have been using this model for decades. It's about time that kind of engineering rigor came to software.
I'll hijack your OT comment to just mention that all this talk of Cathedrals and Master Builders reminds me of the historical novel "Pillars of the Earth" which I found a truly fascinating read.
When reading the Mythical Man Month I got the impression that their idea of a "chief architect" is more like today's "product manager", not today's "chief architect".
That's a very good point - I've always felt it was a shame that in order for a thesis to be "publishable" you have to frame it in 300 to 500 pages. The Design of Everyday Things by Don Norman is a great example. It's an awesome book, and anyone who's designing user-facing-anything should read it, but I think the story could have been told in about half as many words.
I felt this way about Thinking: Fast and Slow. One of my favorite books, but could have been half the length. At least they included the original white paper in the back.
No, I liked the book. My point is that there are other great ideas out there that we're not exposed to because they're 100 page ideas rather than 300 page ones. It's less of an issue as we go more digital and rely less on traditional publishing channels, but there's still a knee there right now.
I've thought this way about other books, and seen this thought expressed about The Checklist Manifesto ("it should have been a 20 page pamphlet"). But I feel that most of the length (say, at least 75% of the book's published length) is necessary and justified.
Not because the concept itself requires so many pages to understand, but because it takes several repeated high-profile examples of checklists making a big difference, before the feeling of "it's just a checklist, what's the big deal" - that a lot of people express upon hearing the idea first - is replaced with understanding and internalization.
Something to consider, modern buildings are often terrible. Checklists are great for building the minimal possible thing that works, but deeper insight is often necessary for enduring projects.
Microsoft could probably rewrite Windows at great expense to use 10% as much code and it would still be a bloated mess. But, well why would they waste the money?
Instead, in construction, they use a system of checks to ensure that different experts consult one another so that every decision is reviewed by a relevant expert.
I suspect that the "chief architect" approach that Brooks advocates may have become obsolete as well since the Mythical Man Month was written. Perhaps software developers could learn something from the newer methods that replaced the "master builder" model in construction.