I'm at a mandatory-pairing-multiple-times a day place right now and I've made the same observations you are identifying here.
Pairing may be good to some degree for knowledge sharing but it works against good architectural design, which requires the kind of holistic thinking you describe, as well as kind of micro-iterative whittling away at the problem that interactive pairing couldn't ever achieve. In addition, the n^2 communication lines yield the worst effects of Conway's Law https://en.wikipedia.org/wiki/Conway's_law
Nearly all of the computational problems we face in industry (save a narrow class of specialty problems) can be pretty easily articulated in the "macro" sense: "oh, we'll just have this distributed actor invoke that one and he'll reply with the blah, blah state".
Look, we just paired and collaborated on a design, cool! But the reality is that none of the hard parts have been solved. The hard parts are always the unanticipated errors, edge cases, invariant violations that beset every system--a large set of issues that need to be made airtight when it comes down to actually putting the code and tests together, and these require the holistic thinking you're talking about or either you get subtle bugs or otherwise a mess is made in the code.
As an industry we've gone so hard on coming up with social processes to "knowledge share" (pairing etc) and do things like audit for defects (code review process), but all of these are ultimately compensatory measures for bad design and bad code; and, in fact, as you suggest, they kind of encourage and trend toward bad code.
A different approach to building systems socially would focus on the code being decoupled and articulated such that the barriers to entry would not _require_ pairing and these other processes. Decouple and compose components that can be understood at face value and in situ.
But this is very hard and requires extensive practice and expertise, something our industry isn't keen on talking about. We like that quick race to "expert beginner" wherein we can say "Phew, now I'm an expert". So instead of focusing on mastering these we go for least-common denominator processes, which if you think about it is what pairing, and many of our processes are,- least common denominator.
The truth is we're social creatures, so I don't think we'll ever get out of this state. Social processes will win not because they optimize the problem space but because most of us love and crave the interaction. My only lament is that we don't call the spade a spade and we try to argue that these processes are necessary for optimization when they really impeded optimizations and create downstream frustration and limitations.
IMO, pair programming is most useful for short design discussions and mentoring, like explaining a better way to architect a feature to a junior engineer. I look back on these experiences fondly, and this type of collaboration helped me grow a lot as a developer.
I don't feel like I get the concentration and focus required to "put the whole code implementation in my head" when I'm pairing, so I don't really see how it can be used 100% of the time to build systems efficiently.
And regardless of whether it's a good way to work or not, many engineers just won't want to have someone looking over their shoulder 24/7. That kind of environment would drive me mad. I need to be able to drown out the world and bury my head in the code, and most of us don't work on code the entire day. I like having the freedom to read an article or browse the web for a bit whenever I want to fuck off.
Pair programming and mob programming are definitely not for me, and there's no way I would take a job that requires doing either most of the time.
Good design and getting everything right up works right up until the requirements change and the design no longer meets the ask. Unsurprisingly, teams might not even know all the requirements when building something new. Having everything orderly up front is a pipe dream.
That said, there are definitely cases where sitting back and thinking through a design is a really good idea. In my experience, it's 1/10th of the job.
Good design can handle changing requirements, otherwise it isn't good design. Good design requires a lot of domain expertise and not just being a good programmer since you have to be aware of what kinds of things the system should be able to handle, but it isn't impossible.
By domain expert I mean someone who has worked in a single domain for many years. Lets say you have written healthcare backend software for 10 years, then if they tell you to design a new healthcare software system then you will be able to predict most future requirements the system will have even if they don't spell those out explicitly to you. Sometimes laws changes and you have to redesign stuff, but having to adapt code due to new laws isn't a common occurrence.
Edit: And 10 years in a single domain really isn't a lot of time. In a sane world you would be called an apprentice until you reach that level, that is the only way to make robust products. This is how almost every other field works, but in programming people seem to think that such expertise is impossible, at least at scale.
Pairing may be good to some degree for knowledge sharing but it works against good architectural design, which requires the kind of holistic thinking you describe, as well as kind of micro-iterative whittling away at the problem that interactive pairing couldn't ever achieve. In addition, the n^2 communication lines yield the worst effects of Conway's Law https://en.wikipedia.org/wiki/Conway's_law
Nearly all of the computational problems we face in industry (save a narrow class of specialty problems) can be pretty easily articulated in the "macro" sense: "oh, we'll just have this distributed actor invoke that one and he'll reply with the blah, blah state".
Look, we just paired and collaborated on a design, cool! But the reality is that none of the hard parts have been solved. The hard parts are always the unanticipated errors, edge cases, invariant violations that beset every system--a large set of issues that need to be made airtight when it comes down to actually putting the code and tests together, and these require the holistic thinking you're talking about or either you get subtle bugs or otherwise a mess is made in the code.
As an industry we've gone so hard on coming up with social processes to "knowledge share" (pairing etc) and do things like audit for defects (code review process), but all of these are ultimately compensatory measures for bad design and bad code; and, in fact, as you suggest, they kind of encourage and trend toward bad code.
A different approach to building systems socially would focus on the code being decoupled and articulated such that the barriers to entry would not _require_ pairing and these other processes. Decouple and compose components that can be understood at face value and in situ.
But this is very hard and requires extensive practice and expertise, something our industry isn't keen on talking about. We like that quick race to "expert beginner" wherein we can say "Phew, now I'm an expert". So instead of focusing on mastering these we go for least-common denominator processes, which if you think about it is what pairing, and many of our processes are,- least common denominator.
The truth is we're social creatures, so I don't think we'll ever get out of this state. Social processes will win not because they optimize the problem space but because most of us love and crave the interaction. My only lament is that we don't call the spade a spade and we try to argue that these processes are necessary for optimization when they really impeded optimizations and create downstream frustration and limitations.