It's been a while since I read the Mythical Man Month, but as I recall his answer to your point is to look at things through the perspective of systems and subsystems. While Linus is responsible for the overall Linux kernel/system, someone else may be responsible for the X subsystem in the Linux kernel. Which, from what I'm aware, is an accurate representation of how Linux is developed in practice.
