Posts like these really show the problems in the current free software development world; balkanization between different stakeholders and bizarre processes that alternate between bureaucracy and anarchy. Watching the tug-of-war between Red Hat and Canonical play itself out in the form of dysfunctional community development process is depressing. I'm aware that watching sausage being made never makes you very hungry, but as a user of free software, I don't feel my needs are served very well by the territorial and political battles being fought.
What makes it even more frustrating to me is that not only are KDE, Gnome 3, and Unity all failing to cooperate successfully on shared resources, but they are all marching resolutely in the wrong direction, regardless of the pleas of the user community. So far as I can tell, everyone hates Gnome Shell, everyone hates Ubuntu Unity, and everyone thinks that KDE 4 has been a big mess compared to KDE 3. I see all of these projects as engaging in a slappy fight as they march together off the cliff of unusability and misguided redesigns.
I think to be fair to all three, you have to give them six months after they are actually released before we judge them. As is typical in OS Linux community, a very vocal minority is probably speaking up and "hating" all three. I would wager that the majority opinion is much different than what you hear from the vocal minority.
"Everybody" doesn't hate anything, perhaps all you hear are the people complaining. These projects are being accepted by various Linux users reasonably well. Any type of change is going to inspire some ire among longtime users.
And I'm afraid these types of conflicts happen in the non-free-software world, too, the most easy examples being recent stories about internal projects in Nokia and Microsoft. Things get messier when you want standards and interop.
This type of lazy thinking in generalizations amazes me. Inefficient at what? If you mean it's inefficient for rapid decision making, I wholeheartedly agree. If you mean inefficient at allowing people to express themselves, I disagree.
Fair enough - Open systems explore the landscape of all possible systems through diverse implementation and designs. This diversity sometimes leads to open systems finding the global optima, but even in less ideal circumstances the various different approaches reach a local optima for some subset of user needs.
Closed systems on the other hand try to reach the global maxima by analysis, design and 'vision'.
The original poster laments the competing implementaions, back stabbiness and ideologies. From my perspective that is how open source is supposed to work - people who are full of it trying to take their vision to its maximum limit. Competing implementaions and backstabbiness is not a modern phenomenon - bsd vs linux, emacs vs vi, xemacs vs emacs, n implementaions of scheme, n implementations of common lisp, ironpython/jython/python/pypy.
You bring up an interesting phenomenon: well-developed open source environments share a cut-throat competitive culture for consumer mindshare rather similarly to multinational corporations. I wonder if there is a scalability issue in human management where at some point we go berserk trying to define the mainstream.
As far as open source is concerned, it might be helpful to split the projects into two categories before examining their motives
1) Corporate driven projects. Here the need to reinvent might be due to profit based motives, or it could be because someone is trying to create work for themselves.
2) Volunteer driven projects. In this case I think it's the geeky need to understand things from first principle that drives the reinvention. The fact that for poorly documented projects the cost of reimplementation early on is comparable to rewriting from scratch doesn't help. Also large projects seem disorganized and bloated when compared against a new naive implementation with few features.
What makes it even more frustrating to me is that not only are KDE, Gnome 3, and Unity all failing to cooperate successfully on shared resources, but they are all marching resolutely in the wrong direction, regardless of the pleas of the user community. So far as I can tell, everyone hates Gnome Shell, everyone hates Ubuntu Unity, and everyone thinks that KDE 4 has been a big mess compared to KDE 3. I see all of these projects as engaging in a slappy fight as they march together off the cliff of unusability and misguided redesigns.