I think the underlying reason long term plans don't work in most commercial software development is that accurately estimating software development effort or schedule is the same order of magnitude complexity (CS big O) as implementing that software. If people could efficiently generate accurate estimates (and therefore schedules and plans), they would. It turns out to be hard even in a static world, and impossible in an unpredictable dynamic one. The fundamental issue is that developing software is 'design all the way down', it rarely turns into 'mechanical assembly' - in general, each line of code written is designed. That was one of the insights of the agile movement - the implementation 'stage' required considerable 'design' (or re-design, as the case may be). The difference between the coders and the architects is simply the scope of the abstractions. A successful architecture can reduce the scope of abstractions necessary when implementing specific bits - but it can't eliminate them.
