"Thinking ahead" is often a great excuse to design an overengineered mess of a solution that can't be tweaked and doesn't really properly solve the problem either. To be pithy - see Java.
Sometimes, exploring the problem space can give you a fuller understanding of a problem faster by forcing you to confront pitfalls that may not be obvious until you try a solution. We use all kinds of wonderful terms for this - "Agile", "Prototyping", etc.
Both extremes - fetishizing planning and up front design, or fetishizing short term iteration and poking things without deeper thought - have their problems, and occur too often. Neither tool is a panacea, but both have their place.
There's a difference between "thinking ahead into the next problem", i.e. premature generalisation, and "thinking ahead into the details of the current problem".
It's good that you mentioned Java, because it is a language which I find extremely IDE-centric, and I suspect that's also what causes easy premature generalisation --- creating new classes with tons of boilerplate automatically generated by the IDE is so easy that it encourages programmers to. That doesn't help one bit with the details of the algorithm, unfortunately; it often gets "OOP-ified" into a dozen classes and much-too-short methods created as a result of the "fiddle with it until it works" mentality.
'Fiddle with it until it works' has to be done when you are working with a product that isn't documented well enough. If that mentality is used in general for programming it is bad, but there are some situations where experimentation has to be done to work out how parts the product work.
Java as a language is producing more value to actual businesses than most other popular languages. Where you see an over engineered mess, others see valuable abstractions, extensibility, compatibility and self documentation. Unfortunately, understanding this so called mess requires knowledge of the lingua franca of object oriented design which has fallen out of favour by the new generation.
I'm not saying that there are no unjustifiable over engineered java libraries, but the current hype cycle of web frameworks seem to indicate the burden of proof of good design should lie with current technologies as well as previous.
> Java as a language is producing more value to actual businesses than most other popular languages. Where you see an over engineered mess, others see valuable abstractions, extensibility, compatibility and self documentation.
"Everyone uses it" or "it's producing value" doesn't mean it's not an overengineered mess that everyone recognizes as such - it just means that imperfect code still trumps no code. Switching languages usually means tossing out your old codebase, leaving you at "no code".
I have worked on such messes, created such messes (oops!), and cleaned up such messes.
That said, I'm sure there is a Java project out there which actually benefits from stereotypical levels of Java abstraction and patterns - and I'm sure there's a few codebases out there where "my" and "others" opinions differ exactly as you say.
> I'm not saying that there are no unjustifiable over engineered java libraries, but the current hype cycle of web frameworks seem to indicate the burden of proof of good design should lie with current technologies as well as previous.
100% agreed - not that I'm qualified enough at web dev to have much of an opinion on this. If anything, the churn of web frameworks smacks of being both overengineered (do you really need a whole framework for that?) and underengineered (wait why are we replacing things yet again?) simultaneously.
Spring comes to mind as a widely used framework that benefits from those "stereotypical levels of Java abstraction and patterns."
But it's the exception rather than the rule. Once you have something like Spring in your codebase, to take care of modularity and reuse, everything else should be coded with as little "abstraction and patterns" as possible.
Sometimes, exploring the problem space can give you a fuller understanding of a problem faster by forcing you to confront pitfalls that may not be obvious until you try a solution. We use all kinds of wonderful terms for this - "Agile", "Prototyping", etc.
Both extremes - fetishizing planning and up front design, or fetishizing short term iteration and poking things without deeper thought - have their problems, and occur too often. Neither tool is a panacea, but both have their place.