The challenge is that it's often not possible to understand which few extra hours at the whiteboard will save you the refactoring, or even which code will be worth keeping around long enough to be worth refactoring.
I've experienced both of these extremes - days of refactoring that could've been avoided by a few extra hours thinking about the problem, and days of refactoring that could've been avoided entirely by shipping the product early and realizing it was a bad idea to begin with. I think the optimal approach is actually to sketch out a simple solution first, ship it with bugs and all to a group of small trusted testers (or even just use it yourself), and then reevaluate which parts of the product are actually useful. Then throw out the 80% that's of marginal utility and really think through how to implement the remaining 20% in a solid way that you can build upon. Repeat once you have that launched, building out additional features in the same way.
I've experienced both of these extremes - days of refactoring that could've been avoided by a few extra hours thinking about the problem, and days of refactoring that could've been avoided entirely by shipping the product early and realizing it was a bad idea to begin with. I think the optimal approach is actually to sketch out a simple solution first, ship it with bugs and all to a group of small trusted testers (or even just use it yourself), and then reevaluate which parts of the product are actually useful. Then throw out the 80% that's of marginal utility and really think through how to implement the remaining 20% in a solid way that you can build upon. Repeat once you have that launched, building out additional features in the same way.