>more similar to writing a software project for the second time after you throw away your first version - but now with more resources
All aircraft design is basically this. You don't/can't ever really start from scratch, there's an enormous volume of historical aircraft designs and all new planes look very similar to old planes. There are very rarely opportunities to solve problems in new ways.
even if you have a novel design your customers are rarely interested.
Airlines are extremely conservative because pilots are expensive and hard to train, which is how we got the whole 737MAX fiasco instead of a clean-sheet, optimized design. Though i wonder why they didn't go for commonality with the 787 similar to how the a330 and a350 are common type ratings.
The problem Boeing had was that there is a massive amount of 737:s, and a correspondingly massive pool of crew qualified for those planes. Customers really wanted not just a plane that was compatible with something, but specifically compatible with the 737. And this was a problem because the 737 is too old of a design.
If Boeing attempted to sell these customers a new type, even if there is some compatibility with some other plane, it creates an opportunity for them to freely compete bids between Airbus and Boeing, and even when Boeing wins that bid, it's a loss to them. (Because they are going to win at a lower price point than if the customer has to account for the switchover cost only for Airbus planes.)
All aircraft design is basically this. You don't/can't ever really start from scratch, there's an enormous volume of historical aircraft designs and all new planes look very similar to old planes. There are very rarely opportunities to solve problems in new ways.