To use coding terminology, there's a major distinction between "refactoring" vs "rewriting". Refactoring involves many incremental and iterative changes to an existing system. Whereas rewriting is discarding all incremental changes, and instead building a new system from scratch.
Refactoring is generally the most reliable way to improve a system. But rewriting is the best way to get promoted, and do work that feels personally "fun" and "impactful". I suspect this is the decisive factor in a large fraction of UI overhauls
If people thought of pointy-clicky UIs in similar terms to the way they build CLIs there would be more refactoring and less rewriting. "Breaking the UX" in a CLI tool means that untold numbers of its dependents are broken. It's a breaking change to your public API, and it shouldn't be done. The only reason you'd do it is if your first attempt was a complete failure. We should treat user interaction with the same level of respect we treat all the other APIs.
Refactoring is generally the most reliable way to improve a system. But rewriting is the best way to get promoted, and do work that feels personally "fun" and "impactful". I suspect this is the decisive factor in a large fraction of UI overhauls