What tends to happen as you refactor bad code is that you gain some intuition about the way the code needs to flow. The longer you spend grinding away at the existing code, the more likely it is that rewriting it will work, because you'll have pent-up "architectural energy" waiting to be used, and good, already-debugged code from the previous version that can be copied in.
The most likely causation for crossing a threshold from refactor to rewrite, while steering clear of the "big bang rewrite", is that you have to ship a feature that triggers an end-run around some of the existing architecture. So you ship both new architecture and the new feature, and then it works so well that you can deprecate the old one almost immediately, eliminating entire modules that proved redundant.
Edit: And if you don't really know where to start when refactoring, start by inlining more of the code so that it runs straightline and has copy-pasted elements(you can use a comment to note this: "inlined from foo()"). This will surface the biggest redundancies at a minimum of effort.
The most likely causation for crossing a threshold from refactor to rewrite, while steering clear of the "big bang rewrite", is that you have to ship a feature that triggers an end-run around some of the existing architecture. So you ship both new architecture and the new feature, and then it works so well that you can deprecate the old one almost immediately, eliminating entire modules that proved redundant.
Edit: And if you don't really know where to start when refactoring, start by inlining more of the code so that it runs straightline and has copy-pasted elements(you can use a comment to note this: "inlined from foo()"). This will surface the biggest redundancies at a minimum of effort.