Well, though I’ve also seen people do the same thing inside of existing engines, for example trying to use Unity but become so frustrated with its design so much that they begin to spending their time building their own external systems and tools. It seems more of a mindset problem than what engine you actually use (whether you build your own or not).
If you have some amount of self-discipline and implement your custom engine to be minimal and only tailored to your game (as many indie games have demonstrated), I don’t see how it can’t be done.
Unity is used more than any other game engine in the world to release successful commercial games. If you can’t figure out how to use Unity to make commercial games then the problem is you not Unity. I have released commercial games using 3D game engines I wrote from scratch years ago before Unity existed. I would never recommend doing that today when Unity is available and so easy to use compared with having to write everything from scratch.
If you have some amount of self-discipline and implement your custom engine to be minimal and only tailored to your game (as many indie games have demonstrated), I don’t see how it can’t be done.