I've only used a debugger a few times, when the program did not seem to behave according to the source code. That was invariably due to a third-party bug (compiler, library, OS) or to me failing to understand a subtle point about the language or library I was using.
But in general I agree with you.