Refactoring always requires a deeeep muscle-memory of the codebase, getting to that point at a job is a great feeling. Sometimes (always?) making things pretty pays off, too! :)
It depends on how the code is structured. Our apps use a micro kernel / plugin architecture of loosely coupled modules so most of the time it is sufficient to know only a few modules to be able to refactor. Fundamental changes require a lot more knowledge - this is true - but they are not common.
In my experience so far it is the structuring for maintainability that really shows the skill of the programmer - writing the logic itself is the easy part.
