For me the key is that I can lay things out in memory exactly the way I want, if necessary to the point where I can fit things in cache entirely when I need the performance and only to break out of the cache when I'm done with the data. This is obviously not always possible but the longer you can keep it up the faster your code runs and the gains you can get like that are often quite unexpected. I spend a lot more time with gprof than with a debugger.
