Whether or not you use an ORM, you ideally have your own abstraction on top of it, defining a set of known queries. This avoids the classic Rails problem where the User model has a million methods, and you don't know which of them might make a SQL query.
Also, ORM caching always finds a way to become a huge pain. It's easy to wind up with multiple objects representing the same underlying DB row, but updating one in memory won't update the others, so you update them all from DB just to be safe... You might say "find a better ORM" but I'm pretty sure they all run into this problem at some point. IMO you're better off doing it yourself and making it explicit.
Also, ORM caching always finds a way to become a huge pain. It's easy to wind up with multiple objects representing the same underlying DB row, but updating one in memory won't update the others, so you update them all from DB just to be safe... You might say "find a better ORM" but I'm pretty sure they all run into this problem at some point. IMO you're better off doing it yourself and making it explicit.