Really I think the point is to know what you are abstracting for and not simply dogmatism abstraction for the sake of abstraction or "purity".
If you expect to have a system which handles the low level details that is fine. If you don't want it to possible to mess up state consistency at the cost of possible performance fine.
However, if you expect abstraction to mean latency doesn't vary at all regardless of if it is on chip cache, RAM, hard drive you are going to have a bad time unless you are dealing with a library that specifically tries to mask timing attacks.
If you expect to have a system which handles the low level details that is fine. If you don't want it to possible to mess up state consistency at the cost of possible performance fine.
However, if you expect abstraction to mean latency doesn't vary at all regardless of if it is on chip cache, RAM, hard drive you are going to have a bad time unless you are dealing with a library that specifically tries to mask timing attacks.