Hitting Escape in normal mode takes you out of editing the cell and into "notebook manipulation mode" instead. This is so counter to the way Vim normally works - Esc should leave you in normal mode no matter where you started - that I found it almost unusable until I realised I could just remap that binding. I made it Shift-Esc and am very happy with it now.
