For me it's not about wall clock quicker, but perceived quicker. I use a couple of different keyboards throughout the day, and they all have the arrows in slightly different places (more variation than letter keys...) which makes it similar to the mouse...
Namely, the amount of concentration required to hit an arrow or get the mouse to do what I want feels slower because I have to pause my thoughts, rather than just queue up a nerve message to my fingers to "then go over there and...". Stopping the train of thought to manipulate the mouse - which requires a lot more visual feedback and focus than just knowing that the cursor will get there - makes me overall slower because I'm context switching.
Namely, the amount of concentration required to hit an arrow or get the mouse to do what I want feels slower because I have to pause my thoughts, rather than just queue up a nerve message to my fingers to "then go over there and...". Stopping the train of thought to manipulate the mouse - which requires a lot more visual feedback and focus than just knowing that the cursor will get there - makes me overall slower because I'm context switching.