Wild guess: The libvorbis, the "official" codec is using floating point numbers, hence the decreased performance on mobile devices that don't have native floating point support. There are integer-based codecs for MP3 (such as libmad) so it's not an issue with MP3.
I suppose the solution would be writing an integer-based codec for Vorbis as well.
There is an integer implimentation called tremor. I believe it is more to do with the fact that there is often a dedicated mp3 decoder in these portable devices, and it has to burn the cpu for decoding vorbis.
I suppose the solution would be writing an integer-based codec for Vorbis as well.