I might be talking from ignorance here, but if you put the iPhone chips in a different environment that included the same active cooling that desktop CPUs enjoy, would this still be true?
Modern X64 cores are super smart. Not just on _easy_ things, like predicting branches, and how data flows though instructions. But with caching, memory access, inter-core communication, cache snooping, and cache synchronization/invalidation.
The real issue for modern system is RAM access. Everything you CPU does between hitting RAM may as well be instant how expensive a RAM hit is.
X64 has a large cache hierarchy to make this rarer. A11/A10X/A10 less so.