I agree with you, but the issue here is that in traditional phone architectures, the baseband processor could interrupt and monitor the execution of the main CPU (for efficiency purposes). So theoretically you couldn't be certain that your plaintext wasn't being exfiltrated. With a separate baseband, it's a lot harder (but possible) for a blackbox CPU to exfiltrate your info.
