What happens when you lose connection mid-maneuver, or there's congestion on the network causing high latency?
The vehicle would have to be smart enough to execute all the tactical parts of driving anyway. Adding remote piloting just adds more failure cases and complexity.
I'm not talking about remote driving all the time - I'm talking about recovering from weird edge-cases, like the car gets confused by a traffic cone on a flat bed truck and stops and needs to be reversed 3 feet and pointed in a slightly different direction to get it running again.
The vehicle would have to be smart enough to execute all the tactical parts of driving anyway. Adding remote piloting just adds more failure cases and complexity.