You haven't taken into account car pooling. Driverless cars can do adhoc car pooling. I can even see things like local 4 seater driverless cars picking you up, and taking you to another 10 (or more) seater vehicle for the rest of your journey. (By definition the congestion is because a lot of people have overlapping routes.)
Additionally you are no longer to blame for your commute since the driverless system would be responsible for timing, in much the same way trains are to blame for timing now. I'd also expect the driverless system to have variable pricing - need to get to work by 9am instead 9.15am and it is $5 more out of your pocket. Ultimately your employer is paying that and they have an incentive to only do so if it is valuable.
It wouldn't be long before every car used in this way became like a small limousine. The efficiency of driverless cars would easily outweigh the cost of such a redesign.
Most of the reason for that (at least in my experience) isn't about some deep desire for privacy, but for the prosaic reason that it's a nuisance to arrange with someone else to meet at a certain time and place, to make sure that everyone in the car pool gets out of the house in the right 10 minute window, etc
Being able to just call a car to drive you to work, while an automatic system figures out the car sharing arrangements in real time, could remove a lot of the barriers.
Current cars are personalized which makes sense given who uses it and how much they cost. Sharing is annoying for logistical reasons and because you don't want people messing with your stuff.
Driverless cars have many possible solutions. A simple one is pricing - it costs $5 more if you refuse to share. (And another $5 more if you want massage seats.) It is possible to completely partition the seats from each other so each one is effectively its own cocoon where you can play your own music as loud as you want and are completely isolated from the other passengers.
Heck it is even possible like many current electrics to have a base chassis that includes the motors and batteries in the floor, and have pods that mount on top. You could have your own personalized pod you keep at home.
Additionally you are no longer to blame for your commute since the driverless system would be responsible for timing, in much the same way trains are to blame for timing now. I'd also expect the driverless system to have variable pricing - need to get to work by 9am instead 9.15am and it is $5 more out of your pocket. Ultimately your employer is paying that and they have an incentive to only do so if it is valuable.