I'd say the reasoning is that you then have to trust there are no privesc/bypass opportunities in your environment. Trusting that all your dbus/pulseaudio/network-manager/cups/fuse/display manager & friends aren't going to give your rogue chrome process on one account some kind of access to another (thanks to X11/XDMCP, they'll at least have keylogging) - that's a big surface area, aas in: space is big. Really big. You just won't believe how vastly, hugely, mind-bogglingly big it is.
Compared to the few hundred lines in the hypervisor providing VM-level isolation you'd be a bit mad to say that these are equivalent means of isolation.
Compared to the few hundred lines in the hypervisor providing VM-level isolation you'd be a bit mad to say that these are equivalent means of isolation.