It can be pretty similar for ICs too though. Even if everyone came into the office, there's almost no one I regularly work with who is in the office I would go into.
> there's almost no one I regularly work with who is in the office I would go into
Then you already fundamentally have a remote job. A non-remote job would organize business units / teams by location so that you're in the same office as people you're working with. And yes, this would require relocations and imply a smaller hiring pool. That's the tradeoff of not having remote work, not just spending more time in the office. You're not comparing remote to non-remote work, you're just comparing two flavors of remote work.
At large companies, co-location seems to drift over time. My company has long been pretty remote-friendly and I expect most teams are pretty distributed at this point.
ADDED: You can force co-location but, as you say, it reduces company options as well as employee options. And may not even be very practical; I work extensively with other teams both in another state and in Europe.