A prestigious school like Harvard has all the best professors, all the best resources (libraries, computer labs, machine shops, laboratories), and all the best opportunities (more research positions, more interesting academic work to be done), etc. A small school just can't afford all that stuff.
So you're right that a good school is much more than just the things you learn in class. But to boil that down to some "all the elites go here just to meet the other elites" is closer to a conspiracy theory than the truth.
The undergrads don't see the best professors. The best professors there, as everywhere, often do what they can to evade teaching. Many of the real-world resources are, of course, unavailable online, (computers being a likely exception).
A prestigious school like Harvard has all the best professors, all the best resources (libraries, computer labs, machine shops, laboratories), and all the best opportunities (more research positions, more interesting academic work to be done), etc. A small school just can't afford all that stuff.
So you're right that a good school is much more than just the things you learn in class. But to boil that down to some "all the elites go here just to meet the other elites" is closer to a conspiracy theory than the truth.