No one is really getting rich off of the endowment though. It's not like the endowment pays billions in distributions to the board of trustees or faculty. Sure the guy that manages Harvard's $53.2B is paid $6 million a year (and appears to be worth every penny of it), but face it. If Harvard had no endowment, he'd be on wall street making at least as much anyway. It's not some scheme to enrich him or anyone.
I feel that if I could be 100% certain every dime went to pay for exceptionally talented students to study, full time, for free or to upgrade facilities, I would be less disgruntled.
I don't think that its possible to be even 90% certain of this with our current financial technologies