Well, historically it's been the rich and landed though that's not a part I'd advocate for keeping.
I actually think a lot of the problems with total direct democracy come not purely from hot tempered reactions of the majority (though that certainly is a problem at times), but from the mere fact that a lot of governing decisions need to get made by someone who really studies the details of a problem closely and makes an informed decision.
Let's say as a society we're making a complicated financial decision (like, is it in our long-term financial interest to build a bridge linking Point A and Point B). That's a very complex decision, where it's extremely hard to pin down with specifics both the costs and the eventual benefits of the bridge.
So, as a society we put together a group that studies the issue in depth and puts together a 100-page report that's the distillation of their research. It nails down the costs and the benefits in about as concise a form as it can be, while still capturing the full complexity of the project.
Now here's the question: what's the process for deciding how to build the bridge? Do we:
- ask everyone in society to read and digest the 100 page report, make their own informed decision, and then vote on the outcome?
- Or do we as ask everyone to select the person they trust to read and digest the report, make an informed decision that would align with their interests, and then we vote for the person we want to make the decision?
I think there's some benefits to each approach that aren't captured by the other. In the former approach there's a lot of redundant work that we're asking from every member of society. It's much easier on each member of the democracy to make an informed decision about selecting a single person to make a decision on their behalf than it is to make an informed decision on the project directly.
Multiply this across every decision at the city, county, state, and federal government and suddenly the decision burden on the individual becomes a really good argument on its own for representative democracy!
I have a solution for this: immediately revocable vote vouchers. You pick your rep from the pool of all people willing to be a representative, and they vote on your behalf until they make a decision you don't like, at which moment you are free to revoke their control of your vote and give that to someone else. We could compensate representatives based on their rolling average of people represented.
You decide who represents you, directly (rather than pooling geographically, which is patently stupid) and representatives are operating with consistent incentive to act in the interest of their direct constituents.
You pick the tool for the job. A planner to run something relating to planning. An engineer running something related to engineering. A scientist to run something relating to science. Domain experts, who have spent the years needed to actually understand the nuances of the problem. Too often we just get the loudest mouth in the room and their gang of friends looking to make their buck.