I'd agree that there are some investment opportunities that can "beat the market".
But typically, the average retail investor doesn't get access to them. There are some hedge funds or VC/PE firms that might have alpha, active management skills, the golden touch (or just such a reputation that they get the best deals), but they tend to be closed by now or require huge investments. If you're a Goldman partner, you'll come across some excellent investment opportunities.
Furthermore, as you highlight, just like it is difficult to evaluate single stocks ex ante, it is difficult to evaluate active managers ex ante. Thus, as their expected contribution is negative, I think the advice to most retail investors to go for cheap, passive funds is sound.
Cheap passive funds are good advice if you want to maximize your return in the most frequent outcomes, but it ignores some powerful psychological stuff that's real and in play.
An index fund is like a bus: everyone is going to the same place, and you're not driving. Many investors just can't deal with that, and may rationally and happily take bigger risks seeking excess returns, even if they know the odds are against them.
Most investors I know, even beginners, believe they can beat the market with minimal effort and essentially zero knowledge about it. The usual results are, however:
That's a good point, though I would think you can handle a lot of that with the macro asset allocation (shares, bonds, alternative; domestic, international; etc.), which is still something one needs to do and look at every year or so.
What are you talking about? That gives you opportunity to gamble and speculate, not to invest.
Of course, if many people gamble, some will win, and they might post somewhere about it, but it is foolish to suggest that you can beat the market by significant amounts as a retail investor with options and margins without a commensurate increase in risk (and, because the options and margins are not provided by your friendly investment banker for free, with an increase in cost reducing your expected risk adjusted return).
Gambling, you bet (might win or lose) on something very random (dice, roulette wheel).
Speculating, you bet (might win or lose) on something "economic", "business"-like.
Those two terms have a connotation of imprudence (at least for me): You might win, but that's because you're lucky, not because you're smart. And you might lose.
If you invest, you try to take luck out of it as much as possible, for example, by investing in something that you have a very deep understanding of, or investing (unleveraged) very broadly in the market.
It's basically a continuum, with investing on the lower return, lower variance, less luck, more prudent end of things, and speculation at the other end.
Sorry if this isn't the best article on the topic, it was a quick google search.
My friend got bit hard by this. He had a HFT algorithm that was essentially guaranteed to at least break even by providing liquidity in the market by doing some fancy tricks that he's used for embedded digital signal processing. Problem was, his bids weren't being fulfilled even though he'd made them hours, sometimes days, previously. Given the amount of information available to him it looked like his orders were first in line at a given strike price to have his orders filled once the price moved to him. The only thing he could figure out was that someone was able to submit subpenny orders that would fill before his.
I was skeptical initially, then some weeks later it started making the news that dark pools were managing to make subpenny bids on various digital exchanges. Far as I know the SEC turned a blind eye since the dark pools already had special privelidge in exchange for being market makers and liquidity providers.
But typically, the average retail investor doesn't get access to them. There are some hedge funds or VC/PE firms that might have alpha, active management skills, the golden touch (or just such a reputation that they get the best deals), but they tend to be closed by now or require huge investments. If you're a Goldman partner, you'll come across some excellent investment opportunities.
Furthermore, as you highlight, just like it is difficult to evaluate single stocks ex ante, it is difficult to evaluate active managers ex ante. Thus, as their expected contribution is negative, I think the advice to most retail investors to go for cheap, passive funds is sound.