I quite clearly remember seeing Google ads on many of these sites, as they were unintrusive enough that I didn't block them. My theory is still "not SEO'd enough", or perhaps due to the huge amount of text and links that they tended to contain relative to styling elements, they appeared linkfarm-ish enough to Google's algorithms to get penalised.
Or SEO'd at all. It's easy, and maybe appropriate, to hate on SEO, in part because it works. If you have a bunch of people pouring money, thought, and energy into optimizing crap content, and many people producing high-quality /niche/non-commercial content giving it little to no thought, it's not surprising that the crap floats to the top. It doesn't require malfeasance on Google's part.
The other thing to consider is that Google has to optimize for the general case. If their mission was The Best Physics Search Engine or The Best Academic Search Engine, they might do a better job with more esoteric material. But it's meant for everyone, and most people want less detailed, more digestible content.
