Most of those high-quality sites that contain detailed technical information have something in common: they don't carry Google's advertising, so Google earns no revenue when you visit them. Despite Google's protestations over the last few years, I haven't heard a better explanation for the shift in the nature of the results they return.
In reality, the issue is simply that the sites that contain in-depth writings don't get updated as often as the blogs, forums, and content farms that contain superficial information. Google's engines are tweaked to always prefer sites that are updated very regularly, so as to sift out obsolete information.
Google is a tech company, tech is seemingly outdated as soon as it arrives. They push the tech line of thinking everywhere they go, even if it doesn't belong.
Well to be fair, trying to differentiate between information which does become outdated and information which is static is non-trivial to do algorithmically. Choosing between promoting new and updated information and promoting static information I think the former is a better choice for most things.
I quite clearly remember seeing Google ads on many of these sites, as they were unintrusive enough that I didn't block them. My theory is still "not SEO'd enough", or perhaps due to the huge amount of text and links that they tended to contain relative to styling elements, they appeared linkfarm-ish enough to Google's algorithms to get penalised.
Or SEO'd at all. It's easy, and maybe appropriate, to hate on SEO, in part because it works. If you have a bunch of people pouring money, thought, and energy into optimizing crap content, and many people producing high-quality /niche/non-commercial content giving it little to no thought, it's not surprising that the crap floats to the top. It doesn't require malfeasance on Google's part.
The other thing to consider is that Google has to optimize for the general case. If their mission was The Best Physics Search Engine or The Best Academic Search Engine, they might do a better job with more esoteric material. But it's meant for everyone, and most people want less detailed, more digestible content.
I have yet to see any evidence for this and it's a wild accusation that I'm seeing a lot on HN without merits. Occam's razor applies here as well, with the simplest explanation being that search is a difficult problem to solve. And I've written here before that for non-US users at least, Google's results are vastly better than everything else, so it isn't that competition doesn't exist, but hey, you're free to try and solve this problem in a better way.
Evidence would be difficult to offer given that their algorithm is a changing black box.
I also don't see how "search is hard, which is why Google's search results have issues," is a simpler explanation than "Google has some bias towards results that they profit from."
Why not both? Search is really easy if you have a corpus of static information. Search is very hard when you have a huge amount of information that changes, many people are looking for the newest information, and outside influences want to bias search in their favor for profit too.
