At least in part this seems to be due to the profiles that Google has built. Whenever I try to use Google in anonymous mode, the results become noticeably worse.
Which, of course, means that it is yet another barrier to entry for any would-be competitors.
This could also be a perverse incentive reflected algorithmically (whether on purpose or not). Google has better data and makes more money when users are identified. So they have a vested interest in making users think that they will get worse results in Incognito mode.
Personalization is a double-edged sword. @pg once wrote about Google results becoming "what's true for you" rather than what's objectively true. And filter bubbles and subtle "personalization censorship" are also dangers. I think it's possible to have high quality results and privacy/anonymity, and it's not a binary choice. It's a challenge worth figuring out.
Totally agree that personalization is a double edged sword. Problem is getting people to actually want to change their personalized feeds/results. Most people have no clue what they’re missing and this issue has huge “Medium is the message” like implications.
Which, of course, means that it is yet another barrier to entry for any would-be competitors.