> DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (our crawler) and crowd-sourced sites (like Wikipedia, stored in our answer indexes). We also of course have more traditional links in the search results, which we also source from a variety of partners, including Oath (formerly Yahoo) and Bing.
How does that work with their privacy stance? Do Yahoo/Bing get to keep and use that search data and it's just anonymized, or does DDG pay to keep it untracked?
Kind of disheartening regardless. I assumed they had their own scrappy, independent tech stack.
Headless Chrome is available and widely-used, and commonly you can get around the JS thing by simply waiting a few seconds before scraping. I'd assume the crawling itself isn't the hard part (aside from maybe just the raw compute time it takes).
For the most part yes. They could be getting search results from other paid search engine APIs but you have to balance cost of providing results with ad/affiliate revenue.
You can get paid API search results from Google and Yandex for example just like with Bing (similar prices, different limits). And you can even use Wolfram Alpha API for certain types of queries ("what is apple's average revenue per employee?").
Doing all these would allow you to surface better results than using just one. But it comes at a cost.
Google and Bing are the only ones that matter and you can’t compete with Google by paying them for their search results.
Yandex is Russian and has pretty poor results. And you’d be naive to think that those of us concerned about privacy would ever touch something built on top of it.
In other words there aren’t paid search engines that DuckDuckGo could turn to. Unless they build their own crawler, the only game in town is Bing.
There is no difference in terms of building something on top of Bing or Yandex as your private data never touches their servers. All they get is anonymized stream of queries, in this case from DuckDuckGo.
And yes you can compete with Google (for a certain target group) by paying them for their search results. Results are just a distribution channel, it is what you do with them that matters. For example Google and DuckDuckGo both choose to show you ads and affiliate links but that is hardly the only option.
They're desperate to collect user click data because they know that's the only way they'll have any chance of success. Even anonymized, that's very valuable data.
