At some point, we're going to cut the branding middle-man, and put Bing on the front page of HN as a viable alternative to Google Search.
Not as sexy as the high-minded front organizations that reskin it, but it'd be nice if we could be honest with ourselves about the beasts we were feeding.
Is that meant to imply that the privacy protection and un-bubbling of the "middle men" is bogus? (I don't know about Qwant, but DDG isn't solely Bing, and it feeds some worthwhile organizations with donations.)
I think DDG is more of a legitimate competitor in the search space than other engines running off the Bing index. My understanding is that they've been pretty transparent about using Bing and Yandex as sources for its product, have done quite a bit of work to tailor the results they spit back, and maintain their own knowledge graph for features like instant answers. They seem to function more as a metasearch engine.
From ehat I've seen, search engines like Ecosia and Qwant aren't very clear about what they bring to the table, beyond serving as a proxy for Bing that doesn't store as much data on you.
I don't think there's anything explicitly wrong with that. But I don't think they really merit anybody's attention either. Especially since they don't explicitly advertise what they are.
Ecosia explicitly specifies "Results from Microsoft" on every results page, so they seem pretty transparent there. I don't think DuckDuckGo and Qwant are as upfront about that.
Why are so many search engines trying to piggy back off other search engines? Can they not crawl sites themselves and cut out the extra layer? In a sea of bad websites I feel like if you started with a whitelist approach it could be a good basis.
I’ve worked in this space for over two years. They can’t. It’s simply not possible. The only feature that’s feasible to offer is privacy. You’ll never be able to catch up to Google or Bing with crawling.
In your opinion, do you think a new original search engine can survive in this landscape? Of course it would always be behind Google and Bing by about 15 years.
It's a pretty herculean task, moreso no than ever before. On top of the sheer volume of the internet, and the technical challenges of performant search, many sites are hostile to unknown scrapers, and automation.
I just did a search on our business and the results were not the same, just like DDG and Bing isn't the same. From best to worst result on the same search string:
Google >
Startpage >
Qwant >
Bing >
DDG
But even if they were 100% identical, it has some privacy unlike 3 out of the other 4.
If the name is quite unique it's a good indicator because some site with high pagerank and lot of seo that list other websites/companies/software are sometime higher than the original site (stackshare.io or alternativeto.net for example)
I use Bing (and Edge, and Windows 10...) but for the average HN reader that's not cool enough. Tech is more and more about personal branding and following the hype train. DDG is in. Bing is not.
Also, for the older crowd who came from slashdot (and still says 'Micro$oft') it'd be morally unacceptable to use Bing directly. They prefer either a rebrand, or to directly give all their data to Google (maybe because they still see it as fresh and new?)
DDG isn't just a rebrand of Bing (and wouldn't be even if they did all searches via Bing); they don't pass personal information along to Bing that would allow Microsoft to track the user.
We don’t know that. I mean all they have to show for it is a privacy policy and a few articles saying “we value privacy”. If they truly cared about privacy then why wouldn’t they be open source?
I feel DDG can be compared to Apple in many ways, but here’s it’s important to look at what Apple is doing to its Chinese users by letting the CCP spy on all their iCloud data (sacrifice all the privacy values to obtain access to China’s market). Perhaps DuckDuckGo have also been forced to sacrifice their users’ privacy to join Microsoft Advertising as well? It’s difficult to tell since they’re closed source, and Microsoft Advertising is a private ad network (anyone besides DDG/Ecosia that’s been invited?) that doesn’t make any API documentation publicly available. At least with Bing’s API we can tell that Microsoft just need the search query for it to work, but how about their Microsoft’s advertising API?
Qwant, i.e., lite.qwant.com, prefixes search result URLs to point to Qwant servers. Qwant redirects www.qwant.com to lite.qwant.com when Javascript is disabled.
Bing does not prefix search results.
Bing requires sign-up in order to use their API.
Qwant's undocumented API is freely accessible, no sign-up.
Qwant, i.e., lite.qwant.com, requires a User-Agent header.
Bing does not require a UA header.
Speaking as someone who ran operations in a search company that had an api for accessing their search results, that api will get slammed. But not in a good way.
Crooks use search engines to find pages that have exploitable js code or sql injection vulnerabilities. They use them to find unprotected comment sections so they can inject spam into them. They use them to build dossiers on people by scraping public information sites. And all that API use, they never pay a dime for access. Nor will they reveal who they are in order to get access. It just isn't how they operate.
At its peak I had over 2.6 MILLION internet hosts black listed from using the Blekko API. Exactly zero reached out to any of the easily found contact addresses and said "Hey your API seems to be unresponsive" :-)
Query per second or qps limits is a good start, but I quickly learned that a 1qps limit to "free" access meant that all of a sudden it was a 1000 unique IPs that were making one request each.
Lots of botnets were highlighted this way, when one address searches for 'joomla v2.3', and the next IP searches for 'joomla v2.3', page=2, and the next IP searches for 'joomla v2.3', page=3, etc.
It was annoying but an interesting problem. We could implement any arbitrary policy and then watch as the bots adjusted to come in just at that policy limit. We banned entire Ukrainian ISPs (they were a big source at the time) and have VPN providers become the big users. We put in limits per day, I tried a "thermal" system where IPs gained "heat" by queries and "cooled" by idle time. We built a server with a "broken" IP stack that we could send the initial TCP connect to, the server would accept the connection and then never respond. A "black hole" if you will. The trick was we didn't actually keep[ sockets open we just pretended like we it was the other end of a TCP connection. It did everything correctly except complete the connection. That would cause any client using off the shelf IP stacks to hang indefinitely.
This was posted in another thread, but I'd never seen it before. French-made.
Results seem just as good as Google, PutLocker searchers work, and it's as privacy-respecting as DDG. I've been using it all day and I'm happy with the results, so I thought I'd share.
It's a bit more than just hype: it's rebranding Bing results to make them more palatable to a EU audience, while guzzling on public subsidies to pretend it's a EU unicorn and gather support from nationalists.
There are actual results - as a public subsidies scam!
>while guzzling on public subsidies to pretend it's a EU unicorn and gather support from nationalists.
This seems like a weird accusation given that government VC arms like In-Q-Tel or institutions like DARPA invest heavily in Silicon Valley tech firms, not to mention that the national science foundation supported the development of pagerank itself[1]. One of Apple's first investors was the Small Business Investment Company, an investment arm of the federal government. The EU would be stupid to not support the development of an independent technology sector.
I self-host SearX, and so can you. Its fairly easy with Docker [1]. However via self-hosting you don't get the anonymity benefits from their upstream (Bing, Yandex, whatever else you enable) which you do get via a public SearX. But with a public SearX instance you don't get the privacy benefits from that instance (ie. all searches can be traced to you).
So I end up using both public and self-hosted SearX, DDG, and Google as last resort.
Maybe this is naive, but I don't understand what Google's impenetrable moat is around search.
It's a hard technical problem, but not one with a concrete barrier to entry. And there are lots of really smart people solving really hard technical problems out in the open. On top of that, much of the recent progress in AI is open-source. Surely that could help?
I guess indexing at that scale takes a lot of hardware. This is the most plausible barrier that I can see, but it still doesn't seem insurmountable.
What if you rethought what a search engine is? Does it really have to cover all of the text on every page on the internet? Could it be more focused?
This just seems like a solvable problem once there are enough smart, motivated people. Which there appear to be.
Indexing isn't the hard part. Ranking is. For any given query, there are probably millions of web pages that match. How do you pick the best ones to show the user? How do you do that even when most people are really bad at expressing what they want?
Even if all websites were acting in good faith, this would be a really hard problem. Now add in the fact that there's an $80 billion industry devoted to gaming search results, and it suddenly becomes much harder.
Would it be impossible to build a search engine as good as Google? No. But you could very well spend billions of dollars to match or even slightly surpass Google's performance and still lose. Why? Because unless it's clear and obvious that you're superior to Google, people are still going to use Google because that's what they're used to.
That's why the smart players in the space like DDG aren't competing head-to-head in search, and are instead focusing on areas, like privacy, where Google can't compete. As for the others, I suggest you try Googling "cuil" some time.
Then Google’s ranking algo must be pretty bad because all I have is a first page full of bullshit Advertorial articles when I’m looking for serious stuff.
(Maybe that’s what the average users are looking for, though.)
And of course there are lots of other issues. For example the Web is very hostile when you are not Google Bot (i.e. there are a lot of big sites that will forbid you from crawling their content, unless you are Google, or Bing).
The money mentioned in that article is about being the default search engine, not being a functional one. Basically the question I'm asking is why DDG has to outsource to Bing.
> there are a lot of big sites that will forbid you from crawling their content, unless you are Google, or Bing
If you're talking about robots.txt, that's just a suggestion. It doesn't hold any actual preventative power.
IIRC that thought process was where DDG started some years back now. It was developed in the open wrt HN, by a then HN regular.
My recollection is they started by doing their own crawling.
(I want to say the username was Weinberg?? But honestly can't recall if that's right.)
TBH it's probably also the thought that Google started on, as principally we had manual indexes back then.
Again IIRC, DDG got along way with private indexing and then realised it didn't really gain them much and they instead wanted to focus on being a Google competitor. They did that by focusing on instant answers as a distributing feature and by buying results to get closer to the coverage they needed.
Lucky for them Google made their search a lot worse so DDG's quality caught up.
Yeah I saw that on the front page a few months back, and people seemed impressed with it. Although, I also remember seeing Qwant on the front page, or at least in the comments of another search engine post and people were impressed with it. Seems the tide has turned!
What runnaroo seems to do that is working for me is to somehow prioritize programmer type hits. Maybe it's just the stack overflow source. The "skinned" Bing sites seems to localize too hard and prioritize shopping and big-news type hits.
What I don't really like about Runnaroo is that it's hard to pin down who's behind it. It's seems to be just the one guy and I can relate to his need for privacy but on the other hand I can't 100% trust the "We don't save or sell your data"-claim when there is no one to hold accountable.
Looks like an interesting project! It would still need serious audits since it's all browser-based (arbitrary JS could be added at any time and snoop on the results page) and there's a black box element at the point of retrieving results: we're hoping that the search provider is only sending encrypted results back, but I don't see how we have a guarantee that this is the case and that the query and results aren't being logged in various places and cross-referenceable. Basically, we receive an encrypted result, but where is the non-black-box, non-trust-based guarantee that this is the only result seen?
It's not the job of each individual application to do everything. If you want privacy of your web activities, use Tor, which will work for every website. That's the best you can, even though it is not perfect.
I use the putlocker test to determine if/when a search engine ceases to provide useful results. If a search for putlocker only yields .com domains and the like, it no is longer prioritizing the most useful results.
Google long ago stopped returning useful results for my academic research. For example, it never returns Wikipedia articles unless I explicitly use Wikipedia as a keyword, which I now have to do for most of my searchers.
I used it for a long time (and 2 years as my main search engine) for 3 reasons...
- it is french
- it is working in Europa/USA/China (I studied in Paris/San Francisco/Tianjin)
- my best friend used to works their.
We had some good talks about Qwant, but it will not be relevant for my user review.
So here we are:
1. They are not Open-Source (as opposite to Searx).
Only the plugins are.
Can we trust them when they say to respect our privacy ?
2. They are not TOR friendly.
Full of captchaS !!!
3. They have no onion site.
DDG have one.
4. They run some kind of weird analytics.
Each time we click on a search, the JS code trigger a fetch to `https://api.qwant.com/api/action/url` and include:
- our current language
- the query we searched
- the link we clicked
- etc...
They were backed by Bing before, is it still the case or are they running their own stats engine? I do not know.
If we trigger ourself some false fetches, can we show "twitter" as first result when someone searches for "facebook"?
5. Their lite version is not lite (=/= duckduckgo.com/html/).
In sum up, in a customer point of view, Qwant is just a frenchy Google hosted mainly in Europa and allowed in China. Nothing new here, I recommend to stick to DDG for the moment.
Edit: fix minor typo + add proof for QwantLite analytics
* Their home page boasts to be "The only search engine that respects your privacy.". DDG has been there for much longer. I could not trust them after such obviously false claims.
* They promised (to the public and their investors, which includes the state funds) that if they used Bing, they were improving they own engine which was handling more and more queries. Studied showed it was not the case, that many searches showed outdated results and repeating entries (to make think they have many search results).
* Their previous boss was a tyrannic Jobs wannabe. He wanted Qwant to become as trendy as Google so he kept launching half-baked products (Qwant maps! Qwant mail! etc). Constant chance of priorities and new projects were devastating for developers (especially when the search engine was still very weak). He was finally pushed out of the company.
Google doesn't work for me anyway because Google blocks VPNs (interferes with its spying I guess), but latest Firefox has a nice feature where you can conveniently select where your search goes to. So now 50% of my searches go directly to wikipedia without any middle-man, russian queries go to yandex, and the rest go to qwant, and if this doesn't work - to startpage, which feeds from google.
I've been trying to get search results for months now, and Qwant just fails everytime with a white screen of death. See screenshot: https://imgur.com/a/jUv6vm0
I've even tried with different IPs / VPN connections and it still refuses to show results.
Surely I'm not the only user who experiences this?
But why does one have to resort to hacks like this? I want the English version to work. Obviously Qwant have to iron this out, but I've waited months and they haven't even fixed it. It's a huge bug. It's like they just abandoned the project.
Oh I definitely agree with you, I just found that workaround above because I was really frustrated as I don't see any advantage to having javascript enabled on my searches and was used to not needing it in Google, Duckduckgo, Ecosia, and other search engines...
I've been generally happy with startpage.com, they seem to mirror Google's results. And it works with dark mode. You don't get the contextual niceties that you get with Google though e.g. results for "70F in C".
As I posted on another comment I think that happens because you have disabled Javascript and there seems to be a bug for English.
If you use JS (as done for your Google search) it returns results as expected.
Qwant is Springer. Springer is the company that tries to blackmail google and all others by lobbying into EU law (article 17, former 13) fighting a free and open internet. I don't know which is worse. Springer is the german Apple in terms of privacy.
In our (US) current political climate I can't help but cringe a bit at the prominent use of the letter Q and the claim of privacy.
I doubt there is a relation with the conspiracy theory group, but the marketing seems a bit awkward.
I think Qwant predates the Q-phenomenon becoming mainstream by several years. Other than that, I agree with the other answer – Q is a perfectly good letter and we shouldn't let them steal it from us.
Not as sexy as the high-minded front organizations that reskin it, but it'd be nice if we could be honest with ourselves about the beasts we were feeding.