I could be wrong, but I believe Shodan actually portscans the entire internet, whereas Google only crawls known URLs. They also index HTTP headers, which Google doesn't do.
No! I do NOT try to authenticate with username/ password! The only exception to that is for FTP, where I try to do an anonymous/anonymous connection (identical to what Firefox etc. do). I put a lot of effort into making the crawling as benign and unobtrusive as possible, so I definitely do NOT try to brute force devices.
Is that legal? I've seen all kinds of analogies like "if your neighbor leaves the front door unlocked..." or "but if you go down the street testing each lock..." but never anyone who really knew what actual criminal law says.
It is a grey area, at least in the US. The main federal law for computer crimes is the ancient Computer Fraud and Abuse Act. The provisions of the act state all work off the concept of "exceeding authorized access" - but the law never defines what authorized access actually is. Logging in with a default username and password has never been tested in court, as far as I know, and I think there are arguments to be made for both sides about whether that counts as authorized access.
I'm not an expert in it by all means, but from what I've seen it is like having Google log all the http headers and servers connected to requested as well. This means that it is incredibly easy to, for example, track down certain servers with a certain exploit that you know about [1], or complete systems that shouldn't really be attached to the internet in their current state [2]. Not sure either of those are possible with Google.
A "bad" search engine should treat robots.txt pretty much in reverse: Anything disallowed should go to the top of the list of things to index.. There are sites out there that uses robots.txt rules to prevent Google from indexing things that should be password protected but isn't...
The irony is that robots.txt doesn't even prevent things from being indexed. The files can still be indexed if there's a link to them on the Internet; that's what <meta noindex> is for. (Which, ironically, requires that the page not be robotted, because if it is it can't be crawled, which means the meta tag can't be discovered.)
it was an honest question and i don't think your reaction is appropriate behaviour for this site.
If most think my comment is worthless, the voting system will make it enter the void. If others think it isn't it will be upvoted. That's how this site works.
Just responding with "Shut up" adds nothing to the discussion and is something i am shocked to see on this site :(
I used to be able to tell people like you to go back to Reddit. Unfortunately the quality of HN has declined far enough that your content-free insulting of a decent question is not immediately recognizable as something with no place here.
I consider that fact a sad commentary on how far HN has fallen.
Responding to trolls probably does more to decrease the quality of discourse than the original troll does, since those typically get voted into oblivion relatively quickly.
Tossing in a belittling jab at another website to boot doesn't help either.
I'm a relatively new HN reader (~1 year) and have taken much away from my time here (much reading, few comments). I understand where you're coming from with concerns about quality; however, I resent the fact that I may be considered part of the increased readership responsible for "HN's decline"
Hopefully knowing my fuller opinion will decrease your resentment.
I suspect that new users follow something like an 90/10 rule. 90% of them are good to have around, and contribute more than they detract. New blood is good. But 10% contribute junk that is like virtual cholesterol, it builds up, clogs the system, and if left untreated eventually will be lethal to the community. of that 10%, perhaps 10% are simply toxic waste that you want to get rid of, and 90% just need encouragement to fit in better.
I sometimes comment on egregious comments by the 10%. You've reminded me that I should more often acknowledge the existence of the 90%, and on my hope that the 10% I'm looking at are part of the redeemable 9%.
There is an observation that goes back centuries, which applies here. The observation is that if you pack a barrel of apples and there is even a single bad one, the whole barrel will spoil. But if every apple is good, the barrel will remain good for the entire winter. Thus, "don't let a few bad apples spoil the barrel". We want the apples, but none of the bad ones.
Unfortunately the advent of refrigeration has caused us to forget the original wisdom and the saying is currently used as the exact reverse of its original meaning ("oh, it was just a few bad apples").
During packing we were still checking all the pears for injuries ("stem punch", caused by other pears), since apparently it would spoil the whole box if there was a bad one. (Packing pears in New Zealand for export to Europe and the US) - On a commercial and longterm scale you apparently still have to take care.
Rotting fruits release ethylene gas which is a ripening agent. This causes fruit next to rotting fruit to ripen then rot.
An interesting mechanism I think. Before looking that up I would have suspected a biological transmission of infection or something, not a chemical transmission.
That's how all communities work though. As a community grows and attracts new members, the old guard moan about how it was better when they were noobs.
In fact this is true for real -"offline"- life as well.
meaning that with a specific search in google i can find for example all kinds of cameras or systems one shouldn't find, e.g.:
-) http://preview.tinyurl.com/34959u
Maybe Shodan "focuses" on that, but they can't possible index more of those things than Google already has...
Can you find one single thing over Shodan you can't with a specific Google search? (maybe you find such things more easily with Shodan...)
EDIT: More information on Shodan:
-) Defcon Presentation [pdf]
https://www.defcon.org/images/defcon-18/dc-18-presentations/...
-) Secanalysis.com
http://secanalysis.com/a-brief-analysis-of-shodan/
JoshuaRedmond beneath also provided interesting links