It's called fair use but people have this idea that Robots.txt was a bill passed by Congress. But yes, I totally agree, and sites like Stack Overflow are doing this very well. It's a shame that Archive.org was not allowed to crawl my profile before I left last year but that they allowed Google/Bing/etc to crawl it.
