On: Addressing Recent Claims of “Manipulated” Blog Pos...

[icebraining]

As per that post, they only ignored robots.txt for .gov and .mil sites.

[aero-002 → icebraining]

IA disallow in robots.txt will still block archive.org, the blog post was about ignoring parts that were meant for search engines.

[klez → icebraining]

Yes, but it also says

> We are now looking to do this more broadly.

That's the part I'm asking about.

[icebraining → klez]

Right, but it doesn't mean they reverted it, they are probably still looking into it.