On: AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt

[rossdavidh]

It's a radar gun/radar detector kind of situation. You can always change your strategies; countless top-level links that are shown on the menu in a way that no human sees them (due to font color or size or etc). Small number of pages with a few that go on forever, in a way that any human would stop reading but a bot may have trouble detecting. Real (for humans) text in images, with endless invisible text for bots. Etc.

I think it will probably make it harder for screen readers, unfortunately.

[dylan604 → rossdavidh]

> countless top-level links that are shown on the menu in a way that no human sees them (due to font color or size or etc).

wouldn't this lower their page rankings? that's the kind of shenanigans of the old days with meta key word stuff and what not

[smaudet → dylan604]

I'm not sure the old search engines are any good anymore, though, so we don't really care...

That's kind of the point. AI has just been more terrible news after terrible news.

It is NOT a good thing. Unless you know, you like being covered in oil and lit on fire...

[rossdavidh → dylan604]

If you've got your robots.txt set to tell search engines to go away, or at least don't look at those pages, then you probably aren't worried about that, I assume?

[dylan604 → rossdavidh]

there's a difference between an ai-bot and a search engine bot though