If you publish to make information known, then your incentive is that it helps spread that information to people who may not otherwise visit your site. If you are trying to make money off of search engine traffic then you might not like this much at all. I think most people would rather not be pointed to those sites in the first place, so it’s a win-win if they block crawlers.
This is a bad take. I don't make money off search engine traffic (beyond the occasional donated dollar), and yet I would quite rather[0] that AI doesn't visit my site.
Imagine for a second a world where instead of publishing directly to your own website (home), you publish into the LLM's knowledge base. Nobody comes to you for information, they come to the LLM. Nobody cares about you, or the work you put in. Your labour is just a means to their end. To some extent, you could argue that Wikipedia works in this way, but it really doesn't. The work you put in is reproduced verbatim, and the work is collaborative. You get the joy of seeing your writing being used to help other people in a very direct sort of way, as opposed to being aggregated, misinterpreted and generally warped by a non-intelligent LLM.
In other words, you cannot possibly expect others to want to work in a sweat shop, toiling away to provide you with instant gratification. We must leave room for human expression.
Okay, I’ve imagined the world you described. I don’t care for it. I also don’t think that’s a likely outcome of LLMs. Why would somebody continue “toiling away to provide you with instant gratification” if there’s nothing in it for them?
