> The interesting thing about robots.txt is that there really isn't a standard for it. This [0] is the closest thing to one and almost every modern website deviates from it.
Yeah, my first reaction to Google heading yet another standard was to cringe but this is one of the situations where I think it makes a lot of sense. They're dominant in the search industry and most other engines tend to take their cue so having them spearhead it seems like a good move.
That is changing, and was announced today: https://news.ycombinator.com/item?id=20326067