On: Only Google is really allowed to crawl the web

[rezonant]

Will it be a flat fee, so that I, a lowly one-man crawler developer will not be able to afford it? Will it be that only Google can afford it, thus making their monopoly position even stronger?

Is there a Wikipedia crawling "welfare" program if I'm not a trillion dollar mega company?

[dannyw → rezonant]

Actually, yes! The current API is free. The new Enterprise API is paid.

[aabhay → rezonant]

Sure! Apply to become a crawler. And if you meet certain criteria and your crawlers don’t exceed a quota then have at it. The key is not to make it technically challenging, but to erect a legal barrier.