Almost every one of those books you find in the library is on the internet, the reason you're not finding them is payment.
Publishers don't put their books for free in on the net, they want to make a living. Libraries on the other hand, pay publishers for the books, but only have a limited number of copies. You can probably access most of these books behind a paywall, but paywalls filter out search engines like Google.
It is kind of silly comparing the modern internet to a library, because technologically, the Internet and computer wins by a large margin. Every book in my local library can be digitized and put on my computer, and I'll still have a huge amount of space available. My computer can make unlimited copies of that information. My computer can OCR the information and the content of the books can be searchable, something a physical copy library cannot do. I can share my entire library with everyone on the planet with an internet connection.
I'll repeat, this is not a problem with webcrawling. It's a problem with financial incentives.
Publishers don't put their books for free in on the net, they want to make a living. Libraries on the other hand, pay publishers for the books, but only have a limited number of copies. You can probably access most of these books behind a paywall, but paywalls filter out search engines like Google.
It is kind of silly comparing the modern internet to a library, because technologically, the Internet and computer wins by a large margin. Every book in my local library can be digitized and put on my computer, and I'll still have a huge amount of space available. My computer can make unlimited copies of that information. My computer can OCR the information and the content of the books can be searchable, something a physical copy library cannot do. I can share my entire library with everyone on the planet with an internet connection.
I'll repeat, this is not a problem with webcrawling. It's a problem with financial incentives.