Sorry, they've crawled trillions of pages, and narrowed it down to an index of 1...

karpierz on May 11, 2023 | parent | context | favorite | on: Ask HN: What is new in algorithms and data structu...

Sorry, they've crawled trillions of pages, and narrowed it down to an index of 100s of billions. Conveniently, the link answers your question of "can you have PB sized indices?" to which we can clearly say, yes.