Yeah, the Pile is approximately the size of the GPT-3 training data, which is no...

leogao on Jan 2, 2021 | parent | context | favorite | on: The Pile: An 800GB Dataset of Diverse Text for Lan...

Yeah, the Pile is approximately the size of the GPT-3 training data, which is not a coincidence--one major reason we created the Pile (though certainly not the only one) was for our GPT-3 replication project.