The Pile
The Pile was one of the first serious attempts to build a diverse open pretraining corpus. Rather than relying on web crawl alone, EleutherAI combined 22 sources into one dataset, including PubMed and arXiv papers, books, GitHub code, Stack Exchange, subtitles, and filtered web text. The idea was that a mix of domains produces more capable and better-rounded models than web data on its own.
It totals about 825 GB and is distributed as JSON Lines, where each line is one document with metadata marking which sub-dataset it came from. That tagging makes it easy to include or exclude specific domains for your own use.
It trained models such as GPT-Neo, GPT-J, and GPT-NeoX, and remains a common research baseline. Licensing is mixed: MIT covers the compilation, but individual subsets keep their own licences, so check terms before redistributing.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
DCLM-Baseline (DataComp-LM)
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
Dolma
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.