C4 (Colossal Clean Crawled Corpus)
C4 was created for the T5 paper as a cleaned single-snapshot version of Common Crawl. The cleaning is rule-based: it removes pages without proper sentences, strips boilerplate and navigation text, drops lines that do not end in punctuation, and filters out offensive and placeholder content. What remains is fairly readable English prose.
It is a good middle ground between raw crawl data and heavily curated corpora: small enough to handle without a cluster, clean enough to use directly. The Allen Institute for AI (Allen AI) hosts it on HuggingFace as Parquet files, with language variants and a noisier `en.noclean` split alongside the standard one.
Access it by streaming with the `datasets` library or downloading individual shards. It remains a common baseline for pretraining experiments and a handy source of general English text for RAG.
Related sources
Common Crawl
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
DCLM-Baseline (DataComp-LM)
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
Dolma
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.
FineWeb / FineWeb-Edu
A 15T token English web corpus distilled from Common Crawl by HuggingFace, filtered and deduplicated for language model training. FineWeb-Edu is a subset filtered for educational content. One of the higher-quality open web datasets for pretraining and broad-coverage RAG.