Common Crawl
Common Crawl is one of the foundational open data projects for AI. Since 2007 it has crawled the web and made the results freely available. Each monthly snapshot contains raw page data, extracted plain text, and metadata, published in three formats: WARC (the full archived response), WET (extracted text), and WAT (metadata).
The data is hosted on AWS as a public dataset, so the cheapest way to work with it is from inside AWS, where there is no transfer cost. It forms the base layer of most major open pretraining corpora, including C4, FineWeb, RedPajama, and Dolma, and has been cited in thousands of academic studies.
Related sources
C4 (Colossal Clean Crawled Corpus)
A 750 GB English corpus derived from Common Crawl using heuristic cleaning to keep natural language and drop gibberish, boilerplate, and placeholder text. Built to train Google's T5 and later used for MPT-7B and others.
DCLM-Baseline (DataComp-LM)
A filtered English web dataset from the DataComp-LM benchmark project, produced by running model-based quality filtering over Common Crawl. Built to show which data-curation choices most improve language model training.
Dolma
A 3T token open corpus from Allen AI (its name stands for "Data for Open Language Models' Appetite") combining web text, scientific papers, code, public-domain books, Reddit posts, and Wikipedia. Built to train the OLMo models.
FineWeb / FineWeb-Edu
A 15T token English web corpus distilled from Common Crawl by HuggingFace, filtered and deduplicated for language model training. FineWeb-Edu is a subset filtered for educational content. One of the higher-quality open web datasets for pretraining and broad-coverage RAG.