Standardized Project Gutenberg Corpus (SPGC)
The Standardized Project Gutenberg Corpus takes the raw Project Gutenberg collection and turns it into something you can analyse straight away. It strips the boilerplate headers and footers, applies consistent formatting, and attaches clean metadata plus word and token counts (a token is roughly a word or word-piece) for each title.
The processing code lives on GitHub, and prebuilt versions of the corpus are published on Zenodo, an open research archive, so you can either regenerate it yourself or download a ready-made snapshot. This makes it a good fit for reproducible research and for building literary RAG systems without writing your own cleaning pipeline.
The book content stays in the public domain, while the processing code is released under the MIT licence.
Related sources
Google Books Ngrams
Word and phrase frequency data drawn from Google's digitised book collection, spanning centuries of published text. It counts how often words and short phrases appear by year, which is useful for historical language analysis and time-aware RAG features.
LibriVox
A volunteer project providing free public domain audiobooks. Volunteers record themselves reading works whose copyright has expired, which makes it a handy source of speech data for training speech-to-text models and building multimodal RAG.
Open Library
An Internet Archive project building a web page for every book ever published. It holds more than 20 million catalogue records and lends many titles digitally, making it a rich source of book metadata for RAG systems.
Project Gutenberg
A volunteer effort to digitise and archive public domain books, with more than 70,000 free ebooks. Mostly English, but it covers many languages. One of the oldest digital library projects, so it is a clean, permissively licensed source of full-text literature for RAG.