A curated directory of open datasets, knowledge bases, and data repositories for building Retrieval-Augmented Generation (RAG) applications: apps that retrieve relevant documents, then use a language model to answer from them.
78
data sources
14
categories
71
open access
Large-scale web-crawled text datasets
Wikipedia, Wikidata, and structured knowledge bases
Research papers, preprints, and citation graphs
Semantic knowledge graphs and ontologies
Source code repositories and developer Q&A
Court opinions, legislation, and regulatory filings
Official government open data portals
Geographic data, maps, and place databases
News archives, event databases, and media datasets
Public domain books and literary corpora
Clinical data, drug databases, and medical literature
Platforms for discovering and trading datasets
Benchmark datasets designed for RAG evaluation
Awesome-lists and dataset directories
A good place to start, whatever you are building.
An open-access preprint server for physics, mathematics, computer science, quantitative biology, statistics, and more, holding over 2.5 million papers. You can pull the full archive in bulk from Amazon S3, or harvest the metadata through OAI-PMH, a standard protocol for sharing records between repositories.
A nonprofit that crawls the web and freely provides its archives and datasets. Petabytes of raw web data from billions of pages, with a new snapshot each month. Used in the training of GPT-3, LLaMA, T5, and many other large language models.
The largest open hub for machine learning datasets, with well over 100,000 datasets you can search, stream, and version. Many RAG-ready corpora live here, and its Python library lets you pull data straight into a pipeline in a few lines.
A crowdsourced map of the world, built and maintained by a community of volunteers. It holds detailed geographic data on roads, buildings, land use, points of interest, and much more, which you can download as a full planet file or as smaller regional extracts.
A volunteer effort to digitise and archive public domain books, with more than 70,000 free ebooks. Mostly English, but it covers many languages. One of the oldest digital library projects, so it is a clean, permissively licensed source of full-text literature for RAG.
The US National Library of Medicine's database of biomedical and life sciences literature. PubMed indexes over 36 million citations and abstracts, and PubMed Central (PMC) adds free full-text access to a growing subset. You can download the data in bulk over FTP.
Allen AI's academic graph, covering hundreds of millions of papers linked by billions of citations. S2ORC, the Semantic Scholar Open Research Corpus, is the downloadable dataset with full text and parsed references, drawn from arXiv, PubMed, Crossref, publishers, and web crawlers.
Periodic XML dumps of every site in the Stack Exchange network, including Stack Overflow. Each dump holds questions, answers, comments, votes, and user profiles. One of the richest question-and-answer datasets you can freely download, and a natural fit for building technical support and developer RAG systems.
A free, collaborative knowledge base that holds structured facts for Wikipedia and the other Wikimedia projects. More than 100M items, each one machine-readable with statements, relationships, and identifiers, so you can pull clean facts instead of parsing article prose. Released into the public domain under CC0.
Complete database dumps of Wikipedia and the other Wikimedia projects, in every language, refreshed roughly twice a month. The most widely used knowledge source for RAG systems, and an easy first corpus to build a retrieval pipeline on.
RAG Repo is a community directory. Every source has a plain-English description, a clear licence, and a note on how ready it is for retrieval, so you can decide what fits before you download anything. We link only to primary sources, and advertising never influences what we list.