API

The whole directory is published as static JSON so you can build on it. It is free and read-only, needs no API key or sign-up, and is served with cross-origin (CORS) access, so you can fetch it from a browser, a script, or a backend. If you are building a tool that discovers or ingests datasets, this is the front door.

Endpoints

• GET /sources.json — the full catalogue: all 264 sources, each with its category, formats, access patterns, size tier, licence and licence-clarity flags, links, and canonical page. Includes alegend that documents every field.
• GET /updates.json — recently added and reviewed sources, newest first. A lean feed for detecting changes without downloading the whole catalogue.
• GET /updates.xml — the same "what's new", as an RSS feed.

Quick start

# Fetch the catalogue
curl https://rag-repo.org/sources.json

// In JavaScript
const res = await fetch('https://rag-repo.org/sources.json');
const { count, sources } = await res.json();
console.log(count, 'sources');

The shape

/sources.json returns a top-level object with catalog,site, generated (the build date, YYYY-MM-DD),count, a licence note, a legend, and asources array. Each source looks like this (fields are omitted ornull when not known):

{
  "slug": "common-crawl",
  "name": "Common Crawl",
  "page": "https://rag-repo.org/source/common-crawl",
  "url": "https://commoncrawl.org",
  "category": "web-corpora",
  "categoryName": "Web Corpora",
  "accessType": "open",
  "accessPatterns": ["cloud-bucket", "bulk-download"],
  "ragReadiness": "raw",
  "format": ["WARC", "WET", "WAT"],
  "size": "Petabytes (each monthly crawl is ~200-400 TB compressed)",
  "sizeTier": "huge",
  "licence": "Open (content subject to original site terms)",
  "commercialUse": "permitted",
  "shareAlike": false,
  "attributionRequired": false,
  "contentLicenceDiffers": true,
  "version": null,
  "lastReviewed": "2026-08-06",
  "supersedes": [],
  "supersededBy": null,
  "maintainer": "Common Crawl Foundation",
  "tags": ["web-crawl", "multilingual", "pretraining"],
  "links": { "download": "https://commoncrawl.org/the-data/" }
}

The legend object in the response documents every enumerated field:accessType, accessPatterns (how you actually get the bytes), sizeTier (a coarse download-volume gate), and the versioning fields. You do not need to hard-code these meanings; read them from the file.

Filtering

There are no query parameters: fetch once and filter client-side. Thesejqexamples show the fields most people want.

# Open sources you can use commercially
curl -s https://rag-repo.org/sources.json \
  | jq '.sources[] | select(.accessType=="open" and .commercialUse=="permitted") | .name'

# Everything on the Hugging Face Hub
jq '.sources[] | select(.accessPatterns | index("huggingface")) | .slug'

# Small enough to pull onto a laptop
jq '[.sources[] | select(.sizeTier=="tiny" or .sizeTier=="small")] | length'

# One category
jq '.sources[] | select(.category=="legal") | {name, licence}'

Watching for changes

Rather than diffing the whole catalogue, poll /updates.json (or subscribe to /updates.xml). It is ordered by lastReviewed, newest first, so you can compare against the last date you saw to pick up new or re-reviewed sources cheaply.

Terms and good practice

• Free, read-only, no authentication.
• The files are static and served through a CDN with a one-hour cache. Please cache your copy and poll no more than hourly. There are no hard rate limits, just be reasonable.
• Catalogue metadata is reusable with attribution to RAG Repo (rag-repo.org). Each dataset keeps its own licence, given in thelicence field alongside our commercialUse,shareAlike, attributionRequired andcontentLicenceDiffers flags. Check those before you build on a source, and treat them as a shortlist, not legal advice: confirm the terms at the source.
• Fields may be added over time; we will not remove or repurpose existing ones without notice. The generated field dates each build.

For how sources are selected and reviewed, see ourmethodology. Need a field the API does not expose, or spotted something wrong? Get in touch via thecontribute page.