Hugging Face Datasets
Hugging Face Datasets is both a website and a Python library. On the site you can browse datasets by task, language, size, and licence, preview the data in your browser, and read community documentation. The library then lets you load any of them with a single call, with support for streaming so you can work with corpora far larger than your local disk.
For RAG it is often the fastest route to data: pre-parsed Wikipedia, filtered web corpora like FineWeb, code datasets, and purpose-built RAG evaluation sets are all hosted here. Datasets are versioned with git, so you can pin an exact revision for reproducibility.
Licences vary from dataset to dataset, so always check the individual dataset card before you use its data in production.
Related sources
AWS Data Exchange
A marketplace for finding, subscribing to, and using third-party data inside the AWS cloud. It carries both free open datasets and paid commercial data products, so you can pull licensed data straight into your AWS workflows without setting up separate transfers.
AWS Open Data Registry
A registry of high-value datasets hosted on AWS and made publicly available, covering genomics, geospatial data, climate, satellite imagery, and more. Over 300 PB of data in total, free to access: you pay only for the compute you use to process it.
Datahub.io
A platform for publishing and finding open data packages: datasets bundled with consistent metadata in standardised formats. It hosts curated collections of widely used reference data, from country codes to exchange rates, ready to drop into a pipeline.
Google Dataset Search
A search engine for datasets that indexes millions of them from thousands of repositories worldwide. It does not host data itself: it points you to wherever each dataset lives, which makes it a fast first stop when you are hunting for a source on a specific topic.