Open RAGBench (Vectara)
Open RAGBench takes 1,000 papers from arXiv, the open preprint server for scientific research, and turns them into an evaluation set for RAG systems. The content extraction is multimodal, meaning it pulls not just body text but also material from tables and figures, which reflects how messy real documents actually are.
The benchmark ships 3,045 question-and-answer pairs spanning several scientific domains, so you can measure how well a system retrieves the right passage and produces a grounded answer over long, technical source documents. It is published on Hugging Face under an Apache 2.0 licence.
This is a good choice when your target use case is scientific or technical document search, where questions often depend on precise details buried deep in a paper. It is more demanding than small Wikipedia-based sets and closer to production conditions.
Related sources
FlashRAG Benchmark Datasets
A research toolkit that bundles standardised, pre-processed benchmark datasets for RAG evaluation across many scenarios. Every dataset comes in a single unified JSONL format, so you can swap benchmarks in and out without rewriting your data loading each time.
PleIAs RAG-Resources
A curated collection of datasets and resources assembled by PleIAs for building and evaluating RAG applications. A useful starting point when you want ready-made material to test retrieval pipelines without hunting for datasets one by one.
RAG-Mini-Wikipedia
A small evaluation set that pairs 918 question-and-answer pairs with a matching corpus of 3,200 Wikipedia passages. Built specifically for testing RAG systems, it is small enough to run quick evaluation loops while still covering realistic retrieval questions.