Semantic Scholar / S2ORC
Semantic Scholar is a free search and discovery service for scientific literature, built by the Allen Institute for AI. Behind it sits a large academic graph that connects papers, authors, venues, and the citations between them, which is what makes it useful for building knowledge-aware retrieval over research.
S2ORC is the bulk dataset you download to work with locally. It gives you cleaned full text where the licence allows, along with parsed section structure, inline citations, and reference lists, so you can retrieve at the passage level and follow citation links between papers. Papers are pulled together from arXiv, PubMed, Crossref, direct publisher feeds, and web crawls.
You can reach the data two ways: the S2ORC snapshot on Hugging Face for bulk use, or the Semantic Scholar API for live lookups of papers, authors, and citations. The bulk corpus is released under ODC-By, which asks you to credit the source, while live API access follows separate API terms.
Related sources
arXiv
An open-access preprint server for physics, mathematics, computer science, quantitative biology, statistics, and more, holding over 2.5 million papers. You can pull the full archive in bulk from Amazon S3, or harvest the metadata through OAI-PMH, a standard protocol for sharing records between repositories.
CORE
An aggregator of open-access research papers that harvests from thousands of repositories and journals worldwide. It holds over 300 million metadata records and more than 40 million full-text articles, all reachable through one search API.
Crossref
A nonprofit DOI registration agency that publishes metadata for over 150 million scholarly works, including journal articles, books, and conference proceedings. A DOI is the permanent identifier assigned to each work, and Crossref shares its metadata through a free API.
OpenAlex
A free, open catalogue of the world's scholarly works, authors, venues, institutions, and research topics. It succeeds Microsoft Academic Graph and holds hundreds of millions of works with rich metadata and citation links.