UltraRAG
by OpenBMB
UltraRAG is a low-code RAG framework that turns retrieval, generation, and evaluation into standalone MCP servers you wire together with YAML instead of glue code.
Install from source
git clone https://github.com/OpenBMB/UltraRAG.git --depth 1 && cd UltraRAG && uv sync --all-extras
About
UltraRAG, built by THUNLP, NEUIR, OpenBMB, and AI9stars, breaks a retrieval-augmented generation pipeline into separate MCP servers, including Retriever, Generation, Evaluation, Corpus, Router, and Note, instead of one monolithic script. A YAML file describes how those servers connect, including loops, conditional branches, and early stopping, so changing your pipeline's control flow doesn't mean rewriting Python.
Because each component is its own MCP server, you can swap a retriever or generation backend without touching the rest of the pipeline, and the same servers work whether you're running a quick local experiment or a full research benchmark. A built-in UI can turn a finished YAML pipeline into an interactive web interface for demoing or debugging, and the project includes standardized evaluation workflows for comparing RAG configurations against each other.
Key features
- RAG pipeline stages (retrieval, generation, evaluation, corpus prep) run as independent MCP servers
- YAML-defined orchestration with loops, conditional branches, and early stopping
- One-click generation of an interactive web UI from a pipeline definition
- Built-in standardized evaluation and benchmarking workflows
- Modular design lets you swap retrievers or generators without rewriting the pipeline
- Docker image available alongside the source install
Use cases
- Running head-to-head RAG benchmark comparisons across different retriever/generator pairings
- Prototyping a multi-step RAG pipeline with branching logic without hand-writing the control flow
- Standing up a demo UI for a RAG pipeline for stakeholders without building a frontend
- Researching new RAG architectures where components need to be swapped in isolation
Available tools
Retriever server
Handles document retrieval (dense, sparse, or hybrid) as a standalone MCP server the pipeline calls into.
Generation server
Wraps an LLM backend to handle the generation step of the pipeline.
Evaluation server
Runs standardized scoring and benchmarks against a pipeline's outputs.
Corpus server
Handles document ingestion and corpus preparation feeding the retriever.
ultrarag run <yaml>
CLI command that executes a full pipeline defined in a YAML configuration file.
Frequently asked questions
Do I need to write Python to build a RAG pipeline with UltraRAG?
Not for the control flow. Sequencing, branching, and loops are described in YAML. You still supply or configure the underlying retriever, generator, and evaluation servers, but wiring them together is low-code.
Can I swap out the retriever without rebuilding the whole pipeline?
Yes. Retrieval, generation, and evaluation are separate MCP servers, so replacing one only means pointing the YAML config at a different server.