AI & Agents · Advanced
Agentic RAG Knowledge Platform
Build a hybrid-search RAG service that routes queries across retrieval, web fallback, and a reranker, with citations and an offline eval harness that scores faithfulness.
You build a production-grade Retrieval-Augmented Generation service that combines dense vector search (pgvector or Qdrant) with BM25 sparse retrieval, then routes low-confidence queries to a live web fallback before a BGE cross-encoder reranker selects the final context passages. A LangGraph agent orchestrates the routing logic, the FastAPI layer serves grounded answers with inline citations, and a Ragas eval harness runs offline to score faithfulness, context precision, and answer relevancy on a held-out question set. This stack covers the full lifecycle that teams actually deploy: ingestion, hybrid retrieval, agentic routing, reranking, citation formatting, and quantitative evaluation.
What you build
- Hybrid retrieval that merges dense (pgvector or Qdrant) and sparse (BM25) results using Reciprocal Rank Fusion
- LangGraph agent that decides whether to answer from the index, trigger a web search fallback, or escalate to a human
- BGE cross-encoder reranker that rescores candidate passages before they reach the LLM context window
- FastAPI endpoint that returns structured JSON with inline source citations and confidence metadata
- Chunking pipeline with configurable overlap that ingests PDFs, Markdown, and plain text into the vector store
- Ragas offline eval harness that scores faithfulness, context recall, and answer relevancy against a golden QA set
- Streaming response support so the UI can progressively render the answer as tokens arrive
What it teaches
- Hybrid search architecture: combining dense embeddings with BM25 and fusing results via Reciprocal Rank Fusion
- LangGraph stateful agent design: conditional routing nodes, tool invocation, and state persistence across graph steps
- Cross-encoder reranking mechanics and when it outperforms bi-encoder cosine similarity for passage selection
- Citation grounding: extracting source metadata from retrieved chunks and propagating it through the LLM output
- RAG evaluation methodology using Ragas metrics (faithfulness, context precision, answer relevancy) on a golden dataset
- Production API patterns: streaming responses, structured JSON output schemas, and confidence metadata in FastAPI
How it works
- 1
Ingest
- PDF / Markdown / Text
- Chunker + Embedder
- pgvector / Qdrant
↓ indexed
- 2
Hybrid Retrieve
- Dense search
- BM25 sparse
- RRF fusion
↓ candidates
- 3
Agent Route
- LangGraph node
- confidence check
- web fallback tool
↓ passages
- 4
Rerank
- BGE cross-encoder
- top-K selection
↓ context
- 5
Generate
- LLM + citations
- FastAPI stream
↓ answer
- 6
Evaluate
- Ragas harness
- faithfulness score
Sign in to open the build guide
Free account. Get the step-by-step build and every resource link.
Take it further
- Add a self-hosted Qdrant collection alongside pgvector and implement an A/B router that splits queries between the two stores so you can compare retrieval quality metrics.
- Build a feedback loop: capture thumbs-up/down signals from the API response and fine-tune the reranker on hard negatives collected from low-rated answers.
- Containerize the full stack with Docker Compose, add a Prometheus metrics endpoint tracking query latency and retrieval hit rate, and wire a Grafana dashboard.
