ProofRAG

ProofRAG started from a practical RAG problem: teams can change a chunker, retriever, reranker, prompt, model, or context-packing strategy, but without a stable evaluation set they are mostly comparing impressions. The hard part is not only running metrics; it is producing a useful golden set from the actual corpus and keeping the evaluation loop repeatable.

The tool packages that loop as both a Python CLI and an agent skill. It reads a corpus, generates and validates corpus-grounded test cases, calls the user's RAG system through HTTP or a Python callable, judges answers with a pinned LLM-as-judge, computes retrieval metrics, and emits a self-contained HTML scorecard.

From Corpus to Gate

The workflow is intentionally explicit. Every stage leaves behind an artifact that can be reviewed, committed, compared, or uploaded by CI.

1. 01
   Generategoldenset.jsonl

   Build corpus-grounded questions and expected contexts.

2. 02
   Validatevalidation.json

   Check coverage, duplicates, sources, and corpus drift.

3. 03
   Runpredictions.jsonl

   Call the RAG app through HTTP or a Python callable.

4. 04
   Evaluateresults.json

   Score retrieval and generation with a pinned judge.

5. 05
   Reportscorecard.html

   Package the run into a static, reviewable scorecard.

$ proofrag generate --corpus ./docs --out goldenset.jsonl --n 20
20 grounded cases · 14 sources · corpus fingerprint saved
$ proofrag validate --goldenset goldenset.jsonl --corpus ./docs
validation passed · coverage and source references are consistent
$ proofrag run --goldenset goldenset.jsonl --endpoint http://localhost:8000/ask
predictions written to predictions.jsonl
$ proofrag evaluate --goldenset goldenset.jsonl --predictions predictions.jsonl
retrieval and generation scores written to results.json
$ proofrag report --results results.json --out scorecard.html
scorecard.html is ready to review

Golden Set Design

A benchmark is only useful when its cases stay grounded in the material the RAG system is meant to serve. ProofRAG therefore treats the golden set as a versioned evaluation asset, not temporary prompt output.

Reports You Can Review

Reports are static HTML files with no external assets. They can be opened locally after an agent run, sent to a teammate, or retained as a CI artifact without a hosted dashboard.

Evaluation report Blind A/B

CI Gates

The GitHub Action turns the same evaluation into a merge gate. An absolute mode enforces a score floor; a regression mode compares the candidate run with a committed baseline and fails only when a metric moves beyond the configured tolerance.

› uses: unshDee/[email protected]
goldenset: eval/goldenset.jsonl
predictions: predictions.jsonl
baseline: eval/baseline.json
fail-under: 0.7
› proofrag diff --baseline eval/baseline.json --candidate results.json
checking score floors and tolerated metric deltas
gate passed · no metric regressed beyond tolerance

Three Ways In

ProofRAG exposes one evaluation model through three entry points. The interface changes, but the data contracts, metrics, reports, and gates do not.

Scoring Backends

The default path uses ProofRAG's pinned LLM judge. DeepEval and Ragas can replace generation scoring while retrieval metrics, reporting, diffing, and CI behavior stay on the same shared surface.

Case Studies

Release 0.8.0 applies that evaluation loop to three controlled retrieval questions. Each study freezes the rest of the stack and retains official hash-checked sources, reviewed golden data, predictions, scorecards, costs, failure analysis, judge disagreements, and reproduction steps.

Study Index

The three experiments cover retriever choice, index enrichment, and context depth. None produced a universal optimization rule; together they show why retrieval and generation need separate measurements.

What the Three Studies Say Together

Better retrieval metrics do not guarantee better answers. Positive deltas do not automatically clear a materiality threshold. A blind judge can disagree with an absolute judge—and a project audit can disagree with both. ProofRAG keeps those disagreements visible instead of compressing them into a single leaderboard number.

Related RAG Utilities

I also maintain rag-utils^[1], an intentionally loose collection of scripts I reach for while building RAG systems rather than a second end-to-end product.