ProofRAG

ProofRAG started from a practical RAG problem: teams can change a chunker, retriever, reranker, prompt, model, or context-packing strategy, but without a stable evaluation set they are mostly comparing impressions. The hard part is not only running metrics; it is producing a useful golden set from the actual corpus and keeping the evaluation loop repeatable.

The tool packages that loop as both a Python CLI and an agent skill. It reads a corpus, generates and validates corpus-grounded test cases, calls the user's RAG system through HTTP or a Python callable, judges answers with a pinned judge or the optional typed Jev backend, computes retrieval metrics, and emits a self-contained HTML scorecard.

Engineering

ProofRAG grew out of the same evaluation problems turning up again and again in enterprise RAG work, so it's built as a tool other people can drop into their pipelines, not a notebook.

• Swappable providers and scorers. Anthropic, OpenAI and any OpenAI-compatible endpoint (Ollama, vLLM, LM Studio) sit behind one client, and the built-in judge can be swapped for DeepEval, Ragas or Jev. Retrieval metrics, scorecards, diff and the CI gates keep one output contract across all of them. Your RAG app is called over HTTP or as a Python callable, so changing its model or vector store doesn't touch the evaluation code.
• Judge output is validated, not trusted. Judge JSON is parsed strictly: duplicate keys, malformed objects, non-finite numbers and boolean or string scores are rejected, and truncated or filtered provider responses are errors. A bad judgment fails the run instead of quietly becoming a zero or a tie. The Jev backend keeps its typed rubric probabilities in the results.
• Fingerprints and baseline-drift protection. Each results file records the golden-set fingerprint, matcher, cutoff, backend, metric schema and judge identity. proofrag diff exits with an error when those differ between baseline and candidate, and a different judge needs an explicit --allow-judge-mismatch.
• CI gates. An absolute floor (--fail-under) and a regression gate against a committed baseline. The composite GitHub Action runs both, writes a job summary, and uploads the scorecard and results from the current run, including when a gate fails.
• Release hygiene. 188 pytest tests, run in CI on Python 3.11 to 3.14 alongside ruff, codespell and basedpyright. Versions come from git tags, and a GitHub release publishes to PyPI through trusted publishing after checking that the tag matches the changelog, the Action's default version and the plugin manifests. v0.9.0 shipped on 4 October 2026.

Design Decisions

• Fail rather than score partial data. Missing, duplicate or unexpected prediction ids, provider failures and invalid judgments end the run with an error. A flaky provider can fail a CI job as a result; a scorecard built on half the cases would be worse.
• A dependency-free core. The base install has no dependencies. Anthropic, OpenAI, DeepEval, Ragas and PDF support are extras, and Jev is called over its HTTP API without adding one.
• Static reports. Scorecards and A/B reports are self-contained HTML with no JavaScript or external assets, so they work as CI artifacts and open anywhere. Artifact data is escaped before it is written.
• No retriever of its own. ProofRAG scores what your system returns. Adapters accept text answers and text contexts only and reject ambiguous objects instead of coercing them to strings.

From Corpus to Gate

The workflow is intentionally explicit. Every stage leaves behind an artifact that can be reviewed, committed, compared, or uploaded by CI.

1. 01
   Generategoldenset.jsonl

   Build corpus-grounded questions and expected contexts.

2. 02
   Validatevalidation.json

   Check coverage, duplicates, sources, and corpus drift.

3. 03
   Runpredictions.jsonl

   Call the RAG app through HTTP or a Python callable.

4. 04
   Evaluateresults.json

   Score retrieval and generation with a pinned judge.

5. 05
   Reportscorecard.html

   Package the run into a static, reviewable scorecard.

$ proofrag generate --corpus ./docs --out goldenset.jsonl --n 20
20 grounded cases · 14 sources · corpus fingerprint saved
$ proofrag validate --goldenset goldenset.jsonl --corpus ./docs
validation passed · coverage and source references are consistent
$ proofrag run --goldenset goldenset.jsonl --endpoint http://localhost:8000/ask
predictions written to predictions.jsonl
$ proofrag evaluate --goldenset goldenset.jsonl --predictions predictions.jsonl
retrieval and generation scores written to results.json
$ proofrag report --results results.json --out scorecard.html
scorecard.html is ready to review

Golden Set Design

A benchmark is only useful when its cases stay grounded in the material the RAG system is meant to serve. ProofRAG therefore treats the golden set as a versioned evaluation asset, not temporary prompt output.

Reports You Can Review

Reports are static HTML files with no external assets. They can be opened locally after an agent run, sent to a teammate, or retained as a CI artifact without a hosted dashboard.

Evaluation report Blind A/B

CI Gates

The GitHub Action turns the same evaluation into a merge gate. An absolute mode enforces a score floor; a regression mode compares compatible candidate and baseline runs against the configured tolerance. Gates use full-precision scores even when reports round the displayed values. Invalid judgments and incompatible runs fail the check.

› uses: unshDee/[email protected]
goldenset: eval/goldenset.jsonl
predictions: predictions.jsonl
baseline: eval/baseline.json
fail-under: 0.7
› proofrag diff --baseline eval/baseline.json --candidate results.json
checking score floors and tolerated metric deltas
gate passed · no metric regressed beyond tolerance

Three Ways In

ProofRAG exposes one evaluation model through three entry points. The interface changes, but the data contracts, metrics, reports, and gates do not.

Scoring Backends

The default path uses ProofRAG's pinned LLM judge. DeepEval and Ragas can replace generation scoring while retrieval metrics, reporting, diffing, and CI behavior stay on the same shared surface.

Typed Jev Judging

Version 0.9.0 adds evaluate --backend jev through the official TypeSafe API. It needs a TYPESAFE_API_KEY and no extra Python dependency. The pinned jev-1.13.0 model scores groundedness, correctness, completeness, and citation quality using four ordered five-level rubrics in one request per case. Probability-weighted scores are normalized from 0–4 to ProofRAG's 0–1 scale.

Results retain raw probabilities, confidence, the resolved model, token usage, and a rubric fingerprint. Invalid distributions, provider failures, or model drift invalidate the run. Confidence describes rubric uncertainty; hosted RAG accuracy and dataset calibration have not been benchmarked by ProofRAG. Thresholds still need review against representative human labels.

Jev is available for evaluation only. Golden-set generation and blind A/B comparison continue to use Anthropic or an OpenAI-compatible provider.

Upgrading Baselines

Native judging now includes every retrieved context and JSON-encodes the evaluated text as data. Native evaluation and comparison fingerprints move to v3; DeepEval and Ragas fingerprints move to v2. Rerun older baselines with the same package version, backend, and judge before using them to gate new results. Keep baselines separate across scoring backends, even when their metric names match.

Case Studies

Introduced in v0.8.0, these three controlled studies apply the evaluation loop to retriever choice, index enrichment, and context depth. Each study freezes the rest of the stack and retains official hash-checked sources, reviewed golden data, predictions, scorecards, costs, failure analysis, judge disagreements, and reproduction steps. These are the retained v0.8.0 results, not new v0.9.0 or Jev benchmarks.

Study Index

The three experiments cover retriever choice, index enrichment, and context depth. None produced a universal optimization rule; together they show why retrieval and generation need separate measurements.

What the Three Studies Say Together

Better retrieval metrics do not guarantee better answers. Positive deltas do not automatically clear a materiality threshold. A blind judge can disagree with an absolute judge—and a project audit can disagree with both. ProofRAG keeps those disagreements visible instead of compressing them into a single leaderboard number.

Related RAG Utilities

I also maintain rag-utils^[1], an intentionally loose collection of scripts I reach for while building RAG systems rather than a second end-to-end product.