rag-utils
Open source · Standalone Python scripts
rag-utils is deliberately not a library. Each script stands alone, with no framework and no shared package to install, so you copy the one you need into your own pipeline. ProofRAG is the packaged, tested tool; this is the toolbox around it.
Where each script fits
- Ingest
- docx_to_chunks · pdf_to_markdown
- Chunk
- semantic_chunker · overlap merger · quality scorer · dedup
- Enrich
- contextual_retrieval · late_chunking · embedding_cache
- Index
- binary_quant_index
- Retrieve
- hybrid_retriever · query_expander
- Rerank
- cross_encoder_reranker · context_pruner
- Pack
- context_window_packer
- Gate
- sufficient_context
- Evaluate
- eval_set_generator · retrieval_eval · answer_eval
- Observe
- rag_tracer
Grouped by pipeline stage, as in the repository README.
A Few Worth Opening
Hybrid retrieval. BM25 and dense search, with FAISS or a NumPy fallback, fused by reciprocal-rank fusion. RRF combines ranks rather than scores, so the two systems never need their scores normalized. The fusion function is exported on its own for custom pipelines.
Embedding cache. A SQLite cache keyed on a hash of model id and text, so the same text is never embedded twice across runs. It takes any embedder callable, supports a TTL, and uses WAL mode with thread-local connections to stay safe under threads.
Tracing. Named spans with latency, token counts and custom metadata, usable as a context manager or a decorator, written out as JSON Lines. Setting TRACING_ENABLED=false turns it into a no-op.
Evaluation. A generator builds a golden set from your chunks, so the relevant ids come for free, then retrieval_eval scores Hit@k, Precision@k, Recall@k, MRR, NDCG@k and MAP, and answer_eval scores what the model did with the retrieved context.
Trying Them
Several scripts need only the standard library. Many take a --demo or --generate-sample flag, so they run without your own data first. Usage and dependencies for each are in the README.