RAG Evaluator

A local RAG evaluation tool with a Python/ML backend and browser frontend.

Setup

cd backend
pip install -r requirements.txt
python app.py

Then open frontend/index.html in your browser (or use the Live Server extension in VS Code).

How it works

The evaluation model uses three classical NLP techniques:

Metric	Method
Faithfulness	TF-IDF cosine similarity per answer sentence vs context
Answer relevance	TF-IDF similarity between question and answer
Context utilization	IDF-weighted vocabulary recall from context
Groundedness	Weighted combination of faithfulness + full-text cosine
Accuracy	Token-level F1 + TF-IDF similarity vs ground truth

VS Code

Press F5 to launch the backend via the included .vscode/launch.json.

Project structure

rag-evaluator/
├── backend/
│   ├── app.py          # Flask API
│   ├── evaluator.py    # ML evaluation model
│   └── requirements.txt
├── frontend/
│   └── index.html      # Single-file UI
└── .vscode/
    └── launch.json     # VS Code debug config

API

POST /evaluate

{
  "question": "...",
  "context": "...",
  "answer": "...",
  "ground_truth": "..." // optional
}