PureLLM

PureLLM is a benchmark for testing whether an LLM API router or relay is serving the model it claims to serve. It focuses on authenticity signals rather than broad model capability, then compares a routed provider run against a baseline captured from the official provider API.

The project can be used from either a local web interface or the command line. The web interface is the easiest workflow for recording official baselines, selecting router presets, loading available models, and browsing saved runs. The CLI remains available for scripted baseline and relay comparisons.

You can also use the deployed version here: https://purellm.ai.studio/ The UI is different from the current code, since AI Studio 'improved' it when code is uploaded. API Keys are not persisted in the backend, if you worry about key security, generate a temporary key and invalidate it after the test. Thanks to AI Studio for providing free deployment and a free domain name.

Getting Started

Web Interface

Start the local web server:

python web_app.py --host 127.0.0.1 --port 8787

Open:

http://127.0.0.1:8787

Typical workflow:

1. Open the Baseline page.
2. Choose an official provider preset, enter the provider API key, load models if needed, and run a baseline against the official model.
3. Open the Test Provider page.
4. Choose the saved official baseline, select a router provider, enter the router API key, choose or type the routed model identifier, and start the provider test.
5. Review saved runs on the Leaderboard page.

The web interface stores local state in data/purellm.sqlite3 and writes full run JSON files under outputs/web/.

CLI Interface

Record an official baseline:

python -m llm_benchmark official \
  --api-type openai \
  --api-key "$OPENAI_API_KEY" \
  --model gpt-4o \
  --output baseline_gpt4o.json

Test a router or relay against that baseline:

python -m llm_benchmark test \
  --api-type openai \
  --api-key "$RELAY_API_KEY" \
  --model gpt-4o \
  --base-url https://relay-service.example.com/v1 \
  --baseline baseline_gpt4o.json \
  --output relay_results.json

Anthropic-compatible baseline example:

python -m llm_benchmark official \
  --api-type anthropic \
  --api-key "$ANTHROPIC_API_KEY" \
  --model claude-3-5-sonnet-20241022 \
  --output baseline_claude.json

You can also provide a JSON config file with shared settings:

python -m llm_benchmark --config config.json official

Methodology

PureLLM runs in two modes:

1. Official mode records a baseline from the official model provider. The baseline captures outputs, token usage, scores, and per-test details.
2. Test mode runs the same active suite against a router or relay and compares the result with the official baseline to produce an authenticity score and verdict.

The benchmark is designed around signals that are hard for a relay to fake accidentally. Tokenizer behavior, deterministic outputs, knowledge boundaries, alignment style, and sampling preferences tend to vary across model families and provider implementations. A routed model can match generic capability while still diverging on these fingerprints.

Dynamic text comparisons are judged semantically where exact matching would be brittle. Deterministic probes still use deterministic evaluators, including token counts, strict identity formats, exact low-temperature outputs, and preference distribution checks.

Test Categories

PureLLM currently uses 41 active authenticity tests across six categories:

Category	Count	Purpose
Identity	8	Checks model self-description, creator, dates, parameter claims, and architecture claims.
Tokenizer	15	Compares API-reported prompt token counts across texts that expose tokenizer differences.
Determinism	5	Compares temperature-zero outputs against the official baseline.
Knowledge Boundary	5	Checks whether the model shares the official model's public-event knowledge boundary.
Alignment Fingerprints	5	Compares refusal, redirection, and safety-boundary behavior.
Logit Preference Fingerprints	3	Samples ambiguous completions repeatedly and compares output distributions.

Older programming and instruction-following tests remain in the codebase, but they are skipped by the default active suite because they mainly measure capability and compliance rather than authenticity.

Project Files

• tests.py defines the benchmark probes and evaluators.
• runner.py executes benchmark runs and baseline comparisons.
• api_client.py provides OpenAI-compatible and Anthropic-compatible client paths.
• reporter.py prints summaries and writes JSON output.
• web_app.py serves the local web interface.
• web_db.py stores web-interface providers, official models, runs, and results.
• providers.json defines official provider and router presets.

License

PureLLM is released under the MIT License. See LICENSE for details.