feat: reproducible benchmark suite; align published perf claims with measured results#164
Merged
Merged
Conversation
…ith measured results Every performance number DataFog publishes is now reproducible with one command: python benchmarks/run.py. Suites: core scan/redact throughput, LiteLLM guardrail (DataFogGuardrail), Claude Code hook (datafog-hook subprocess incl. startup), plus clearly-labeled comparisons against presidio-analyzer and spaCy NER on identical pinned payloads (manifest-verified detection counts before timing; en_core_web_sm pinned, favorable to comparison targets, so ratios are lower bounds). Claims aligned with measured results (Apple M5 Pro reference numbers in benchmarks/README.md): - LiteLLM guardrail: '~31µs per request' was really per-scan; reworded to ~40µs per message scanned (full 2-msg request: ~120µs clean / ~220µs with redaction+litellm logging). - Claude Code hook: ~70ms confirmed (69-89ms median incl. startup); reworded to ~70-90ms. - '190x performance advantage' tagline not reproducible (measured 103-170x vs Presidio, 114-140x vs spaCy); reworded to '100x+ faster than NER-based detection' in setup.py, setup_lean.py, __init__ docstrings, AGENTS.md, docs/roadmap.rst. Also refresh docs/roadmap.rst to current state (4.1-4.7.0 shipped, 4.7.x precision patches, v5.0.0 themes from the public roadmap) and add a ROADMAP.md stub pointing at it as the single source of truth (removing the stale case-insensitive Roadmap.md gitignore entry that swallowed it). Benchmark deps stay out of the core package (benchmarks/requirements.txt); benchmarks/ has no __init__.py so nothing ships in the wheel.
1 task
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every performance number DataFog publishes should be reproducible by a skeptic with one command. This PR adds that command and aligns the published copy with what it actually measures.
New:
benchmarks/python benchmarks/run.py— five suites: core (scan/redact throughput), guardrail (DataFogGuardrailvia LiteLLM, in-process), hook (datafog-hookas a real subprocess incl. Python startup, exactly as Claude Code invokes it), spacy and presidio comparisons on identical payloads and entity types.manifest.json-recorded detection counts, verified before timing — the suite fails loudly rather than benchmark an engine that stopped detecting. All PII values are standard synthetic test data.en_core_web_sm(smallest model, favorable to spaCy/Presidio) and exclude their setup cost, so measured ratios are defensible lower bounds.timeitonly for the core/hook path; guardrail + comparison deps live inbenchmarks/requirements.txt, out of the package. No__init__.py, nothing ships in the wheel.Claim changes backed by the measurements (Apple M5 Pro reference numbers in
benchmarks/README.md)Roadmap consolidation
docs/roadmap.rstrefreshed to reality: 4.1–4.7.0 marked shipped with actual release content, 4.7.x precision-patch focus, v5.0.0 themes from the public roadmap text.ROADMAP.mdadded as a two-line pointer todocs/roadmap.rst(single source of truth). Removed the case-insensitiveRoadmap.mdgitignore entry that swallowed it. Supersedes docs: add public v5.0.0 roadmap #153.Test plan
benchmarks/README.md)--quick,--suitesubset, unknown-suite error path,--jsonexport exercisedpytestgreen — 646 passed; only pre-existingen_core_web_lg-missing failures (unrelated, env-only)black/isort/flake8clean onbenchmarks/