Problem
The frozen external benchmark still has one honest top-5 miss: the Zod #5944 case. The fixing file is packages/zod/src/v4/core/regexes.ts, while FixMap currently ranks JSON-schema consumer modules above the definition site.
Current v0.5.1 result: Top-1 3/6, Top-3 5/6, Top-5 5/6. Exact outputs are recorded in benchmarks/external/results.json.
The benchmark deliberately uses the issue title plus the first 600 body characters. That excerpt ends inside the reported pattern, so the fix should improve general definition-site evidence rather than edit the task, expected label, or case.
Desired behavior
When a task contains a distinctive identifier or literal pattern, the file that defines it should receive useful evidence without letting broad consumers or generated/example files dominate.
Acceptance criteria
- Add a focused synthetic regression test for the generalized signal.
- Move
packages/zod/src/v4/core/regexes.ts into the top 5 for the frozen Zod case.
- Do not special-case Zod, issue 5944, repository names, or expected paths.
- Do not edit the frozen task text or expected labels to manufacture a hit.
- Keep all existing core tests, internal evaluation, and the other five external cases green.
- Refresh
benchmarks/external/results.json only with npm run evaluate:external:record and document the measured result truthfully.
- Keep ranking deterministic and bounded on large repositories.
Reproduce
npm ci
npm run build:core
node scripts/evaluate-external.mjs --gate --check-recorded
Implementation ideas are welcome, but evidence from definitions or re-exports should remain explainable in each ranked file's reasons.
Problem
The frozen external benchmark still has one honest top-5 miss: the Zod #5944 case. The fixing file is
packages/zod/src/v4/core/regexes.ts, while FixMap currently ranks JSON-schema consumer modules above the definition site.Current v0.5.1 result: Top-1 3/6, Top-3 5/6, Top-5 5/6. Exact outputs are recorded in
benchmarks/external/results.json.The benchmark deliberately uses the issue title plus the first 600 body characters. That excerpt ends inside the reported pattern, so the fix should improve general definition-site evidence rather than edit the task, expected label, or case.
Desired behavior
When a task contains a distinctive identifier or literal pattern, the file that defines it should receive useful evidence without letting broad consumers or generated/example files dominate.
Acceptance criteria
packages/zod/src/v4/core/regexes.tsinto the top 5 for the frozen Zod case.benchmarks/external/results.jsononly withnpm run evaluate:external:recordand document the measured result truthfully.Reproduce
Implementation ideas are welcome, but evidence from definitions or re-exports should remain explainable in each ranked file's reasons.