Skip to content

Improve definition-site ranking for exact identifiers (Zod #5944) #59

Description

@aryamthecodebreaker

Problem

The frozen external benchmark still has one honest top-5 miss: the Zod #5944 case. The fixing file is packages/zod/src/v4/core/regexes.ts, while FixMap currently ranks JSON-schema consumer modules above the definition site.

Current v0.5.1 result: Top-1 3/6, Top-3 5/6, Top-5 5/6. Exact outputs are recorded in benchmarks/external/results.json.

The benchmark deliberately uses the issue title plus the first 600 body characters. That excerpt ends inside the reported pattern, so the fix should improve general definition-site evidence rather than edit the task, expected label, or case.

Desired behavior

When a task contains a distinctive identifier or literal pattern, the file that defines it should receive useful evidence without letting broad consumers or generated/example files dominate.

Acceptance criteria

  • Add a focused synthetic regression test for the generalized signal.
  • Move packages/zod/src/v4/core/regexes.ts into the top 5 for the frozen Zod case.
  • Do not special-case Zod, issue 5944, repository names, or expected paths.
  • Do not edit the frozen task text or expected labels to manufacture a hit.
  • Keep all existing core tests, internal evaluation, and the other five external cases green.
  • Refresh benchmarks/external/results.json only with npm run evaluate:external:record and document the measured result truthfully.
  • Keep ranking deterministic and bounded on large repositories.

Reproduce

npm ci
npm run build:core
node scripts/evaluate-external.mjs --gate --check-recorded

Implementation ideas are welcome, but evidence from definitions or re-exports should remain explainable in each ranked file's reasons.

Metadata

Metadata

Assignees

No one assigned

    Labels

    evaluationBenchmarks, datasets, and ranking qualityhelp wantedMaintainer welcomes community implementation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions