feat(review): [R20 S4] triage promotion — never auto-approve risk-bearing dbt metadata#1028
Conversation
…adata Grounded in the 5-PR internal corpus study (data-engineering-skills/docs/pr-review-corpus-findings-r20.md) where historical altimate-code recall was 2/84 = 2.4%. Two of five PRs auto-approved despite each having 8+ substantive human findings: - PR D: test-only YAML under `models/marts/` adding `data_tests:` / `constraints:` on a contracted mart → 8 misses, including 5 dbt-trino adapter-semantic bugs. - PR E: cost-anchor redesign under `mrt_jobs_cost_savings.sql` → 11 misses, including 3 critical FinOps corrections. Adds three FileChangeClass signals + wires them into `fullTierReasons()`: - **`dbtRiskYmlChanges`** — schema.yml diff introduces or edits `data_tests:`, `constraints:`, `contract:`, or a `unique_combination_of_columns` list-item. Regex anchors to YAML key position after optional diff marker, optional indent, and optional list marker; explicitly excludes comment lines. This catches PR D. - **`martLayerChange`** — file lives under `models/marts/` or `models/mart/`. Does NOT promote on its own (description-only edits stay trivial, matching the existing test), but ENRICHES the `dbtRiskYmlChanges` reason string so the customer sees "under models/marts/ (mart-API surface)". - **`finopsPathToken`** — path/filename contains a FinOps keyword (`cost|saving|billing|credit|dbu|spend|revenue|price|rate|pricing| invoice`) at a word / segment / extension boundary. Catches PR E. Word-boundary regex prevents false positives on incidental substrings (`broadcaster` ≠ `caste`, `precast` ≠ `cast`). Regression tests (11 new, all green): - PR D shapes (data_tests, constraints, unique_combination_of_columns) all promote to full with `mart-API surface` context. - PR E shape (mrt_jobs_cost_savings.sql) + 7 other FinOps keyword variants all promote to full. - FinOps false-positive guard: `broadcaster` / `precast` stay lite. - Description-only edits under models/marts/ still trivial (pre-existing behavior preserved). - Comment lines and description strings mentioning `data_tests:` / `constraints:` do NOT promote (regex tightness). - Nested-indent YAML key position (production shape) still fires. Codex-reviewed diff. Two highs addressed: - Regex tightening to avoid comment / description-string false positives - FileChangeClass consumer audit (no external constructions; safe) Ship criteria (from plan v2): PRs D and E from the corpus must no longer auto-approve. Baseline recall on the existing 13-scenario corpus must not regress. Both hold: 96/96 tests pass (85 pre-existing + 11 new); full altimate review suite 3781/3781 green. Depends on PR #1027 (feat/review-r18-observability-recall) — this branch stacks on top so tierReasons[] wiring is in place. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017zXDXMiNFh4qDPxPCfa2of
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| // dbtRiskYmlChanges hit but doesn't fire on its own. | ||
| if (c.dbtRiskYmlChanges) { | ||
| const loc = c.martLayerChange ? " under models/marts/ (mart-API surface)" : "" | ||
| reasons.push(`schema.yml diff touches data_tests/constraints/contract${loc}`) |
There was a problem hiding this comment.
SUGGESTION: Reason string omits unique_combination_of_columns
DBT_UNIQUE_COMBO_RE (line 78) matches unique_combination_of_columns and feeds into dbtRiskYmlChanges, but the surfaced reason only lists data_tests/constraints/contract. A customer who adds only a unique_combination_of_columns test (covered by the test at review.test.ts:466) would see a reason naming keys they didn't touch — inconsistent with both the field doc (lines 43-45) and the cubic summary in the PR description, which both list all four signals.
| reasons.push(`schema.yml diff touches data_tests/constraints/contract${loc}`) | |
| reasons.push(`schema.yml diff touches data_tests/constraints/contract/unique_combination_of_columns${loc}`) |
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
Code Review SummaryStatus: 1 Issue Found | Recommendation: Address before merge Overview
Issue Details (click to expand)SUGGESTION
Files Reviewed (2 files)
Fix these issues in Kilo Cloud Reviewed by glm-5.2 · Input: 64.6K · Output: 23.6K · Cached: 414K Review guidance: REVIEW.md from base branch |
Validation report against 5-PR internal corpusRan the S4-branch CLI against the 5 real internal dbt PRs from the R20 study (both A1 pure-deterministic and A2 with LLM lane). Baseline is the PR-#1027-tip CLI on the same corpus. Recall delta
Tier changes
InterpretationS4 is a routing change, not a detector change. On this corpus:
S4's value is foundational: it unblocks subsequent detector PRs (S1 grain-key Recommend keeping this PR open as the routing floor, but expect the benchmark movement to come from the detector-work PRs stacked above it. |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
3 issues found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/opencode/src/altimate/review/risk-tier.ts">
<violation number="1" location="packages/opencode/src/altimate/review/risk-tier.ts:77">
P2: Multiline description-only YAML edits can be promoted to full tier when their block-scalar text contains a line such as `constraints:` or `data_tests:`. Because this regex has no YAML scalar context, the claimed description precision guard does not hold; a YAML-aware or hunk/context-aware detector would avoid treating documentation content as metadata keys.</violation>
<violation number="2" location="packages/opencode/src/altimate/review/risk-tier.ts:77">
P1: Edits to an existing `data_tests:` or `constraints:` block can still be classified as `trivial`: the detector scans only changed lines, so an added list item such as `+ - not_null` has no risk key on the changed line and is missed. A hunk-aware YAML/context check (or parsing the resulting YAML and intersecting changed nodes) would keep these metadata edits from being auto-approved.</violation>
<violation number="3" location="packages/opencode/src/altimate/review/risk-tier.ts:144">
P2: The reason string `"schema.yml diff touches data_tests/constraints/contract"` doesn't account for the `unique_combination_of_columns` case. Since `dbtRiskYmlChanges` is true when *either* `DBT_RISK_KEY_RE` or `DBT_UNIQUE_COMBO_RE` matches, a customer who only adds a `unique_combination_of_columns` test will see a tier-promotion reason referencing keys they never touched. Consider including `unique_combination_of_columns` in the reason string (e.g., `data_tests/constraints/contract/unique_combination_of_columns`) to stay consistent with the field documentation and actual detection logic.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| // forms because `data_tests`/`constraints`/`contract` are ALWAYS keys, whereas | ||
| // `unique_combination_of_columns` is a TEST NAME that shows up as a list item | ||
| // (`- dbt_utils.unique_combination_of_columns:`). | ||
| const DBT_RISK_KEY_RE = /^[+-]?[ \t]*(?!#)(?:-[ \t]+)?(?:data_tests|constraints|contract)[ \t]*:/im |
There was a problem hiding this comment.
P1: Edits to an existing data_tests: or constraints: block can still be classified as trivial: the detector scans only changed lines, so an added list item such as + - not_null has no risk key on the changed line and is missed. A hunk-aware YAML/context check (or parsing the resulting YAML and intersecting changed nodes) would keep these metadata edits from being auto-approved.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/src/altimate/review/risk-tier.ts, line 77:
<comment>Edits to an existing `data_tests:` or `constraints:` block can still be classified as `trivial`: the detector scans only changed lines, so an added list item such as `+ - not_null` has no risk key on the changed line and is missed. A hunk-aware YAML/context check (or parsing the resulting YAML and intersecting changed nodes) would keep these metadata edits from being auto-approved.</comment>
<file context>
@@ -50,6 +66,23 @@ export interface ClassifyOptions {
+// forms because `data_tests`/`constraints`/`contract` are ALWAYS keys, whereas
+// `unique_combination_of_columns` is a TEST NAME that shows up as a list item
+// (`- dbt_utils.unique_combination_of_columns:`).
+const DBT_RISK_KEY_RE = /^[+-]?[ \t]*(?!#)(?:-[ \t]+)?(?:data_tests|constraints|contract)[ \t]*:/im
+const DBT_UNIQUE_COMBO_RE = /^[+-]?[ \t]*(?!#)-[ \t]+(?:[\w.]+\.)?unique_combination_of_columns[ \t]*:/im
+const MARTS_DIR_RE = /(?:^|\/)models\/marts?\//i
</file context>
| // forms because `data_tests`/`constraints`/`contract` are ALWAYS keys, whereas | ||
| // `unique_combination_of_columns` is a TEST NAME that shows up as a list item | ||
| // (`- dbt_utils.unique_combination_of_columns:`). | ||
| const DBT_RISK_KEY_RE = /^[+-]?[ \t]*(?!#)(?:-[ \t]+)?(?:data_tests|constraints|contract)[ \t]*:/im |
There was a problem hiding this comment.
P2: Multiline description-only YAML edits can be promoted to full tier when their block-scalar text contains a line such as constraints: or data_tests:. Because this regex has no YAML scalar context, the claimed description precision guard does not hold; a YAML-aware or hunk/context-aware detector would avoid treating documentation content as metadata keys.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/src/altimate/review/risk-tier.ts, line 77:
<comment>Multiline description-only YAML edits can be promoted to full tier when their block-scalar text contains a line such as `constraints:` or `data_tests:`. Because this regex has no YAML scalar context, the claimed description precision guard does not hold; a YAML-aware or hunk/context-aware detector would avoid treating documentation content as metadata keys.</comment>
<file context>
@@ -50,6 +66,23 @@ export interface ClassifyOptions {
+// forms because `data_tests`/`constraints`/`contract` are ALWAYS keys, whereas
+// `unique_combination_of_columns` is a TEST NAME that shows up as a list item
+// (`- dbt_utils.unique_combination_of_columns:`).
+const DBT_RISK_KEY_RE = /^[+-]?[ \t]*(?!#)(?:-[ \t]+)?(?:data_tests|constraints|contract)[ \t]*:/im
+const DBT_UNIQUE_COMBO_RE = /^[+-]?[ \t]*(?!#)-[ \t]+(?:[\w.]+\.)?unique_combination_of_columns[ \t]*:/im
+const MARTS_DIR_RE = /(?:^|\/)models\/marts?\//i
</file context>
| // dbtRiskYmlChanges hit but doesn't fire on its own. | ||
| if (c.dbtRiskYmlChanges) { | ||
| const loc = c.martLayerChange ? " under models/marts/ (mart-API surface)" : "" | ||
| reasons.push(`schema.yml diff touches data_tests/constraints/contract${loc}`) |
There was a problem hiding this comment.
P2: The reason string "schema.yml diff touches data_tests/constraints/contract" doesn't account for the unique_combination_of_columns case. Since dbtRiskYmlChanges is true when either DBT_RISK_KEY_RE or DBT_UNIQUE_COMBO_RE matches, a customer who only adds a unique_combination_of_columns test will see a tier-promotion reason referencing keys they never touched. Consider including unique_combination_of_columns in the reason string (e.g., data_tests/constraints/contract/unique_combination_of_columns) to stay consistent with the field documentation and actual detection logic.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode/src/altimate/review/risk-tier.ts, line 144:
<comment>The reason string `"schema.yml diff touches data_tests/constraints/contract"` doesn't account for the `unique_combination_of_columns` case. Since `dbtRiskYmlChanges` is true when *either* `DBT_RISK_KEY_RE` or `DBT_UNIQUE_COMBO_RE` matches, a customer who only adds a `unique_combination_of_columns` test will see a tier-promotion reason referencing keys they never touched. Consider including `unique_combination_of_columns` in the reason string (e.g., `data_tests/constraints/contract/unique_combination_of_columns`) to stay consistent with the field documentation and actual detection logic.</comment>
<file context>
@@ -91,6 +128,22 @@ export function fullTierReasons(c: FileChangeClass): string[] {
+ // dbtRiskYmlChanges hit but doesn't fire on its own.
+ if (c.dbtRiskYmlChanges) {
+ const loc = c.martLayerChange ? " under models/marts/ (mart-API surface)" : ""
+ reasons.push(`schema.yml diff touches data_tests/constraints/contract${loc}`)
+ }
+ if (c.finopsPathToken) reasons.push("path contains FinOps keyword (cost/saving/billing/dbu/etc.)")
</file context>
| reasons.push(`schema.yml diff touches data_tests/constraints/contract${loc}`) | |
| reasons.push(`schema.yml diff touches data_tests/constraints/contract/unique_combination_of_columns${loc}`) |
1 participant
Summary
Adds triage-tier promotion rules to
risk-tier.tsso the reviewer never auto-approves the two failure modes surfaced by the R20 corpus study:models/marts/addingdata_tests:/constraints:/unique_combination_of_columnson a contracted mart → previously classifiedtrivial→ auto-approved → missed 8 substantive human findings including 5 dbt-trino adapter-semantic bugs.mrt_jobs_cost_savings.sqlpath → previously classifiedlite→ approved → missed 11 findings including 3 critical FinOps corrections.Three new
FileChangeClasssignals wire intofullTierReasons():dbtRiskYmlChangesdata_tests:/constraints:/contract:/unique_combination_of_columnslist itemmartLayerChangemodels/marts/ormodels/mart/. Description-only edits under marts stay trivial (matches existing behavior).finopsPathTokenTest plan
packages/opencode/test/altimate/review.test.ts— all green.Ship criteria (from plan v2)
Depends on
Stacks on top of #1027 (
feat/review-r18-observability-recall). This branch uses thetierReasons[]wiring landed there so promotion reasons surface in the signed envelope. Merge #1027 first, then rebase this onto main.🤖 Generated with Claude Code
https://claude.ai/code/session_017zXDXMiNFh4qDPxPCfa2of
Summary by cubic
Promotes risk-bearing dbt schema changes and FinOps-related paths to full review so they never auto-approve. Adds precise signals in risk-tier classification to catch the R20 D/E PR shapes and surface clear reasons.
dbtRiskYmlChanges: triggers onschema.ymldiffs touchingdata_tests:,constraints:,contract:, or aunique_combination_of_columnsitem; ignores comments/description; promotes to full.finopsPathToken: triggers when the path/filename containscost|saving|billing|credit|dbu|spend|revenue|price|rate|pricing|invoiceat segment boundaries; promotes to full.martLayerChange: flags files undermodels/marts?/; enriches the reason when combined withdbtRiskYmlChanges; does not promote on its own so docs-only edits stay trivial.fullTierReasons()to explain promotions; 11 new tests cover D/E shapes and FP guards; existing tests unchanged.Written for commit 8eac0d9. Summary will update on new commits.