Docs/discoverability improvements to help agents (and humans) do non-trivial work
Feedback from an AI (Claude) doing fairly sophisticated work in a pgxntool
consumer project (cat_tools). The docs here are actually good — this is mostly
about discoverability plus one concrete inaccuracy. Filed at the
maintainer's request to improve AI results with pgxntool.
1. CLAUDE.md mis-describes make test's exit behavior (please fix — highest impact)
CLAUDE.md (Key Implementation Details → Test System) says:
.IGNORE: installcheck allows make test to handle errors (show diffs, then exit with error status)
That's backwards. .IGNORE: installcheck makes Make ignore installcheck's
non-zero exit, and the test recipe only cats regression.diffs — so
make test returns 0 even when tests fail. Verified in practice: a consumer's
CI showed "2 of 13 tests failed" while the job's make test step still passed
(green). verify-results is the actual gate, and only guards make results, not
make test — so CI that runs make test does not fail on a regression
unless it also runs a check (e.g. make verify-results).
This is the single most consequential thing for an agent to get right (it
determines whether CI catches test regressions), and the current wording asserts
the opposite. Suggest correcting it and, ideally, a one-liner: "to make CI fail
on a regression, run make verify-results after make test (or check for
regression.diffs)."
2. Discoverability: agents in a consumer repo don't see these docs
pgxntool is embedded as a nested pgxntool/ subtree. An agent working in the
consumer project loads the project's CLAUDE.md, not pgxntool/CLAUDE.md or
pgxntool/README.asc (nested, not auto-loaded). So for non-trivial build/test
work it tends to reverse-engineer base.mk and gets things wrong. Concretely, I
conflated DATA as a pgxntool variable when it's PGXS's (pgxntool only seeds
it), and I re-derived test/install's "state persists into the suite" behavior
that README.asc already documents.
Ideas (leaving specifics to you):
- Have
setup.sh add (or suggest) a line in the consumer's own CLAUDE.md like:
"For any non-trivial build/test work, read pgxntool/README.asc and
pgxntool/CLAUDE.md first." A pointer from the auto-loaded file is what closes
the gap.
- A short, prominent "mental model for agents" section distinguishing what's
PGXS (DATA, MODULES, DOCS, installcheck, …) from what pgxntool
adds/wraps (test, test-build, test/install, verify-results, version-file
generation, pgxntool-sync). Misattributing that boundary was my main source
of confusion.
3. Related, already filed
Net: the framework is well documented; the gap is (a) the one wrong line about
make test exit status, and (b) getting an agent to actually read the shipped
docs before doing sophisticated work.
Docs/discoverability improvements to help agents (and humans) do non-trivial work
Feedback from an AI (Claude) doing fairly sophisticated work in a pgxntool
consumer project (cat_tools). The docs here are actually good — this is mostly
about discoverability plus one concrete inaccuracy. Filed at the
maintainer's request to improve AI results with pgxntool.
1.
CLAUDE.mdmis-describesmake test's exit behavior (please fix — highest impact)CLAUDE.md(Key Implementation Details → Test System) says:That's backwards.
.IGNORE: installcheckmakes Make ignore installcheck'snon-zero exit, and the
testrecipe onlycatsregression.diffs— somake testreturns 0 even when tests fail. Verified in practice: a consumer'sCI showed "2 of 13 tests failed" while the job's
make teststep still passed(green).
verify-resultsis the actual gate, and only guardsmake results, notmake test— so CI that runsmake testdoes not fail on a regressionunless it also runs a check (e.g.
make verify-results).This is the single most consequential thing for an agent to get right (it
determines whether CI catches test regressions), and the current wording asserts
the opposite. Suggest correcting it and, ideally, a one-liner: "to make CI fail
on a regression, run
make verify-resultsaftermake test(or check forregression.diffs)."2. Discoverability: agents in a consumer repo don't see these docs
pgxntool is embedded as a nested
pgxntool/subtree. An agent working in theconsumer project loads the project's CLAUDE.md, not
pgxntool/CLAUDE.mdorpgxntool/README.asc(nested, not auto-loaded). So for non-trivial build/testwork it tends to reverse-engineer
base.mkand gets things wrong. Concretely, Iconflated
DATAas a pgxntool variable when it's PGXS's (pgxntool only seedsit), and I re-derived
test/install's "state persists into the suite" behaviorthat README.asc already documents.
Ideas (leaving specifics to you):
setup.shadd (or suggest) a line in the consumer's own CLAUDE.md like:"For any non-trivial build/test work, read
pgxntool/README.ascandpgxntool/CLAUDE.mdfirst." A pointer from the auto-loaded file is what closesthe gap.
PGXS (
DATA,MODULES,DOCS,installcheck, …) from what pgxntooladds/wraps (
test,test-build,test/install,verify-results, version-filegeneration,
pgxntool-sync). Misattributing that boundary was my main sourceof confusion.
3. Related, already filed
DATA's parse-time$(wildcard)misses generated (gitignored) SQL.test/installALTER-UPDATE equivalence pattern.Net: the framework is well documented; the gap is (a) the one wrong line about
make testexit status, and (b) getting an agent to actually read the shippeddocs before doing sophisticated work.