Skip to content

Docs/discoverability for agents: fix make-test exit-status claim; help AIs read the shipped docs #45

Description

@jnasbyupgrade

Docs/discoverability improvements to help agents (and humans) do non-trivial work

Feedback from an AI (Claude) doing fairly sophisticated work in a pgxntool
consumer project (cat_tools). The docs here are actually good — this is mostly
about discoverability plus one concrete inaccuracy. Filed at the
maintainer's request to improve AI results with pgxntool.

1. CLAUDE.md mis-describes make test's exit behavior (please fix — highest impact)

CLAUDE.md (Key Implementation Details → Test System) says:

.IGNORE: installcheck allows make test to handle errors (show diffs, then exit with error status)

That's backwards. .IGNORE: installcheck makes Make ignore installcheck's
non-zero exit, and the test recipe only cats regression.diffs — so
make test returns 0 even when tests fail. Verified in practice: a consumer's
CI showed "2 of 13 tests failed" while the job's make test step still passed
(green). verify-results is the actual gate, and only guards make results, not
make test — so CI that runs make test does not fail on a regression
unless it also runs a check (e.g. make verify-results).

This is the single most consequential thing for an agent to get right (it
determines whether CI catches test regressions), and the current wording asserts
the opposite. Suggest correcting it and, ideally, a one-liner: "to make CI fail
on a regression, run make verify-results after make test (or check for
regression.diffs)."

2. Discoverability: agents in a consumer repo don't see these docs

pgxntool is embedded as a nested pgxntool/ subtree. An agent working in the
consumer project loads the project's CLAUDE.md, not pgxntool/CLAUDE.md or
pgxntool/README.asc (nested, not auto-loaded). So for non-trivial build/test
work it tends to reverse-engineer base.mk and gets things wrong. Concretely, I
conflated DATA as a pgxntool variable when it's PGXS's (pgxntool only seeds
it), and I re-derived test/install's "state persists into the suite" behavior
that README.asc already documents.

Ideas (leaving specifics to you):

  • Have setup.sh add (or suggest) a line in the consumer's own CLAUDE.md like:
    "For any non-trivial build/test work, read pgxntool/README.asc and
    pgxntool/CLAUDE.md first." A pointer from the auto-loaded file is what closes
    the gap.
  • A short, prominent "mental model for agents" section distinguishing what's
    PGXS (DATA, MODULES, DOCS, installcheck, …) from what pgxntool
    adds/wraps (test, test-build, test/install, verify-results, version-file
    generation, pgxntool-sync). Misattributing that boundary was my main source
    of confusion.

3. Related, already filed

Net: the framework is well documented; the gap is (a) the one wrong line about
make test exit status, and (b) getting an agent to actually read the shipped
docs before doing sophisticated work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions