Skip to content

DATA's parse-time $(wildcard) misses generated (gitignored) versioned SQL #44

Description

@jnasbyupgrade

DATA's parse-time $(wildcard sql/*--*--*.sql) misses generated (gitignored) scripts

Context

Some extensions generate their versioned SQL (the install and upgrade
scripts) from .sql.in at build time and gitignore the generated .sql (so the
repo tracks only the .sql.in sources). cat_tools does this.

Problem

DATA is a PGXS variable, but pgxntool's base.mk seeds it with:

DATA = $(EXTENSION_VERSION_FILES) $(wildcard sql/*--*--*.sql)

$(wildcard ...) is expanded — and GNU Make caches the sql/ directory listing
— at parse time, which is before make all generates the .sql. On a
clean tree (fresh clone / after make clean) the generated upgrade scripts don't
exist yet, so the wildcard returns nothing for them, they never enter DATA, and
make install silently omits them. Because DATA is recursive, one might expect
the recipe-time re-expansion to pick them up once all has generated them, but
Make's cached directory listing means the wildcard stays empty for the run.

Concrete impact (cat_tools)

  • A clean make install omits the generated upgrade scripts, so a user who
    installed an older version cannot ALTER EXTENSION ... UPDATE — the upgrade
    path files simply aren't installed.
  • On a branch that also has an upgrade build test (test/build/upgrade.sql doing
    CREATE EXTENSION VERSION '<old>' + ALTER EXTENSION UPDATE), a clean
    make test fails with ERROR: no update path from "X" to "Y".
  • Tracked as cat_tools#28; worked around per-project by listing the generated
    scripts in DATA explicitly from their .sql.in source names (which are
    parse-stable) — cat_tools PR Clarify debug() level scheme: use ranges, not multiples of 10 #32.

Scope note

DATA itself is PGXS, so there's a limit to what pgxntool can change — but the
choice to populate it via a parse-time $(wildcard) over generated output is
pgxntool's, so this seems like the right place to raise it. The underlying issue
is really "pgxntool + generating .sql", not a bug in any one line.

An idea (leaving the design to you)

It would be nice if pgxntool offered first-class support for generated SQL — for
example, a convention where an extension drops generated .sql into a
.generated/ directory that pgxntool knows to build first and then install, so
DATA never depends on parse-time globbing of not-yet-generated files. Not
prescribing a solution; just flagging the problem and this direction.

Companion to #42 (test/install ALTER-UPDATE pattern), which came out of the same
cat_tools work.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions