Add @fedify/netlify for async workloads#934
Conversation
Introduce @fedify/netlify for durable Fedify queue delivery through Netlify Functions. Preserve per-key FIFO processing with crash-safe CAS state, durable workload sleeps, dead-letter recovery, and conservative handling of ambiguous sends. Make PostgresKvStore CAS-capable and logged by default so ordering state survives PostgreSQL crash recovery. Add portable unit coverage, a live Netlify Dev integration test, an Astro example, and deployment docs. fedify-dev#930 Assisted-by: Codex:gpt-5.6-sol
The Netlify Async Workloads dependency emits the same non-actionable binary setup warning from every Deno validation process. Suppress that diagnostic chatter during code generation, linting, and type checking while leaving command failures and error output intact. fedify-dev#930 Assisted-by: Codex:gpt-5.6-sol
✅ Deploy Preview for fedify-json-schema canceled.
|
|
@codex review |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdds the ChangesNetlify integration
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant FederationApp
participant NetlifyMessageQueue
participant AsyncWorkloads
participant QueueHandler
participant PostgresKvStore
FederationApp->>NetlifyMessageQueue: enqueue message
NetlifyMessageQueue->>PostgresKvStore: reserve ordering sequence
NetlifyMessageQueue->>AsyncWorkloads: send workload event
AsyncWorkloads->>QueueHandler: deliver event
QueueHandler->>PostgresKvStore: wait for prior sequence
QueueHandler->>FederationApp: process queued task
QueueHandler->>PostgresKvStore: complete or skip sequence
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the new @fedify/netlify package, which integrates Fedify's message queue with Netlify Async Workloads, and adds a corresponding Astro example. It also updates @fedify/postgres's PostgresKvStore to support atomic Compare-And-Swap (cas()) operations and to use crash-safe logged tables by default. The review feedback highlights three key improvements: correcting a premature sequence-skipping condition in the Netlify workload handler due to 1-based attempt counting, updating the corresponding unit test to reflect this fix, and replacing to_regclass with a direct ::regclass cast in PostgreSQL to robustly handle mixed-case table names.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1642f88ef8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Codecov Report❌ Patch coverage is
... and 9 files with indirect coverage changes 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/netlify-astro/src/lib/federation.ts`:
- Around line 52-53: Replace the comma operator in the created-value assignment
within the federation key-pair creation logic with a clearer if/else structure:
when context.data.kv.cas is null, await context.data.kv.set(keyPairKey,
serialized) and set created to true; otherwise preserve the existing behavior.
In `@packages/netlify/src/handler.ts`:
- Around line 146-157: Wrap the await options.queue.skipOrderingSequence call in
the catch block of the handler with its own try-catch, preserving and rethrowing
the original processing error even if skipping the ordering sequence fails;
optionally log the skip failure without allowing it to replace the original
error.
In `@packages/postgres/src/kv.ts`:
- Around line 244-262: Document the operational cost in the initialization path
around PostgresKvStore.initialize: note that converting an existing unlogged
table via SET LOGGED rewrites the table and takes an ACCESS EXCLUSIVE lock, and
that the relpersistence probe plus CREATE TABLE check run whenever `#initialized`
is false. Also document that per-invocation callers such as
createNetlifyQueueHandler should pass initialized: true when initialization is
already known to avoid repeated startup round trips.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8acfed68-5cf9-4709-93f8-cd8032a86521
⛔ Files ignored due to path filters (2)
deno.lockis excluded by!**/*.lockpnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (49)
.github/CODEOWNERS.gitignore.hongdown.tomlAGENTS.mdCHANGES.mdCONTRIBUTING.mdcspell.jsondeno.jsondocs/manual/deploy.mddocs/manual/mq.mddocs/package.jsonexamples/netlify-astro/.gitignoreexamples/netlify-astro/README.mdexamples/netlify-astro/astro.config.tsexamples/netlify-astro/netlify.tomlexamples/netlify-astro/netlify/functions/fedify-queue.tsexamples/netlify-astro/package.jsonexamples/netlify-astro/src/lib/federation.tsexamples/netlify-astro/src/lib/runtime.tsexamples/netlify-astro/src/middleware.tsexamples/netlify-astro/src/pages/index.astroexamples/netlify-astro/tsconfig.jsonexamples/test-examples/mod.tsmise.tomlpackages/fedify/README.mdpackages/netlify/README.mdpackages/netlify/deno.jsonpackages/netlify/fixtures/netlify-dev/.gitignorepackages/netlify/fixtures/netlify-dev/netlify.tomlpackages/netlify/fixtures/netlify-dev/netlify/functions/enqueue.mtspackages/netlify/fixtures/netlify-dev/netlify/functions/status.mtspackages/netlify/fixtures/netlify-dev/netlify/functions/workload.mtspackages/netlify/fixtures/netlify-dev/netlify/lib/runtime.tspackages/netlify/fixtures/netlify-dev/public/index.htmlpackages/netlify/package.jsonpackages/netlify/src/handler.tspackages/netlify/src/mod.tspackages/netlify/src/mq.tspackages/netlify/src/types.tspackages/netlify/test/handler.test.tspackages/netlify/test/integration/netlify.test.tspackages/netlify/test/mq.test.tspackages/netlify/test/package.test.tspackages/netlify/tsdown.config.tspackages/postgres/README.mdpackages/postgres/src/kv.test.tspackages/postgres/src/kv.tspnpm-workspace.yamlscripts/check_types.ts
Do not let a partially accepted Async Workloads batch masquerade as an atomic enqueue when Fedify applies one deduplication marker. Preserve ordered progress for malformed events, and resolve quoted PostgreSQL table names correctly during logged-table migration. fedify-dev#934 Assisted-by: Codex:gpt-5.6-sol
A ParallelMessageQueue that fans out to a backend without enqueueMany is not atomic itself. Report that inherited limitation directly so every caller, not only Fedify's task enqueue path, sees the correct capability. fedify-dev#934 Assisted-by: Codex:gpt-5.6-sol
PostgreSQL and AMQP enqueueMany implementations send messages one at a time. Declare that limitation so batch-level task deduplication rejects before either backend can accept only part of a batch. fedify-dev#934 Assisted-by: Codex:gpt-5.6-sol
A queue without enqueueMany cannot pass an atomic batch capability through the parallel wrapper. Make the shared adapter test assert the wrapper's effective capability instead of requiring a literal property copy. Assisted-by: Codex:gpt-5.6-sol
Keep the ordering-key support table consistent with the new Netlify queue section, including the CAS-capable store required for strict sequencing. fedify-dev#934 Assisted-by: Codex:gpt-5.6-sol
|
/gemini review |
|
@codex review |
There was a problem hiding this comment.
Code Review
This pull request introduces the @fedify/netlify package, integrating Fedify message queues with Netlify Async Workloads, and adds a corresponding Astro example. It also implements PostgresKvStore.cas() for atomic compare-and-swap operations, changes PostgreSQL tables to be logged by default with automatic migration, and introduces the atomicEnqueueMany property to enforce atomic batch enqueues during deduplication. The review feedback suggests simplifying the PostgreSQL table persistence check by using a native ::regclass cast, which would also allow the removal of the custom quoteIdentifier helper.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 775f583a04
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/netlify/README.md (1)
33-38: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winKeep installation instructions consistent with the examples.
Both documentation paths omit dependencies required by their PostgreSQL/Netlify Database examples:
packages/netlify/README.md#L33-L38: add@netlify/database,@fedify/postgres, andpostgres, or mark them as optional example dependencies.docs/manual/mq.md#L664-L686: add@fedify/postgresandpostgres, or separate the PostgreSQL setup into an optional installation path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/netlify/README.md` around lines 33 - 38, Update the installation instructions in packages/netlify/README.md (lines 33-38) to include `@netlify/database`, `@fedify/postgres`, and postgres, or clearly mark those dependencies as optional example requirements. Also update docs/manual/mq.md (lines 664-686) to include `@fedify/postgres` and postgres, or separate the PostgreSQL setup into an optional installation path, keeping both documentation flows consistent with their examples.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/netlify/src/mq.ts`:
- Line 307: Add a concise JSDoc comment directly above the public
atomicEnqueueMany property, documenting that it indicates whether atomic
enqueue-many operations are supported. Keep the existing readonly property and
value unchanged.
---
Outside diff comments:
In `@packages/netlify/README.md`:
- Around line 33-38: Update the installation instructions in
packages/netlify/README.md (lines 33-38) to include `@netlify/database`,
`@fedify/postgres`, and postgres, or clearly mark those dependencies as optional
example requirements. Also update docs/manual/mq.md (lines 664-686) to include
`@fedify/postgres` and postgres, or separate the PostgreSQL setup into an optional
installation path, keeping both documentation flows consistent with their
examples.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: ab77acde-c3fd-4624-8374-1b1f2d994cf2
📒 Files selected for processing (20)
CHANGES.mddocs/manual/mq.mddocs/manual/tasks.mdexamples/netlify-astro/src/lib/federation.tspackages/amqp/src/mq.test.tspackages/amqp/src/mq.tspackages/fedify/src/federation/mq.test.tspackages/fedify/src/federation/mq.tspackages/fedify/src/federation/tasks/enqueue.test.tspackages/fedify/src/federation/tasks/enqueue.tspackages/netlify/README.mdpackages/netlify/src/handler.tspackages/netlify/src/mq.tspackages/netlify/test/handler.test.tspackages/netlify/test/mq.test.tspackages/postgres/README.mdpackages/postgres/src/kv.test.tspackages/postgres/src/kv.tspackages/postgres/src/mq.test.tspackages/postgres/src/mq.ts
Keep the base queue installation small while listing every dependency used by the PostgreSQL ordering examples. Separate commands make it clear that the database packages are needed only for that setup. fedify-dev#934 (review) Assisted-by: Codex:gpt-5.6-sol
|
Addressed the optional PostgreSQL installation documentation in 723d4bd. Both examples now list the dependencies they import without making them mandatory for the base Netlify queue setup. |
|
@codex review |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces the @fedify/netlify package, which integrates Fedify's message queue with Netlify Async Workloads for Netlify Functions. It also adds PostgresKvStore.cas() to @fedify/postgres for atomic key-value operations and updates PostgresKvStore to use logged tables by default. Additionally, a new atomicEnqueueMany property is added to the MessageQueue interface to indicate atomic batch capabilities. A critical PostgreSQL syntax issue was identified in the PostgresKvStore.cas() implementation where schema-qualified table names in the ON CONFLICT DO UPDATE WHERE clause will fail at runtime; a suggestion has been provided to remove the table qualification.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Separate sequence counters for queues that share a key-value store but route to different workload event names. Normalize the case-insensitive event name so equivalent routes still share one ordering namespace. fedify-dev#934 (comment) Assisted-by: Codex:gpt-5.6-sol
Expose the inherited MessageQueue documentation on the public capability flag so generated API references explain why Netlify batches are non-atomic. fedify-dev#934 (comment) Assisted-by: Codex:gpt-5.6-sol
Alias the INSERT target before the conflict update so expired-row replacement can reference the existing row without emitting an invalid schema-qualified FROM-clause reference. fedify-dev#934 (comment) Assisted-by: Codex:gpt-5.6-sol
|
/gemini review |
|
@codex review |
There was a problem hiding this comment.
Code Review
This pull request introduces the @fedify/netlify package, which integrates Fedify's message queue with Netlify Async Workloads, and includes a new Astro example. It also implements PostgresKvStore.cas() for atomic key-value operations, changes PostgresKvStore to use logged tables by default, and introduces the atomicEnqueueMany property to the MessageQueue interface to prevent non-atomic batch deduplication. Feedback on the changes suggests optimizing cancelOrderingSequence in NetlifyMessageQueue to avoid redundant database writes when a sequence is already completed or cancelled.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf18ddfed7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Let ordering-state updates report that no write is needed, so repeated completion or cancellation does not issue a redundant CAS operation. fedify-dev#934 (comment) Assisted-by: Codex:gpt-5.6-sol
Run ordered task processing as a durable Async Workloads step so a transient ordering-state completion failure can retry cleanup without repeating the task side effects. fedify-dev#934 (comment) Assisted-by: Codex:gpt-5.6-sol
|
@codex review |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces the @fedify/netlify package, enabling integration with Netlify Async Workloads via NetlifyMessageQueue and createNetlifyQueueHandler. It also adds PostgresKvStore.cas() for atomic operations, updates PostgreSQL tables to be logged by default, and introduces the MessageQueue.atomicEnqueueMany property. Feedback on the changes highlights potential runtime errors when handling Temporal.DurationLike plain objects for delay and TTL options, which should be normalized using Temporal.Duration.from. Additionally, a compatibility concern was raised regarding the ALTER TABLE ... SET LOGGED migration, which requires PostgreSQL 15 or later.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| message: Message, | ||
| options?: MessageQueueEnqueueOptions, | ||
| ): Promise<void> { | ||
| const delay = options?.delay?.total("milliseconds"); |
There was a problem hiding this comment.
If options.delay is passed as a Temporal.DurationLike plain object (e.g., { seconds: 5 }), options.delay?.total will be undefined, and invoking it as a function will throw a TypeError: options.delay.total is not a function at runtime. Normalizing it with Temporal.Duration.from first ensures safe handling of both Temporal.Duration instances and plain objects.
const delay = options?.delay == null
? undefined
: Temporal.Duration.from(options.delay).total("milliseconds");| options?: KvStoreSetOptions, | ||
| ): Promise<boolean> { | ||
| await this.initialize(); | ||
| const ttl = options?.ttl == null ? null : options.ttl.toString(); |
There was a problem hiding this comment.
If options.ttl is passed as a Temporal.DurationLike plain object (e.g., { seconds: 10 }), calling options.ttl.toString() will return "[object Object]". This is an invalid interval format for PostgreSQL and will cause a database error at runtime. Normalizing it with Temporal.Duration.from first ensures that we always get a valid ISO 8601 duration string (e.g., PT10S).
const ttl = options?.ttl == null
? null
: Temporal.Duration.from(options.ttl).toString();| if (persistence[0]?.relpersistence === "u") { | ||
| await this.#sql` | ||
| ALTER TABLE ${this.#sql(this.#tableName)} SET LOGGED; | ||
| `; | ||
| } |
There was a problem hiding this comment.
Note that ALTER TABLE ... SET LOGGED is only supported in PostgreSQL 15 and later. If users are running PostgreSQL 14 or older, this migration will fail with a syntax error on startup. Consider documenting this requirement or gracefully handling the error if older PostgreSQL versions need to be supported.
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Closes #930.
Why this integration has two halves
Netlify Functions can already serve Fedify's HTTP endpoints, and Netlify Database can provide persistent state through PostgreSQL. The missing part was queue consumption: a request-scoped Function cannot keep
MessageQueue.listen()running between invocations.This integration follows Netlify's execution model instead of trying to recreate a long-lived worker.
NetlifyMessageQueuesubmits Fedify jobs as Async Workloads events, including delayed jobs, whilecreateNetlifyQueueHandler()turns a Function under netlify/functions/ into the consumer. The handler builds a freshFederationfor each invocation, passes the job toprocessQueuedTask(), and lets processing errors escape so Async Workloads owns retries and dead-letter handling. Invalid envelopes are marked non-retryable because another attempt cannot repair them.Ordering without expiring locks
Async Workloads routes events in FIFO order, but it may process several events concurrently. A simple lease would still allow a later job to overtake an earlier one, expire during a long task, or spend every retry attempt merely waiting for another invocation.
For each
orderingKey, the producer instead reserves a monotonic sequence in a CAS-capableKvStore. Consumers wait for the preceding sequence with the workload's durablestep.sleep(). Waiting therefore does not hold a Function open or consume the failure-retry budget, and exclusivity lasts for the whole task rather than for a guessed lease duration.The PostgreSQL adapter now supplies the CAS operation needed by this protocol. Its table is logged by default, and an existing unlogged table is migrated during initialization, because losing sequence state during PostgreSQL crash recovery could leave accepted events blocked indefinitely. Applications that only store disposable data can still opt into an unlogged table.
Recovery favors delivery safety
An ordered job that reaches its last configured processing attempt releases its sequence before Netlify dead-letters it, which lets later jobs continue. Timeouts and abrupt termination cannot run that cleanup, so
skipOrderingSequence()provides an explicit recovery path after an operator has confirmed that the event is permanently dead-lettered.The producer takes the opposite approach when a send is not acknowledged. A missing response does not prove that Netlify rejected the event, so its sequence remains reserved and
NetlifyMessageQueueSendErrorexposes the ordering metadata for later investigation. Advancing automatically could let subsequent work pass an accepted event and then discard that event when it eventually arrives.Verification and operating guidance
Portable unit tests use a narrow
AsyncWorkloadsClientinterface and cover event encoding, delays, retries, malformed input, concurrent ordering, dead-letter progress, ambiguous sends, and manual recovery. A Node.js integration test starts Netlify Dev against a fixture and exercises real delivery, retry, FIFO processing, and continuation after dead-lettering. It loadsNETLIFY_AUTH_TOKENandNETLIFY_SITE_IDfrom the root .env file and skips itself when either value is absent. Deno and Bun report a conditional skip through the same test command.The package README.md, message queue manual, and deployment manual explain provisioning, retry configuration, the 500 KB event limit, and the need to keep preview federation isolated or disabled. The Astro example applies the same production guard to both its web middleware and workload consumer so a deploy preview cannot publish as the production actor or enqueue work that no consumer will process.
The branch was checked with
mise run check, the cross-runtime package suites, the live Netlify Dev integration test, and the package build.