feat(mothership): Chat-scoped outputs/ folder in VFS + Fork/Duplicate Chat

[j15z]

Summary

Adds two related capabilities on the chat/VFS surface, plus the field fixes found while testing them. (1) A chat-scoped outputs/ namespace for one-off generated files (images, audio, video, ffmpeg results) — separate from workspace files/, living with the chat, saveable to the workspace on request; agent-side steering is behind the mothership chat-scoped-outputs flag, so it dark-launches. (2) Fork and Duplicate — one engine, two gestures. The Fork button on each assistant reply copies the conversation up to and including that message into a fresh chat (Fork | <name>); the right-click Duplicate action is the same fork route with no cut point (upToMessageId is now optional) — a whole-chat copy titled <name> (Copy) that keeps every message and copies agent outputs too. In both modes, copied files get fresh row ids and storage keys, bytes are physically copied and quota-charged, and every in-transcript file reference is re-pointed at the copies so the copy survives deletion of the original. Duplicates ask the mothership fork endpoint for its whole-chat clone mode (compacted working memory preserved verbatim); branch forks keep the truncate-and-rebuild path. Schema: two additive migrations — 0254 adds nullable workspace_files.message_id, the provenance column that lets a branch fork copy only the chat-owned files (uploads AND outputs) born at-or-before the cut, and 0255 adds a partial unique index on output display names (see review-response changes below).

Field fixes included (found via live testing of the above):

• Ghost resources: a branch fork copied the chat's resources jsonb wholesale, keeping chips for files it doesn't have; file resources whose chat-owned file wasn't copied are now dropped.
• Fork on a live message: a just-streamed reply carries a synthetic live-assistant:<streamId> id and forking it 400'd; the Fork button now waits for the persisted id.
• Media-tool reference inputs: generate_image/video/audio and ffmpeg resolved inputs.files workspace-only, so uploads//outputs/ references never loaded — and generate_image silently generated from the prompt alone. New shared resolveToolInputFile covers all three VFS namespaces, and unresolvable explicit references now fail the call instead of being silently skipped.
• Phantom-stream reconnect wedge: an outputs/ link click on the home surface minted an empty-id resource that 400'd the next send and wedged the UI in "running" for ~3 minutes; empty-id resources are now rejected and a failed POST rolls back instead of retry-reconnecting.

Review-response changes (e43fe9c34)

Addresses all 8 findings from code review:

• Write-once in both modes: outputs/ + overwrite is now rejected in headless runs too — the files/ redirect can no longer silently replace a same-named workspace file.
• Branch forks now copy pre-cut outputs (design change from review discussion): outputs are stamped with message_id at creation and join the same timeline cut as uploads, so inline embeds are re-pointed and forks fully survive source-chat deletion. The rule for every chat-owned file: it travels with the fork iff the user message that carried/requested it is kept.
• Resolver adoption: open_resource, table import, and knowledge add-document resolve outputs//uploads/ refs and bare wf_ ids (new chat-scoped by-id fallback in resolveToolInputFile); presigning and the background CSV import honor chat-scoped storage contexts (TableImportPayload.fileContext, backward-compatible).
• create_file guard: checks the raw fileName before files/ prefixing, so outputs/notes.md can no longer create a literal files/outputs/ folder.
• Migration 0255: partial unique index on (chat_id, display_name) WHERE context='output' + a 23505 retry in uploadChatOutput, closing the concurrent same-name generation race (mirrors the existing uploads index/retry; separate index so the two namespaces stay independent).
• Reader consolidation: upload-file-reader + output-file-reader merged into one context-parameterized chat-file-reader (~200 duplicated lines removed, all public names preserved); shared isOutputsPath/leaf helpers replace 5 inline prefix checks.
• Fork blob copy: headObject replay guard (no double-copy/double-charge on retry) + bounded concurrency (4) + a single workspace_files read per fork with the branch cut applied in memory.
• Panel actions: EmbeddedFile/EmbeddedFileActions share one resolver hook (list → by-id → chat outputs by leaf name), so Download/Open work on path-referenced outputs.

Deploy note: run a one-off duplicate check for (chat_id, display_name) output rows before applying 0255 — the unique index build fails if race-produced dupes already exist (unlikely: young, flag-gated feature).

Post-review rounds (after e43fe9c34)

Four more review/babysit rounds plus a max-effort adversarial pass landed on the branch; full detail lives in the two tracked handoff docs below.

• Rounds 2–5 (babysit + Bugbot): materialize's import resolves outputs/ like save; the twin mothership↔copilot resource schema families are synced and parity-pinned by test; four round-5 Bugbot findings fixed; the two earlier review fixes that landed without tests are now pinned (tool-call-state.test.ts, contracts/copilot.test.ts).
• Migration 0255 now builds its partial unique index CONCURRENTLY (the deploy-note pre-check still applies).
• Max-effort review pass (2026-07-05): 35 verified findings. Root cause of the worst cluster: the by-id resolvers pinned context='workspace', hiding output rows from every path not explicitly rewired for the third namespace. Fixed on-branch: model payload now includes output tabs; output tabs render read-only (no doomed editor); >5MB CSV outputs preview instead of 404ing; presign-flow UUID upload ids resolve as tool inputs. A client-side send-probe fix for the double-send was attempted and reverted after the gate re-review refuted its premise (see R0 in ISSUES.md).
• Release triage (2026-07-05): all remaining findings ship as tracked fast-follows behind the mothership flag — the decision and its conditions are recorded at the top of ISSUES.md.

Handoff docs (tracked on this branch)

• ISSUES.md — the live issue tracker: 11 triaged findings (quota ratchet, delete-path blob orphaning, fork crash window, …), review fast-follows R0–R8, the cleanup batch, and the accepted-behavior record. Newest is R8 (2026-07-06, found by manual testing): the flag-on create_file schema advertises outputs/<name> while the sim server tool rejects it, so an explicit text-output request deterministically lands in files/ — fix touches mothership catalog/files/create_file.go plus a prompt/product call on how text outputs should be created at all.
• MANUAL-TESTING.md — a 39-test ordered manual plan (smoke → edge) with per-test expectations and [KNOWN — …] markers tying expected failures back to ISSUES.md entries. Test 2 documents the R8 behavior and the function_execute workaround that unblocks the tests depending on outputs/notes.md.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• Other: ___________

Testing

• Full affected sweep green (vitest run lib/copilot lib/table lib/uploads app/api/mothership from apps/sim): 107 files, 1067 tests passing, including fork route, fork-chat-files, rewrite-file-references, effective-transcript, resolve-input-file, chat-file-reader (the merged upload/output readers), vfs, materialize-file, create-file, resources, and track-chat-upload. New coverage from the review pass: write-once rejection in both interactive and headless modes, the headless files/ redirect, the create_file raw-fileName guard, the wf_ id chat-scoped fallback, outputs joining the fork timeline cut (pre-cut copied + re-pointed, post-cut ghosts dropped), the blob-copy replay guard, and the bounded concurrency pool — alongside the earlier whole-chat duplicate, ghost-resource, and resolver cases.
• tsc --noEmit ✅ 0 errors · biome check ✅ clean · check:api-validation ✅ · check:react-query ✅. (apps/docs type-check noise is pre-existing — missing generated .source artifact; this branch touches no docs files.)
• Manual (local dev, three rounds of field-testing): uploads copy with their bytes and re-pointed references; branch forks carry pre-cut outputs with re-pointed embeds and leave post-cut files behind; duplicates open as <name> (Copy) with every message; reference-image generation actually uses the uploaded reference.
• Reviewers should focus on: the two-mode branch in fork/route.ts (isWholeChatDuplicate drives the message cut, file listing, title, Go body, and analytics — now a single workspace_files read cut in memory by filterForkableChatFiles); the ghost-resource drop rule in rewriteResourceFileRefs (chat-owned ∧ not-copied ⇒ dropped; shared workspace files pass through); the deliberate quota divergence from the workspace-fork precedent (forked bytes ARE counted — see the comment at the increment site); the write-once check ordering in resource-writer.ts (rejection must precede the headless redirect); and the two migrations (0254: additive, no backfill, NULL = birth-unknown ⇒ included in every fork; 0255: partial unique index, see deploy note).

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

Companion PR

Companion (mothership): https://github.com/simstudioai/mothership/pull/342

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Building	Preview, Comment	Jul 6, 2026 6:32pm

[cursor]

PR Summary

High Risk
Touches auth for a new storage context, billing quota on fork copies, and large fork/blob-copy paths with documented crash and refund gaps; client stream/resource changes affect core chat reliability.

Overview
Introduces chat-scoped outputs/ (alongside uploads) end-to-end: merged chat-file-reader, GET …/chats/[chatId]/outputs, storageContext: 'output' on records, and UI hooks so the resource picker, tabs, and /api/files/view can list, resolve, and preview outputs that never appear in the workspace Files list. materialize_file can promote uploads or outputs into workspace files (with name disambiguation); tool writes thread chatId / interactive / messageId so interactive outputs/ paths don’t silently land in files/.

Fork and duplicate share one route: optional upToMessageId for branch vs whole-chat copy, quota gate on copyable bytes, planChatFileCopies + post-commit blob copies (quota-charged, concurrency-bounded), transcript/resource rewrites, ghost resource drops after a branch cut, and cleanup when blob copy fails. Sidebar Duplicate and message Fork surface failedFileCopies warnings; fork is disabled on live assistant message ids until persisted.

Security & reliability: output files require chat owner, not just workspace membership, on serve/view paths; stream resume returns 503 on DB lookup failure instead of 404 (avoids terminal reconnect wedge). Chat client hardens resource ids (reject empty, filter placeholders, legacy cleanup) and pre-stream send failures (rollback + no bogus reconnect). Contracts sync mothership/copilot resource types; ISSUES.md documents known follow-ups (quota refund, delete orphan blobs, fork OOM window).

^{Reviewed by Cursor Bugbot for commit 4a2333a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR adds two major capabilities to the chat/VFS surface: (1) a chat-scoped outputs/ namespace for agent-generated one-off files (images, audio, video), and (2) Fork and Duplicate — a single fork endpoint driving two gestures. It also ships a batch of field fixes found during testing (silent reference-image skip, phantom-stream reconnect wedge, ghost file resources on forks, live-message fork 400).

• Chat-scoped outputs (outputs/<name>, context='output'): write-once, flat, tagged to the owning chat via FK. Two migrations: 0254 adds nullable workspace_files.message_id for timeline-cut on branch forks; 0255 adds a CONCURRENTLY-built partial unique index on (chat_id, display_name) WHERE context='output' with a 23505 retry in uploadChatOutput, mirroring the existing uploads index.
• Fork / Duplicate: one POST endpoint, two modes (upToMessageId set → branch fork titled "Fork | …", omitted → whole-chat duplicate titled "… (Copy)"). A single workspace_files read per fork, cut in memory by filterForkableChatFiles. Blob copies are bounded-concurrency, best-effort post-commit; failed copies clean up their dead DB rows, surface failedFileCopies in the response, and show a toast warning. Ghost file resources (uploads/outputs not copied in a branch fork) are dropped from the new chat's resource chips.
• Resolver consolidation: upload-file-reader + output-file-reader merged into a single context-parameterized chat-file-reader; a new resolveToolInputFile covers all three VFS namespaces (files/, uploads/, outputs/) and bare wf_ IDs; generate_image, generate_video, generate_audio, and ffmpeg now fail instead of silently skipping an unresolvable explicit reference input.

Confidence Score: 5/5

Safe to merge. The fork/duplicate logic, authorization changes, and outputs namespace are well-structured, consistently applied, and backed by thorough test coverage. The two migrations are additive and carefully constructed.

No data-correctness or security defects were found. The authorization additions — ownership check for output files in both verifyWorkspaceFileAccess and verifyRegularFileAccess, WORKSPACE_FILE_LOOKUP_CONTEXTS filter, getPreviewableWorkspaceFile userId guard — are consistently applied and correct. The fork transaction correctly rolls back on any failure; post-commit blob copies are best-effort with proper dead-row cleanup and user-visible warnings. The rewrite functions correctly handle the ghost-resource drop and never leave cross-chat dangling references. The only findings are a missing 'output' entry in the schema comment and a missing isPending guard on the Duplicate context-menu action.

No files require special attention. The schema.ts context comment omission and the sidebar double-click guard are the only items worth addressing before shipping.

Important Files Changed

Filename	Overview
apps/sim/app/api/mothership/chats/[chatId]/fork/route.ts	Implements the unified fork/duplicate endpoint. Two-mode branch (isWholeChatDuplicate) correctly drives message cut, file listing, title, Go body, and analytics from a single workspace_files read. Failed blob copies clean up dead rows and return failedFileCopies to the UI. Logic is sound.
apps/sim/lib/copilot/chat/fork-chat-files.ts	planChatFileCopies inserts copy rows in-transaction then defers blob I/O. filterForkableChatFiles correctly includes NULL-messageId rows in every fork. executeChatFileBlobCopies uses bounded concurrency (4) and best-effort error handling with per-task failure tracking.
apps/sim/lib/copilot/chat/rewrite-file-references.ts	rewriteMessageFileRefs and rewriteResourceFileRefs correctly use idMap/keyMap for rewrites and drop ghost file resources (chat-owned but not copied) via the dropFileIds set. Pure functions, well-tested.
apps/sim/lib/copilot/vfs/resource-writer.ts	isOutputsPath guard applied before files/ prefix (preventing outputs/notes.md → files/outputs/ escape). Write-once rejection now precedes headless redirect. Correct chatId + interactive gate for uploadChatOutput calls.
apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts	uploadChatOutput follows the same retry+cleanup pattern as trackChatUpload. mapWorkspaceFileRecord now propagates the real storageContext instead of hardcoding 'workspace'. getPreviewableWorkspaceFile adds correct userId ownership check for output rows. Logic is sound.
apps/sim/app/api/files/authorization.ts	lookupWorkspaceFileByKey now covers both 'workspace' and 'output' contexts. outputOwnershipSatisfied check is applied in both verifyWorkspaceFileAccess and verifyRegularFileAccess, correctly preventing cross-user output access by workspace membership alone.
apps/sim/lib/copilot/tools/handlers/chat-file-reader.ts	Merges upload-file-reader and output-file-reader into a single parameterized namespace. resolveChatFileRecordById correctly scopes to chatId + context IN ('mothership','output'). Normalized name comparison handles encoded vs raw segment variants.
packages/db/schema.ts	Adds nullable message_id text column and chatOutputDisplayNameUnique partial unique index. The context column inline comment still lists only the original contexts and omits 'output', which is now a first-class context written by uploadChatOutput.
packages/db/migrations/0255_chat_output_display_name_unique.sql	Uses CONCURRENTLY + explicit COMMIT (runner convention) to avoid ACCESS EXCLUSIVE on the live table. Deploy note in PR description calls out the pre-migration duplicate check requirement. Pattern mirrors the existing uploads index migration.
apps/sim/app/workspace/[workspaceId]/home/hooks/use-chat.ts	isPersistedChatResource predicate consistently filters empty-id and streaming-file placeholders from send, reorder, and persist paths. StreamNotFoundError breaks the reconnect loop for 404 runs immediately. streamStarted gate correctly prevents rollback of a never-registered stream.
apps/sim/app/workspace/[workspaceId]/w/components/sidebar/sidebar.tsx	handleDuplicateChat wires useForkMothershipChat for the right-click Duplicate action and shows a toast for partial file-copy failures. Missing forkChatMutation.isPending guard could allow concurrent duplicate requests on rapid double-click.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[POST /fork chatId] --> B{upToMessageId set?}
    B -- No --> C[Whole-chat duplicate\nAll files, title: name Copy]
    B -- Yes --> D[Branch fork\nSlice messages to cut point\ntitle: Fork name]

    C --> E[listDuplicableChatFiles\nALL chat-owned files\ncontext IN mothership,output]
    D --> F[listDuplicableChatFiles then filterForkableChatFiles\nmessageId IN keptSet OR NULL]

    E --> G[checkStorageQuota]
    F --> G

    G --> H[DB Transaction: INSERT copilot_chats]
    H --> I[planChatFileCopies\nINSERT workspace_files copies\nfresh id + key per row]
    I --> J[rewriteResourceFileRefs\nremap chips to new ids\ndrop ghost chat-owned refs]
    J --> K[rewriteMessageFileRefs\nremap transcript URLs and attachments]
    K --> L[appendCopilotChatMessages in tx]
    L --> M[COMMIT]

    M --> N[executeChatFileBlobCopies\nbounded concurrency 4\nbest-effort post-commit]
    N --> O{Any failed?}
    O -- Yes --> P[DELETE dead workspace_files rows\nremoveChatResources\nreturn failedFileCopies in response]
    O -- No --> Q[Fork copilot-service Go endpoint\nbest-effort]
    P --> Q

    Q --> R[publishStatusChanged chatPubSub]
    R --> S[Return id and optional failedFileCopies]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[POST /fork chatId] --> B{upToMessageId set?}
    B -- No --> C[Whole-chat duplicate\nAll files, title: name Copy]
    B -- Yes --> D[Branch fork\nSlice messages to cut point\ntitle: Fork name]

    C --> E[listDuplicableChatFiles\nALL chat-owned files\ncontext IN mothership,output]
    D --> F[listDuplicableChatFiles then filterForkableChatFiles\nmessageId IN keptSet OR NULL]

    E --> G[checkStorageQuota]
    F --> G

    G --> H[DB Transaction: INSERT copilot_chats]
    H --> I[planChatFileCopies\nINSERT workspace_files copies\nfresh id + key per row]
    I --> J[rewriteResourceFileRefs\nremap chips to new ids\ndrop ghost chat-owned refs]
    J --> K[rewriteMessageFileRefs\nremap transcript URLs and attachments]
    K --> L[appendCopilotChatMessages in tx]
    L --> M[COMMIT]

    M --> N[executeChatFileBlobCopies\nbounded concurrency 4\nbest-effort post-commit]
    N --> O{Any failed?}
    O -- Yes --> P[DELETE dead workspace_files rows\nremoveChatResources\nreturn failedFileCopies in response]
    O -- No --> Q[Fork copilot-service Go endpoint\nbest-effort]
    P --> Q

    Q --> R[publishStatusChanged chatPubSub]
    R --> S[Return id and optional failedFileCopies]

Loading

_{Reviews (9): Last reviewed commit: "test(contracts): pin mothership↔copilot ..." | Re-trigger Greptile}

[j15z]

@greptile

[j15z]

@cursor review

[j15z]

@greptile

[j15z]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9d8756c. Configure here.}

[j15z]

@greptile

[j15z]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

5 issues from previous reviews remain unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8afecd5. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4a2333a. Configure here.}

[cursor]

Outputs API hides list failures

Medium Severity

The new chat-outputs HTTP route calls listChatOutputs, which catches database errors and returns an empty array. The route then responds with success: true and files: [], so transient DB failures look like “no outputs” instead of a 500. That undermines useChatOutputs throwing on failure, because the client never sees an error to retry.

Additional Locations (1)

• apps/sim/app/api/mothership/chats/[chatId]/outputs/route.ts#L40-L42

 

^{Reviewed by Cursor Bugbot for commit 4a2333a. Configure here.}

[cursor]

Errored outputs query shows not found

Medium Severity

After useChatOutputs fails (network or 500), callers still default data to [] and never read isError. useResolvedEmbeddedFile then stops resolving once list and by-id lookups miss, so the embedded preview shows “File not found” instead of loading or error/retry—even when outputs exist.

Additional Locations (1)

• apps/sim/hooks/queries/workspace-files.ts#L96-L114

 

^{Reviewed by Cursor Bugbot for commit 4a2333a. Configure here.}