Scale media collection concurrency with available resources#11
Merged
Conversation
railway-app
Bot
temporarily deployed
to
impartial-energy / media-scraper-pr-11
July 18, 2026 03:13
Destroyed
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughChangesCollection concurrency
Railway CLI workflow documentation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant WorkerConfig
participant WorkerEntrypoint
participant BullMQWorker
WorkerConfig->>WorkerEntrypoint: provide COLLECTION_CONCURRENCY
WorkerEntrypoint->>WorkerEntrypoint: derive CPU and memory-based concurrency
WorkerEntrypoint->>BullMQWorker: configure collectionConcurrency
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
This PR increases the worker’s throughput by replacing a hard-coded single-collection BullMQ concurrency with a bounded, resource-aware default, while keeping storage retention serialized to avoid quota-decision overlap.
Changes:
- Adds
COLLECTION_CONCURRENCYto worker config with validation (optional; max 8). - Implements automatic worker concurrency sizing using container CPU/memory (capped at 4) and logs the chosen source.
- Updates local/dev documentation and Compose/env examples to describe and expose the new override.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents concurrent collection processing and clarifies retention serialization + sizing rationale. |
| packages/config/src/index.ts | Adds validated COLLECTION_CONCURRENCY support to the worker environment schema. |
| docker-compose.yml | Wires COLLECTION_CONCURRENCY into the worker service environment. |
| apps/worker/src/index.ts | Computes and applies dynamic BullMQ worker concurrency based on CPU/memory (with override). |
| .pi/skills/railway-cli/SKILL.md | Adds a project-local Railway CLI skill documenting safe operational workflows. |
| .env.example | Documents the COLLECTION_CONCURRENCY override behavior for local setups. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.pi/skills/railway-cli/SKILL.md:
- Line 75: Update the inspection workflow command list to remove the raw JSON
variable listing from railway variable inspection, or replace it with a
key-only/redacted variant that cannot expose secret values. Preserve the
remaining generic inspection commands unchanged.
- Around line 90-96: Update the detached deployment workflow to capture the
deploymentId returned by railway up --detach --json and poll that specific
deployment directly, rather than selecting the newest result from railway
deployment list. Preserve the existing SUCCESS, FAILED, CRASHED, and other-state
reporting behavior for the tracked deployment.
- Around line 110-120: Update the provisioning guidance in the Services and
infrastructure section so railway add --service and railway add --database
require explicit user confirmation immediately before execution. Preserve the
existing instruction to inspect services first, and make clear that confirmation
is required to prevent unintended billed resources.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1ec087ca-3632-4377-8bba-b831ff74838e
📒 Files selected for processing (6)
.env.example.pi/skills/railway-cli/SKILL.mdREADME.mdapps/worker/src/index.tsdocker-compose.ymlpackages/config/src/index.ts
railway-app
Bot
temporarily deployed
to
impartial-energy / media-scraper-pr-11
July 18, 2026 03:20
Destroyed
|
🚅 Deployed to the media-scraper-pr-11 environment in impartial-energy
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
railway-cliagent skill with current, safety-focused Railway workflowsCOLLECTION_CONCURRENCYoverride (1-8) for explicit tuningWhy
The worker used
concurrency: 1because collection completion triggers storage retention. The collection path itself is already safe to parallelize:Railway CLI inspection of the production
backendservice showed:Automatic sizing budgets two CPU threads and 1.5 GiB per collection, reserves 1.5 GiB for the co-located API/worker overhead, and caps concurrency at four. This resolves to four on the current Railway allocation and remains conservative on smaller containers.
Validation
pnpm lintpnpm typecheckpnpm buildgit diff --checkdocker compose config --quietcould not run because Docker is not installed in the current WSL environment; the Compose YAML passed Prettier validation.Summary by CodeRabbit
New Features
COLLECTION_CONCURRENCYoverride, capped at 8.Documentation
COLLECTION_CONCURRENCY(including the commented example).