Skip to content

feat: add external python module deployment#2869

Draft
TomCC7 wants to merge 18 commits into
mainfrom
cc/feat/external-python-module
Draft

feat: add external python module deployment#2869
TomCC7 wants to merge 18 commits into
mainfrom
cc/feat/external-python-module

Conversation

@TomCC7

@TomCC7 TomCC7 commented Jul 11, 2026

Copy link
Copy Markdown
Member

Problem

DimOS modules currently run inside the main Python environment, which makes it hard to integrate modules that need conflicting or heavyweight package dependencies while still using coordinator wiring, RPC, streams, lifecycle management, and docs-tested examples.

Solution

  • Add an experimental local packaged-Python external-python deployment path with coordinator-visible declarations and runtime launch envelopes.
  • Add WorkerManagerExternal, deployment planning/reference resolution, runtime entrypoint, and a temporary plan/prepare/run launcher.
  • Support uv-only and Pixi+uv local package layouts; prepare materializes the external package environment before run.
  • Add a runnable example package proving dependency isolation with humanize, declared RPC calls, module refs, and stream roundtrip behavior.
  • Document authoring, contributor testing, and coding-agent conventions for external Python modules.

Testing

  • uv run ruff check dimos/core/deployment dimos/core/coordination/worker_manager_external.py examples/external-python-module/**/*.py
  • uv run pytest dimos/core/deployment/test_deployment.py dimos/core/coordination/test_module_coordinator.py::test_build_happy_path -q
  • PYTHONPATH="examples/external-python-module" uv run python -c 'from dimos.core.coordination.module_coordinator import ModuleCoordinator; from dimos.core.deployment.ref import resolve_deployment_ref; from deployment import ExampleClient, ExampleExternalDeclaration; spec = resolve_deployment_ref("deployment:deployment_spec"); coordinator = ModuleCoordinator.build_deployment(spec); print(coordinator.get_instance(ExampleExternalDeclaration).greet("qa")); print(coordinator.get_instance(ExampleClient).call_external_dependency("qa")); print(coordinator.get_instance(ExampleClient).roundtrip_stream("stream-qa")); coordinator.stop()'
  • timeout 10s env PYTHONPATH="examples/external-python-module" uv run python -m dimos.core.deployment.launcher run deployment:deployment_spec
  • uv run --with mypy mypy dimos/core/module.py dimos/core/deployment dimos/core/coordination/worker_manager_external.py
  • uv run doclinks --check docs/usage/modules.md docs/usage/external-python-modules.md docs/development/external_python_modules.md docs/coding-agents/index.md docs/coding-agents/external-python-modules.md examples/external-python-module/README.md
  • openspec validate "external-python-module-deployment" --type change --strict --no-interactive

@mintlify

mintlify Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
dimensional 🟢 Ready View Preview Jul 11, 2026, 6:56 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@codecov

codecov Bot commented Jul 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 65.39589% with 236 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/core/coordination/external_worker.py 30.53% 157 Missing ⚠️
dimos/core/coordination/worker_manager_external.py 69.79% 23 Missing and 6 partials ⚠️
dimos/core/deployment/test_deployment.py 87.09% 18 Missing and 2 partials ⚠️
dimos/core/deployment/models.py 89.24% 10 Missing ⚠️
dimos/core/deployment/ref.py 55.55% 5 Missing and 3 partials ⚠️
dimos/core/coordination/module_coordinator.py 57.14% 4 Missing and 2 partials ⚠️
dimos/core/deployment/planner.py 91.04% 3 Missing and 3 partials ⚠️
@@            Coverage Diff             @@
##             main    #2869      +/-   ##
==========================================
+ Coverage   72.21%   72.35%   +0.13%     
==========================================
  Files        1001     1005       +4     
  Lines       89755    89915     +160     
  Branches     8196     8388     +192     
==========================================
+ Hits        64818    65055     +237     
+ Misses      22718    22646      -72     
+ Partials     2219     2214       -5     
Flag Coverage Δ
OS-ubuntu-24.04-arm 65.22% <65.39%> (+0.12%) ⬆️
OS-ubuntu-latest 67.62% <65.39%> (+0.13%) ⬆️
Py-3.10 67.61% <65.39%> (+0.13%) ⬆️
Py-3.11 67.62% <65.39%> (+0.12%) ⬆️
Py-3.12 67.62% <65.39%> (+0.13%) ⬆️
Py-3.13 67.61% <65.39%> (+0.11%) ⬆️
Py-3.14 67.62% <65.39%> (+0.12%) ⬆️
Py-3.14t 67.61% <65.39%> (+0.12%) ⬆️
SelfHosted-Large 30.19% <37.09%> (+0.12%) ⬆️
SelfHosted-Linux 37.56% <36.51%> (+0.15%) ⬆️
SelfHosted-macOS 36.41% <36.51%> (+0.59%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/core/deployment/target_session.py 100.00% <100.00%> (ø)
dimos/core/module.py 77.82% <100.00%> (ø)
dimos/core/coordination/module_coordinator.py 83.42% <57.14%> (-0.74%) ⬇️
dimos/core/deployment/planner.py 91.04% <91.04%> (ø)
dimos/core/deployment/ref.py 55.55% <55.55%> (ø)
dimos/core/deployment/models.py 89.24% <89.24%> (ø)
dimos/core/deployment/test_deployment.py 87.09% <87.09%> (ø)
dimos/core/coordination/worker_manager_external.py 69.79% <69.79%> (ø)
dimos/core/coordination/external_worker.py 30.53% <30.53%> (ø)

... and 22 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread dimos/core/deployment/launcher.py Outdated
Comment thread dimos/core/deployment/models.py Outdated
Comment thread dimos/core/deployment/models.py Outdated
Comment thread dimos/core/deployment/ref.py
Comment thread dimos/core/coordination/module_coordinator.py Outdated
Comment thread dimos/core/coordination/module_coordinator.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant