feat(hosted-teleop): Go2 teleoperation over Cloudflare/LiveKit broker#2798
Open
ruthwikdasyam wants to merge 232 commits into
Open
feat(hosted-teleop): Go2 teleoperation over Cloudflare/LiveKit broker#2798ruthwikdasyam wants to merge 232 commits into
ruthwikdasyam wants to merge 232 commits into
Conversation
…e SFU Implements a new pubsub transport backed by WebRTC DataChannels over Cloudflare's Realtime SFU. Two new classes in dimos/protocol/pubsub/impl/webrtcpubsub.py: - CloudflareSession: manages the WebRTC PeerConnection lifecycle. Opens two CF sessions (publisher + subscriber) so a single process can do loopback pubsub. Runs aiortc on a dedicated background asyncio thread with its own ThreadPoolExecutor (so we don't leak asyncio_N worker threads). Uses negotiated=True placeholder DCs with id=100 during transport establishment to avoid stream-id collisions with CF-assigned ids. - WebRTCPubSub: bytes-on-the-wire pubsub facade matching the LCMPubSubBase / BytesSharedMemory interface (string topics, bytes payloads). Lazily creates pub/sub DataChannel pairs on first publish/subscribe per topic. Also adds: - WebRTCTransport in dimos/core/transport.py (mirrors LCMTransport pattern, no encoding - bytes only). - WebRTC benchmark testcase in dimos/protocol/pubsub/benchmark/testdata.py, gated on aiortc + CF_TELEOP_APP_ID / CF_TELEOP_APP_SECRET env vars. - Integration test in dimos/protocol/pubsub/impl/test_webrtcpubsub.py covering basic pub/sub, latency, and throughput (all live tests skip without CF credentials). - aiortc + httpx as new 'webrtc' optional extra in pyproject.toml. Live benchmark (us-east-2 -> CF edge): - 64-256B: ~10K msgs/s, 0% loss - 1KiB: ~7K msgs/s, 0% loss - >= 64KiB: dropped (above SCTP message size) - Median single-RTT: ~2.5 ms
…sport - Add BrokerProvider: DataChannelProvider that works through the hosted teleop broker (dimensional-teleop) instead of directly with CF credentials. Handles session registration, heartbeat loop, and DataChannel creation when an operator joins via the broker's bridge-datachannel API. - Extend WebRTCTransport with optional msg_type parameter for typed LCM encode/decode with fingerprint-based filtering. Multiple transports can share a single multiplexed DataChannel and each receives only its type. - Add hosted teleop blueprints (dimos/teleop/hosted/) demonstrating the module-free architecture: make_teleop_hosted_go2() uses pure transport (zero modules), make_teleop_hosted_go2_scaled() adds a thin TeleopScalerModule for speed scaling only. - Add unit tests for typed mode, fingerprint filtering, multiplexed dispatch, and BrokerProvider credential validation.
- Rebase on main and regenerate uv.lock (resolve conflict) - Add _LoopbackProvider (in-process, no network) to benchmark testdata - Enables local WebRTC transport benchmarking without CF credentials - All 12 message sizes pass locally (2.78s total)
The previous lock regen dropped the `exclude-newer-span` marker, leaving only the frozen `exclude-newer` timestamp. uv then treats every resolve as "cooldown was newly added" and forces a re-resolve against today minus 7 days — which currently excludes md-babel-py 1.2.0 (published 2026-05-15) and breaks `uv sync --extra all` / `uv lock`. Re-adding the span line tells uv the lock was generated with P7D semantics, so the existing pinned versions are honored.
- Remove __init__.py files (project policy: no init files) - Remove section markers from test_webrtcpubsub.py - Regenerate all_blueprints.py (adds TeleopScalerModule) - Fix WebRTCTransport.__reduce__ to preserve msg_type across pickle - Fix CloudflareProvider.publish() race: snapshot loop ref before use - Fix CloudflareProvider.subscribe() race: check sub_channels inside lock - Add comment clarifying TwistStamped→Twist type safety in blueprint
- Add return type annotations to CloudflareProvider event handlers - Fix type: ignore codes to match actual mypy errors (attr-defined) - Add type annotation to __setstate__ dict parameter - Add type: ignore[arg-type] for WebRTCPubSub→PubSub duck typing in benchmark - Remove TwistStamped subclass comment from blueprint
ruthwikdasyam
requested review from
leshy,
mustafab0,
paul-nechifor and
spomichter
as code owners
Contributor
Greptile SummaryThis PR introduces robot-side hosted teleoperation for the Unitree Go2 over WebRTC via the dimensional-teleop broker, supporting both Cloudflare Realtime and LiveKit SFUs. It replaces the monolithic
Confidence Score: 4/5Safe to merge with one targeted fix to the drive guard in Go2CommandModule — a NaN twist timestamp silently disables the out-of-order filter for the rest of the session. The drive guard in dimos/teleop/hosted/go2_command.py — Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant Op as Operator Browser
participant Broker as CF/LiveKit Broker
participant Prov as BrokerProvider
participant Cmd as Go2CommandModule
participant Mux as CameraMuxModule
participant MM as MovementManager
participant Go2 as GO2Connection
Op->>Broker: cmd_unreliable (TwistStamped)
Broker->>Prov: DataChannel
Prov->>Cmd: _on_cmd_vel_in (stale/future/reorder guard)
Cmd->>MM: tele_cmd_vel (guarded Twist)
MM->>Go2: cmd_vel
Op->>Broker: state_reliable (estop / sport_cmd)
Broker->>Prov: DataChannel message
Prov->>Cmd: _on_state_json
Cmd->>Go2: sport_command() via RPC
Go2->>Mux: color_image
Mux->>Prov: mux_image set_video_frame
Prov->>Broker: video track
Broker->>Op: WebRTC video
Op->>Broker: ping
Broker->>Prov: state_reliable
Prov->>Prov: _maybe_answer_ping inline
Prov->>Broker: pong state_reliable_back
Broker->>Op: pong
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant Op as Operator Browser
participant Broker as CF/LiveKit Broker
participant Prov as BrokerProvider
participant Cmd as Go2CommandModule
participant Mux as CameraMuxModule
participant MM as MovementManager
participant Go2 as GO2Connection
Op->>Broker: cmd_unreliable (TwistStamped)
Broker->>Prov: DataChannel
Prov->>Cmd: _on_cmd_vel_in (stale/future/reorder guard)
Cmd->>MM: tele_cmd_vel (guarded Twist)
MM->>Go2: cmd_vel
Op->>Broker: state_reliable (estop / sport_cmd)
Broker->>Prov: DataChannel message
Prov->>Cmd: _on_state_json
Cmd->>Go2: sport_command() via RPC
Go2->>Mux: color_image
Mux->>Prov: mux_image set_video_frame
Prov->>Broker: video track
Broker->>Op: WebRTC video
Op->>Broker: ping
Broker->>Prov: state_reliable
Prov->>Prov: _maybe_answer_ping inline
Prov->>Broker: pong state_reliable_back
Broker->>Op: pong
Reviews (7): Last reviewed commit: "fix(hosted-teleop): publish cmd_vel_stam..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
github-actions
Bot
added
the
ready-to-merge
…nd filter + video encoding, truthful obstacle-avoidance ack, camera_select guard
github-actions
Bot
added
ready-to-merge
spomichter
requested changes
Jul 10, 2026
| teleop_hosted_go2_transport = ( | ||
| autoconnect( | ||
| unitree_go2_basic.disabled_modules(GO2Connection), | ||
| Go2HostedConnection.blueprint(), |
Contributor
There was a problem hiding this comment.
No need for hostedconnection module? handled at transport layer thats the point of webrtctransport
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
Contributor
There was a problem hiding this comment.
can move this file to somewhere other than /quest_hosted, since blurpints in here ar enot quest related
| @@ -15,20 +15,18 @@ | |||
| """Hosted Teleop Module — Cloudflare Realtime SFU client. | |||
Contributor
There was a problem hiding this comment.
Remember to delete this in a later PR
spomichter
requested changes
Jul 10, 2026
| allow_acrobatics: bool = False # let the operator trigger FrontJump/Pounce | ||
|
|
||
|
|
||
| class Go2HostedConnection(GO2Connection, HostedConnectionMixin): |
Contributor
There was a problem hiding this comment.
SHould be deleted. Should be a HostedStatsModule that just does lightweight stats publishing to webrtc. The issue is from ruthwik is that if this is a seperate module it starts two cloudflare sessions, whichis a problem on my end. Also This file handles audio, which should be handled seperately as well. May also need to paramertize cloudflare datachannels at the blueprint layer which was on the TODO list anyway
github-actions
Bot
removed
the
ready-to-merge
Comment on lines
+409
to
+422
| ts = float(twist.ts) | ||
| age = time.time() - ts | ||
| if age > self.config.cmd_stale_after_sec: | ||
| logger.debug("dropping stale cmd_vel: age=%.3fs", age) | ||
| return | ||
| if age < 0: # future-stamped (clock skew / corrupt ts): don't let it | ||
| # advance _last_cmd_ts, or every in-order frame until wall-clock | ||
| # catches up would be dropped as "out-of-order" (drive stall). | ||
| logger.debug("dropping future-stamped cmd_vel: age=%.3fs", age) | ||
| return | ||
| if ts <= self._last_cmd_ts: | ||
| logger.debug("dropping out-of-order cmd_vel: ts=%.3f last=%.3f", ts, self._last_cmd_ts) | ||
| return | ||
| self._last_cmd_ts = ts |
Contributor
There was a problem hiding this comment.
If twist.ts decodes as NaN (e.g. a browser bug, or a malformed LCM field), then ts = float(NaN) propagates through every comparison silently: NaN > 0.5 → False (not stale), NaN < 0 → False (not future), NaN <= self._last_cmd_ts → False (not out-of-order), so the frame is forwarded and self._last_cmd_ts = NaN. From that point on, ts <= NaN is False for every valid timestamp, so the reorder filter is permanently disabled for the rest of the session. Reordered UDP drive frames that arrive late will no longer be dropped and may drive the robot in an unexpected direction.
By contrast, _handle_nav_goal and _handle_light both call math.isfinite on operator-supplied floats. The same guard is missing here. Add if not math.isfinite(ts): logger.debug("dropping non-finite ts"); return immediately after ts = float(twist.ts) to keep the safety property intact across all inputs.
github-actions
Bot
added
the
ready-to-merge
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Teleoperating a robot over the internet (not just LAN) needs a dial-out architecture: robots behind NAT/cellular can't accept inbound connections, so they must connect out to a broker, and browser/VR operators reach them through it. dimos had no such path — only the LAN-only quest WebSocket teleop.
Closes DIM-XXX
Solution
Robot-side hosted teleoperation over WebRTC, via the dimensional-teleop broker (Cloudflare Realtime or LiveKit SFU). One session per robot carries video, commands, telemetry and safety.
Transport layer - CloudflareTransport / LiveKitTransport (+ *VideoTransport) bind directly to blueprint streams; a per-process BrokerProvider (CF) / LiveKitBrokerProvider owns the single session. Commands, state and video all ride it - no separate teleop module owning its own PeerConnection.
Go2HostedConnection - colocates every broker-bound stream on one module (the driver is dedicated_worker, so they must share a process to share a session). Adds:
Clock-sync ping/pong is answered inline in the provider (no module hop, so RTT/offset stay jitter-free). Legacy HostedTeleopModule is deprecated. Arm/manipulation hosted teleop was split out to a follow-up branch to keep this Go2-focused.
Coverage: unit tests for the command plane, camera mux, clock-sync ping, stream/video stats, speaker track, and the LiveKit frame packing. The live WebRTC session paths are exercised by the opt-in broker e2e/benchmark cases (they need a real SFU + operator), not unit tests.
How to Test
Contributor License Agreement