robotics-connect
Agent skills + verified-on-hardware control stacks that let an AI agent discover, operate, and
real-to-sim-stage a humanoid robot — from plugged-in on the bench, to driving it through a real task
with a human in the loop, to a sim-to-real-valid Isaac Lab RL job on a DGX Spark.
A projected ~8× fewer tokens to build the full bed-making demo with robotics-connect — an illustrative model, not a measurement (how it adds up).
⚠️ Before running any on-hardware control, read SAFETY.md. It is the
non-negotiable safety layer for commanding a real humanoid's motors — the stop hierarchy, the
de-risk ladder for deploying a whole-body policy, and the cardinal rule (never kill -9 a
low-level control process — it latches the last high-gain command and the robot runs away).
Every motor-command process must wrap its loop in lib/safe_stop.py.
robotics-connect is a collection of Claude Code Agent Skills plus the
verified-on-hardware control stacks they wrap. It has two pillars, both driven by the same robot
descriptor:
- Operate the real robot. Hardware-verified capability stacks an agent uses to actually run a task:
perceive a scene (depth + LiDAR, body-frame geometry), walk and navigate to it, speak to and
hear from a human partner over Device Connect, grip with the hands, and abort from the
controller — the layers behind a real bed-making G1 that asks a person for help when it gets stuck.
- Real-to-sim staging. Discover and characterize the robot's real sensor + effector envelope
into a machine-readable descriptor, stage a sensor-calibrated, free-base Isaac Lab RL job from it
(with deploy-safe observations — privileged terms to the critic only), and deploy the trained policy
back out into a control loop — and, through a safety-gated de-risk ladder, onto the real motors.
The same flow generalizes to a new humanoid by re-running the skills.
Token-efficient by design
A third goal, and a frequent ask from Arm's Physical AI partners: keep the agent's token and
context budget small. Operating a robot shouldn't mean re-deriving DDS plumbing, control loops,
joint-order maps, and safety procedures token-by-token on every run. robotics-connect pushes that
work out of the prompt and into deterministic, hardware-verified code:
- Progressive disclosure. Capabilities are Agent Skills: the agent sees a
one-line description and a concise
SKILL.md, and loads the full detail only for the skill it
actually invokes — the working context stays small no matter how large the stack grows.
- Code, not tokens. The verified
lib/ and capability stacks are the control, perception,
locomotion, deploy, and safety logic. The agent calls proven code instead of regenerating (and
re-reasoning about) it each session — fewer tokens per task, and more reliable behavior.
- One descriptor, reused. A single machine-readable
robot descriptor is the compact source
of truth every skill consumes, so the robot's envelope is characterized once, not re-prompted per
task or per robot.
The result is lower token cost and latency per task, and a control stack that fits comfortably
alongside on-device and smaller-context models — without giving up hardware-verified behavior.
How the ~8× in the headline chart adds up (a projected model, not a measurement): without
robotics-connect an agent re-derives each capability from primary sources and on-hardware debugging,
reloading context every session; with it, the agent loads a concise skill on demand, calls
verified code, and reuses one robot descriptor.
The real-to-sim thesis
Isaac Sim does not fully expose a real robot's sensor + effector envelope. The sensors are tilted,
range-limited, and occluded by the robot's own body in ways the sim won't tell you; the real degrees of
freedom, gains, and hand morphology differ from whatever asset you downloaded. robotics-connect
characterizes those on the hardware, and that characterization is calibration data an agent builds
the sim from — so a sim-trained policy/detector transfers, and the RL training cycles are spent on a
sim that already matches the robot. robotics-connect is the real-to-sim half that feeds sim-to-real.
