Skip to content

Capability · applied inside an engagement

Robotics, VLMs & VLAs

From perception to action — vision-language and vision-language-action models, on real robots, with a deterministic safety boundary.

Hyperion does not position itself as a general engineering bench. Technical fluency is used to make better product decisions, challenge assumptions, define evidence and coordinate the critical specialists required by the mandate.

Pixel-art scene of a robot arm turning a language instruction into a grounded pick-and-place action.
  1. Perception

  2. Policy (VLA)

  3. Control

  4. Safety monitor

From sensors to motion, behind an independent safety monitor.

A four-stage pipeline: perception, then the policy (a vision-language-action model), then control, then an independent safety monitor.

A robot that works in a demo is not one you can deploy. The consequential choices — VLM or VLA, what a simulation is allowed to prove, where the deterministic safety boundary sits — are product decisions before they are engineering ones. Hyperion owns them, names the evidence that closes each one, and coordinates the specialists who implement the result.

The sensor-to-action pipeline (perception → policy → control), the VLM-vs-VLA split (a VLM perceives and reasons; a VLA outputs actions), a training and evaluation loop with simulation and sim-to-real, and a deterministic safety boundary — the policy proposes, an independent monitor constrains.

ROS 2; vision-language-action policies (π-family, SmolVLA, NVIDIA Isaac GR00T) and VLMs; LeRobot for data, training and evaluation; teleoperation for demonstrations; simulation (Isaac Sim/Lab) and sim-to-real; fine-tuning (LoRA/QLoRA) and closed-loop evaluation; runtime safety monitor, operating envelope and emergency stop.

Perception to action, with a boundary that fails safe

Learned models propose; a deterministic boundary disposes. Every motor command passes an independent safety monitor before it reaches hardware.

  • Input
  • Deterministic
  • Learned
  • Safety
  • Output
Perception to action, with a boundary that fails safeA pixel topology: sensors feed perception; learned VLM and VLA models propose an action; a deterministic controller and an independent safety monitor, inside a safety and cybersecurity membrane, accept or block every command before it reaches the robot; results feed back to perception.

Flow from sensors through perception, VLM and VLA, planner and controller, to an independent safety monitor that gates the actuators, with feedback to the sensors.

  1. InputSensorsCameras, depth, force-torque, proprioception
  2. DeterministicPerceptionFusion, SLAM, 6-DoF pose
  3. LearnedVLMGrounds the goal — "what am I looking at?"
  4. LearnedVLAGenerates action chunks on hardware
  5. DeterministicPlanner & ControlTrajectory and ~1 kHz control
  6. SafetySafety monitorIndependent, out-of-band envelope and e-stop
  7. OutputActuatorsMotion in the physical world

Deterministic safety boundary

Only the safety monitor's ACCEPT lets a command reach the actuators; REJECT overrides to a safe state.

Closed loop: actuator and environment state feed back to the sensors.

Illustrative control-loop budgets: reflex/safety ~1 kHz · perception ~10–40 ms · policy ~20–100 ms · VLM ~1 Hz.

VLM reasons. VLA acts.

Two model classes with different jobs. Only the VLA's output reaches motors — so only the VLA's output must pass the deterministic safety boundary.

VLM grounds the goal the VLA executes. Model examples are illustrative of open classes (e.g. Pixtral-class VLMs; π0 / SmolVLA-class VLAs).
DimensionVLM — Vision-Language ModelVLA — Vision-Language-Action
OutputText / structured languageRobot actions / action chunks
Question it answers"What am I looking at?""What should the joints do now?"
RoleHigh-level perception, reasoning, groundingCloses the loop on hardware
Typical rate~1 Hz / on-demand~10–50 Hz
Trained onImage-text pairsTeleoperation demos (+ Open X-Embodiment)
Failure modeHallucinated planUnsafe motion — mandates a safety monitor

What we measured

  • Measured

    68.71 GB

    π0.5 full fine-tune — peak memory
  • Measured

    1.08 → 0.13 (300 steps)

    Real fine-tune (LIBERO, language-conditioned) — loss
  • Simulation

    0% → 96%

    Simulation success (LIBERO-Spatial) — base → fine-tuned
  • On-robot, dry-run

    12 / 12

    On-robot cognitive benchmark (12 scenes) — perceive · understand · interpret
  • Measured

    2.3× – 14.3×

    bf16 throughput vs FP32 (ACT)

Not yet measured

  • On-robot task success rate — the arm loop is dry-run (no physical motion yet).
  • NPU (XDNA2) execution — the INT8 toolchain validates, but no DPU subgraph compiled; iGPU/CPU fallback only.
  • Convergence (the fine-tune stops at 300 steps), energy, and multi-task generalisation.

Discuss embodied-AI work