How to Set Up LeRobot: Expert AI Guide 2024

TL;DR

• Install LeRobot in 2 minutes with pip install lerobot and test with a pretrained policy on a simulated Franka arm.
• Record teleoperation data from a real robot (e.g., UR5) using the lerobot.record CLI tool.
• Train a custom policy from demonstrations with 3 lines of code using lerobot.train.
• Evaluate in <a href="/services/digital-twin-consulting">simulation</a> (Isaac Gym) or deploy to hardware (SO-100) with minimal code changes.
• Share datasets/models on Hugging Face Hub with lerobot.push_to_hub.

1. Installation and the LeRobot Dataset Format

Install LeRobot

Start with the official installation. For most users, the Python package is the fastest option:

pip install lerobot -U

Verify installation:

python -c "import lerobot; print(lerobot.__version__)"

Expected output:

0.5.0

GitHub Releases

Dataset Format Overview

LeRobot datasets follow the RLDS (Robot Learning Dataset Standard) format, but with Hugging Face-specific extensions. A minimal dataset directory structure looks like this:

my_dataset/
├── scenes/               # Optional: 3D scene configs
│   └── table.yaml
├── trajectories/         # Recorded demonstrations
│   ├── demo_0.h5         # HDF5 format (required)
│   ├── demo_1.h5
│   └── ...
└── metadata.json         # Dataset description (required)

Key fields in metadata.json:

{
  "description": "UR5 pick-and-place dataset",
  "robot": "ur5e",
  "tasks": ["pick", "place"],
  "sensors": ["rgb_camera", "joint_states"],
  "version": "0.1.0"
}

Dataset Guide

Common errors:

• Missing metadata.json: LeRobot will raise ValueError: Dataset metadata not found.
• Incorrect HDF5 structure: Ensure each .h5 file has /observations and /actions groups.

2. Loading Pretrained Policies

LeRobot provides 3 ready-to-use policy types: ACT (Action Chunk Transformer), Diffusion Policy, and SmolVLA (Small Vision-Language-Action). Here’s how to load and test them.

Load an ACT Policy

from lerobot import ACTPolicy

# Load pretrained ACT model (Franka arm, table-top manipulation)
policy = ACTPolicy.from_pretrained("lerobot/act3d-franka-table")

# Simulate a step (input: dict of observations)
observation = {
    "image": torch.randn(3, 256, 256),  # RGB image
    "joint_states": torch.randn(7),     # Arm joint angles
    "ee_pose": torch.randn(7)            # End-effector pose
}
action = policy(observation)
print(action.shape)  # Expected: torch.Size([7]) for joint torques

Model Cards

Load a Diffusion Policy

from lerobot import DiffusionPolicy

# Load diffusion policy for grasping
policy = DiffusionPolicy.from_pretrained("lerobot/diffusion_policy-grasp")

# Sample an action (stochastic)
action = policy.sample(observation)
print(action)  # Output: {"action": tensor([...]), "log_probs": tensor([...])}

Model Cards

Load SmolVLA

from lerobot import SmolVLA

# Load VLA model for language-conditioned tasks
vla = SmolVLA.from_pretrained("lerobot/smolvla-aloha")

# Query with text and image
query = {
    "image": torch.randn(3, 224, 224),
    "text": "pick up the red block",
    "joint_states": torch.randn(7)
}
action = vla(query)
print(action)  # Output: tensor([...]) for joint actions

Hugging Face Blog

Gotchas:

• ACT policies require observations in normalized format (check the model card for scaling).
• Diffusion policies are slower (~100ms per step) but more sample-efficient.
• SmolVLA needs text embeddings precomputed (use lerobot.tokenizers if missing).

3. Recording Teleoperation Datasets on a Real Arm

Use the lerobot.record CLI tool to collect data from a real robot. Below is a UR5 example.

Prerequisites

1. Install ROS 2 (Humble) and the UR driver:

   sudo apt install ros-humble-ur-driver
   

   Copy
2. Ensure your robot is on the network and reachable via ROS 2.

Record a Dataset

lerobot record \
  --robot ur5e \
  --sensors rgb_camera joint_states \
  --output_dir ./ur5_demos \
  --max_episodes 10

Hardware Docs

Expected output:

Recording dataset for UR5e...
Episode 1/10: Waiting for teleop input...
[ROS 2 node started: /lerobot_recorder]
[Recording to: ./ur5_demos/trajectories/episode_0.h5]
[Press Ctrl+C to end episode]

Teleoperation Modes

Common errors:

• ROS 2 connection failed: Verify ROS_DOMAIN_ID and ROS_MASTER_URI are set. Fix: Run export ROS_DOMAIN_ID=0 before recording.
• Sensor calibration: If RGB images are distorted, recalibrate the camera with ros2 camera_calibration.

4. Training a Policy from Demonstrations

Train a custom policy using the lerobot.train API. Below is a BC (Behavior Cloning) example.

Train from Scratch

from lerobot import BehaviorCloningTrainer
from lerobot.datasets import load_dataset

# Load your dataset
dataset = load_dataset("path/to/ur5_demos")

# Initialize trainer
trainer = BehaviorCloningTrainer(
    model="lerobot/act3d-base",  # Start from a pretrained base
    dataset=dataset,
    batch_size=32,
    epochs=50,
    lr=1e-4
)

# Train!
trainer.train()
trainer.push_to_hub("my-ur5-policy")  # Save to Hugging Face Hub

API Docs

Hyperparameter Tuning

Key parameters to adjust:

Gotchas:

• Overfitting: If training loss << validation loss, reduce model capacity or add data augmentation.
• Action scaling: Ensure actions in your dataset match the robot’s command range (e.g., [-1, 1] for torques).

5. Evaluating in Simulation and on Hardware

Evaluate in Isaac Gym

from lerobot.sim import IsaacGymEnv
from lerobot import ACTPolicy

# Load policy and env
policy = ACTPolicy.from_pretrained("my-ur5-policy")
env = IsaacGymEnv(robot="ur5e", task="pick_place")

# Run evaluation
success_rate = env.evaluate(policy, episodes=10)
print(f"Success rate: {success_rate * 100:.2f}%")

NVIDIA Developer

Expected output:

Episode 1/10: Success = True
Episode 2/10: Success = False
...

Deploy to Hardware (UR5 Example)

from lerobot.hardware import UR5Controller
from lerobot import ACTPolicy

# Initialize robot and policy
robot = UR5Controller()
policy = ACTPolicy.from_pretrained("my-ur5-policy")

# Run closed-loop control
while True:
    obs = robot.get_observations()
    action = policy(obs)
    robot.send_action(action)

Hardware Docs

Hardware Support Table:

Gotchas:

• Latency: Real-world deployment requires <50ms loop time. Use robot.set_control_rate(20) for 20Hz updates.
• Safety: Always enable force/torque limits and emergency stops in hardware mode.

6. Pushing Datasets and Models to the Hub

Share your work with the community using Hugging Face Hub.

Push a Dataset

lerobot push_to_hub \
  --path ./ur5_demos \
  --repo_id my-username/ur5-pick-place-demos \
  --private  # Optional: Set to False for public

Datasets Guide

Push a Model

from lerobot import ACTPolicy

# Load your trained policy
policy = ACTPolicy.from_pretrained("lerobot/act3d-base")
policy.train()  # Assume you've trained it

# Push to Hub
policy.push_to_hub("my-username/ur5-act-policy")

Model Hub

Hub Features:

• Model Cards: Auto-generated from your code (edit README.md in the repo).
• Datasets: Versioned and searchable (e.g., huggingface.co/datasets/my-username/ur5_demos).
• Spaces: Deploy interactive demos (see LeRobot Spaces).

Gotchas:

• Large files: Datasets >10GB may fail. Use lerobot push_to_hub --chunk_size 1G.
• License: Always specify a license (e.g., mit or apache-2.0) in metadata.json.

7. Hardware-Specific Guides

SO-100 (Official Support)

LeRobot includes native support for the SO-100 mobile manipulator.

Install SO-100 drivers:

pip install lerobot[so100]

Hardware Docs

Example: Teleoperate SO-100:

lerobot record \
  --robot so100 \
  --sensors rgb_camera base_pose \
  --output_dir ./so100_navigation

Koch <a href="/services/physical-ai"><a href="/services/physical-ai-robotics">robotics</a></a> Arms

For Koch K1/K2 arms, use the ROS 2 interface:

from lerobot.hardware import KochArm

robot = KochArm()
robot.move_to_joint_positions([0.5, -0.3, 0.0,