TL;DR
- Install LeRobot in 2 minutes with
pip install lerobotand test with a pretrained policy on a simulated Franka arm. - Record teleoperation data from a real robot (e.g., UR5) using the
lerobot.recordCLI tool. - Train a custom policy from demonstrations with 3 lines of code using
lerobot.train. - Evaluate in simulation (Isaac Gym) or deploy to hardware (SO-100) with minimal code changes.
- Share datasets/models on Hugging Face Hub with
lerobot.push_to_hub.
1. Installation and the LeRobot Dataset Format
Install LeRobot
Start with the official installation. For most users, the Python package is the fastest option:
pip install lerobot -U
Verify installation:
python -c "import lerobot; print(lerobot.__version__)"
Expected output:
0.5.0
Dataset Format Overview
LeRobot datasets follow the RLDS (Robot Learning Dataset Standard) format, but with Hugging Face-specific extensions. A minimal dataset directory structure looks like this:
my_dataset/
├── scenes/ # Optional: 3D scene configs
│ └── table.yaml
├── trajectories/ # Recorded demonstrations
│ ├── demo_0.h5 # HDF5 format (required)
│ ├── demo_1.h5
│ └── ...
└── metadata.json # Dataset description (required)
Key fields in metadata.json:
{
"description": "UR5 pick-and-place dataset",
"robot": "ur5e",
"tasks": ["pick", "place"],
"sensors": ["rgb_camera", "joint_states"],
"version": "0.1.0"
}
Common errors:
- Missing
metadata.json: LeRobot will raiseValueError: Dataset metadata not found. - Incorrect HDF5 structure: Ensure each
.h5file has/observationsand/actionsgroups.
2. Loading Pretrained Policies
LeRobot provides 3 ready-to-use policy types: ACT (Action Chunk Transformer), Diffusion Policy, and SmolVLA (Small Vision-Language-Action). Here’s how to load and test them.
Load an ACT Policy
from lerobot import ACTPolicy
# Load pretrained ACT model (Franka arm, table-top manipulation)
policy = ACTPolicy.from_pretrained("lerobot/act3d-franka-table")
# Simulate a step (input: dict of observations)
observation = {
"image": torch.randn(3, 256, 256), # RGB image
"joint_states": torch.randn(7), # Arm joint angles
"ee_pose": torch.randn(7) # End-effector pose
}
action = policy(observation)
print(action.shape) # Expected: torch.Size([7]) for joint torques
Load a Diffusion Policy
from lerobot import DiffusionPolicy
# Load diffusion policy for grasping
policy = DiffusionPolicy.from_pretrained("lerobot/diffusion_policy-grasp")
# Sample an action (stochastic)
action = policy.sample(observation)
print(action) # Output: {"action": tensor([...]), "log_probs": tensor([...])}
Load SmolVLA
from lerobot import SmolVLA
# Load VLA model for language-conditioned tasks
vla = SmolVLA.from_pretrained("lerobot/smolvla-aloha")
# Query with text and image
query = {
"image": torch.randn(3, 224, 224),
"text": "pick up the red block",
"joint_states": torch.randn(7)
}
action = vla(query)
print(action) # Output: tensor([...]) for joint actions
Gotchas:
- ACT policies require observations in normalized format (check the model card for scaling).
- Diffusion policies are slower (~100ms per step) but more sample-efficient.
- SmolVLA needs text embeddings precomputed (use
lerobot.tokenizersif missing).
3. Recording Teleoperation Datasets on a Real Arm
Use the lerobot.record CLI tool to collect data from a real robot. Below is a UR5 example.
Prerequisites
- Install ROS 2 (Humble) and the UR driver:
sudo apt install ros-humble-ur-driver - Ensure your robot is on the network and reachable via ROS 2.
Record a Dataset
lerobot record \
--robot ur5e \
--sensors rgb_camera joint_states \
--output_dir ./ur5_demos \
--max_episodes 10
Expected output:
Recording dataset for UR5e...
Episode 1/10: Waiting for teleop input...
[ROS 2 node started: /lerobot_recorder]
[Recording to: ./ur5_demos/trajectories/episode_0.h5]
[Press Ctrl+C to end episode]
Teleoperation Modes
| Mode | Command Flag | Description |
|---|---|---|
| Keyboard | --teleop_mode keyboard | Use arrow keys to control the robot. |
| Joystick | --teleop_mode joystick | Requires python-joystick package. |
| Gamepad | --teleop_mode gamepad | Works with XInput-compatible controllers. |
| Sensors Guide |
Common errors:
- ROS 2 connection failed: Verify
ROS_DOMAIN_IDandROS_MASTER_URIare set. Fix: Runexport ROS_DOMAIN_ID=0before recording. - Sensor calibration: If RGB images are distorted, recalibrate the camera with
ros2 camera_calibration.
4. Training a Policy from Demonstrations
Train a custom policy using the lerobot.train API. Below is a BC (Behavior Cloning) example.
Train from Scratch
from lerobot import BehaviorCloningTrainer
from lerobot.datasets import load_dataset
# Load your dataset
dataset = load_dataset("path/to/ur5_demos")
# Initialize trainer
trainer = BehaviorCloningTrainer(
model="lerobot/act3d-base", # Start from a pretrained base
dataset=dataset,
batch_size=32,
epochs=50,
lr=1e-4
)
# Train!
trainer.train()
trainer.push_to_hub("my-ur5-policy") # Save to Hugging Face Hub
Hyperparameter Tuning
Key parameters to adjust:
| Parameter | Default | Recommended Range | Description |
|---|---|---|---|
batch_size | 32 | 16–128 | Larger batches = faster training. |
lr | 1e-4 | 1e-5 to 1e-3 | Too high = instability; too low = slow. |
epochs | 50 | 20–200 | Early stopping if validation loss plateaus. |
kl_weight | 0.1 | 0.01–0.5 | Only for RL fine-tuning. |
Gotchas:
- Overfitting: If training loss << validation loss, reduce model capacity or add data augmentation.
- Action scaling: Ensure actions in your dataset match the robot’s command range (e.g.,
[-1, 1]for torques).
5. Evaluating in Simulation and on Hardware
Evaluate in Isaac Gym
from lerobot.sim import IsaacGymEnv
from lerobot import ACTPolicy
# Load policy and env
policy = ACTPolicy.from_pretrained("my-ur5-policy")
env = IsaacGymEnv(robot="ur5e", task="pick_place")
# Run evaluation
success_rate = env.evaluate(policy, episodes=10)
print(f"Success rate: {success_rate * 100:.2f}%")
Expected output:
Episode 1/10: Success = True
Episode 2/10: Success = False
...
Deploy to Hardware (UR5 Example)
from lerobot.hardware import UR5Controller
from lerobot import ACTPolicy
# Initialize robot and policy
robot = UR5Controller()
policy = ACTPolicy.from_pretrained("my-ur5-policy")
# Run closed-loop control
while True:
obs = robot.get_observations()
action = policy(obs)
robot.send_action(action)
Hardware Support Table:
| Robot | LeRobot Support | Notes |
|---|---|---|
| UR5/UR10 | ✅ | ROS 2 driver required. |
| Franka Emika | ✅ | Works with franka_ros2 package. |
| SO-100 | ✅ | Official support (see below). |
| Stretch RE1 | ✅ | Requires stretch_ros2 driver. |
| Aloha | ✅ | Experimental (2026-05). |
| Hardware Docs |
Gotchas:
- Latency: Real-world deployment requires <50ms loop time. Use
robot.set_control_rate(20)for 20Hz updates. - Safety: Always enable force/torque limits and emergency stops in hardware mode.
6. Pushing Datasets and Models to the Hub
Share your work with the community using Hugging Face Hub.
Push a Dataset
lerobot push_to_hub \
--path ./ur5_demos \
--repo_id my-username/ur5-pick-place-demos \
--private # Optional: Set to False for public
Push a Model
from lerobot import ACTPolicy
# Load your trained policy
policy = ACTPolicy.from_pretrained("lerobot/act3d-base")
policy.train() # Assume you've trained it
# Push to Hub
policy.push_to_hub("my-username/ur5-act-policy")
Hub Features:
- Model Cards: Auto-generated from your code (edit
README.mdin the repo). - Datasets: Versioned and searchable (e.g.,
huggingface.co/datasets/my-username/ur5_demos). - Spaces: Deploy interactive demos (see LeRobot Spaces).
Gotchas:
- Large files: Datasets >10GB may fail. Use
lerobot push_to_hub --chunk_size 1G. - License: Always specify a license (e.g.,
mitorapache-2.0) inmetadata.json.
7. Hardware-Specific Guides
SO-100 (Official Support)
LeRobot includes native support for the SO-100 mobile manipulator.
Install SO-100 drivers:
pip install lerobot[so100]
Example: Teleoperate SO-100:
lerobot record \
--robot so100 \
--sensors rgb_camera base_pose \
--output_dir ./so100_navigation
Koch Robotics Arms
For Koch K1/K2 arms, use the ROS 2 interface:
from lerobot.hardware import KochArm
robot = KochArm()
robot.move_to_joint_positions([0.5, -0.3, 0.0,
