How to Set Up LeRobot (Hugging Face): A Practical Guide

TL;DR

Install LeRobot in 2 minutes with pip install lerobot and test with a pretrained policy on a simulated Franka arm.
Record teleoperation data from a real robot (e.g., UR5) using the lerobot.record CLI tool.
Train a custom policy from demonstrations with 3 lines of code using lerobot.train.
Evaluate in simulation (Isaac Gym) or deploy to hardware (SO-100) with minimal code changes.
Share datasets/models on Hugging Face Hub with lerobot.push_to_hub.

1. Installation and the LeRobot Dataset Format

Install LeRobot

Start with the official installation. For most users, the Python package is the fastest option:

pip install lerobot -U

Verify installation:

python -c "import lerobot; print(lerobot.__version__)"

Expected output:

0.5.0

GitHub Releases

Dataset Format Overview

LeRobot datasets follow the RLDS (Robot Learning Dataset Standard) format, but with Hugging Face-specific extensions. A minimal dataset directory structure looks like this:

my_dataset/
├── scenes/               # Optional: 3D scene configs
│   └── table.yaml
├── trajectories/         # Recorded demonstrations
│   ├── demo_0.h5         # HDF5 format (required)
│   ├── demo_1.h5
│   └── ...
└── metadata.json         # Dataset description (required)

Key fields in metadata.json:

{
  "description": "UR5 pick-and-place dataset",
  "robot": "ur5e",
  "tasks": ["pick", "place"],
  "sensors": ["rgb_camera", "joint_states"],
  "version": "0.1.0"
}

Dataset Guide

Common errors:

Missing metadata.json: LeRobot will raise ValueError: Dataset metadata not found.
Incorrect HDF5 structure: Ensure each .h5 file has /observations and /actions groups.

2. Loading Pretrained Policies

LeRobot provides 3 ready-to-use policy types: ACT (Action Chunk Transformer), Diffusion Policy, and SmolVLA (Small Vision-Language-Action). Here’s how to load and test them.

Load an ACT Policy

from lerobot import ACTPolicy

# Load pretrained ACT model (Franka arm, table-top manipulation)
policy = ACTPolicy.from_pretrained("lerobot/act3d-franka-table")

# Simulate a step (input: dict of observations)
observation = {
    "image": torch.randn(3, 256, 256),  # RGB image
    "joint_states": torch.randn(7),     # Arm joint angles
    "ee_pose": torch.randn(7)            # End-effector pose
}
action = policy(observation)
print(action.shape)  # Expected: torch.Size([7]) for joint torques

Model Cards

Load a Diffusion Policy

from lerobot import DiffusionPolicy

# Load diffusion policy for grasping
policy = DiffusionPolicy.from_pretrained("lerobot/diffusion_policy-grasp")

# Sample an action (stochastic)
action = policy.sample(observation)
print(action)  # Output: {"action": tensor([...]), "log_probs": tensor([...])}

Model Cards

Load SmolVLA

from lerobot import SmolVLA

# Load VLA model for language-conditioned tasks
vla = SmolVLA.from_pretrained("lerobot/smolvla-aloha")

# Query with text and image
query = {
    "image": torch.randn(3, 224, 224),
    "text": "pick up the red block",
    "joint_states": torch.randn(7)
}
action = vla(query)
print(action)  # Output: tensor([...]) for joint actions

Hugging Face Blog

Gotchas:

ACT policies require observations in normalized format (check the model card for scaling).
Diffusion policies are slower (~100ms per step) but more sample-efficient.
SmolVLA needs text embeddings precomputed (use lerobot.tokenizers if missing).

3. Recording Teleoperation Datasets on a Real Arm

Use the lerobot.record CLI tool to collect data from a real robot. Below is a UR5 example.

Prerequisites

Install ROS 2 (Humble) and the UR driver:
```
sudo apt install ros-humble-ur-driver
```
Ensure your robot is on the network and reachable via ROS 2.

Record a Dataset

lerobot record \
  --robot ur5e \
  --sensors rgb_camera joint_states \
  --output_dir ./ur5_demos \
  --max_episodes 10

Hardware Docs

Expected output:

Recording dataset for UR5e...
Episode 1/10: Waiting for teleop input...
[ROS 2 node started: /lerobot_recorder]
[Recording to: ./ur5_demos/trajectories/episode_0.h5]
[Press Ctrl+C to end episode]

Teleoperation Modes

Mode	Command Flag	Description
Keyboard	`--teleop_mode keyboard`	Use arrow keys to control the robot.
Joystick	`--teleop_mode joystick`	Requires `python-joystick` package.
Gamepad	`--teleop_mode gamepad`	Works with XInput-compatible controllers.
Sensors Guide

Common errors:

ROS 2 connection failed: Verify ROS_DOMAIN_ID and ROS_MASTER_URI are set. Fix: Run export ROS_DOMAIN_ID=0 before recording.
Sensor calibration: If RGB images are distorted, recalibrate the camera with ros2 camera_calibration.

4. Training a Policy from Demonstrations

Train a custom policy using the lerobot.train API. Below is a BC (Behavior Cloning) example.

Train from Scratch

from lerobot import BehaviorCloningTrainer
from lerobot.datasets import load_dataset

# Load your dataset
dataset = load_dataset("path/to/ur5_demos")

# Initialize trainer
trainer = BehaviorCloningTrainer(
    model="lerobot/act3d-base",  # Start from a pretrained base
    dataset=dataset,
    batch_size=32,
    epochs=50,
    lr=1e-4
)

# Train!
trainer.train()
trainer.push_to_hub("my-ur5-policy")  # Save to Hugging Face Hub

API Docs

Hyperparameter Tuning

Key parameters to adjust:

Parameter	Default	Recommended Range	Description
`batch_size`	32	16–128	Larger batches = faster training.
`lr`	1e-4	1e-5 to 1e-3	Too high = instability; too low = slow.
`epochs`	50	20–200	Early stopping if validation loss plateaus.
`kl_weight`	0.1	0.01–0.5	Only for RL fine-tuning.

Gotchas:

Overfitting: If training loss << validation loss, reduce model capacity or add data augmentation.
Action scaling: Ensure actions in your dataset match the robot’s command range (e.g., [-1, 1] for torques).

5. Evaluating in Simulation and on Hardware

Evaluate in Isaac Gym

from lerobot.sim import IsaacGymEnv
from lerobot import ACTPolicy

# Load policy and env
policy = ACTPolicy.from_pretrained("my-ur5-policy")
env = IsaacGymEnv(robot="ur5e", task="pick_place")

# Run evaluation
success_rate = env.evaluate(policy, episodes=10)
print(f"Success rate: {success_rate * 100:.2f}%")

NVIDIA Developer

Expected output:

Episode 1/10: Success = True
Episode 2/10: Success = False
...

Deploy to Hardware (UR5 Example)

from lerobot.hardware import UR5Controller
from lerobot import ACTPolicy

# Initialize robot and policy
robot = UR5Controller()
policy = ACTPolicy.from_pretrained("my-ur5-policy")

# Run closed-loop control
while True:
    obs = robot.get_observations()
    action = policy(obs)
    robot.send_action(action)

Hardware Docs

Hardware Support Table:

Robot	LeRobot Support	Notes
UR5/UR10	✅	ROS 2 driver required.
Franka Emika	✅	Works with `franka_ros2` package.
SO-100	✅	Official support (see below).
Stretch RE1	✅	Requires `stretch_ros2` driver.
Aloha	✅	Experimental (2026-05).
Hardware Docs

Gotchas:

Latency: Real-world deployment requires <50ms loop time. Use robot.set_control_rate(20) for 20Hz updates.
Safety: Always enable force/torque limits and emergency stops in hardware mode.

6. Pushing Datasets and Models to the Hub

Share your work with the community using Hugging Face Hub.

Push a Dataset

lerobot push_to_hub \
  --path ./ur5_demos \
  --repo_id my-username/ur5-pick-place-demos \
  --private  # Optional: Set to False for public

Datasets Guide

Push a Model

from lerobot import ACTPolicy

# Load your trained policy
policy = ACTPolicy.from_pretrained("lerobot/act3d-base")
policy.train()  # Assume you've trained it

# Push to Hub
policy.push_to_hub("my-username/ur5-act-policy")

Model Hub

Hub Features:

Model Cards: Auto-generated from your code (edit README.md in the repo).
Datasets: Versioned and searchable (e.g., huggingface.co/datasets/my-username/ur5_demos).
Spaces: Deploy interactive demos (see LeRobot Spaces).

Gotchas:

Large files: Datasets >10GB may fail. Use lerobot push_to_hub --chunk_size 1G.
License: Always specify a license (e.g., mit or apache-2.0) in metadata.json.

7. Hardware-Specific Guides

SO-100 (Official Support)

LeRobot includes native support for the SO-100 mobile manipulator.

Install SO-100 drivers:

pip install lerobot[so100]

Hardware Docs

Example: Teleoperate SO-100:

lerobot record \
  --robot so100 \
  --sensors rgb_camera base_pose \
  --output_dir ./so100_navigation

Koch Robotics Arms

For Koch K1/K2 arms, use the ROS 2 interface:

from lerobot.hardware import KochArm

robot = KochArm()
robot.move_to_joint_positions([0.5, -0.3, 0.0,

How to Set Up LeRobot (Hugging Face): A Practical Guide

1. Installation and the LeRobot Dataset Format

Install LeRobot

Dataset Format Overview

2. Loading Pretrained Policies

Load an ACT Policy

Load a Diffusion Policy

Load SmolVLA

3. Recording Teleoperation Datasets on a Real Arm

Prerequisites

Record a Dataset

Teleoperation Modes

4. Training a Policy from Demonstrations

Train from Scratch

Hyperparameter Tuning

5. Evaluating in Simulation and on Hardware

Evaluate in Isaac Gym

Deploy to Hardware (UR5 Example)

6. Pushing Datasets and Models to the Hub

Push a Dataset

Push a Model

7. Hardware-Specific Guides

SO-100 (Official Support)

Koch Robotics Arms

The 30% Report

Related Articles

Want to Discuss These Ideas?

Sources

How to Set Up Hugging Face Hub: A Practical Guide

How to Set Up LangChain: A Practical Guide for Enterprise AI Teams