Get Started#

Prerequisites#

  • Ubuntu operating system

  • Basic familiarity with terminal commands

  • Miniconda (recommended)

Installation#

1. Create a Conda Environment#

If you already have a lerobot conda environment, you can use that. Otherwise, create a new environment:

conda create -y -n lerobot python=3.11
conda activate lerobot

Note: It’s recommended to have ManiSkill and lerobot code in the same environment for future sim2real deployment.

2. Install ManiSkill#

Follow the official installation instructions for ManiSkill.

# Basic installation
pip install mani-skill

# Download scene dataset
python -m mani_skill.utils.download_asset "ReplicaCAD"

If the dataset downloading goes wrong, you can directly download the dataset from this google drive link. And then folder contents should go to

~/.maniskill/data/scene_datasets/replica_cad_dataset

Familiarize yourself with ManiSkill using the quickstart guide and demo scripts.

Try this command to test whether you have successfully installed Maniskill:

Note

Change shader=”rt-fast” to “default” for faster rendering or if your computer doesn’t support ray-tracing rendering.

python -m mani_skill.examples.demo_random_action -e "ReplicaCAD_SceneManipulation-v1" \
  --render-mode="human" --shader="rt-fast"

And you should be looking at something like this:

image

3. Additional Dependencies#

We use pygame to read keyboard input and show the control panel in real time, and Rerun to visualize and collect camera data.

pip install pygame
pip install rerun-sdk

4. Put XLeRobot Files to Maniskill#

Note

We are expecting Maniskill to offer official support for XLeRobot soon, after that this step can be skipped.

Navigate to the ManiSkill package folder in your conda environment:

cd ~/miniconda3/envs/lerobot/lib/python3.10/site-packages/mani_skill

Put the XLeRobot files into Maniskill folders:

  1. Download the XLeRobot Robot files here:

  2. Put the files in /agents/robots, /assets/robots, and /envs/scenes: image

    image

    image

    After putting XLeRobot robot description files to /agents/robots, add this line in /agents/robots/init.py image

  3. Add the control codes to /examples:

    image

  4. Modify utils/scene_builder/replicacad/scenebulider.py: For new robot ids, you need to add them to ReplicaCAD scene first.

    image

Get Moving#

Note

Change shader=”default” to “rt-fast” for photo-realistic ray-tracing rendering (but slower).

Joint Control#

python -m mani_skill.examples.demo_ctrl_action -e "ReplicaCAD_SceneManipulation-v1" -r "xlerobot"  --render-mode="human" --shader="default" -c "pd_joint_delta_pos_dual_arm"

image

End Effector Control#

Original dual-arm version:

python -m mani_skill.examples.demo_ctrl_action_ee_keyboard -e "ReplicaCAD_SceneManipulation-v1" -r "xlerobot"  --render-mode="human" --shader="default" -c "pd_joint_delta_pos_dual_arm"

Single-arm version:

python -m mani_skill.examples.demo_ctrl_action_ee_keyboard_single -e "ReplicaCAD_SceneManipulation-v1" -r "xlerobot_single"  --render-mode="human" --shader="default" -c "pd_joint_delta_pos"

With Camera Visualization via Rerun:

python -m mani_skill.examples.demo_ctrl_action_ee_cam_rerun -e "ReplicaCAD_SceneManipulation-v1" -r "xlerobot"  --render-mode="human" --shader="default" -c "pd_joint_delta_pos_dual_arm"

image