AutoDex

Dataset

Real-robot dexterous grasp executions with synchronized multi-view video, calibrated cameras, 6-DoF object poses, and executed grasp annotations. The release has 2,610 trials collected with the Allegro and Inspire hands; each subset follows the same per-episode directory structure.

# one grasp trial
{hand}/{object}/{timestamp}/
├── pose_world.npy          # 6-DoF object pose as a 4×4 transform (T_world_object)
├── R2C.npy                 # (4,4) T_world_robot (robot base → world)
├── cam_param/              # intrinsics.json + extrinsics.json (per serial)
├── videos/{serial}.avi     # 24 synchronized camera videos (undistorted)
├── arm/                    # executed arm trajectory: action / state .npy
├── hand/                   # executed hand trajectory (16 Allegro | 6 Inspire)
├── executed_grasp/         # the grasp the robot actually performed
│   ├── wrist_se3.npy       # (4,4) wrist pose in the OBJECT frame
│   ├── grasp_pose.npy      # (16|6) finger joints at grasp
│   └── meta.json           # grasp_frame / grasp_time / success / states
├── object_tracking/        # 6-DoF object trajectory (gotrack subsets)
│   └── gotrack_output/world_pose_records.json
└── recompute_pose.json     # {sil_loss, reject} pose-quality record

• pose_world = T_world_object (object → world). For an object-frame point p: world_point = pose_world @ p.
• Project into camera s: p_camera = extrinsics[s] @ pose_world @ p, then u ~ K @ p_camera[:3] (divide by depth; use intrinsics_undistort).
• R2C = T_world_robot: robot-base → world is R2C @ p.
• executed_grasp/wrist_se3 is the executed wrist in the object frame, so a grasp replays relative to the object.

pip install huggingface_hub

# Python — dataset only (skips the web-gallery assets)
from huggingface_hub import snapshot_download
snapshot_download("willi19/autodex-gallery", repo_type="dataset", local_dir="autodex",
                  allow_patterns=["allegro/*", "inspire/*"])

# CLI — both hands, or just one subset / object
hf download willi19/autodex-gallery --repo-type dataset --local-dir autodex \
  --include "allegro/*" "inspire/*"
hf download willi19/autodex-gallery --repo-type dataset --include "allegro/attached_container/*"

import numpy as np, json

trial = "allegro/attached_container/20260330_164351"
pose_world = np.load(f"{trial}/pose_world.npy")          # (4,4) object → world
wrist_obj  = np.load(f"{trial}/executed_grasp/wrist_se3.npy") # (4,4) wrist in object frame
fingers    = np.load(f"{trial}/executed_grasp/grasp_pose.npy") # (16,) Allegro
meta       = json.load(open(f"{trial}/executed_grasp/meta.json")) # grasp_frame, success, states