Start a project

Work Humanoid robotics Simulation

Humanoid control

Unitree G1 whole-body control with GR00T and SONIC

Eight episodes from the egocentric manipulation dataset, each captioned with its task instruction.Simulation

This work builds a loco-manipulation stack for the Unitree G1 humanoid. NVIDIA GR00T acts as the vision-language-action brain and outputs latent motion tokens, while GEAR-SONIC acts as the whole-body controller that turns those tokens into coordinated leg, waist, arm and hand motion.

Period
2026, ongoing
Role
Architecture, simulation and controller development
Stack
Unitree G1, NVIDIA GR00T, GEAR-SONIC, Isaac Sim and Isaac Lab, MuJoCo, Genesis, LeRobot, unitree_sdk2_python, unitree_rl_gym, ROS 2

Challenge

Humanoid loco-manipulation needs a policy that understands tasks and a controller that keeps the robot balanced while the arms do the work. Neither can be tuned safely on hardware first.

Approach

Development follows a simulation-first path: bring up the SONIC sim-to-sim loop, generate demonstrations in simulation, fine-tune the policy and evaluate before moving to hardware. The work included research into motion and retargeting datasets (BONES-SEED, GRAIL, OmniRetarget) and the Unitree SDK ecosystem. Alongside it, motion-capture retargeting drives upper-body motion on the simulated G1.

In brief

What was built.

  • GR00T VLA with the SONIC whole-body controller on the Unitree G1
  • Simulation-first pipeline in Isaac Lab and MuJoCo
  • Synthetic egocentric manipulation dataset of 82 validated episodes
  • Dataset research: BONES-SEED, GRAIL, OmniRetarget
  • Motion-capture retargeting for upper-body motion

Egocentric manipulation dataset

To train the vision-language-action policy, the lab built a synthetic egocentric manipulation dataset: 82 validated episodes recorded from the G1's own head camera while the whole-body controller keeps the robot balanced. Tasks include picking fruit and placing it on a plate or tray, stacking and dropping onto a block, left-hand variants, lifting a board with both hands, pouring from a bottle or mug into a bowl, and opening a drawer. Each episode carries its language instruction, so the same data trains instruction following and manipulation together.

Manipulation under whole-body control

Arm and hand tasks run on top of the whole-body controller, which keeps the robot balanced while the policy commands the motion.

Locomotion and whole-body control

The whole-body controller handles balance, walking and turning while the policy commands the task; the same stack walks the robot to a goal before manipulation starts.