SMR

Learning human motion. Realizing robot motion.

SMR

Learning streaming motion retargeting
for humanoids from human semantic

Robot-aware semanticsPhysics-grounded learningStrictly causal inference
Upper-body motionSMR · Physical Unitree G1 · Source playback
Explore the project

Closed-loop performance

97.0% motion completion

400 test motions · Frozen Unitree G1 controller · Simulation

Streaming inference

0.37 ms per model frame

50 Hz input · Reported model-step timing

Physics-grounded motion

No future human frames

Current and past observations → Robot references

arXiv preprint

SMR

Learning streaming motion retargeting for humanoids from human semantic

Chun Gu1,2,3, Juntong Zhang2,3, Jingyu Li1,2,3, Li Zhang1,2,3*

1School of Data Science, Fudan University2Shanghai Innovation Institute3Logos Robotics Group

*Corresponding author (lizhangfd@fudan.edu.cn).

PDF GitHub BibTeX
SMR pipeline: human motions are converted to robot-aware semantics, tracked by physics experts, and distilled into a causal retargeter

SMR turns human motion into robot-aware task-space targets. Physics experts execute these targets in simulation. Accepted state rollouts train a strictly causal transformer that generates motion references from current and past human observations.

The problem & our approach

Abstract

Retargeting human motion to humanoids requires preserving motion semantics while accounting for the robot's physical constraints. Kinematic pipelines prescribe a robot trajectory through geometric matching, without establishing whether it can be executed under the robot's dynamics. Prior physics-supervised neural methods train bidirectional full-sequence models on physics-refined kinematic trajectories, remaining tied to prescribed kinematic solutions and requiring noncausal context.

We propose SMR, a streaming neural motion retargeter learned from semantic-conditioned physics experts. We convert human motion into robot-aware task-space targets for body landmark positions, orientations, and velocities. Physics experts track these targets directly in simulation under contact, dynamics, and actuator constraints. We train a single strictly causal Transformer on their executed state trajectories that complete the motion and pass semantic and physical validity checks. At each incoming frame, the student uses only current and past human observations to generate a robot motion reference for a downstream whole-body controller.

Experiments demonstrate that semantic-conditioned physics experts generate motion with higher semantic fidelity and physical validity than direct optimization-based retargeting, while the resulting streaming retargeter achieves improved closed-loop performance over kinematic and physics-refined baselines under a common whole-body controller.

02 / From semantics to references

Method

Eight paired source SMPL-H and Unitree G1 poses with semantic landmark targets overlaid on Unitree G1
Qualitative examples of robot-aware semantic tracking across eight motions. Each example pairs a source SMPL-H pose (inset) with the corresponding Unitree G1 pose produced by a physics expert in simulation. Colored markers and axes depict the transported landmark targets, while the Unitree G1 poses preserve distinctive features of the source postures despite differences in morphology.

Semantic-conditioned physics experts

Let the robot realize the motion.

Expert policies track semantic task-space targets directly in simulation. Contact, balance, and actuator constraints shape the executed trajectory. Only expert rollouts that pass semantic and physical validity checks become student supervision. IK is used only to initialize expert episodes.

  • 01 Robot-aware body landmark commands express the human motion task.
  • 02 Accepted simulator state rollouts provide dynamics-grounded labels.
  • 03 A causal student uses current and past human observations, with no future human frames.
Complete SMR training and deployment pipeline
The student outputs robot motion references; a downstream whole-body controller executes them.

03 / Evaluation

Closed-loop execution on Unitree G1

All methods are compared on the same 400 motions in simulation using the same frozen MimicLite whole-body controller. Success means completing a motion without the paper's pelvis-height or projected-gravity failure conditions. Errors include all motions, including failures.

GMR
90%
NMR
92%
GMR-supervised Student
91.5%
GMR-Expert Student
93%
SMR (Ours)
97%
Closed-loop evaluation on 400 motions using the same frozen whole-body controller. SMR achieves the highest success rate and lowest tracking and human-target errors, demonstrating more reliable execution and better preservation of human motion semantics.
Reference source Success ↑ MPJPE ↓ E-MPJPE ↓
GMR 90.0% 0.060 m 0.115 m
NMR 92.0% 0.057 m 0.111 m
GMR-supervised Student 91.5% 0.055 m 0.111 m
GMR-Expert Student 93.0% 0.051 m 0.112 m
SMR (Ours) 97.0% 0.039 m 0.080 m

MPJPE measures tracking error relative to the robot reference; E-MPJPE measures executed wrist and ankle errors relative to the human targets.

At 50 Hz, the SMR model step takes 0.37 ms per frame, compared with 4.87 ms for the warm-started GMR solve under the reported protocol.

04 / Results

Qualitative comparisons

Eight matched human poses and corresponding Unitree G1 reference poses from GMR, NMR, and SMR
Qualitative comparison of Unitree G1 reference poses at matched timestamps from eight human motions. The rows show source SMPL poses and the corresponding references produced by GMR, NMR, and SMR. Compared with GMR and NMR, SMR better preserves the source whole-body posture, especially in the crouching, kneeling, and floor-level examples.
SMR data and model scaling curves
Data and model scaling. Gains diminish beyond 50% training data, while the 12.7M model achieves the lowest held-out error.
Whole-body controller training curves for different reference sources
MimicLite training with different motion references. SMR achieves longer episodes and higher rewards by 4,000 iterations.

Motion reference comparisons

Human motion → robot references

Four qualitative reference replays compare the source human motion with GMR, NMR, and SMR. These visualize the generated motion references, separately from the frozen-controller completion results above.

01

Reference comparison 1

Human · GMR · NMR · SMR

02

Reference comparison 2

Human · GMR · NMR · SMR

03

Reference comparison 3

Human · GMR · NMR · SMR

04

Reference comparison 4

Human · GMR · NMR · SMR

Videos retain their embedded panel labels. The bottom-right NeuralRetarget panel denotes SMR. Playback timing is preserved from the supplied clips.

Streaming / physical deployment

Streaming on a physical Unitree G1

Human motion is translated into robot references for execution through MimicLite. The recordings below compare GMR-driven and SMR-driven execution across three demonstrations.

GMR and SMR are independent recordings. Each clip has its own playback controls; they are qualitative examples, not synchronized trials or an additional success-rate evaluation.

01

Upper-body motion

Streaming arm motion from a human operator to Unitree G1.

GMR

Physical Unitree G1 · independent recording

SMR (Ours)

Physical Unitree G1 · independent recording

02

Crouching & kneeling

Lower-body transitions with coordinated arm motion.

GMR

Physical Unitree G1 · independent recording

SMR (Ours)

Physical Unitree G1 · independent recording

03

Additional demo: motion with a hand-held bag

Upper-body motion while the robot holds a bag.

GMR

Physical Unitree G1 · independent recording

SMR (Ours)

Physical Unitree G1 · independent recording

06 / Citation

BibTeX

@article{gu2026smr,
  title         = {SMR: Learning streaming motion retargeting for humanoids from human semantic},
  author        = {Gu, Chun and Zhang, Juntong and Li, Jingyu and Zhang, Li},
  journal       = {arXiv preprint},
  year          = {2026}
}