ECCV 2026 World Model Robot Learning

Learning Transferable Dynamics Priors from Action to World Modeling

A2World pretrains action-conditioned multi-view world models on large-scale robot manipulation trajectories, then transfers the learned dynamics prior to simulation rollouts and instruction-conditioned control.

Ze Huang*, Jiahui Zhang*, Hairuo Liu*, Chenxi Zhang, Ran Cheng, Li Zhang
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University · McGill University

ECCV 2026 Code Try Interactive Demo

Overview

Action-conditioned world modeling as a reusable dynamics prior.

A2World learns how robot actions drive visual scene evolution. Instead of treating video generation as an isolated rendering task, it uses action-conditioned future prediction to build a dynamics prior that can be reused by both simulator-centric and policy-centric robot learning.

Large-scale pretraining 2.1M+ robot manipulation trajectories spanning 20+ robot embodiments.
Two downstream transfers A2World-sim rolls out long-horizon observations; A2World-policy jointly predicts video and actions.
Real robot validation Flexiv dual-arm tasks covering insertion, reorientation, switch interaction, lifting, and deformable-object handling.
A2World overview: world model pretraining and prior transfer

Method

One pretrained dynamics prior, two robot-learning interfaces.

A2World starts from action-to-video world model pretraining, then adapts the same prior into a simulator and a policy. The shared representation connects visual rollout quality with downstream robot behavior.

A2World model extensions for simulator and policy transfer
01 Pretrain

A2World base model

Predicts future multi-view manipulation videos from initial observations and future action chunks.

02 Simulate

A2World-sim

Uses pose-guided history and autoregressive rollout to replace expensive real-robot evaluation.

03 Control

A2World-policy

Performs joint video-action diffusion for instruction-conditioned real-robot execution.

Interactive Demo

Choose actions. Generate a rollout.

Shared initial observation

Select actions, then generate.

World Model Rollouts

Generated rollouts visualize the learned action-conditioned dynamics.

These videos show A2World world model rollouts on real-robot manipulation scenarios. They are predictions of future interaction dynamics, not direct camera recordings of policy execution.

Flip small box

Reorientation under contact-rich manipulation.

world rolloutOOD dynamics

Insert RAM module

Precision alignment and insertion dynamics.

world rolloutprecision

Lift box high

Longer-horizon object lifting and transport.

world rolloutlifting

Put chain in the box

Deformable-object handling with container interaction.

world rolloutdeformable

Toggle power switch

Interactive switch manipulation with small contact changes.

world rolloutswitching

A2World-policy

Real-robot executions test whether the learned prior transfers to control.

A2World-policy is evaluated on a Flexiv dual-arm suite. These clips show real executions on the same task families used to stress precision, contact, object lifting, switch interaction, and deformable-object handling.

Put chain in the box

Deformable-object handling in a real execution.

Toggle power switch

Small-contact switching with a dual-arm platform.

Flip small box

Object reorientation with contact-rich dynamics.

Insert RAM module

Precision insertion under visual and instruction conditioning.

Lift box high

Object lifting with visible progress toward the target state.

Citation

Reference

If this project is useful for your research, please cite the paper, in Proceedings of ECCV 2026.

@inproceedings{huang2026a2world,
  title={Learning Transferable Dynamics Priors from Action to World Modeling},
  author={Huang, Ze and Zhang, Jiahui and Liu, Hairuo and Zhang, Chenxi and Cheng, Ran and Zhang, Li},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2026},
}