Drawer packing ARX-X5 · 1× speed
1 Fudan University2 Shanghai Innovation Institute3 Imperial College London4 University of Surrey
Future world states are a rich learning signal. FDM uses them to train action features, while keeping predicted futures out of the action path.
Direct action prediction
Future-conditioned actions
Action-to-future learning
Predict actions from observations and language, without explicit future-state supervision.
Generate futures to predict actions. Visual prediction errors can affect the policy.
Predict futures from action features. Learn from future states; deploy with action-only inference.
Three expert roles, one directional connection. The future-prediction loss flows back through the features that produce actions.
Understand the scene
and the task.
Predict the robot’s
next actions.
Predict the consequences
from action features.
Train jointly with action imitation and future prediction. Asymmetric attention blocks future information from reaching actions, including through context.
Scroll horizontally to follow the full pipeline.
A π₀.₅ / PaliGemma policy with direct regression of future-image latents. One future image per camera view, supervised with MSE.
VLM implementationA Wan2.2 video backbone with flow matching on future-video latents. Current-observation processing remains in the action path.
VGM implementationEvaluated in simulation and on a dual-arm ARX-X5 robot. Results below are reported in the paper.
FDM w/ VLM · four task suites
Up to 13 percentage points over compared baselines
FDM w/ VGM vs. FastWAM-Joint · one H100
Average · % · higher is better
FDM is 13 percentage points above π₀.₅.
FDM is 5 percentage points above FastWAM-Joint.
Dual-arm ARX-X5 · 20 trials per method per task · averages across five tasks.
| Backbone | Method | Setup | Drawer packing (%) | Box packing (%) | Shuttlecock insertion (%) | Cup stacking (%) | Block stacking (%) | Average (%) |
|---|---|---|---|---|---|---|---|---|
| VLM | π₀ | VLA baseline | 60 | 65 | 55 | 80 | 75 | 67 |
| VLM | π₀.₅ | VLA baseline | 75 | 80 | 75 | 95 | 90 | 83 |
| VLM | FDM w/ VLM | Our model | 95 | 95 | 90 | 100 | 100 | 96 |
| VGM | DreamZero | WAM baseline | 65 | 70 | 65 | 75 | 80 | 71 |
| VGM | FastWAM-Joint | Future-conditioned WAM | 70 | 70 | 75 | 90 | 85 | 78 |
| VGM | FDM w/ VGM | Our model | 70 | 75 | 70 | 100 | 100 | 83 |
At perturbation strength α = 1.0, FastWAM-Joint’s action error increases by 4.44% on LIBERO and 5.52% on RoboTwin. FDM’s actions remain unchanged when model parameters, inputs and action-sampling randomness are fixed.
The FDM zero line is an architectural property under fixed inputs and randomness, rather than a fitted experimental curve.
The paper measures cosine similarity between action-imitation gradients and future-loss gradients through the action-feature interface. Alignment is predominantly positive for VLM and positive in most VGM settings after initial fluctuations.
The controlled ablations above isolate the behavioral contribution: full FDM improves average success over stop-gradient by 7 points for VLM and 5 points for VGM.
A shared robot setup, five manipulation tasks, and explicit success criteria. Explore how the demonstrations are collected and what each task asks the policy to do.
Two ARX-X5 arms observe the workspace from three viewpoints. Demonstrations are collected through VR teleoperation.
150 trajectories per task: 147 for training and 3 for offline validation. Object placements vary across demonstrations and evaluation trials.
20 trials per method per task. Human evaluators are blinded to model identity and judge whether the task goal is met. Timeouts and trials judged unable to continue count as failures.
Hardware, collection protocol and task definitions: Appendix D of the paper.
Inspect six chronological frames from each task in the paper, or jump to the full policy comparisons.
Put a charger and a block inside an initially closed drawer, then close it again.
Success conditionBoth objects are inside and the drawer is fully closed.
Charger and block positions are randomized; the drawer base shifts slightly.
Pack a scarf and a flower into an open box, then close the lid.
Success conditionBoth objects are enclosed and the lid is closed.
The positions of the box, scarf and flower vary slightly.
Insert two shuttlecocks into a tube using both arms together.
Success conditionBoth shuttlecocks are inside the tube.
The tube shifts slightly; shuttlecock positions and orientations vary.
Build a two-tier structure: three cups below and two cups above.
Success conditionThe five-cup structure stands freely and stays stable.
Two cups start on the left and three on the right, with randomized positions on each side.
Stack the small red and yellow blocks on the larger blue base.
Success conditionA stable three-block tower: blue below, red in the middle, yellow above.
Small-block positions are randomized; the blue base shifts slightly.
Five manipulation tasks. Four methods. Three camera views. Explore the original real-robot recordings side by side.
Place the objects in the drawer and close it.
Select a task, then play the four recordings together.
Each video combines a 640 × 480 third view above two 320 × 240 wrist views. All views start together; shorter recordings hold their final frame. Displayed duration is the recording length, not model inference latency. Individual clips illustrate behavior; aggregate success uses 20 trials per method per task.
The main findings, in brief. Each answer links back to the evidence.
It helps across simulation and real-robot tests. On the five real tasks, FDM improves average success from 83% to 96% for the VLM family and from 78% to 83% over FastWAM-Joint for the VGM family. The gains are not universal: on LIBERO, FDM w/ VGM reaches 98.1%, slightly below LingBot-VA’s 98.5%.
Explore task successThose gradients make the action features more useful for control. Compared with keeping the same forward connection but stopping its gradients, full FDM gains 7 percentage points for VLM and 5 for VGM on the real tasks.
Compare the ablationsIn FDM, actions never read predicted futures, so changing a future prediction leaves actions unchanged when model parameters, inputs and sampling randomness are fixed. FastWAM-Joint depends on those futures: at the strongest tested perturbation, its action error rises by 4.44% on LIBERO and 5.52% on RoboTwin.
See the perturbation analysisThe measured gradients generally point in compatible directions. Their alignment is mostly positive for VLM and becomes positive in most VGM settings after early fluctuations, supporting future prediction as a useful training signal for action learning.
See the gradient analysisFor VLM, training steps take about 42% longer, while inference stays close to π₀.₅ at about 203 ms per action chunk. For VGM, training cost is nearly unchanged and inference drops from 446.55 to 253.45 ms per chunk, about 1.8× faster than FastWAM-Joint on one H100.
Compare training and inference costsCondensed from the paper’s experimental questions Q1–Q5. Read the full experiments.
Code, training configurations, and evaluation instructions are available in the repository.
View on GitHub
@misc{ma2026forward,
title = {Forward Dynamics Models},
author = {Ma, Zipei and Jiang, Junzhe and
Zhang, Jiahui and Zhang, Juntong and
Gu, Chun and Zhang, Bozhou and
Deng, Jiankang and Zhu, Xiatian and Zhang, Li},
year = {2026},
eprint = {2610.13117},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
doi = {10.48550/arXiv.2610.13117},
url = {https://arxiv.org/abs/2610.13117}
}