Predict & plan
A video expert predicts future visual and tactile observations while an action expert jointly generates an action chunk.
WORLD-ACTION MODELS · REAL-WORLD TOUCH
Tactile Prediction and
Real-Time Action Correction
in World-Action Models for Contact-Rich Manipulation
1 School of Data Science, Fudan University2 Shanghai Innovation Institute3 NeoteAI
The full story, in three minutes.
ENGLISH NARRATION / 03:08
01 / PRECISIONCharger plug ↗
02 / ALIGNMENTExpansion card ↗
03 / FRAGILITYPotato chip ↗
04 / GRIP CONTROLEmpty bottle ↗
05 / LONG HORIZONFruit transfer ↗
Five real-world tasks. One prediction-informed tactile feedback loop.
64%
Average task success+16 percentage points over T-Rex
30.4ms
Per tactile correction32.9 Hz with a reusable cache
20.7×
Lower inference latencyCompared with regenerating a chunk
01 / OVERVIEWPredict contact.
Predict contact.
Correct in real time.
A robot can plan what happens next.
Touch tells it what is happening now.
Small contact errors can jam an insertion or break a fragile object. Future tactile predictions provide useful expectations, but those predictions are made before an action chunk begins.
TacPAC connects this anticipated contact to live tactile feedback. A tactile expert reads each new tactile observation against a cached representation of the prediction and plan, then corrects the actions that have yet to execute. The robot keeps moving throughout.
THE IDEA / EXPECTATION MEETS TOUCHFollow the signal ↓
02 / METHODFrom expected contact
From expected contact
to corrected action.
Plan once per chunk. Read new touch against the plan. Update what comes next.
FIGURE 2 / THREE EXPERTS, ONE FEEDBACK LOOP
01
Predict and plan together.
The video and action experts jointly generate future visual and tactile observations and an action chunk through shared self-attention.
Architecture redrawn from the supplied pipeline artwork and paper Figure 2. Image tiles indicate visual and tactile modalities.
Cache the expectation
A clean prefill pass stores predicted tactile and action representations as a reusable, layer-wise key–value cache.
Correct with live touch
Each new tactile frame triggers one expert pass, producing delta actions for the unexecuted part of the chunk.
FIGURE 3 / ASYNCHRONOUS INFERENCE
The plan updates.
The robot keeps moving.
Observe at m. Compute while the arm moves. Apply the delta only from m + d onward.
Already executedCommitted during latencyEligible for correction
0m = 16m + d = 18H = 48
32 actions remain; the correction can update 30 actions after the 2-step latency window.
30 / 48 actionsInteractive timing illustration, slowed for clarity. H = 48; robot control = 30 Hz. d is measured from recent correction latency, not fixed. The paper reports d = 0 for most execution; only 1.6% of calls span ≥1 control step.
03 / REAL-ROBOT EXPERIMENTS
Contact makes the difference.
Explore complete rollouts across precision insertion, fragile-object handling, and force-sensitive manipulation.
Charger-plug insertion
Precise alignment under constrained contact.
iVideos show selected successful or failed rollouts, not all trials. Success rates summarize 20 trials per method per task. Speed applies equally to all three videos; each stops at its final frame.
Figure 6 · Inside the experimental setupROBOT + SENSORS +
One arm. Vision and touch.
A Flexiv Rizon 4 uses two InTac S1 tactile sensors in its gripper, a wrist-mounted Intel RealSense D405, and a third-person Intel RealSense D435i.
TacPAC takes a task instruction, two camera views, proprioception, and two tactile views. All methods are evaluated on the same five tasks with 20 trials per task.
Full evaluation protocol in the paper ↗04 / RESULTS
Better across all five tasks.
64% average success, with prediction and correction working together.
FIGURE 4
Seven methods. Five contact-rich tasks.
+16 pp
Average gain over the strongest baseline, T-Rex.
20 trials per method per task · 100 trials per method overall
Exact values for every methodSUCCESS RATE (%) +
| Method | Plug | Fruit | Chip | Bottle | Card | Average |
|---|
TABLE 1
Which tactile pathways matter?
Prediction alone raises average success from 22% to 37%. Giving the corrector access to predicted touch raises 47% to 64%.
Pathways available to each variant
Complete component-ablation table5 VARIANTS / ALL TASKS +
| Variant | Prediction | Correction | Tactile cache | Plug | Fruit | Chip | Bottle | Card | Average |
|---|
FIGURE 5
Brief contact transitions. Outsized tactile change.
4.9%
→of all frames
18.6%
of total tactile change
Contact transitions are the most active phase in every task, yet account for only a small fraction of the prediction target.
(a) Relative tactile change
(b) Share of episode frames
View digitized phase values and source notesAPPROXIMATE VALUES +
Bar heights and error bars were recovered from the original PDF vector paths, not raw measurements. Values below are approximate; error bars retain the published geometry (the caption does not specify their statistical definition). The 4.9% and 18.6% callouts are reported directly in Section 4.3.
Download digitized values (JSON) ↓TABLE 2
Fast enough to act on new touch.
20.7×
lower latency per correction
than regenerating a chunk
| Inference stage | Latency ↓ | Rate ↑ |
|---|---|---|
| Chunk generation · ten flow steps | 628.6 ms | 1.59 Hz |
| Cache prefill · one clean pass | 62.6 ms | 15.97 Hz |
| Tactile correction · one pass | 30.4 ms | 32.90 Hz |
FIGURES 7–11 / SUPPLEMENT
Contact, through the course of execution.
VISION
TOUCH
TOUCHApproachContact transitionSustained contactPost-contact
Dark line: tactile activity cₜ (left axis). Rust line: measured gripper width (right axis). Curves preserve the original PDF vector paths. Download trace geometry ↓ The representative training-episode trace and the successful-rollout image strip are from separate episodes, as stated in the supplement; the time cursor applies to the trace only.
05 / CITATION
Build on TacPAC.
If you find this work useful,
please consider citing our paper.
@article{ma2026tacpac,
title={TacPAC: Tactile Prediction and Real-Time Action
Correction in World-Action Models for
Contact-Rich Manipulation},
author={Ma, Zipei and Wei, Xiaofei and Jiang, Junzhe
and Lu, Shunlin and Zhang, Li},
journal={arXiv preprint arXiv:2609.05266},
year={2026},
url={https://arxiv.org/abs/2609.05266}
}