One failure. A reusable idea.
Follow the four moments that turn a brittle robot into a planner that learns.
The robot knows Pick and Insert. A block against the wall makes Pick fail.
Parallel simulation starts at the failure. PPO learns a relational Poke policy; success means planning can resume.
Counterfactual dreaming separates the useful relation from accidental details, then updates the planning model.
The goal changes. ReSYNC reasons backward from Insert, schedules Poke first, and avoids failure altogether.
One framework, many physical worlds
Hover over a domain—or tap it—to replay its learned behaviors.
Drive through unknown ice and mud.
Punch, wiggle, and fiddle around obstructions.
Open articulated furniture and clear clutter.
Reuse tools to clear obstructed goals.
Why recovery alone is not enough
A reactive policy waits for insertion to fail. By then another block is involved, the scene is out-of-distribution, and recovery cannot restore a solvable state.
Empirical results
Success rate across four simulated domains. Gray columns are held-out, compositionally novel tasks.
| Method | Icy Transport | Cluttered Drawer | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| T₁ train | T₁ test | T₂ train | T₂ test | T₁ | T₁ train | T₁ test | T₂ train | T₂ test | T₁ | |
| GNN | .98±.03 | .01±.01 | .18±.13 | .18±.13 | .38±.18 | .86±.04 | .00±.00 | .69±.07 | .48±.10 | .00±.00 |
| TF | .98±.03 | .00±.00 | .26±.14 | .17±.12 | .52±.16 | .11±.07 | .00±.00 | .03±.03 | .02±.02 | .00±.00 |
| VLM-OS | .39±.13 | .00±.00 | .30±.14 | .00±.00 | .24±.12 | .68±.06 | .29±.03 | .18±.04 | .25±.06 | .50±.10 |
| DSG-S | .67±.12 | .00±.00 | .24±.07 | .02±.02 | .44±.08 | .73±.03 | .04±.01 | .67±.04 | .04±.01 | .38±.02 |
| DSG-M | .61±.04 | .33±.04 | .15±.07 | .01±.02 | .21±.05 | .13±.02 | .03±.01 | .00±.00 | .00±.00 | .09±.04 |
| RC | .71±.11 | .00±.00 | .34±.16 | .00±.00 | .50±.18 | .68±.03 | .01±.01 | .62±.04 | .14±.02 | .35±.15 |
| RC+IVNTR | .50±.20 | .44±.18 | .18±.12 | .08±.08 | .66±.16 | .69±.03 | .59±.15 | .64±.03 | .27±.18 | .58±.14 |
| ReSYNC | .92±.05 | .76±.16 | .85±.09 | .68±.13 | .87±.09 | .95±.01 | .96±.02 | .64±.02 | .72±.03 | .76±.03 |
| Method | Blocked Stacking | Rearrange | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| T₁ train | T₁ test | T₂ train | T₂ test | T₁ | T₃ train | T₃ test | T₁,₂ | T₁ train | T₁ test | |
| GNN | .74±.03 | .08±.06 | .55±.11 | .12±.10 | .00±.00 | .56±.15 | .07±.06 | .03±.02 | .39±.12 | .00±.00 |
| TF | .74±.03 | .05±.10 | .66±.05 | .11±.12 | .68±.02 | .72±.05 | .08±.06 | .35±.16 | .15±.14 | .00±.00 |
| VLM-OS | .74±.01 | .04±.03 | .19±.15 | .02±.01 | .21±.10 | .00±.00 | .09±.08 | .23±.14 | .26±.02 | .00±.00 |
| DSG-S | .66±.05 | .04±.03 | .52±.14 | .01±.02 | .35±.19 | .68±.04 | .00±.00 | .17±.07 | .35±.03 | .00±.00 |
| DSG-M | .38±.06 | .05±.03 | .01±.02 | .01±.01 | .07±.10 | .00±.00 | .05±.03 | .05±.02 | .18±.03 | .15±.00 |
| RC | .75±.03 | .04±.04 | .61±.11 | .24±.18 | .39±.17 | .10±.03 | .16±.12 | .17±.14 | .00±.00 | .00±.00 |
| RC+IVNTR | .74±.02 | .70±.03 | .64±.04 | .17±.13 | .72±.02 | .64±.04 | .37±.17 | .09±.10 | .47±.12 | .48±.12 |
| ReSYNC | .73±.04 | .76±.05 | .67±.10 | .84±.06 | .76±.02 | .69±.05 | .72±.08 | .80±.06 | .59±.03 | .57±.03 |
Table 1. Success rates in four simulated domains, averaged over five random seeds (± standard deviation), 50 tasks per scenario. ReSYNC is competitive in seen tasks and consistently strongest on unseen compositions while retaining earlier skills.
Acknowledgements
This work was supported by AFRL and DARPA (FA8750-23-2-1015), DSTA, the ACCESS program, and a Princeton SEAS Innovation grant. This work is also supported by the NVIDIA Academic Grant Program using NVIDIA RTX PRO 6000 GPUs. We thank Patrick Yin for support on the real-robot and sim-to-real experiments, and Jiayuan Mao for valuable early feedback.