Accepted to CoRL 2026

Recover, Discover, Plan:
Learning Skills and Concepts from Robot Failures

Bowen Li^1,2Mayank Mishra¹ Y. Isabel Liu³Stone Tao⁴ Nishanth Kumar⁵Alexander Gray² Ruwan Wickramarachchi⁶Jonathan Francis^1,6 Sebastian Scherer¹Tom Silver³

¹CMU²Centaur AI³Princeton⁴UCSD⁵MIT⁶Bosch Center for AI

Paper Code See how it works

ReSYNC turns a recovery learned from one failure into an abstract skill the robot can plan with anywhere.

One failure. A reusable idea.

From failure to foresight

Follow the four moments that turn a brittle robot into a planner that learns.

Deployment

Initial skills meet a hard corner.

The robot knows Pick and Insert. A block against the wall makes Pick fail.

Pick(block)Insert(block, block)×

Pick failed

Recovery learning

Spawn worlds. Learn one way out.

Parallel simulation starts at the failure. PPO learns a relational Poke policy; success means planning can resume.

π_poke

Many robot simulations running in parallel

GPU parallel simulation

Abstract planning

A new task. The right skill, earlier.

The goal changes. ReSYNC reasons backward from Insert, schedules Poke first, and avoids failure altogether.

PokePickInsert✓

Replanned before failure

One framework, many physical worlds

Skills emerge where plans break

Six worlds. Familiar recoveries, new plans. Hover over a domain—or tap it—to replay both scenarios.

Icy Transport

IcyDrive and MuddyDrive cross slippery or high-friction regions. The previews show IcyDrive.

2D transport

trainIce on the way to an object

testIce on the way to the delivery goal

Blocked Stacking

Punch, Wiggle, and Fiddle dislodge different obstructions. The previews show Punch.

2D manipulation

trainClear the block to be grasped

testClear the stacking target

Cluttered Drawer

Pull opens the drawer; Wiggle frees a cluttered grasp. The previews show Pull.

ManiSkill · mobile manipulation

trainOpen the drawer to retrieve the hammer

testOpen the drawer to reach the placement target

Cluttered Room

Push moves a can out of an occupied goal before object placement. Complete episodes use a wide view, zooming in on the obstructed goal during the slow-motion RL push.

ManiSkill · rearrangement

trainClear the box’s placement goal

testReuse Push at the bowl’s placement goal

Flipped Stacking

Flip turns an inverted block upright before pick-and-place. RL at 1×; planned motions accelerated.

IsaacLab · manipulation

trainFlip one block, then stack two

testFlip two blocks, then build a four-block tower

Cornered Stacking

A learned corner recovery repositions a block near the enclosure walls for stacking. RL at 1×; planned motions accelerated.

IsaacLab · manipulation

trainRecover the block to be picked up

testRecover the block used as the stacking support

Selected successful inference rollouts · Train: familiar recovery setup · Test: transfer or composition · IsaacLab RL at 1×; planned motions accelerated. IsaacLab examples are demonstrations, separate from the four benchmark evaluations.

Reactive recovery

Why recovery alone is not enough

Too late is a different state.

A reactive policy waits for insertion to fail. By then another block is involved, the scene is out-of-distribution, and recovery cannot restore a solvable state.

Insert failsRecover lateOOD state

ReSYNC plans Poke first.
It reuses recovery while the state is still recoverable.

Empirical results

Generalization is the test.

Success rate across four simulated domains. Gray columns are held-out, compositionally novel tasks.

70%average success
on novel tasks

Method	Icy Transport					Cluttered Drawer
Method	T₁ train	T₁ test	T₂ train	T₂ test	T₁	T₁ train	T₁ test	T₂ train	T₂ test	T₁
GNN	.98±.03	.01±.01	.18±.13	.18±.13	.38±.18	.86±.04	.00±.00	.69±.07	.48±.10	.00±.00
TF	.98±.03	.00±.00	.26±.14	.17±.12	.52±.16	.11±.07	.00±.00	.03±.03	.02±.02	.00±.00
VLM-OS	.39±.13	.00±.00	.30±.14	.00±.00	.24±.12	.68±.06	.29±.03	.18±.04	.25±.06	.50±.10
DSG-S	.67±.12	.00±.00	.24±.07	.02±.02	.44±.08	.73±.03	.04±.01	.67±.04	.04±.01	.38±.02
DSG-M	.61±.04	.33±.04	.15±.07	.01±.02	.21±.05	.13±.02	.03±.01	.00±.00	.00±.00	.09±.04
RC	.71±.11	.00±.00	.34±.16	.00±.00	.50±.18	.68±.03	.01±.01	.62±.04	.14±.02	.35±.15
RC+IVNTR	.50±.20	.44±.18	.18±.12	.08±.08	.66±.16	.69±.03	.59±.15	.64±.03	.27±.18	.58±.14
ReSYNC	.92±.05	.76±.16	.85±.09	.68±.13	.87±.09	.95±.01	.96±.02	.64±.02	.72±.03	.76±.03

Method	Blocked Stacking								Rearrange
Method	T₁ train	T₁ test	T₂ train	T₂ test	T₁	T₃ train	T₃ test	T₁,₂	T₁ train	T₁ test
GNN	.74±.03	.08±.06	.55±.11	.12±.10	.00±.00	.56±.15	.07±.06	.03±.02	.39±.12	.00±.00
TF	.74±.03	.05±.10	.66±.05	.11±.12	.68±.02	.72±.05	.08±.06	.35±.16	.15±.14	.00±.00
VLM-OS	.74±.01	.04±.03	.19±.15	.02±.01	.21±.10	.00±.00	.09±.08	.23±.14	.26±.02	.00±.00
DSG-S	.66±.05	.04±.03	.52±.14	.01±.02	.35±.19	.68±.04	.00±.00	.17±.07	.35±.03	.00±.00
DSG-M	.38±.06	.05±.03	.01±.02	.01±.01	.07±.10	.00±.00	.05±.03	.05±.02	.18±.03	.15±.00
RC	.75±.03	.04±.04	.61±.11	.24±.18	.39±.17	.10±.03	.16±.12	.17±.14	.00±.00	.00±.00
RC+IVNTR	.74±.02	.70±.03	.64±.04	.17±.13	.72±.02	.64±.04	.37±.17	.09±.10	.47±.12	.48±.12
ReSYNC	.73±.04	.76±.05	.67±.10	.84±.06	.76±.02	.69±.05	.72±.08	.80±.06	.59±.03	.57±.03

Table 1. Success rates in four simulated domains, averaged over five random seeds (± standard deviation), 50 tasks per scenario. ReSYNC is competitive in seen tasks and consistently strongest on unseen compositions while retaining earlier skills.

Acknowledgements

Built with generous support.

This work was supported by AFRL and DARPA (FA8750-23-2-1015), DSTA, the ACCESS program, and a Princeton SEAS Innovation grant. This work is also supported by the NVIDIA Academic Grant Program using NVIDIA RTX PRO 6000 GPUs. We thank Patrick Yin for support on the real-robot and sim-to-real experiments, and Jiayuan Mao for valuable early feedback.

Recover, Discover, Plan:Learning Skills and Concepts from Robot Failures