CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates

Shresth Grover¹, Priyank Pathak², Akash Kumar², Yogesh S Rawat²

¹University of California San Diego, ²University of Central Florida,

ECCV'26

Try CoSPlan Yourself
Paper (Coming Soon) 🤗 Dataset Code arXiv

Model Rankings on CoSPlan

Models ranked by average Step Completion accuracy across all CoSPlan tasks (using Scene Graph method)
Rank	Model	Robo-VQA-E	Shuffle-E	Maze-E	Blocks-World-E	Average Step Completion
1	GPT-4o	52.2	30.1	46.1	54.3	45.7
2	Intern-VLM	25.1	23.4	41.2	18.9	27.1
3	CoG-VLM	21.5	23.7	26.5	26.7	24.6
4	Janus-pro-7B	21.3	23.5	21.7	25.1	22.9
5	Qwen2-VL-8B	18.9	25.1	28.3	18.8	22.8
6	Random	20	20	20	20	20.0

Corrective Sequential Planning: Given the initial and final states, with already performed actions with some errors (initial context), model identifies errors in the provided context, and picks the optimal action steps to reach the final goal, correcting the error.

Abstract

Vision Language Models (VLMs) have shown promising planning capabilities, yet their success remains confined to the text domain, leaving visual decision-making relatively underexplored. Addressing this gap, we introduce Corrective Sequence Planning (CoSPlan) benchmark, where VLMs must plan a sequence of visual actions from an initial scene to a target scene. CoSPlan evaluates models on their ability to imagine and execute a coherent set of visual steps required to reach the goal (Step Completion). To prevent any shortcuts that simply describe the final scene, we introduce an erroneous action in decision making, which must be detected (Error Detection) and corrected to reach the goal, enabling a deeper understanding of the task. CoSPlan spans across 4 tasks: maze navigation, block re-arrangement, image reconstruction, and object re-organization. Despite using advanced reasoning strategies such as Chain-of-Thought and Scene Graphs, VLMs struggle on CoSPlan, while still showing promising performance in the text domain. Addressing this, we propose Scene Graph Incremental updates (SGI), a novel training-free method to transform images into ‘textual’ scene graphs, enabling step-by-step reasoning through iterative scene graph refinement. SGI yields an average of ~4.4% ↑ on CoSPlan w/ generalization on PlanBench and VQA. Link for solving puzzles above.

CoSPlan Benchmark

We introduce CoSPlan (Corrective Sequence Planning), a benchmark designed to study VLMs' planning capabilities in erroneous scenarios. CoSPlan focuses on 2D spatial vision tasks guided by text-based instructions, requiring models to plan a temporal sequence of actions toward a goal (temporal), while detecting and correcting an erroneous action.

CoSPlan includes four diverse tasks:

Maze-E: Navigation in a 2D maze with obstacles and erroneous moves.
Blocks-World-E: Re-arranging colored blocks into a target configuration.
Shuffle-E: Reconstructing shuffled image tiles to form the original image.
Robo-VQA-E: Re-organizing real-world objects based on instructions.

Dataset Stats
Model	Task	Type	Size	Initial Context (# of already performed actions)	Remaining Steps (#remaining steps to reach goal)	Source
Maze-E	Navigation	Path Planning	5000	2.0	4.6	Synthetic
Blocks-World-E	Re-arrange	Blocks	5000	2.0	3.8	Synthetic
Shuffle-E	Re-construct	Puzzle	1000	3.7	7.1	ImageNet
Robo-VQA-E	Re-organize	Real-world	350	5.5	4.1	ROM

(a) Maze E (Navigating): Only navigating (→, ↑, ←, ↓) from green to blue cell, avoiding cells marked in red

(b) Blocks World (Rearrangement): X from (a) → (b) rearranges block ‘X’ from column ‘a’ to column ‘b’.

(c) Shuffle E (Reconstruction): Re-constructing target image Ig by swapping (↔) image patches.

(d) Robo VQA (Reorganization): Re-organizing real-world objects

Overview of CoSPlan Benchmark Datasets: Maze-E, Blocks-World-E, Shuffle-E, and Robo-VQA-E.

Analysis

Why Model Give wrong Results (Generic)

(a) As number of MCQ option goes up accuracy falls

(b) Models have a strong bias towards picking option A

(c) Models dont reason well with visual problems

Why Handling Error is difficult (Ours)

(a) Without Error models have a higher accuracy

(b) Error related to objects in the scene (in-context) are harder to handle

(c) The higher the intial context (already performed actions) better the accuracy

(d) Number of steps required to reach the goal. Models dont seem to take advantage of projected paths towards goal in picking up the option.

Solution : Scene Graph Incremental Update (SGI)

Motivation: With only initial and final images, models internally interpolate missing states (CoT & SG), a process they struggle in the visual domain, even though they perform with relative ease in the text domain

Solution: We leverage SG to transform visual images into textual representations, and apply an iterative step-by-step reasoning on text space instead of the visual domain. Instead of reasoning within a single static scene, SGI explicitly derives next-time-frame scene graphs as actions unfold in evolving scenes (See Gif below). This incremental formulation bridges the gap between the initial and final states via explicit intermediate states, and decomposing reasoning into smaller transitions, thereby improving corrective sequence planning and error detection.

SGI consists of three main steps:

Vanilla Scene Graphs (SG): Generate initial and goal Scene Graphs.
Incremental Scene Update: Simulate each action to update the Scene Graph incrementally, creating intermediate representations.
Similarity Comparison: Compare the resulting Scene Graph with the goal Scene Graph to select the correct sequence of actions.

SGI Method: 1) Initial and Goal Scene Graphs (SG) are generated. 2) Incremental Scene Update sequentially modifies SG for each action. 3) Similarity Comparison matches the resultant SG with Goal graph for searching for the best-aligned sequence.

Scene Graph Incremental Update (SGI) Results

SGI improvement relative to vanilla Scene Graph (SG) method. All values show percentage accuracy (↑ higher is better).

Step Completion Performance

Method	Robo-VQA-E		Shuffle-E		Maze-E		Blocks-World-E
Method	SG	SGI	SG	SGI	SG	SGI	SG	SGI
Intern-VLM - 2	25.1	32.1 (+7.0)	23.4	25.2 (+1.8)	41.2	43.2 (+2.0)	18.9	29.2 (+10.3)
Intern-VLM - 3	31.4	33.7 (+2.3)	27.1	28.6 (+1.5)	50.1	54.8 (+4.7)	29.4	30.6 (+1.2)
GPT-4o	52.2	56.4 (+4.2)	30.1	37.0 (+6.9)	46.1	56.1 (+10.0)	54.3	55.3 (+1.0)

Error Detection Performance

Method	Robo-VQA-E		Maze-E		Blocks-World-E
Method	SG	SGI	SG	SGI	SG	SGI
Intern-VLM - 2	26.1	31.5 (+5.4)	33.4	34.8 (+1.4)	37.3	42.9 (+5.6)
Intern-VLM - 3	28.1	29.5 (+1.4)	35.1	36.3 (+1.2)	44.3	45.1 (+0.8)
GPT-4o	44.2	57.4 (+13.2)	35.3	41.1 (+5.8)	42.1	50.7 (+8.6)

Additional SGI Results

SGI improves performance on error-free scenarios

SGI on VQA dataset
Method	Spatial Map			Maze Nav			Spatial Grid
Method	CoT	SG	SGI	CoT	SG	SGI	CoT	SG	SGI
CoG VLM	25.1	36.7	35.8	32.3	32.4	31.2	30.1	34.3	38.2
Janus pro 7B	42.4	47.4	47.8	20.8	27.3	29.3	34.4	35.8	36.3
Intern VLM	36.3	41.3	44.3	28.6	40.5	42.1	33.3	33.8	35.1

SGI (Qwen2 VL 8B) on PlanBench (Task 8)
Method	Score
Vanilla	13.8
CoT	14.1
SG	13.9
SGI (our)	14.7

BibTeX

@misc{grover2026cosplan,
      title={CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates}, 
      author={Shresth Grover and Priyank Pathak and Akash Kumar and Yogesh S Rawat},
      year={2026},
      eprint={2512.10342},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.10342}, 
      }