GSR: Learning Structured Reasoning for Embodied Manipulation

Despite rapid progress, embodied agents still struggle with long-horizon manipulation that requires maintaining spatial consistency, causal dependencies, and goal constraints. A key limitation of existing approaches is that task reasoning is implicitly embedded in high-dimensional latent representations, making it challenging to separate task structure from perceptual variability. We introduce Grounded Scene-graph Reasoning (GSR), a structured reasoning paradigm that explicitly models world-state evolution as transitions over semantically grounded scene graphs. By reasoning step-wise over object states and spatial relations, rather than directly mapping perception to actions, GSR enables explicit reasoning about action preconditions, consequences, and goal satisfaction in a physically grounded space. To support learning such reasoning, we construct Manip-Cognition-1.6M, a large-scale dataset that jointly supervises scene grounding, causal action reasoning, and goal-conditioned planning. Extensive evaluations across RLBench, LIBERO, GSR-benchmark, and real-world robotic tasks show that GSR significantly improves zero-shot generalization and long-horizon task completion over prompting-based baselines. These results highlight explicit world-state representations as a key inductive bias for scalable embodied reasoning.

Overview of GSR Framework. To this end, we introduce Grounded Scene-graph Reasoning (GSR), an embodied reasoning framework based on the principle that agents should plan over abstract world representations rather than raw visual information. GSR leverages semantically grounded scene graphs to extract stable causal structures from observations and explicitly separates high-level conceptual reasoning from low-level action execution. This design enables persistent and task-transferable capabilities, allowing flexible composition of sequenced action and robust adaptation across tasks. To train GSR, we construct the Manip-Cognition-1.6M, which provides joint supervision over world understanding, intention interpretation, and action planning across a diverse set of manipulation tasks.

Given an RGB-D observation \(\mathcal{I}\), we construct a 3D scene graph \(M_{sg} = (O_t, E_t)\) to encode the workspace and object states, where \(O_t = \{o_j\}_{j=1,...,J}\) denotes the set of objects and \(E_t = \{e_k\}_{k=1,...,K}\) denotes the set of relational edges. Figure above illustrates the transformation process from raw visual input to a structured representation and the resulting scene graph. Each object \(o_j\) is represented as a structured entity composed of functional keypoints and, when applicable, articulated child components. For example, a mug includes a "functional keypoint" corresponding to its handle, while a cabinet is modeled as an articulated object with "multiple child elements" such as drawers. Edges \(e_k\) encode spatial relations between object pairs, capturing predicates such as on, inside, or adjacent to (e.g., a mug on a table).

GSR is a fine-tuned Large Language Model (LLM) designed to perform commonsense reasoning over scene-graph representations. To apply GSR in a physical embodiment, we integrate it with a perception front-end and a action expert back-end. The physical system consists of two components: a perception-reasoning module for decision making, and an action expert that executes low-level control. The perception-reasoning module constructs scene graphs from raw observations and enable GSR reasons over these information to generate sequences of actions. To construct scene graphs, an Vision Foundation Model (VFM) is applied. The action expert leverages a meta-skill library, with further details described in the paper.