"Make a short line out of these objects in the center of the table"
StructDiffusion
Building semantically meaningful structures requires satisfying two different sets of constraints: (1) we must place objects in the correct positions to satisfy the desired spatio-semantic relations; (2) we must ensure that objects are not colliding and arrangements are structurally sound.
We propose StructDiffusion, a single framework that jointly optimizes over these two, sometimes contrasting, constraints.
Combining Diffusion Model and Transformer for Object-Centric Prediction
We use unknown object instance segmentation to break our scene up into objects, as per prior work (e.g., [1], [2], [3]). Then, we use a multi-modal transformer to combine both word tokens and object encodings from Point Cloud Transformer in order to make 6-DoF goal pose predictions. These predictions are both refined iteratively via diffusion and selected with a discriminator model that learns to recognize unrealistic samples.
Our diffusion model is integrated with a transformer model that maintains an individual attention stream for each object. This object-centric approach allows us to focus on learning the interactions between objects based on their geometric features as well as the grounding of abstract concepts on spatio-semantic relations between objects (e.g., large, circle, top).
We start from the last step of the reverse diffusion process and jointly predict goal poses for all objects in the scene. This allows our model to reason about object-object interactions in a generalizable way, which outperforms simply predicting goal poses from multi-modal inputs.