Generative 3D Part Assembly via
Dynamic Graph Learning

Jialei Huang1*     Guanqi Zhan1*     Qingnan Fan2     Kaichun Mo2    
Lin Shao2     Baoquan Chen1     Leonidas J. Guibas2 Hao Dong1    
(*: indicates joint first authors)

1 Peking University     2 Stanford University    

Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS) 2020

[ArXiv Preprint] [Code] [BibTex]

Autonomous part assembly is a challenging yet crucial task in 3D computer vision and robotics. Analogous to buying an IKEA furniture, given a set of 3D parts that can assemble a single shape, an intelligent agent needs to perceive the 3D part geometry, reason to propose pose estimations for the input parts, and finally call robotic planning and control routines for actuation. In this paper, we focus on the pose estimation subproblem from the vision side involving geometric and relational reasoning over the input part geometry. Essentially, the task of generative 3D part assembly is to predict a 6-DoF part pose, including a rigid rotation and translation, for each input part that assembles a single 3D shape as the final output. To tackle this problem, we propose an assembly-oriented dynamic graph learning framework that leverages an iterative graph neural network as a backbone. It explicitly conducts sequential part assembly refinements in a coarse-to-fine manner, exploits a pair of part relation reasoning module and part aggregation module for dynamically adjusting both part features and their relations in the part graph. We conduct extensive experiments and quantitative comparisons to three strong baseline methods, demonstrating the effectiveness of the proposed approach.

Coarse-to-Fine Assembly

Figure 1. Coarse-to-Fine Part Assembly. Our iterative part assembly network learns to assemble parts to a shape in a coarse-to-fine manner.

Network Architecture

Figure 2. Network Architecture. The proposed dynamic graph learning framework. The iterative graph neural network backbone takes a set of part point clouds as inputs and conducts 5 iterations of graph message-passing for coarse-to-fine part assembly refinements.

Qualitative Results

Figure 3. Qualitative Comparisons to Baseline Methods. Visual comparisons between our algorithm and the baseline methods.

Quantitative Comparisons

Figure 4. Quantitative Comparisons to Baseline Methods. We show quantitative comparisons between our algorithm and the baseline methods.


This work was supported by the start-up research funds from Peking University (7100602564) and the Center on Frontiers of Computing Studies (7100602567). We would also like to thank Imperial Institute of Advanced Technology for GPU supports.