Neural Wavelet-domain Diffusion for 3D Shape Generation

Ka-Hei Hui¹

Ruihui Li²

Jingyu Hu¹

Chi-Wing Fu¹

SIGGRAPH Asia 2022 (Conference Track)

Our method is able to generate diverse shapes with complex structures and topology, fine details, and clean surfaces.

Overview of our approach. (a) Data preparation builds a compact wavelet representation (a pair of coarse and detail coefficient volumes) for each input shape using a truncated signed distance field (TSDF) and a multi-scale wavelet decomposition. (b) Shape learning trains the generator network to produce coarse coefficient volumes from random noise samples and trains the detail predictor network to produce detail coefficient volumes from coarse coefficient volumes. (c) Shape generation employs the trained generator to produce a coarse coefficient volume and then the trained detail predictor to further predict a compatible detail coefficient volume, followed by an inverse wavelet transform and marching cube, to generate the output 3D shape.

Abstract

This paper presents a new approach for 3D shape generation, enabling direct generative modeling on a continuous implicit representation in wavelet domain. Specifically, we propose a compact wavelet representation with a pair of coarse and detail coefficient volumes to implicitly represent 3D shapes via truncated signed distance functions and multi-scale biorthogonal wavelets, and formulate a pair of neural networks: a generator based on the diffusion model to produce diverse shapes in the form of coarse coefficient volumes; and a detail predictor to further produce compatible detail coefficient volumes for enriching the generated shapes with fine structures and details. Both quantitative and qualitative experimental results manifest the superiority of our approach in generating diverse and high-quality shapes with complex topology and structures, clean surfaces, and fine details, exceeding the 3D generation capabilities of the state-of-the-art models.

Results

Gallery of our generated shapes: Table, Chair, Cabinet, and Airplane (top to bottom). Our shapes exhibit complex structures, fine details, and clean surfaces, without obvious artifacts, compared with those generated by others.

Shape novelty analysis. Top: From our generated shape (in green), we retrieve top-four most similar shapes (in blue) in training set by CD and LFD. Bottom: We generate 500 chairs using our method; for each chair, we retrieve the most similar shape in the training set by LFD; then, we plot the distribution of LFDs for all retrievals, showing that our method is able to generate shapes that are more similar (low LFDs) or more novel (high LFDs) compared to the training set. Note that the generated shape at 50th percentile is already not that similar to the associated training-set shape.

Paper and Supplementary Material

Neural Wavelet-domain Diffusion for 3D Shape Generation
In SIGGRAPH Asia 2022.
[Paper] [supp] [code]

Acknowledgments

We would like to thank the anonymous reviewers for their valuable comments. We also acknowledge help from Tianyu Wang for various visualizations in the paper. This work is supported by Research Grants Council of the Hong Kong Special Administrative Region (Project No. CUHK 14206320 & 14201921) and National Natural Science Foundation of China (No. 62202151).