Neural Template: Topology-aware Reconstruction and Disentangled Generation of 3D Meshes

Ka-Hei Hui*1
Ruihui Li*2,1
Jingyu Hu1
Chi-Wing Fu1
(* joint 1st authors)

CVPR 2022

Our DT-Net learns to construct a topology-aware neural template (b) adapted to the input (a) and then deform it towards an accurate 3D mesh while preserving the initial (learned) topology. This decoupled design enables a disentangled latent representation of \(\textcolor[rgb]{0.00,0.07,1.00}{topology}\) \(Z_T\) and \(\textcolor[rgb]{1.00,0.07,0.00}{shape}\) \(Z_S\), promoting controllable 3D mesh generation, e.g., remixing codes for object re-synthesis.

Abstract

This paper introduces a novel framework called DTNet for 3D mesh reconstruction and generation via Disentangled Topology. Beyond previous works, we learn a topology-aware neural template specific to each input then deform the template to reconstruct a detailed mesh while preserving the learned topology. One key insight is to decouple the complex mesh reconstruction into two sub-tasks: topology formulation and shape deformation. Thanks to the decoupling, DT-Net implicitly learns a disentangled representation for the topology and shape in the latent space. Hence, it can enable novel disentangled controls for supporting various shape generation applications, e.g., remix the topologies of 3D objects, that are not achievable by previous reconstruction works. Extensive experimental results demonstrate that our method is able to produce highquality meshes, particularly with diverse topologies, as compared with the state-of-the-art methods.


Results


Galleries showcasing the results produced by our DT-Net. Each pair shows the learned topology-aware neural template (left) and the associated reconstructed object (right). The produced objects cover various shapes and diverse topologies, ranging from smooth surface (e.g., car and lamp), to complex geometry (e.g., chair and airplane). It is observed that the neural template visually appears like a coarse version of the final shape, even without regularizing the amplitude of the deformation module.
Our DT-Net framework learns a disentangled representation for the topology and shape, thus facilitating novel generation applications via disentangled manipulation on the topology code \(\mathbf{Z_T}\) and/or the shape code \(\mathbf{Z_S}\), e.g., (a) remix the shape and topology of two different objects; (b) object interpolation by manipulating the topology/shape code; and (c) arithmetic operations in the latent space.

Paper and Supplementary Material

Neural Template: Topology-aware Reconstruction and Disentangled Generation of 3D Meshes
In CVPR 2022.
[Paper] [supp] [code]


Video

Bibtex

            @inproceedings{hui2022template,
                title = {Neural Template: Topology-aware Reconstruction and Disentangled Generation of 3D Meshes},
                author = {Ka-Hei Hui* and Ruihui Li and Jingyu Hu and Chi-Wing Fu(* joint first authors)},
                booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
                year={2022}
            }

Acknowledgments

We thank anonymous reviewers for the valuable comments. This work is supported by the Research Grants Council of the Hong Kong Special Administrative Region (Project No. CUHK 14206320 & 14201921).