GSDF: 3DGS Meets SDF for Improved Rendering and Reconstruction

1 Shanghai Artificial Intelligence Laboratory, 2 The Chinese University of Hong Kong, 3 University of Science and Technology of China, 4 Cornell University

TL;DR: We propose GSDF, a dual-branch system that enhances rendering and reconstruction at the same time, leveraging the mutual geometry regularization and guidance between Gaussain primitives and neural surface.

From randomly initialized points, GSDF achieves better rendering quality, especially in the texture-less areas (i.e., grass).

Method Overview

Framework. Our dual-branch framework comprises a GS-branch dedicated to rendering and an SDF-branch focusing on learning neural surfaces. Our design effectively preserves the superiority of rendering with Gaussian primitives in terms of efficiency and fidelity, and also more accurately approximates scene surfaces from an SDF field adapted from NeuS. Concretely, (1) we leverage the efficiency and flexibility advantages of the GS-branch, to render depth maps and guide the ray sampling process of the SDF-branch. For each depth position, we query the SDF-branch to obtain its absolute SDF value |s|, and uniformly sample points within 2k|s| (e.g. k = 4). (2) The predicted SDF values from the SDF-branch are in turn used to guide the density control of the GS-branch to grow Gaussian primitives in near-surface regions and prune the ones that are far-away. (3) We further enforce mutual geometry consistency by comparing the depth and normal maps from each branch to encourage more coherent physical alignment between Gaussian primitives and surfaces.

Reconstruction Results

Neuralangelo Instantnsr GSDF

Reconstruction results with 30k training iterations. The GS-branch accelerates the convergence of the SDF-branch.

Neuralangelo Instantnsr GSDF

Reconstruction results with 500k training iterations. GSDF delivers enhanced precision in reconstruction outcomes.

GSDF yields more surface-aligned primitives with regularized structure.

Rendering Results

Using PhysGaussian to simulate GSDF checkpoints.

GSDF outperforms baselines. In particularly, it can model delicate geometries (1st & 2nd row), anti-blur in texture-less regions and anti-floater in less observations regions (3rd & 4th row).

More obvious improvements for GSDF are shown when the 3D Gaussians are randomly initialized.