ARTDECO: Towards Efficient and High-Fidelity On-the-Fly 3D Reconstruction with Structured Scene Representation

Guanghao Li*1,2,3, Kerui Ren*1,4, Lining Xu1,5, Zhewen Zheng1,6, Changjian Jiang1,7,
Xin Gao1,2, Bo Dai8, Jian Pu$^{\dagger}$2, Mulin Yu$^{\dagger}$1, Jiangmiao Pang1
1Shanghai Artificial Intelligence Laboratory, 2Fudan University, 3Shanghai Innovation Institute,
4Shanghai Jiao Tong University, 5The Chinese University of Hong Kong,
6Carnegie Mellon University 7Zhejiang University 8The University of Hong Kong

* denotes joint contribution. $\dagger$ denotes corresponding authors.
arXiv Code

TL;DR: ARTDECO unifies 3D foundation priors with structured scene representations, enabling robust and generalizable 3D reconstruction of diverse real-world scenes using only monocular video.

Abstract

MY ALT TEXT

We propose ARTDECO, a unified framework that employs 3D foundation models for pose estimation and point prediction, coupled with a Gaussian decoder that transforms multi-scale features into a hierarchical Gaussian representation with a LoD-aware rendering strategy, which improves rendering fidelity while reducing redundancy.

Method

MY ALT TEXT

Frontend and backend modules. (a) Frontend: Images are captured from the scene and streamed into the front-end part. Each incoming frame is aligned with the latest keyframe using a matching module to compute pixel correspondences. Based on the correspondence ratio and pixel displacement, the frame is classified as a keyframe, a mapper frame, or a common frame. The selected frame, along with its pose and point cloud, is then passed to the back-end. (b) Backend: For each new keyframe, a loop-detection module evaluates its similarity with previous keyframes. If a loop is detected, the most relevant candidates are refined and connected in the factor graph; otherwise, the keyframe is linked only to recent frames. Finally, global pose optimization is performed with Gauss–Newton, and other frames are adjusted accordingly. We instantiate the matching module with MASt3R and the loop-detection module with Pi3.

MY ALT TEXT

Mapping process. When a keyframe or mapper frame arrives from the backend, new Gaussians are added to the scene. Multi-resolution inputs are analyzed with the Laplacian of Gaussian (LoG) operator to identify regions that require refinement, and new Gaussians are initialized at the corresponding monocular depth positions in the current view. Common frames are not used to add Gaussians but contribute through gradient-based refinement. Each primitive stores position, spherical harmonics (SH), base scale, opacity, local feature, dmax, and voxel index vid. For rendering, the dmax attribute determines whether a Gaussian is included at a given viewing distance, enabling consistent level-of-detail control.

Results

MY ALT TEXT MY ALT TEXT

Qualitative comparisons. against popular on-the-fly reconstruction baselines across diverse 3D scene datasets. ARTDECO consistently preserves high-quality rendering details in complex and diverse environments, particularly in the regions highlighted with colored rectangles.