Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians

Buildings are primary components of cities, often featuring repeated elements such as windows and doors. Traditional 3D building asset creation is labor-intensive and requires specialized skills to develop design rules. Recent generative models for building creation often overlook these patterns, leading to low visual fidelity and limited scalability. Drawing inspiration from procedural modeling techniques used in the gaming and visual effects industry, our method, Proc-GS, integrates procedural code into the 3D Gaussian Splatting (3D-GS) framework, leveraging their advantages in high-fidelity rendering and efficient asset management from both worlds. By manipulating procedural code, we can streamline this process and generate an infinite variety of buildings. This integration significantly reduces model size by utilizing shared foundational assets, enabling scalable generation with precise control over building assembly. We showcase the potential for expansive cityscape generation while maintaining high rendering fidelity and precise control on both real and synthetic cases.

Our pipeline consists of two stages: (1) Asset Acquisition: We acquire the base assets in the training process of the 3D-GS. These assets are then assembled according to procedural code and add variance assets to create a complete building, which is used for novel view synthesis with Gaussian Splatting. (2) Asset Assembly: We use the building generator and city layout generator to assemble these base assets into a vivid 3D city. Users provide basic urban spatial data (purple city boundary and green primary road) to city layout generator to automatically predict other roads and building layouts. The building generator then assembles base assets into complete buildings using procedural code and predicted building parameters.

Reconstruction Results. The left section shows results from three real-world scenes, while the right section presents results from the MatrixBuilding dataset. Proc-GS achieves rendering quality comparable to 3D-GS. Green boxes in each image highlight pairs of instantiations that share the same base assets, illustrating our method's capability to effectively model variations in geometry and appearance.

City Generation Results. Our Proc-GS framework learns base assets during the training process of 3D-GS using procedural codes, which are then manipulated to assemble these assets into a cohesive 3D city. Compared to other generation-based methods, Proc-GS demonstrates superior visual quality in both aerial and street-level views, especially in architectural details. We recommend zoom-in for detailed inspection.