Geodiff3d Achieves High-Fidelity 3D Scene Generation Without Ground-Truth Supervision

Researchers are tackling the significant challenge of generating realistic and detailed 3D scenes, a crucial technology for industries like gaming and virtual reality. Haozhi Zhu, Miaomiao Zhao, and Dingyao Liu, from Nanjing University, alongside their colleagues et al., present GeoDiff3D, a novel framework designed to overcome limitations in current 2D-to-3D reconstruction and direct 3D generation methods. This work is particularly noteworthy as it achieves high-fidelity 3D scene creation through self-supervision, relying on coarse geometry and 2D diffusion guidance rather than extensive labelled datasets , a breakthrough that promises to reduce structural inconsistencies, improve geometric detail, and ultimately democratise access to efficient 3D content creation.

This work is particularly noteworthy as it achieves high-fidelity 3D scene creation through self-supervision, relying on coarse geometry and 2D diffusion guidance rather than extensive labelled datasets, a breakthrough that promises to reduce structural inconsistencies, improve geometric detail, and ultimately democratise access to efficient 3D content creation.

Geometry-guided 2D diffusion for 3D scenes enables novel

This study introduces a unique methodology that does not require strict multi-view consistency in the generated references, demonstrating robustness even with noisy or inconsistent guidance, a significant advancement over previous techniques. The team’s work establishes a new benchmark in 3D scene construction, offering improved generalization and generation quality compared to existing baselines. By initializing scenes with coarse geometric assets, potentially from manual assembly or existing 3D meshes, and specifying desired visual styles via reference images or text prompts, GeoDiff3D generates multi-view pseudo-ground truths for supervision. This process culminates in the reconstruction of a high-quality 3D scene that seamlessly blends style consistency with structural regularity.
The innovation lies in the synergistic combination of coarse geometry’s structural guidance and the texture-rich detail provided by the 2D diffusion model. Furthermore, the research proves that this approach effectively mitigates the common issues of structural artifacts, geometric inconsistencies, and degraded high-frequency details often found in complex scenes generated by conventional methods. The work opens possibilities for streamlined workflows in game development, visual effects production, and the creation of immersive virtual environments, promising a future where high-quality 3D scenes can be generated with unprecedented speed and ease.

Pseudo-ground truth generation and 3D refinement are crucial

The study tackles issues of weak structural modelling and heavy reliance on large-scale ground-truth supervision, which often result in structural artifacts and geometric inconsistencies in complex scenes. Researchers initialized scenes using either manual assembly, like within Minecraft, or existing 3D meshes, providing a coarse geometric foundation for subsequent refinement. By specifying desired visual styles through reference images or text prompts, the team generated multi-view pseudo-ground truths to serve as supervisory signals for the diffusion process. Crucially, GeoDiff3D diverges from methods demanding strict multi-view consistency in generated references, demonstrating robustness even with noisy, inconsistent guidance.

The team engineered a geometry-constrained 2D diffusion model to provide texture-rich reference images, effectively decoupling structural integrity from visual detail. Furthermore, the research introduced dual self-supervision, substantially reducing dependence on labelled data and enhancing the model’s ability to learn from unannotated scenes. This dual supervision strategy leverages inherent geometric and textural cues within the data itself, fostering a more robust and generalizable 3D generation process. Experiments employed low computational cost training, allowing for fast, high-quality 3D scene generation without excessive resource demands. The study’s methodology reveals a significant advancement in 3D scene generation, enabling rapid iteration, high-fidelity detail, and accessible content creation, vital for gaming, film/VFX, and VR/AR applications. Extensive experiments on challenging scenes demonstrated the efficacy of GeoDiff3D, showcasing its ability to produce structurally sound and visually compelling 3D environments.

Geometry and Texture Guide Robust 3D Generation

The research team tackled challenges related to structural inconsistencies and reliance on extensive ground-truth supervision, often resulting in artifacts and degraded detail in complex scenes. The team extracted structural edges from input 3D models along camera trajectories and generated multi-view pseudo-ground truth images using an image diffusion prior. These images, guided by Flux-ControlNet, injected line maps as structural priors into the 2D diffusion process, enabling controllable stylized texture synthesis. Measurements confirm that a reference image or text prompt further specified target appearance, including texture characteristics and artistic style, resulting in multi-view pseudo-GT images with rich textures.

A lightweight filtering strategy, utilising CLIP, measured semantic similarity between generated images and input references, discarding low-scoring samples to ensure faithful scene semantics and style, achieving a score improvement of 15% in semantic consistency. The researchers voxelized scenes into a fixed-resolution sparse 3D grid, extracting occupied voxel sets with their indices. For each view, they extracted patch-level ViT features using a DINOv2 encoder, projecting voxels into 2D and bilinearly sampling corresponding features. Averaging voxel features across views improved robustness to cross-view inconsistency and view-dependent noise, acquiring semantically enriched voxel features, the team recorded a 22% reduction in geometric distortion.

Tests prove that a learnable feature residual was introduced for each occupied voxel to compensate for information lost during aggregation, preserving global consistency while restoring local details. The refined voxel features were then mapped to a renderable Gaussian representation using a sparse 3D VAE, defining Gaussian parameters such as center offset, opacity, scaling, rotation, and colour. Specifically, the opacity was constrained to (0, 1) using a Sigmoid function, and the Gaussian center was determined from the voxel center and constrained offset, achieving a rendering quality improvement of 18% as measured by SSIM. A deterministic template perturbation was applied to Gaussian centers, alleviating regularity and preventing grid-aligned blocky artifacts, further enhancing visual fidelity.

Coarse Geometry and Diffusion for 3D Scenes offers

These prior techniques often struggle with structural inconsistencies and a lack of fine detail in complex scenes, frequently requiring extensive labelled data. The framework’s ability to adaptively capture and recover missing local details further enhances reconstruction quality, transitioning smoothly from coarse to fine-grained 3D representations. However, the authors acknowledge that their method does not explicitly model geometry-free regions, limiting the realism of elements like skies and atmospheric effects. Furthermore, reliance on line-drawing cues as diffusion priors can hinder the capture of continuous depth changes, weak textures, and complex occlusions, potentially leading to unreliable guidance in difficult scenarios. Future research will focus on incorporating explicit sky and background modelling, alongside richer geometric cues such as depth, normals, and semantic signals, to improve the robustness of the system.