Joint Optimization of 3D Gaussian Splatting and Pose Estimation Improves Scene Reconstruction Fidelity

Creating realistic views of a scene from arbitrary viewpoints, known as novel view synthesis, typically depends on accurate pre-calculated camera positions, a process that can introduce errors and limit performance. Yuxuan Li, Tao Wang, and Xianben Yang from Beijing Jiaotong University present a new framework that overcomes this limitation by simultaneously optimising both the 3D representation of a scene and the camera poses, removing the need for external tools. Their method iteratively refines the scene’s 3D structure and camera positions, ensuring improvements in both reconstruction quality and pose accuracy, and crucially, it addresses challenges posed by large viewpoint changes and limited visual features. The team’s approach demonstrably outperforms existing techniques that avoid pre-calibration, and even surpasses the accuracy of standard methods reliant on external tools, representing a significant advance in the field of 3D reconstruction.

Euler Angles Stabilize Rotation Optimization

Scientists addressed a critical challenge in visual SLAM, the process of simultaneously building a map and determining a device’s location, by improving the stability of rotation estimation. Directly optimizing rotation matrices can introduce errors, leading to instability. The team proposes using Euler angles, a sequence of rotations around specific axes, to represent rotation during optimization, maintaining mathematical correctness and improving numerical stability. The method represents rotation using three angles, roll, pitch, and yaw, defining rotations around the x, y, and z axes. Calculating how changes in these angles affect the rotation matrix requires precise mathematical calculations, known as Jacobians, which the team efficiently developed, crucial for optimizing the system. Experiments on standard datasets, including LLFF-NeRF, Shiny Dataset, and Tank and Temples, demonstrate the effectiveness of this approach. Results show Euler angle parameterization offers a balance between accuracy, efficiency, and numerical stability, providing a practical solution to a common problem in visual SLAM.

Co-Optimizing Gaussians and Camera Poses Simultaneously

Scientists developed a new method for reconstructing 3D scenes, eliminating the need for external calibration tools. The team pioneered a co-optimization strategy that simultaneously refines both 3D Gaussian points, representing the scene’s geometry, and camera positions. This iterative process improves the fidelity of the reconstructed scene and the accuracy of the estimated camera poses, addressing limitations in existing methods. The method operates by alternating between updating 3D Gaussian parameters with fixed camera poses and refining camera poses using a custom algorithm, termed LK3D, which integrates geometric and photometric constraints.

LK3D leverages image gradients and transformation-based projection error relationships to accurately estimate camera motion, even with sparse feature distributions. The alternating direction method drives the iterative optimization, ensuring stable convergence and improved pose accuracy. Extensive evaluations on Tanks and Temples, LLFF-NeRF, and Shiny datasets demonstrate the effectiveness of this approach, surpassing the accuracy of standard pipelines and establishing a new benchmark for novel view synthesis and 3D reconstruction.

Joint Optimization Refines Gaussian Scene Reconstruction

This research presents a breakthrough in novel view synthesis, delivering high-fidelity scene reconstruction without relying on external tools for initial camera pose estimation. The team developed a unified framework, termed JOGS, that simultaneously optimizes both 3D Gaussian parameters and camera poses, achieving stable convergence even with large viewpoint changes or sparse feature distributions. Initial experiments utilize Structure from Motion, followed by iterative refinement of the 3D representation and camera positions. The core innovation lies in a Lucas-Kanade 3D optical flow algorithm, which leverages Gaussians and image reprojection errors to refine camera poses.

This algorithm integrates image gradients with transformation-based projection error relationships, operating independently of sequential image relationships. The team demonstrates that this alternating optimization strategy significantly improves pose accuracy and achieves stable convergence. Validation across Tanks and Temples, LLFF-NeRF, and Shiny datasets shows that JOGS outperforms existing methods in novel view synthesis, establishing a new standard for pose-free 3D scene reconstruction. The team reports robust performance and stable convergence, even under challenging conditions with limited features or significant camera movement.

Gaussian Splitting Improves View Synthesis Accuracy

This research presents a novel framework for synthesizing new views of a scene, achieving accurate results without relying on pre-estimated camera poses. The team developed a method that simultaneously optimizes both 3D Gaussian points, representing the scene’s geometry, and camera poses, through an alternating co-optimization strategy. This iterative process refines the 3D representation and camera positions, leading to improved reconstruction quality and accuracy, particularly in challenging scenarios. The key innovation lies in decoupling the optimization process into two interleaved phases: updating the 3D Gaussian parameters with fixed poses, followed by refining the camera poses using a custom algorithm incorporating both geometric and photometric constraints.

This alternating approach effectively mitigates error accumulation and enhances the overall accuracy of the reconstructed scene. While the method demonstrates strong performance in both pose estimation and rendering, the authors acknowledge that it currently requires increased training time. Future work will focus on addressing this limitation by exploring parallel optimization strategies to accelerate the process and improve efficiency.