Virtual Reality Advances with Nebula, Delivering City-Scale 3D Gaussian Splatting and 1925% Data Savings

Recent advances in 3D Gaussian splatting promise realistic architectural visualisation, but current methods struggle with the demands of very large-scale models and smooth virtual reality experiences. To address these challenges, He Zhu, Zheng Liu, Xingyang Li, and colleagues present Nebula, a new framework that accelerates collaborative rendering for city-scale 3D Gaussian splatting in virtual reality. The team achieves significant performance gains by streaming intermediate rendering results instead of complete video feeds, reducing data communication by an impressive 1925%. Furthermore, Nebula incorporates a temporal-aware approach to optimise data access and a novel stereo rasterisation technique that minimises redundant rendering, ultimately delivering a 2.7x speedup and paving the way for immersive, high-fidelity virtual environments.

Gaussian Splatting Optimisation and Realtime Rendering

Research in rendering technologies focuses on Gaussian Splatting (GS) and Neural Radiance Fields (NeRF) as foundational techniques for image-based rendering and volumetric data processing. Key areas of investigation include optimizing rendering speed, reducing memory footprint, and enabling the rendering of large-scale 3D scenes. This involves advancements in compression, caching, data streaming, and algorithm-hardware co-design to improve performance and power efficiency. A significant trend is the shift towards edge computing, offloading rendering tasks to devices like mobile phones and AR/VR headsets to reduce latency and bandwidth requirements. Researchers are also exploring specialized hardware architectures, including GPUs and custom accelerators, to boost rendering performance. These efforts aim to deliver immersive experiences for applications like virtual reality, augmented reality, mixed reality, 360° video streaming, and the metaverse.

Nebula, Collaborative Gaussian Splatting for Virtual Reality

Researchers developed Nebula, a collaborative rendering framework designed to overcome limitations in delivering large-scale 3D Gaussian Splatting (3DGS) content for virtual reality. By transmitting intermediate rendering results instead of fully rendered images, Nebula reduces data communication by up to 1925%, significantly alleviating bandwidth pressure. A temporal-aware level-of-detail (LoD) search algorithm, deployed on existing GPUs, regulates memory access and leverages consistency between frames to minimize redundant data access. For virtual reality rendering, a novel stereo rasterization pipeline exploits triangulation to share computations between eyes, achieving bit-accurate images with minimal overhead.

This results in a 2.7x improvement in motion-to-photon speed and up to a 52.7x speedup in cloud-based LoD search compared to standard GPU implementations. The system achieves up to a 21.7x speedup compared to a mobile Ampere GPU and a 5.3x speedup compared to existing accelerators, demonstrating its potential for immersive, high-fidelity VR experiences.

Nebula Enables Scalable 3D Gaussian Splatting

Nebula is a collaborative rendering framework designed to address scalability limitations in large-scale 3D Gaussian splatting. By transmitting intermediate results immediately following the level-of-detail (LoD) search, Nebula reduces data communication by up to 1925% and exhibits strong scalability, remaining less sensitive to increases in resolution or frame rate. A temporal-aware LoD search algorithm regulates memory access and leverages similarities between consecutive frames to minimize redundant data access, delivering up to a 52.7x speedup compared to standard GPU implementations. The system also incorporates a runtime Gaussian management system that compresses and transmits only unique Gaussians between frames, further reducing data transfer.

A novel stereo rasterization pipeline, designed for stereo displays, exploits triangulation to share computations and produce bit-accurate images. These advancements result in a 2.7x motion-to-photon speedup and a 1925% reduction in bandwidth compared to conventional lossy video streaming, achieving up to a 21.7x speedup compared to a mobile Ampere GPU and a 5.3x speedup compared to state-of-the-art accelerators.

Nebula Delivers Scalable, High-Fidelity Virtual Reality

Nebula represents a significant step towards realising truly immersive, large-scale virtual reality experiences using 3D Gaussian splatting. The collaborative cloud-client rendering framework overcomes communication bottlenecks by streaming intermediate rendering results instead of complete video feeds, reducing data transfer requirements by up to 1925%. A novel stereo rasterization technique minimises redundancy during the rendering of images for both eyes, further optimising performance. Testing demonstrates that Nebula achieves a 2.7x speedup in rendering while drastically reducing bandwidth demands compared to conventional lossy video streaming. Researchers have successfully demonstrated the feasibility of city-scale 3D Gaussian splatting in virtual reality, paving the way for broader applications of this collaborative rendering approach.