Changing vehicle designs fundamentally alter how an autonomous vehicle perceives the world; moving perception software from an SUV to a sedan shifts the view of critical elements like traffic light placement and curb visibility. NVIDIA Omniverse NuRec addresses the challenge of adapting this software across new carlines without the prohibitive cost of collecting and labeling entirely new real-world datasets, particularly during early development stages.
The system reconstructs real-world driving scenes and renders new camera views specifically for an allowing developers to identify “weak spots” before needing to gather additional data. NuRec uses 3D Gaussian splatting to achieve this, enabling teams to ask questions such as “What would this scenario look like from the target rig?”
NVIDIA Omniverse NuRec Enables Carline Adaptation for AV Perception
NuRec utilizes 3D Gaussian splatting, a technique that reconstructs real-world environments from sensor data and renders them for simulation, to address the challenges of adapting autonomous vehicle perception software to new vehicle platforms. This capability allows for the rendering of scenes with altered camera characteristics, including extrinsics, intrinsics, field of view, and lens models such as pinhole, fisheye, and f-theta configurations.
By reconstructing drives captured by an existing vehicle, NuRec generates new camera streams simulating the viewpoints of a target vehicle, even before physical prototypes are available. The system’s ability to retain original rig trajectories, per-camera calibration, dynamic object tracks, and map data is central to its effectiveness; these assets are used to align and adapt labels, objects, lanes, traffic lights, and road boundaries, within the newly rendered views.
This reusability of scene data and annotations significantly reduces the need for extensive re-labeling efforts, a traditionally expensive and time-consuming process in autonomous vehicle development, NVIDIA says. NVIDIA evaluated this workflow internally within an automated driving program requiring support for a new camera configuration, successfully utilizing an existing library of drives from a different vehicle where target carline data was initially unavailable. The workflow begins with downloading reconstructed scenes from the NVIDIA Physical AI NuRec Dataset, a collection of over 1,500 neural-reconstructed driving scenarios available on Hugging Face.
These scenes are then rendered through the target carline’s sensor rig, and the resulting frames are refined using NVIDIA Harmonizer, a process designed to enhance image quality and realism. “Pairing a reconstructed drive with a target rig configuration” is a critical first step, according to the developers, enabling the system to generate views tailored to the new vehicle’s sensor setup.
To access the NuRec container, users are instructed to pin it to its immutable digest and pull the image using a specific docker command: export NUREC_IMAGE=’nvcr. io/nvidia/nre/nre-ga: @sha256:97f43e7130c5636ce3e80ea3184d97f56a87fdd989b05cce42230881dbdea284′; docker pull. Following rendering and refinement, the generated dataset is prepared for perception-model training, treated similarly to real-world camera data collected from physical vehicles. Developers can then export the trajectory and rendering as separate container operations. The team emphasizes the importance of validating the output before incorporating it into the training pipeline, ensuring the synthetic data accurately reflects real-world conditions.
NuRec also facilitates inspection of the reconstructed sequence using the public NCore viewer, accessible via http://localhost:8080. Users can further define paths for the generated data by setting the NCORE_DIR environment variable and running bazel run //tools/ncore_vis — \ –component-group=. The system’s ability to address carline adaptation challenges stems from its capacity to not only reconstruct environments but also to render them from the perspective of a vehicle that may not yet exist, effectively bridging the gap between simulation and real-world testing and reducing reliance on costly and time-intensive data collection for each new vehicle configuration.
The 26.04 GA image exposes several tools through one entry point, so the workflow explicitly selects the ncore-aux-data subcommand: export NUREC_AUX_IMAGE=nvcr.
3D Gaussian Splatting Reconstructs Environments for Simulation
NuRec uses 3D Gaussian splatting not simply to recreate environments, but to address a fundamental challenge in autonomous vehicle development: the shifting perceptual landscape created by vehicle design changes. Moving perception software between an SUV and a sedan, for example, alters how the system “sees” the world, impacting the placement of critical elements like traffic signals and the visibility of curbs.
This means even identical software requires adaptation to function reliably in different vehicle configurations. NuRec begins with real-world drives, reconstructing each scene using 3D Gaussian splatting and then rendering those scenes from the perspective of a This allows developers to virtually explore scenarios and proactively identify “weak spots” created by geometric differences before committing to costly and time-consuming real-world data collection.
The efficiency of this approach stems from the prohibitive expense of collecting and labeling real-world data for each new vehicle iteration, particularly during early development phases. New vehicle fleets may not be readily available, and capturing rare or edge-case driving conditions presents significant logistical hurdles. While real-world data remains essential for validation, NuRec offers a synthetic alternative to bridge the gap and accelerate the development cycle.
The Physical AI NuRec Dataset itself contains more than 1,500 neural-reconstructed driving scenes, each approximately 20 seconds in length and captured from six camera views: a 120-degree front-wide view, a 30-degree front-telephoto view, 120-degree cross-left and cross-right views, and 70-degree rear-left and rear-right views, providing a substantial foundation for testing and refinement. To facilitate this workflow, NuRec employs a specific process.
Developers first pair a reconstructed drive with the desired target rig configuration. “Validate the NCore sequence Set the generated paths according to the converter’s output contract: export NCORE_DIR=” is a critical step, according to documentation.
NuRec Pipeline: From Real-World Drives to Target Views
NuRec streamlines the adaptation of autonomous vehicle perception systems by reconstructing drives and rendering views tailored to new vehicle configurations, a process NVIDIA validated internally with a program requiring a new camera setup. This approach addresses a critical bottleneck in autonomous development where gathering and labeling data for each new vehicle design is prohibitively expensive, particularly during early stages when vehicle fleets are unavailable or rare conditions must be simulated.
The NuRec pipeline begins with recorded real-world drives which are then reconstructed to create a detailed 3D representation of the environment. The system’s workflow involves downloading a reconstructed scene, exporting the target-camera trajectory, and then performing a sparse render to validate the setup.
Exporting the trajectory requires a specific Docker command, utilizing parameters to define the input scene, rig configuration, and output location for the trajectory data, according to NVIDIA. A sparse render, focused on a single camera, like the in an example target rig, is a critical validation step. This initial render confirms the accuracy of the setup before committing to a full, computationally intensive rendering of the entire drive.
The process allows for early detection of issues with camera positioning or windshield visibility, preventing wasted resources on flawed datasets. Following successful validation, the system refines the rendered frames using NVIDIA Harmonizer, a temporally aware post-processing tool. This refinement prepares the data for training a perception model, effectively treating the newly rendered dataset as equivalent to real-world camera data. According to NVIDIA, the public nurec-skills repository on GitHub contains skills for locating Physical AI datasets, utilizing NCore, running NuRec, and applying DiffusionHarmonizer.
Physical AI NuRec Dataset Provides Reconstructed Driving Scenes
The Physical AI NuRec Dataset facilitates adaptation of perception models by providing reconstructed driving scenes rendered through target vehicle configurations. Access to the dataset is gated, requiring acceptance of a license and authentication via a Hugging Face token before downloading scenes. Teams can use the dataset to render existing reconstructions through a desired vehicle’s sensor setup, refine the resulting imagery, and then prepare the data for training perception models. The process begins with downloading a reconstructed scene, such as from the 26.04 release, using a command like repo_id=.
A coding agent can further automate this process by cloning the NVIDIA/nurec-skills repository, a resource containing tools for locating datasets, running NuRec, and applying DiffusionHarmonizer. This container distinguishes between auxiliary data and dedicated training/rendering images through the NRE_AUX_IMAGE environment variable.
To adapt a scene to a new carline, developers download a USDZ scene and a sample target rig from the Physical AI NuRec Dataset, then replace the rig’s JSON configuration with the target carline’s specifications before rendering again. “)) # Download one reconstructed scene (USDZ) from the 26.04 release repo_id=”,” said a representative, highlighting the initial step in the adaptation process.
Target Rig Configuration Defines Novel Camera Perspectives
Target rig configuration directly influences the perceived world for autonomous systems, as even minor alterations to vehicle geometry impact camera viewpoints. NuRec addresses this by allowing developers to define a target rig and then render existing driving scenarios as if captured by that new platform, effectively simulating data from a vehicle that doesn’t yet exist. The system uses the rig file to define each sensor’s characteristics, including camera name, resolution, lens model, intrinsic parameters, and pose relative to the vehicle.
NuRec converts colon-delimited camera names into logical IDs; for example, transforming into Developers can expand the sensor suite by adding new entries to the rig file, specifying unique names and parameters for each additional camera. Establishing input and output locations is a preliminary step, achieved through commands like export SCENE_DIR=/absolute/path/to/scene and export OUTPUT_DIR=/absolute/path/to/output.
The USDZ scenes used as input are reconstructed from six source cameras: a 120-degree front-wide camera, a 30-degree front-telephoto camera, and cross/rear-facing cameras at various angles. Importantly, a target camera does not require a one-to-one correspondence with these source cameras, offering flexibility in positioning, orientation, resolution, and field of view, as long as its pose is defined within the same rig coordinate frame and its lens model is supported.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
