NVIDIA Omniverse product manager Frank DeLise transformed a SimReady warehouse and humanoid robot into an interactive simulator offering both first- and third-person views using GPT-6 Astra. Developers are increasingly combining advanced AI models with Omniverse libraries to build simulation applications and test physical behavior, streamlining the process of turning ideas into working applications.
DeLise directed Astra to connect NVIDIA Omniverse libraries for physics, scene updates, rendering, and the user interface, creating an environment to explore how the humanoid robot performs warehouse tasks before automation, the company says. “Explore the projects below to see advanced AI models such as GPT-6 Astra at work,” NVIDIA writers state, highlighting a new approach to simulation development.
Astra AI Connects Omniverse Libraries for Warehouse Simulations
DeLise tasked Astra with establishing connections between NVIDIA Omniverse libraries, specifically using ovphysx for physics simulations, ovstage for scene management, ovrtx for rendering and ovui for the user interface. This integration enabled the creation of a dynamic simulation environment centered around a SimReady warehouse and a humanoid robot designed for logistical tasks. The system’s capabilities extend beyond simple visualization; Astra generated the necessary animation and application code to fully integrate these library functions, allowing for comprehensive testing and refinement of robotic behaviors, according to NVIDIA.
In one experiment, a robot successfully navigated a single obstacle in 64 out of 100 simulation runs, providing valuable data for improving its timing and control mechanisms. Jens Jebens, a senior product manager for OpenUSD at NVIDIA, further demonstrated Astra’s versatility by modeling a car suspension within PTC Onshape and configuring it within NVIDIA Isaac Sim.
Astra AI Maps Autonomous-Driving Workflows with Omniverse RTX
NVIDIA used Astra to create a reusable simulation environment based on San Francisco’s Market Street, named Zero to Alpamayo, demonstrating a workflow connecting asset creation, traffic modeling, Omniverse RTX sensor simulation and Alpamayo driving stages. Doyub Kim, a manager on the simulation technology team at NVIDIA, tasked the AI agent with mapping this workflow and verifying each integration step, resulting in a prototype for comparing autonomous vehicle models and tracing the impact of scene or sensor alterations on driving behavior.
Kim further tested the system by varying weather and lighting conditions in recorded simulation videos using a Cosmos3-Nano experiment, enabling a comparison of the driving model’s responses to the same scenario under differing environmental factors.
Astra & Digital Twins: Validating Sensors with ovrtx Outputs
Ashley Reid of NVIDIA directed GPT-6 Astra and Claude Fable 5 agents to compare outputs from ovrtx cameras and raw LiDAR with recorded data, establishing a new method for validating digital twins. The agents autonomously constructed two complete digital twins and refined two pre-existing models, assessing sensor performance against real-world recordings. Over approximately three days, Reid oversaw an iterative process where the agents quantified discrepancies, modified OpenUSD scenes, and verified the results, focusing on elements like missing objects, geometry and material properties.
Acceptance of changes hinged on achieving specific camera and LiDAR metrics, providing a quantifiable pathway for scene improvement. This approach allows developers to use measured differences between simulated and recorded sensor data to guide scene creation, a process detailed in a demonstration showing a comparison of rendered and recorded data.
Nic Johns, an engineering director at NVIDIA, further demonstrated Astra’s capabilities by assembling a NASA asset library into an OpenUSD International Space Station model, complete with telemetry data, all initiated through a single prompt, the firm reports. Johns then refined the scene with a follow-up prompt, shifting its position to display Earth’s daytime side, demonstrating the agent’s responsiveness to directional commands.
The workflow used Blender for initial asset preparation and relied on NVIDIA’s Omniverse libraries, ovrtx for rendering, ovstage for scene runtime, and ovstream for data transmission, to deliver the 3D model and operational data directly within a web browser, the company states. This browser-based application, guided by Johns through prompts and corrections, demonstrates a streamlined process for integrating 3D models with live operational data, offering a new paradigm for visualization and control.
Astra’s integration of NVIDIA Omniverse libraries extended beyond basic scene creation. A separate Cosmos3-Nano experiment varied weather and lighting in recorded simulation videos, allowing Kim to compare the driving model’s responses to the same scenario under different conditions.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
