NVIDIA Holoscan, a platform for building real-time AI applications ranging from medical imaging to robotics, recently saw its development process augmented by an unlikely collaborator: an AI coding agent, HoloHub says. Engineers at NVIDIA utilized Codex with GPT-5.6 sol max mode to construct a real-time endoscopic tool segmentation application, using existing HoloHub examples and documentation.
The development workflow is iterative, with the engineer defining a goal and constraints, then the AI implementing code and using the Holoscan CLI to run development operations; as the team explains, developers should “decompose the ultimate objective into smaller, verifiable engineering iterations guided by uncertainty and evidence.”
Holoscan CLI Enables Agent-Driven Application Development
This demonstration showcases a workflow where the Holoscan CLI functions as a shared execution interface between a human engineer and an artificial intelligence, streamlining the development process. Engineers provided the AI agent with access to the Holoscan CLI alongside the HoloHub repository, progressively revealing documentation, and specific development skills including holohub-app-lifecycle and holohub-debug-build-run. The development process deliberately mirrored a practical engineering task, beginning with a comprehensive goal then breaking it down into smaller, verifiable iterations.
This approach, according to NVIDIA, ensures continuous assessment and refinement throughout the application’s construction. The team initially verified that a pre-existing MONAI endoscopic tool segmentation model and a sample video functioned correctly within a separate application before focusing on building a new application reusing the deep learning pipeline, and adding visualization, telemetry, and benchmarking capabilities.
The agent first executed./holohub create to generate a standard application scaffold, then implemented the application graph, execution modes, tests, and documentation utilizing existing Holoscan operators and assets, according to the company. The resulting application integrated video replay, preprocessing, TensorRT inference, a segmentation postprocessor from the SDK, telemetry data, and HoloViz for visualization; inference and mask postprocessing occurred for each replayed frame, with frame-derived measurements displayed as an overlay.
When provided only with the HoloHub codebase, the AI agent still demonstrated an understanding of the CLI, utilizing it as its primary development tool. This agentic processing took 40 minutes. Comparing three different scaffolding configurations, the team observed variations in processing time and token usage; the most efficient configuration, utilizing the CLI, skills, and documentation, required approximately 40 minutes and 11 million tokens, while a configuration with only the CLI and documentation took approximately 65 minutes.
The resulting applications differed in quality, with the first configuration producing a reviewable application with more host-side validation and retries, and the second requiring rework due to the use of custom PyTorch/MONAI inference instead of Holoscan’s InferenceOp/TensorRT. “The main takeaway is the development loop shared by the developer and the agents,” the team reports; “./holohub supplies consistent operations, the skill encodes project-specific sequences and checks, and examples and documentation provide engineering context.” The agent and the engineer utilized identical CLI commands throughout the process, fostering a collaborative workflow.
The final prototype, built on an existing model, sample video, and Holoscan components, runs end-to-end, incorporates multiple application modes and automated tests, preserves model weights, and generates reproducible benchmark evidence. This demonstration highlights the potential for AI agents to accelerate application development when paired with a robust CLI, well-defined skills, and comprehensive documentation, offering a new paradigm for building real-time AI applications.
Iterative Workflow Guides AI Agent Development
Providing developers with established tools and documentation significantly accelerates AI-driven application creation; the team observed that an agent, when equipped with the Holoscan CLI, development skills, and access to the HoloHub repository, could construct a functional application prototype in under an hour. This contrasts with scenarios where these resources were limited, demonstrating the importance of a well-defined development ecosystem for AI coding agents. Specifically, when lacking HoloHub development skills, the agentic processing extended to approximately 65 minutes, highlighting the efficiency gains from providing specialized guidance.
This collaborative process allowed for continuous review and refinement, ensuring the application met the desired specifications. The agent revised the visual output and added more statistics after each iteration. The core of this accelerated development lies in a shared loop between the engineer and the AI agent.
HoloHub Skills Facilitate Application Lifecycle Management
Providing specific development skills to AI coding agents markedly improves their efficiency when building applications on the NVIDIA Holoscan platform, according to recent experiments. When equipped with HoloHub development skills, including holohub-app-lifecycle and holohub-debug-build-run, an AI agent completed a real-time endoscopic tool segmentation application in 40 minutes, consuming 11 million tokens. This contrasts with an agent lacking these skills, which required approximately 20 million tokens to achieve the same outcome, despite accessing the HoloHub repository and documentation, the firm reports.
The development workflow demonstrated that explicit guidance directs the agent toward optimal development practices and reduces reliance on less efficient, generic tools. The researchers observed that agents without HoloHub skills frequently resorted to unnecessary steps. Providing access to the HoloHub codebase alone, without the accompanying skills and CLI guidance, yielded a 40-minute completion time with 15 million tokens consumed, but resulted in suboptimal code quality.
These results highlight the importance of tailored skills in streamlining the development process and ensuring code integrity. This approach allowed the AI agent to focus on specific tasks and receive feedback, leading to a more efficient and robust application. The resulting application incorporated three distinct execution modes, visual, smoke, and benchmark, each designed for a specific purpose. These modes, along with automated tests, are discoverable and runnable through the Holoscan CLI without requiring detailed knowledge of containerization or application scripting.
“./holohub modes endoscopy_tool_segmentation_dashboard –language python” allows users to view available modes, while “./holohub run endoscopy_tool_segmentation_dashboard benchmark –language python” and “./holohub test endoscopy_tool_segmentation_dashboard –language python” execute specific modes and tests respectively. Benchmark mode used Holoscan Data Flow Tracking to monitor the entire data pipeline, from video replay through preprocessing, inference, telemetry, offscreen HoloViz, and the final rendered-frame sink.
Benchmarking with Holoscan Data Flow Tracking
This granular level of observation allowed for detailed analysis of application-path latency across 300 tracked frames, yielding more repeatable measurements than previous visual assessments provided. These modes, alongside automated tests, were intentionally crafted to be discoverable and runnable without requiring memorization of complex container or application script configurations. Users can view available modes with the command “./holohub modes endoscopy_tool_segmentation_dashboard –language python”, streamlining the development and testing process.
The ability to decompose the overall objective into smaller, verifiable engineering iterations was central to the project’s success. Each iteration produced reviewable code, outputs, and tests, which informed subsequent prompts and design choices. “Show the model-derived mask, coverage and timeline, and useful uncertainty measurements in a polished HoloViz overlay,” was a key instruction guiding the agent’s implementation.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
