NVIDIA is evaluating adaptations to its Blackwell GPUs to maintain AI inference performance while adding a new layer of data privacy. As large language models process increasingly sensitive information, the company is building a trusted environment using memory-encrypted confidential virtual machines, confidential GPUs, and encrypted NVLink interconnects. Maintaining high performance within this system requires co-optimization of AI frameworks and the confidential computing environment; workload characteristics, such as request volume, directly impact how noticeable any performance overhead becomes.
NVIDIA Confidential Computing Protects AI Inference Workloads
NVIDIA Blackwell GPUs are demonstrating the ability to retain between 96.1% and 98.2% of baseline output-token throughput when running AI inference inside memory-encrypted confidential virtual machines, as revealed by recent internal testing. This performance preservation is critical as artificial intelligence increasingly handles sensitive data across personal, enterprise, and regulated sectors, demanding a trusted execution environment for processing. NVIDIA is addressing this need with a system incorporating confidential GPUs and encrypted NVLink interconnects alongside the CVMs, creating a hardware foundation for secure AI workloads.
The company’s confidential computing architecture introduces security paths enforced at the hardware level, protecting data and workloads during use, but does not eliminate the need for careful performance engineering. Framework-level adaptations are therefore essential; NVIDIA’s TensorRT LLM, for example, now incorporates secure data movement, autotuning, and optimized multi-GPU communication to minimize performance impact.
Testing with the DeepSeek-R1 model on eight NVIDIA B200 GPUs showed that these adaptations kept per-token latency overhead within 1.2% to 4.3% of the baseline (lower TPOT is better) while maintaining over 96% of the throughput achieved without confidential computing enabled, the company says. Workload characteristics significantly influence how noticeable any performance overhead from confidential computing becomes, with high request volumes proving particularly effective at masking fixed encryption costs.
By overlapping stalls with other processing tasks, the impact of encryption can be lessened, making it more difficult to observe a direct performance decrease. This means that the specific demands placed on the system, the rate of requests, directly affect how apparent the security measures are to end-users. According to NVIDIA’s documentation, selecting a workload to expose confidential computing overhead is a key step in evaluating the system’s performance.
NVIDIA emphasizes that security configuration and inference optimization should be approached as a unified deployment problem, rather than separate concerns. The company reports that mean TPOT remained within 1.2% to 4.3% when confidential computing was enabled, indicating minimal overhead in controlled tests. To facilitate confidential inference deployments, NVIDIA provides documentation and regularly updates TensorRT LLM with new features. The company also encourages users to benchmark both “CC-on” and “CC-off” configurations using their intended workloads to accurately assess the impact on their specific applications.
As organizations increasingly prioritize data privacy in AI, NVIDIA’s approach aims to deliver a balance between robust security and sustained performance. The company advises users to enable confidential computing, attest the environment, and benchmark CC-on and CC-off configurations using the exact workload they intend to serve, framing it as a full-stack engineering effort.
DeepSeek-R1 Performance: TensorRT LLM with Confidential Computing
NVIDIA’s TensorRT LLM framework, when paired with confidential computing adaptations, sustained between 96.1% and 98.2% of baseline output-token throughput on DeepSeek-R1 inference, even while employing memory encryption and secure data movement protocols. The company’s performance engineering team utilized a specific methodology to quantify confidential computing overhead, selecting a workload characterized by long input contexts, extended output generation, and low concurrency to effectively expose potential bottlenecks.
To isolate the effects of confidential computing, the NVIDIA team deliberately chose a workload that stressed data movement during the prefill stage, amplified per-token overhead during decoding, and limited opportunities for concurrent request processing. This approach allowed for a more granular assessment of the performance impact of secure execution.
The tested configuration employed the DeepSeek-R1-0528-NVFP4 model, utilizing the TensorRT LLM framework with a PyTorch backend, and an input/output sequence length of 32K/1K, respectively, while varying concurrent requests from one to sixteen and maintaining a parallelism of TP=8, EP=1, and PP=1. This commitment to transparency and user empowerment allows AI platform engineers to accurately assess the trade-offs between security and performance in their specific environments, according to NVIDIA. On 2026-09-22, TensorRT and Dynamo-Triton were linked for faster AI on multiple GPUs, demonstrating its commitment to optimizing AI performance across diverse hardware configurations.
CC-Aware Adaptations in TensorRT LLM Reduce Performance Impact
This result highlights a shift toward integrating security considerations directly into inference optimization, rather than treating them as separate challenges. TensorRT LLM addresses altered runtime assumptions introduced by confidential computing through adaptations to data movement, specifically host-to-device transfers. In the B200 system’s confidential computing environment, these transfers now pass through a software-encrypted bounce buffer because the GPU cannot directly access protected virtual machine memory. Consequently, pinned memory, typically used for asynchronous transfers, loses its performance advantage, and some copies can block the calling thread.
To mitigate this, TensorRT LLM employs CC-aware memory selection, opting for pageable memory for affected paths instead of unconditionally using pinned memory. This change allows for more efficient data handling within the secure environment.
Device-to-host transfers also require adaptation, as repeated readback of token and sampling data can create bottlenecks. TensorRT LLM addresses this by moving these operations to an asynchronous worker, preventing protected copies from blocking the main scheduler during the decoding process. This parallelization strategy minimizes the impact of secure data transfer on overall inference speed.
NVIDIA’s approach recognizes that confidential computing does not eliminate the need for performance engineering; instead, it demands greater framework awareness. The company advises that the methodology involves measuring output throughput, the percentage retained when confidential computing is enabled, and latency overhead, expressed as the percentage retained within 1.2% to 4.3% of the baseline.
These metrics provide a quantifiable assessment of the performance trade-offs associated with confidential computing, the company says. These collaborations, alongside the recent report of NVIDIA linking TensorRT and Dynamo-Triton for faster multi-GPU AI, demonstrate a commitment to pushing the boundaries of both performance and security in AI inference.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
