Nvidia boosts portrait relighting to real time with new AI

Nvidia has achieved real-time portrait relighting, with its distilled model processing images in as little as 1.82 milliseconds on an RTX 4090, the company says. The company’s new approach, called Hybrid Domain Knowledge Fusion (HDKF), combines learnings from synthetic data, single-light setups, and real-world images to improve both accuracy and visual realism.

Trained with pixel-aligned RGB, albedo, and normal supervision, HDKF creates a physically grounded relighting effect, allowing for deterministic live video creation where previous methods struggled with speed or fidelity. The framework obtains the best MSE, PSNR, and SSIM scores on a held-out benchmark while maintaining real-time performance at 512×512 resolution.

Hybrid Domain Knowledge Fusion Enables Real-Time Portrait Relighting

The company reports HDKF uniquely combines learnings from synthetic data, data, and images captured “in-the-wild” to improve both the accuracy and realism of relighting effects. This technique moves beyond simple image manipulation by creating a foundation for realistic light interactions. The distilled model also achieved 11.89 milliseconds processing time on an RTX 2060, further solidifying its capacity for real-time applications, according to Nvidia.

Researchers state that HDKF is a relighting-specific training framework that learns complementary physics, reflectance, and realism priors. This fusion of data types allows for a more robust and adaptable system capable of handling diverse lighting conditions and portrait characteristics.