University of Tokyo Study Links Thermodynamics to Optimal Diffusion Models

Diffusion models, a class of generative algorithms increasingly employed in image generation, function by systematically adding noise to data during a training phase – a process termed diffusion dynamics. The subsequent generation of new content relies on reversing this process, effectively removing noise from initially random data. A critical, and previously empirically observed, element in constructing high-quality diffusion models is the selection of an appropriate noise schedule, or diffusion dynamics. Optimal transport theory has demonstrated utility in this context, but lacked a comprehensive theoretical justification until recently.

Joint research conducted at the University of Tokyo has established a link between nonequilibrium thermodynamics – the study of systems undergoing constant change – and the efficacy of optimal transport theory within diffusion models. Researchers derived inequalities connecting thermodynamic dissipation – a measure of energy loss during change – with the robustness of data generation. These inequalities demonstrate that employing optimal transport dynamics ensures the most robust generation of data, providing a theoretical basis for its empirical success. Notably, the derived bounds are accurate to within one order of magnitude for practical image generation scenarios, suggesting their applicability beyond theoretical understanding. This work, building on advancements in thermodynamic trade-off relations, offers a novel thermodynamic approach to machine learning, potentially influencing the development of Diffusion model thermodynamics and related fields. The findings were published in Physical Review X.