Diffusion Sampling Achieves Polylogarithmic Iteration Complexity for High-Accuracy Results

Diffusion models have recently demonstrated impressive capabilities in generating complex and varied data, yet achieving high-quality results typically demands substantial computational effort. Khashayar Gatmiry, Sitan Chen, and Adil Salim present a novel approach to solving the underlying differential equations that govern these models, addressing a critical limitation of existing methods. Their research introduces a solver that significantly reduces the number of iterations needed for accurate sampling, scaling polylogarithmically with desired accuracy , a marked improvement over previous polynomial scaling. Importantly, this new technique achieves dimension-free sampling, meaning its computational cost is determined by the data’s effective radius rather than the potentially very large ambient dimension, representing a substantial advance for high-dimensional generative modelling. This breakthrough promises to make diffusion models more efficient and accessible for a wider range of applications.

Diffusion models currently face a computational bottleneck as the complexity of solving the underlying equations scales polynomially with both the ambient dimension and the desired accuracy, represented as 1/ε. This research introduces a novel solver for diffusion models, built upon a combination of low-degree approximation techniques and the collocation method as described in [LSV18]. The proposed approach achieves a polylogarithmic iteration complexity in 1/ε, establishing the first formal guarantee of high accuracy for a diffusion-based sampler that relies on approximate score access. Importantly, the derived bound demonstrates independence from the ambient dimension, with the dimension’s influence limited to the effective radius of the target distribution’s support. This represents a significant advancement in the efficiency and scalability of diffusion model sampling.

Diffusion Algorithm and Discretisation Technique

Diffusion models represent a leading approach to generative modelling, particularly in image generation, and extend to other data modalities. These models operate by simulating a reverse process, governed by a learned differential equation, to transform noise into meaningful samples. The core principle involves reversing a noise process, allowing the generation of new data points from random noise through accurate simulation of this reverse process. Recent theoretical investigations aim to elucidate the underlying mechanisms enabling diffusion models to efficiently sample from complex, multimodal distributions, building upon established literature in the field.

The research details an algorithm built upon diffusion models and employs a novel approach to discretisation. This method utilises a collocation technique, differing from conventional discretisation schemes, to approximate the reverse process more effectively. The algorithm’s development is underpinned by a low-degree approximation of the true reverse process, aiming to enhance both accuracy and computational efficiency during sampling. This approximation is crucial for ensuring the convergence of the sampler and generating high-quality outputs. A significant aspect of the work focuses on proving the convergence of the proposed sampler.

This involves demonstrating that the algorithm accurately approximates the ideal reverse process, even with the low-degree approximation employed. Mathematical proofs, detailed in appendices C through H, rigorously establish the theoretical foundations of the algorithm, including lemmas and theorems concerning the movement of probability flow and the accuracy of the Langevin Monte Carlo method. These proofs provide a solid basis for understanding the algorithm’s behaviour and performance characteristics. Further supporting material includes related work, detailed notation, and extensive derivative calculations, all provided in supplementary sections. Appendices A and B offer additional context and definitions, while appendices C, H contain the complete mathematical derivations and proofs for the key theoretical results. This comprehensive documentation ensures the reproducibility and verifiability of the research findings, contributing to a deeper understanding of diffusion models and their underlying principles.

Polylogarithmic Scaling Improves Diffusion Model Sampling

Scientists have achieved a breakthrough in sampling from complex, multi-modal distributions, demonstrating a novel solver for differential equations central to diffusion models. Their work addresses a critical limitation in existing discretization methods, which typically exhibit iteration complexity scaling polynomially with both the ambient dimension and the inverse accuracy desired. Experiments reveal that the newly developed solver achieves iteration complexity that scales polylogarithmically in the inverse accuracy, representing a significant advancement in efficiency. This delivers the first “high-accuracy” guarantee for a diffusion-based sampler relying on approximate score access of the data distribution.

The research focuses on distributions satisfying a bounded plus noise assumption, where a compactly supported distribution is convolved with Gaussian noise. Tests confirm that, given access to Lipschitz score estimates with bounded, subexponential error, the algorithm outputs samples closely matching the target distribution in total variation distance. Specifically, the team proved that samples can be generated to within an error of ε in just O((R/σ)2 · polylog(1/ε)) iterations, where R and σ define the parameters of the assumed distribution. Measurements confirm this performance even when the ambient dimension, d, is substantial, as the iteration complexity depends on the effective radius of the distribution’s support rather than d directly.

Further experiments demonstrate the solver’s efficiency in the challenging regime of Gaussian mixtures, where component centers are at a distance of Θ(p log k) from each other. In this scenario, the sampler achieves iteration complexity scaling only polylogarithmically in both k and 1/ε, a substantial improvement over existing diffusion-based methods which would scale polynomially in d and 1/ε. Data shows that the algorithm’s performance is particularly strong when the radius, R, is significantly smaller than the ambient dimension, d, offering a practical advantage in high-dimensional spaces. The breakthrough lies in a key structural property identified by the scientists: the high-order time derivatives of the score function along the reverse process are bounded.

This allows for accurate pointwise approximation of the score function using low-degree polynomials, enabling the adaptation of a specialized ODE solver. Results demonstrate that this approach maintains close alignment with the continuous-time reverse process, ultimately yielding a more efficient and accurate sampling method for diffusion models. This advancement opens possibilities for faster and more reliable generation of complex data in fields like image synthesis and scientific simulation.