Boltzmann Machines Unlock Parallelizable Sampling with Langevin SB, Maintaining Accuracy Comparable to Markov Chain Monte Carlo

Boltzmann machines represent a powerful approach to generative modelling, but their demanding computational requirements have historically limited their application to simpler forms. Kentaro Kubo from Toshiba Corporation and Hayato Goto from RIKEN Center for Quantum Computing, along with their colleagues, now present a significant advance in overcoming these limitations. They introduce a novel sampling method, inspired by principles of combinatorial optimisation, that allows for parallelised computation without sacrificing accuracy, a key challenge for more complex Boltzmann machines. Crucially, the team also developed a technique to precisely control the temperature of the generated distributions, further enhancing performance and establishing a framework that extends the capabilities of Boltzmann machines beyond the constraints of existing methods. This work unlocks the potential for more expressive and powerful energy-based generative models, paving the way for advancements in a range of applications.

Langevin Sampling Accelerates Boltzmann Machine Training

This study tackles a key challenge in Boltzmann Machines (BMs), powerful generative models often limited by computationally intensive training procedures. While Restricted BMs benefit from efficient learning techniques, more complex models with interconnected units require time-consuming Markov chain Monte Carlo (MCMC) sampling due to difficulties in parallel processing. To address this, researchers developed a novel Boltzmann sampler, Langevin SB (LSB), inspired by a quantum-inspired optimization technique called simulated bifurcation. LSB enables parallel sampling for BMs with general connections, including semi-restricted BMs, while maintaining accuracy comparable to sequential MCMC methods.

The team engineered LSB to overcome the sequential update rules that limit the scalability of traditional MCMC approaches. This new sampler allows for simultaneous updates, significantly reducing computation time for complex models. However, LSB initially operates with an unknown effective inverse temperature, hindering performance. To rectify this, scientists developed conditional expectation matching (CEM), an efficient method for estimating this inverse temperature during the learning process. By combining LSB and CEM, the researchers established sampler-adaptive learning (SAL), a framework that unlocks the potential of BMs with greater expressive power than RBMs. This innovative approach allows for efficient training of complex models, paving the way for more powerful and versatile applications in areas like image recognition and natural language processing.

Scalable Boltzmann Machine Training via Langevin Sampling

The research team developed a novel framework, termed sampler-adaptive learning (SAL), to efficiently train complex Boltzmann machines (BMs), overcoming limitations in traditional methods. Conventional BM training relies on Markov chain Monte Carlo (MCMC) sampling, which is computationally expensive and difficult to parallelize. SAL integrates a new sampling technique, Langevin simulated bifurcation (LSB), with a conditional expectation matching (CEM) method to enable scalable training of more expressive models beyond Restricted Boltzmann Machines. LSB functions as a parallelizable sampler, achieving accuracy comparable to MCMC by drawing inspiration from optimization algorithms like simulated bifurcation.

To further refine sampling accuracy, LSB incorporates discretization and stochastic initialization at each iteration. Evaluations on random spin-glass models demonstrated that LSB outperformed conventional Gibbs sampling in many instances. Crucially, SAL addresses the challenge of controlling the inverse temperature of the output Boltzmann distribution, a limitation of LSB. The team developed CEM to estimate this inverse temperature during the learning process, allowing SAL to adapt the model function to the optimal temperature. By combining LSB and CEM, SAL enables the training of Boltzmann machines with greater expressive power, achieving high fidelity in tasks such as image generation and reconstruction, and delivering high classification accuracy on the OptDigits dataset.

Adaptive Langevin Sampling For Boltzmann Machines

The research team developed a new framework for training Boltzmann Machines, addressing limitations in existing methods that hinder their application to complex models. Traditional Boltzmann Machine training relies on computationally expensive Markov chain Monte Carlo sampling, which struggles with parallelization. To overcome this, the researchers introduced a Langevin Simulated Bifurcation sampling method, enabling parallelized sampling while maintaining accuracy comparable to existing techniques. Recognizing that controlling the output temperature is crucial for performance, they further developed a conditional expectation matching method to estimate this temperature during the sampling process.

Combining these innovations into a sampler-adaptive framework, the team demonstrated the ability to train Boltzmann Machines with greater expressive power than previously possible. Experiments on a three-spin model, and image datasets, confirmed the effectiveness of the new approach. The authors acknowledge that the performance of the method is sensitive to the selection of hyperparameters, such as the step size used in sampling. Future work may focus on automating this hyperparameter selection process, and exploring the application of this framework to even more complex generative models and datasets.