QiNN-QJ Quantum-inspired Neural Network Enables Controllable Cross-Modal Entanglement for Multimodal Sentiment Analysis

Multimodal sentiment analysis, the process of understanding emotions from diverse sources like text and video, benefits from techniques inspired by the principles of quantum mechanics. Yiwei Chen from Yunnan University, Kehuan Yan from Fuzhou University, and Yu Pan, along with Daoyi Dong from the University of Technology Sydney, present a novel approach that moves beyond traditional methods by incorporating a concept known as the quantum jump. Their work introduces QiNN-QJ, a quantum-inspired neural network, which generates controllable entanglement between different data types, such as text and facial expressions, using a process that mimics the behaviour of quantum systems. This innovative method not only achieves superior performance on standard benchmarks, including CMU-MOSI, CMU-MOSEI, and CH-SIMS, but also offers improved interpretability by quantifying the degree of connection between different modalities, establishing a principled framework for modelling complex relationships in multimodal data.

Quantum Inspired Multimodal Sentiment Analysis Approaches

The field of Multimodal Sentiment Analysis (MSA) focuses on determining emotions and opinions expressed in data combining multiple sources, such as text, audio, and video. Recent research explores quantum-inspired methods to improve the performance of MSA systems by applying concepts from quantum mechanics to better represent and combine information from different data types. A key challenge in MSA is effectively fusing information from diverse modalities. Researchers are investigating approaches including hierarchical fusion and attention mechanisms, and frequently use Graph Neural Networks (GNNs) to represent relationships between modalities and features.

Quantum-inspired techniques introduce concepts like entanglement to capture complex interactions between data, and researchers are also developing Quantum Neural Networks (QNNs) incorporating quantum principles into their architecture. Several datasets serve as benchmarks for MSA research, including MOSI, CMU-MOSEI, and CH-SIMS. Current findings demonstrate that quantum-inspired models show promise in improving MSA performance, often achieving better results than traditional methods. There is also growing interest in combining large language models (LLMs) with multimodal data, while addressing challenges such as missing modalities and the need for more robust and generalizable models.

Quantum Jumps Model Multimodal Data Entanglement

Scientists engineered a novel Quantum-Inspired Neural Network with Quantum Jump (QiNN-QJ) to model entanglement across multiple data types for sentiment analysis. This approach departs from traditional methods, harnessing dissipative dynamics to achieve both expressive power and stable training. Initially, each modality, such as text, audio, or visual signals, is embedded as a quantum pure state, representing the inherent information within each data type. The core of the method involves a differentiable module simulating a Quantum Jump operator, transforming separate modalities into an entangled representation.

This operator, governed by Hamiltonian and Lindblad operators, generates controllable cross-modal entanglement, allowing researchers to precisely shape the interactions between different data types. By employing dissipative dynamics, the system introduces structured stochasticity and a steady-state attractor, stabilizing the training process and constraining the entanglement shaping. The team validated QiNN-QJ on benchmark datasets including CMU-MOSI, CMU-MOSEI, and CH-SIMS, demonstrating superior performance compared to state-of-the-art models. Furthermore, the method facilitates enhanced interpretability through the calculation of von-Neumann entanglement entropy, providing insights into the relationships between modalities and the network’s decision-making process. This work establishes a principled framework for entangled multimodal fusion and paves the way for quantum-inspired approaches in modelling complex cross-modal correlations.

Entangled Modalities Enhance Neural Network Fusion

This research presents a novel approach to multimodal fusion inspired by principles of quantum mechanics, leveraging concepts like superposition and entanglement to enhance data integration. Researchers developed a Quantum-inspired Neural Network with Jump (QiNN-QJ) that initially encodes each modality, such as text, image, or video, as a pure quantum state, representing the inherent multi-layered information within each data source. A key innovation lies in the use of a differentiable module simulating a “quantum jump” operator, transforming these separate modalities into an entangled representation, effectively creating correlations between them. The QiNN-QJ utilizes both Hamiltonian and Lindblad operators to generate controllable cross-modal entanglement, introducing structured stochasticity and stabilizing the training process.

This allows the model to shape the entanglement between modalities, capturing more complex relationships. The resulting entangled states are then projected onto trainable measurement vectors to produce predictions, enabling optimized data interpretation. Experiments demonstrate that QiNN-QJ outperforms conventional, quantum-inspired, and large language models on benchmark datasets including CMU-MOSI, CMU-MOSEI, and CH-SIMS. Furthermore, the model facilitates enhanced interpretability through von Neumann entanglement entropy, providing a transparent mechanism for multimodal fusion and enabling post-hoc analysis of entanglement strength. The quantum jump method simulates the statistical evolution of quantum systems by tracking individual trajectories with random “jump” events, mirroring irreversible processes like decoherence.

Quantum Inspired Neural Networks Enhance Data Fusion

This research introduces a novel neural network, QiNN-QJ, inspired by principles of quantum mechanics, to improve multimodal data fusion, the process of integrating information from multiple sources like text, images, and video. The team demonstrates that modelling interactions between data modalities using concepts like superposition and entanglement can yield significant improvements over existing methods. QiNN-QJ uniquely employs a ‘quantum jump’ mechanism, simulating how quantum systems evolve with both predictable and random changes, to generate and control entanglement between different data modalities. This approach stabilises the learning process and allows for more nuanced shaping of the relationships between data.

The resulting network achieves superior performance on standard multimodal datasets, surpassing the accuracy of existing state-of-the-art models. Importantly, QiNN-QJ also offers enhanced interpretability, allowing researchers to analyse the strength of entanglement between modalities using a measure called von Neumann entanglement entropy. This provides insights into how the network is integrating information and which relationships it deems most important.