AI Learns from Raw Data Using Adaptable Noise Levels for Better Results

Scientists are increasingly focused on developing effective methods for self-supervised learning from the vast quantities of unlabeled time-series data available today. Duy Nguyen, Jiachen Yao, and Jiayun Wang from Caltech, working with Julius Berner from NVIDIA and Animashree Anandkumar also from Caltech, present a new approach that moves beyond static data corruption techniques. Their research introduces the Flow-Guided Neural Operator (FGNO), a framework combining operator learning with flow matching to dynamically adjust the level of noise applied during training. This innovation allows for the extraction of a richer hierarchy of features, improving representation learning and ultimately achieving significant performance gains across multiple biomedical applications. Evaluations demonstrate FGNO’s superior robustness to limited data, delivering up to 35% gains in signal decoding, 16% reductions in prediction error, and over 20% improvements in accuracy compared to existing state-of-the-art methods.

FGNO combines operator learning with flow matching, a technique that progressively transforms noisy inputs into clean data by predicting intermediate velocities. A crucial element of the work is the use of Short-Time Fourier Transform, which unifies different time resolutions and facilitates the extraction of rich hierarchical features. By learning mappings in functional spaces, FGNO avoids the limitations of traditional methods that often distort data through upsampling or downsampling. Unlike conventional masked autoencoders relying on fixed masking ratios, FGNO learns versatile representations adaptable to specific tasks by leveraging varying strengths of noise applied to input data. This innovation allows the model to capture both low-level patterns and high-level global features within a unified framework. The researchers specifically designed FGNO to use clean inputs for representation extraction during downstream tasks, eliminating randomness inherent in other generative SSL methods and boosting accuracy. Evaluations across three biomedical domains, neural signal decoding using BrainTreeBank, skin temperature prediction with DREAMT, and sleep stage classification with SleepEDF, demonstrate FGNO’s consistent outperformance compared to established baselines. The method achieves up to 35% gains in AUROC for neural signal decoding, 16% reductions in RMSE for skin temperature prediction, and over 20% improvement in both accuracy and macro-F1 scores on SleepEDF, particularly when data is limited. These results underscore FGNO’s robustness to data scarcity and its capacity to learn expressive representations from diverse time-series data, paving the way for more effective analysis in healthcare and beyond. FGNO’s ability to preserve the fidelity of multi-resolution signals while yielding task-adjustable representations represents a significant advancement in the field of self-supervised learning. A Short-Time Fourier Transform (STFT) serves as the initial processing step, converting raw time-series data into spectrograms to reveal local time-frequency features. This transformation allows the model to analyse signals across varying resolutions, capturing both transient events and sustained patterns. The framework’s integration of neural operators and STFT offers a novel approach to time-series modelling, enabling the extraction of both temporal and spectral features at fine-grained resolutions. By varying the strength of noise applied to the input data, the model extracts a rich hierarchy of features from different network layers and at different ‘flow times’, points along the transformation from noise to clean signal. This approach enables the model to capture versatile representations, ranging from low-level patterns to high-level global features, within a single adaptable model. Critically, the team diverges from typical generative SSL methods by utilising clean inputs for representation extraction, despite training the model with noisy data. This design choice eliminates the randomness inherent in using noisy inputs during inference, leading to more stable and accurate results. A shallow spectrogram reconstruction head is employed solely during pre-training to facilitate the flow-matching objective and is subsequently discarded, streamlining the model for downstream tasks. Across three biomedical datasets, the research demonstrates substantial performance gains using the FGNO framework. Specifically, in neural signal decoding with the BrainTreeBank dataset, FGNO achieved up to a 35% increase in Area Under the Receiver Operating Characteristic (AUROC) compared to existing methods. This AUROC metric assesses the model’s ability to distinguish between different neural signals, with higher values indicating better discrimination. Furthermore, FGNO reduced Root Mean Squared Error (RMSE) by 16% when predicting skin temperature using the DREAMT dataset. RMSE quantifies the average magnitude of the error between predicted and actual values; a lower RMSE signifies improved prediction accuracy. On the SleepEDF dataset, FGNO delivered over a 20% improvement in both accuracy and macro-F1 score under conditions of limited data availability. Accuracy represents the proportion of correctly classified sleep stages, while macro-F1 provides a balanced measure of precision and recall across all classes. These gains were achieved despite employing a single model capable of adapting to diverse tasks. The pursuit of genuinely adaptable artificial intelligence often stumbles on the difficulty of extracting meaningful patterns from messy data without painstakingly labelled examples. This work offers a compelling step forward by demonstrating a self-supervised learning approach that actively shapes the learning process through controlled noise injection. It’s not merely about learning from unlabelled data, but about intelligently corrupting that data to force the model to build more robust and versatile representations. By leveraging the Short-Time Fourier Transform and a flow-matching framework, the researchers have created a system that can effectively tap into a hierarchy of features, from fine-grained details to broader contextual information. The consistent performance gains across diverse biomedical datasets, brain signals, skin temperature, and sleep patterns, suggest a level of generalizability often lacking in more specialised approaches. However, the reliance on specific time-series data types and the somewhat opaque nature of the ‘flow time’ parameter remain limitations. Understanding why certain flow times yield optimal performance requires further investigation. Future work could explore the application of this framework to other data modalities, such as video or audio, and delve deeper into the interpretability of the learned representations. More broadly, this approach could pave the way for AI systems that are less reliant on vast labelled datasets and more capable of adapting to new and unforeseen circumstances, a crucial step towards truly intelligent machines.