New Machine Learning Loss Function Overcomes Data Flaws for Faster Training

Loss functions represent a critical element in supervised learning, yet conventional approaches often struggle with high-dimensional data and are susceptible to outliers, hindering performance and slowing training. Soumi Mahato from the National Institute of Technology, Calicut, and Lineesh M. C. address these limitations by introducing a novel robust, bounded, and smooth (RoBoS-NN) loss function. This research is significant because it not only presents a theoretically analysed loss function with demonstrable robustness, but also implements it within a neural network framework to create a new robust algorithm, -NN, which consistently outperforms benchmark models across multiple real-world datasets, even when deliberately challenged with artificially introduced outliers.

Robust outlier handling via a bounded and smooth loss function for time series forecasting

Scientists have developed a novel loss function, termed RoBoS-NN, to address limitations in training machine learning algorithms with high-dimensional and outlier-sensitive datasets. Traditional loss functions often struggle with such data, leading to reduced performance and slower convergence, but this new approach offers a robust alternative.
The research centres on creating a loss function that is not only robust to outliers but also bounded and smooth, characteristics crucial for efficient and reliable training of neural networks. Theoretical analysis confirms the robustness of the newly designed loss function, rigorously justifying its ability to handle challenging data scenarios.

Researchers implemented the RoBoS-NN loss function within a neural network framework to forecast time series data, resulting in a new algorithm named LRoBoS-NN. This integration allows the benefits of the loss function, its robustness and smoothness, to directly impact the predictive capabilities of the neural network.

To thoroughly evaluate LRoBoS-NN, experiments were conducted using multiple real-world datasets, providing a comprehensive assessment of its performance. Furthermore, the study deliberately introduced outliers into these datasets to specifically test the algorithm’s resilience under more demanding conditions.

Numerical results demonstrate that LRoBoS-NN consistently outperforms other benchmark models across various accuracy measures. The RoBoS-NN loss function is defined as L(u) = {λ{1 −(a √ u2 + ε −a√ε + 1)exp(−(a √ u2 + ε −a√ε))} where u = ∣y −y∣, with parameters a, λ, and ε controlling shape, bounds, and stability respectively.

This formulation, building upon previous work with RoBoSS loss in classification, extends its benefits to regression tasks and time series forecasting. The objective function for training the neural network is formulated as minimizing a combination of L2 regularization and the RoBoS-NN loss, expressed as min θ λ 2 ∥θ∥2 + 1 n n ∑ k=1 LRoBoS-NN(yk −f(xk;θ)).

Formulation and implementation of the robust bounded smooth loss function

A robust, bounded, and smooth (RoBoS-NN) loss function underpinned the development of a new forecasting algorithm, -NN, designed to address limitations in high-dimensional and outlier-sensitive datasets. The study began by defining the RoBoS-NN loss, extending a previously established RoBoSS loss function used in support vector machines for classification to the realm of regression and time series forecasting.

This extension involved formulating the loss as L(u) = {λ{1 −(a √ u2 + ε −a√ε + 1)exp(−(a √ u2 + ε −a√ε))}, where ‘u’ represents the absolute error between actual and predicted values, and a, λ, ε 0 are shape, bound, and stability parameters respectively. Subsequently, researchers integrated this RoBoS-NN loss function into a neural network framework, creating LRoBoS-NN, expressed as min θ λ 2 ∥θ∥2 + 1 n n ∑ k=1 LRoBoS-NN(yk −f(xk;θ)).

This formulation minimizes a combination of the RoBoS-NN loss term and a regularization term to prevent overfitting. Optimization was achieved using the Adam algorithm, leveraging its adaptive learning-rate mechanism for efficient large-scale neural network training. The Adam optimizer iteratively updated network parameters through backpropagation, calculating gradients, and applying bias correction to the first and second moment estimates.

To rigorously evaluate the proposed loss function, theoretical analysis focused on bounding the generalization error. The work demonstrated that, given a network of depth ‘d’ with parameters having a Frobenius norm bounded by MF(j), the generalization error R(fp) −Rz(fp) is less than or equal to 2aB( √ 2log(2)d + 1)∏d j=1 MF (j) e√n + √ 8ln 1 ε n, with a confidence level of 1 −ε.

This bound highlights the influence of network depth, parameter magnitude, and sample size on the model’s performance. Experiments were conducted on multiple real-world datasets, with outliers intentionally introduced to assess the robustness of -NN in challenging conditions.

LRoBoS-NN demonstrates improved regression performance across diverse time series datasets

Researchers developed a robust, bounded, and smooth (RoBoS-NN) loss function to address limitations of traditional loss functions in high-dimensional and outlier-sensitive datasets. This new loss function was implemented within a neural network framework, resulting in a robust algorithm termed LRoBoS-NN, designed for time series forecasting.

Experiments conducted on multiple real-world datasets demonstrate that LRoBoS-NN consistently outperforms benchmark models in terms of accuracy measures. The study focused on regression tasks using multilayer perceptrons, where the training dataset is defined as {xk, yk}nk=1, with xk representing the input feature vector and yk denoting the corresponding continuous target value.

Minimization of an empirical risk function, calculated as 1n∑k=1 L(yk −f(xk;θ)), was central to the process, where L(⋅) represents the regression loss function and θ encompasses all network weights and biases. The choice of loss function significantly impacts the robustness and generalization performance of the multilayer perceptron, particularly when dealing with noisy data and outliers.

Mean square loss, a widely used function in forecasting, measures the squared difference between true and predicted values, defined as 1n∑(yi −yi)2. However, this function is highly sensitive to outliers, amplifying larger deviations and potentially degrading predictive performance. Conversely, mean absolute loss quantifies the absolute difference between true and predicted values, expressed as 1n∑∣yi −yi∣, exhibiting improved robustness compared to squared loss, especially in the presence of outliers.

Despite its robustness, the mean absolute loss function lacks smoothness near zero, complicating the optimization procedure. The Huber loss function combines squared and absolute loss functions, introducing a threshold parameter δ to differentiate between regular and outlier samples, mathematically defined by a piecewise function.

Log-cosh loss, defined as the logarithm of the hyperbolic cosine of the prediction error, approximates quadratic loss for small errors and behaves similarly to absolute loss for large errors, providing robustness while maintaining smooth gradients. Although twice differentiable everywhere, the gradient and Hessian of the log-cosh loss can saturate for samples with large errors, potentially reducing optimization efficiency.

Enhanced forecasting accuracy using a robust loss function for time series analysis

A new robust, bounded, and smooth (RoBoS-NN) loss function has been developed to improve the performance of supervised learning algorithms on high-dimensional datasets containing outliers. Traditional loss functions often struggle with these types of data, leading to reduced accuracy and slower training times, but this new function addresses these limitations through its inherent properties.

Theoretical analysis confirms the robustness of the RoBoS-NN loss, establishing a credible foundation for building more resilient supervised learning models. Implementation of the RoBoS-NN loss within a neural network framework, resulting in a new algorithm termed -NN, demonstrates significant improvements in forecasting accuracy across multiple real-world time series datasets.

Experiments, including those with artificially introduced outliers, reveal that -NN outperforms benchmark models, reducing average mean absolute error, root mean squared error, and mean absolute scaled error by substantial margins, approximately 31% to 32% on average across several comparative loss functions. The favourable theoretical characteristics of RoBoS-NN suggest its potential for integration into more complex neural network architectures. Future research will explore the application of this loss function to convolutional and recurrent neural networks, potentially yielding further gains in robustness and performance across diverse applications.