Neuroevolution Achieves Efficient Network Evolution with Novel Inflate and Deflate Operators

The challenge of efficiently evolving artificial neural networks has long been a significant hurdle in machine learning research. Davide Farinati, Frederico J. J. B. Santos, Leonardo Vanneschi, and Mauro Castelli from the Universidade Nova de Lisboa and the University of Trieste present a new approach, termed NEVO-GSPT, designed to overcome these computational limitations. Their work introduces a novel neuroevolution algorithm that leverages geometric semantic operators adapted from genetic programming, alongside a new method for controlled network reduction. This innovation allows for a more focused and cost-effective exploration of potential network architectures, evaluating only the impact of new components rather than the entire structure. Results on standard regression benchmarks demonstrate that NEVO-GSPT consistently produces compact networks with performance matching or exceeding that of established methods like TensorNEAT and SLIM-GSGP.

Slim-GSGP combats bloat in neural networks

This research paper investigates the evolution of artificial neural networks (ANNs) using a neuroevolution method called Geometric Semantic Genetic Programming (GSGP). The study focuses on addressing a common challenge in genetic programming known as bloat, where evolved networks grow unnecessarily large and complex without corresponding improvements in performance. To overcome this, the authors propose a non-bloating variant called SLIM-GSGP, which incorporates specific constraints and mechanisms within its genetic operators to control network growth during evolution.

SLIM-GSGP is evaluated on several benchmark regression datasets, including human oral bioavailability prediction, concrete compressive strength, and building energy efficiency. The results demonstrate that SLIM-GSGP achieves competitive, and in some cases superior, predictive performance compared to standard GSGP and NEAT, while maintaining significantly smaller and more compact network architectures. A key innovation is the deflate mutation, a geometric semantic mutation operator that removes unnecessary nodes and connections, effectively limiting bloat without compromising accuracy.

The study also explores hybridizing GSGP with gradient-based optimization techniques, showing that combining evolutionary search with gradient descent can further enhance network performance. To facilitate adoption and experimentation, the authors have released a Python library implementing SLIM-GSGP, providing a practical tool for researchers interested in neuroevolution and compact neural network design.

Overall, this work presents a scalable and efficient approach to evolving neural networks, producing models that are not only accurate but also more compact and potentially more interpretable. By addressing the bloat problem and providing an accessible software library, SLIM-GSGP offers a valuable contribution to the neuroevolution and machine learning research community.

Neuroevolution via Geometric Semantic Perturbation and Training

The research team developed NeuroEVOlution through Geometric Semantic perturbation and Population based Training (NEVO-GSPT), a novel neuroevolution algorithm designed to overcome computational limitations inherent in evolving neural network architectures. Traditional methods, including grid and random search, often demand exhaustive evaluation of architectural spaces, lacking a clear connection between structural changes and resulting network behaviour. This work addresses these issues by adapting principles from Geometric Semantic Genetic Programming (GSGP) and the Semantic Learning algorithm, SLIM-GSGP, to the evolution of artificial neural networks. Scientists engineered NEVO-GSPT to enhance both computational efficiency and semantic awareness during the evolutionary process.

The study pioneered the use of geometric semantic operators (GSOs), initially from genetic programming, to ensure predictable effects on network semantics, maintaining a unimodal error surface. Crucially, the team introduced the Deflate Geometric Semantic Mutation (DGSM) operator, a novel technique that enables controlled reduction of network size, complementing the existing Inflate Geometric Semantic Mutation (IGSM) which expands networks. This alternating inflation and deflation strategy prevents uncontrolled growth, facilitating the evolution of compact and interpretable networks. Experiments employed a linked list representation of networks as perturbation components, allowing for incremental fitness evaluation.

This innovative approach means that only newly added or removed components require assessment, reusing prior evaluations and drastically reducing computational cost compared to methods requiring full retraining. The experimental setup involved four regression benchmarks, designed to investigate the impact of pre-training the initial population and the benefits of the NEVO-GSPT approach. Results demonstrate that NEVO-GSPT consistently evolves compact networks achieving performance comparable to, or exceeding, established methods like standard neural networks, SLIM-GSGP, TensorNEAT, and SLM. The technique reveals a significant advancement in efficiently exploring the architectural search space, offering a computationally viable alternative to resource-intensive methods such as Neural Architecture Search, which often requires thousands of GPU-hours.

Geometric Operators Speed Neural Network Evolution

Scientists have achieved a breakthrough in neuroevolution with the development of NEVO-GSPT, a novel algorithm for evolving artificial neural networks. This work addresses the computational demands of traditional network evolution methods by introducing geometric semantic operators (GSOs) adapted from genetic programming. These GSOs ensure that structural modifications to networks produce predictable effects, operating within a unimodal error surface, and facilitating a more efficient search process. The team measured a significant reduction in computational cost, as the algorithm only requires evaluating the semantics of newly added components during population-based training.

Experiments revealed that NEVO-GSPT consistently evolves compact neural networks on four regression benchmarks. The research details a methodology incorporating both ‘inflate’ and ‘deflate’ operators; the novel ‘deflate geometric semantic mutation’ (DGSM) controls network size reduction while preserving semantic properties established by the ‘inflate geometric semantic mutation’ (IGSM). Tests prove that this alternating inflation and deflation prevents uncontrolled model growth, resulting in interpretable networks. The study investigated the impact of factors including initial population training, post-evolution fine-tuning, inflate-deflate probabilities, and the complexity of added components, providing a comprehensive analysis of the algorithm’s performance.

Data shows that NEVO-GSPT’s efficient evaluation mechanism drastically reduces fitness computation costs by reusing previous evaluations, a key advancement over methods requiring full network retraining. The researchers conducted experiments designed to assess four key factors, demonstrating the algorithm’s adaptability and robustness. Results demonstrate performance comparable to, or exceeding, established methods like standard neural networks, SLIM-GSGP, TensorNEAT, and SLM. This breakthrough delivers a computationally efficient and semantically aware approach to neural network design, opening possibilities for wider accessibility and application in machine learning tasks.

The work builds upon the Semantic Learning Machine (SLM) framework, extending it with the DGSM operator to achieve controlled network complexity. Scientists recorded that maintaining networks as linked lists of perturbation components further enhances efficiency, as only new or removed components require evaluation during mutation. This incremental mechanism represents a substantial improvement in computational speed and resource utilization, potentially enabling the evolution of more complex and effective neural network architectures.

Geometric Semantics Drive Efficient Neural Architecture Search

This research introduced NEVO-GSPT, a novel neuroevolution algorithm designed to address computational demands and improve understanding of structure-behaviour relationships in neural architecture search. The work centres on two key innovations: the adaptation of geometric semantic operators and the development of a deflationary geometric semantic modification operator, enabling controlled network reduction without compromising semantic properties. Through these advancements, NEVO-GSPT achieves efficient population-based training by evaluating only newly modified components, significantly reducing computational cost. Experimental results across four regression benchmarks demonstrate that NEVO-GSPT consistently achieves performance comparable to, or exceeding, established methods like standard networks, SLM, TensorNEAT, and SLIM-GSGP.

Importantly, this performance is attained with notably more compact networks than those produced by several comparative algorithms. The authors acknowledge that direct runtime comparisons with GPU-accelerated implementations are limited by their CPU-based setup, but highlight the algorithm’s ability to explore numerous architectural solutions rapidly on standard hardware. Future work could focus on exploring the algorithm’s performance with GPU acceleration and investigating its application to more complex problem domains.