Neural Architecture Search Achieves 0.39 Improvement with Dual Contrastive Learning

Automatically designing effective neural networks represents a significant challenge in modern artificial intelligence, and researchers Xian-Rong Zhang, Yue-Jiao Gong, and Wei-Neng Chen from South China University of Technology, along with Jun Zhang from Nankai University and Hanyang University, now present a new approach to this problem. Their work addresses the substantial computational demands of evolutionary neural architecture search, where training numerous networks to guide the design process can be prohibitively expensive. The team developed a method called Dual Contrastive Learning, which efficiently trains a predictor network by first learning meaningful representations without labelled data, and then refining its ability to compare the performance of different network designs, rather than predicting absolute values. This innovation achieves state-of-the-art results on standard benchmarks, improving accuracy by up to 0.39%, and demonstrates a substantial performance gain on a practical medical application, all while dramatically reducing the required computational resources.

Summary of the Research Paper: Self-supervised representation learning for evolutionary neural architecture search (and related works).

This text details research into improving Neural Architecture Search (NAS) through techniques like self-supervised learning (SSL) and efficient search strategies. NAS aims to automatically design optimal neural network architectures, but it is incredibly resource-intensive, requiring the training and evaluation of numerous candidate designs. To address this, researchers are exploring surrogate models, which predict the performance of an architecture without fully training it, and SSL, which learns better representations of network architectures to improve generalization. Multi-fidelity optimization, evaluating architectures at varying levels of detail, allows for quick initial screening, while architecture augmentation creates variations to improve predictor robustness.

Graph Neural Networks (GNNs) effectively represent architecture information, and Bayesian optimization efficiently explores the search space., Key findings demonstrate that SSL improves predictor accuracy, efficient search techniques are crucial for practical NAS, and GNNs are effective for architecture representation. Data augmentation further enhances predictor generalization. In essence, the research focuses on making NAS more efficient and effective by leveraging self-supervised learning to create better representations of neural network architectures and using those representations to guide the search process, ultimately reducing computational cost while maintaining or improving architecture quality.,.

Contrastive Learning for Efficient Architecture Search

The study pioneers a novel approach to Evolutionary Neural Architecture Search (ENAS), termed DCL-ENAS, to address the computational demands of training a high-precision predictor. Recognizing that labeling each architecture requires full training, the team developed a method to maximize predictor accuracy within a limited compute budget. DCL-ENAS employs a two-stage contrastive learning process, initially leveraging contrastive self-supervised learning to extract meaningful representations from neural architectures without labeled data. This allows the system to understand architectural characteristics before performance evaluation, significantly improving efficiency.

Subsequently, the research utilizes fine-tuning with contrastive learning, focusing on predicting the relative performance of architectures rather than absolute values, simplifying the evolutionary search., Experiments employed the NASBench-101 and NASBench-201 datasets, where DCL-ENAS consistently achieved the highest validation accuracy, exceeding existing baselines by 0.05% on the ImageNet16-120 dataset and by 0.39% on NASBench-101. To demonstrate real-world applicability, the team applied DCL-ENAS to an ECG arrhythmia classification task, achieving an approximate 2.5 percentage point performance improvement over a manually designed model, using only 7.7 GPU-days. This work represents a significant step forward in automated machine learning, enabling the discovery of high-performing neural network architectures with substantially reduced computational resources.,.

Contrastive Learning Speeds Neural Network Design

The research team developed DCL-ENAS, a novel method for efficiently designing neural networks, and achieved significant improvements in predictor training for Evolutionary Neural Architecture Search (ENAS). This work addresses the substantial computational costs associated with training predictors, which require fully training numerous network architectures to generate accurate labels, by employing a two-stage contrastive learning process. Initially, the team utilized contrastive self-supervised learning to establish meaningful representations of network architectures without requiring labeled data, learning from the inherent structure of the networks themselves., Experiments demonstrate that this pre-training stage significantly enhances the predictor’s ability to discern subtle differences between architectures, leading to more accurate performance predictions. Following pre-training, a contrastive fine-tuning stage focuses on predicting the relative performance of different networks, rather than absolute values, streamlining the evolutionary search process.

Tests on NASBench-101 and NASBench-201 reveal that DCL-ENAS surpasses existing methods, achieving validation accuracy improvements ranging from 0.05% on the ImageNet16-120 dataset to 0.39% on NASBench-101. Further validation on a real-world ECG arrhythmia classification task showed a performance increase of approximately 2.5 percentage points compared to a manually designed model, all while requiring only 7.7 GPU-days of computation. This breakthrough delivers a substantial reduction in computational burden, enabling the discovery of high-performing neural networks with limited resources, and confirms the effectiveness of the dual contrastive learning strategy in both accuracy and efficiency.,.

Contrastive Learning Streamlines Neural Network Search

The team introduced a new automated network design method called DCL-ENAS, which greatly reduces the high computational cost usually required by evolutionary Neural Architecture Search (ENAS). The method uses a two-stage contrastive learning process to train a performance predictor. First, it learns strong representations from network architectures without using labels. Then, this predictor is refined to accurately compare the relative performance of different architectures.

Instead of estimating exact performance values, DCL-ENAS focuses on predicting which network performs better. This relative evaluation is enough to guide the evolutionary search efficiently and helps identify strong network designs more quickly. Experimental results show that DCL-ENAS achieves state-of-the-art performance on standard benchmarks, outperforming existing methods by up to 0.39% on NASBench-101 and 0.05% on ImageNet16-120.

The approach was also tested in a real-world setting, where it improved performance on an ECG arrhythmia classification task by about 2.5 percentage points compared to a manually designed model, while using much less computational resources—only 7.7 GPU-days. The authors note that the effectiveness of the predictor depends on the quality of the learned network representations and the diversity of explored architectures. They suggest future work could improve the contrastive learning process and expand the search space to further enhance automated network design.