Deepfednas Achieves 61x Faster Federated Neural Architecture Search with 1.21% Improvement

Researchers are tackling the challenge of designing effective machine learning models within the constraints of Federated Learning, a technique prioritising data privacy. Bostan Khan and Masoud Daneshtalab, from Mälardalen University, along with et al., present DeepFedNAS, a unified framework poised to revolutionise this field by addressing key bottlenecks in automated model design, specifically, inefficient supernet training and lengthy post-training subnet discovery. Their innovative two-phase approach, underpinned by a novel fitness function, not only achieves state-of-the-art accuracy , demonstrating up to a 1.21% improvement on the CIFAR-100 dataset , but also dramatically accelerates the process, reducing pipeline times from over 20 hours to just 20 minutes. This significant speedup, coupled with improved efficiency, promises to make practical, hardware-aware Federated Learning deployments a reality.

DeepFedNAS tackles efficient federated architecture search

Scientists have unveiled DeepFedNAS, a novel two-phase framework designed to overcome critical bottlenecks in Federated Neural Architecture Search (FedNAS) for privacy-preserving Federated Learning (FL). The research addresses the challenges of suboptimal models resulting from unguided supernet training and the lengthy pipelines required for post-training subnet discovery. DeepFedNAS introduces a principled, multi-objective fitness function that seamlessly synthesizes mathematical network design with established architectural heuristics, fundamentally improving the efficiency and effectiveness of FedNAS. Enabled by a re-engineered supernet, the team achieved Federated Pareto Optimal Supernet Training, leveraging a pre-computed cache of high-fitness architectures as an intelligent curriculum to optimise shared supernet weights, a significant departure from random sampling approaches.
This innovative approach centres around a unified multi-objective fitness function, F(A), which combines network information theory with empirical architectural heuristics to guide the search process. By operationalising structural guidelines, such as depth uniformity and channel monotonicity, as penalty terms, the researchers created a holistic, single-objective optimisation for desired architectural properties within a federated, weight-sharing context. The resulting Pareto-optimal cache serves as a principled training curriculum, ensuring the supernet’s shared weights are consistently updated by gradients from high-fitness architectures, ultimately producing a superior final supernet. Furthermore, the team introduced a re-engineered, generic ResNet-based supernet framework, expanding the architectural search space and enhancing model fitness substantially.

Subsequently, the study unveils a Predictor-Free Search Method that eliminates the need for costly accuracy surrogates by utilising the fitness function as a direct, zero-cost proxy for accuracy. This allows for on-demand subnet discovery in mere seconds, a dramatic improvement over existing methods. Experiments demonstrate DeepFedNAS achieves state-of-the-art accuracy, with up to a 1.21% absolute improvement on the CIFAR-100 dataset, alongside superior parameter and communication efficiency. The research establishes a substantial ~61x speedup in total post-training search pipeline time, reducing the process from over 20 hours to approximately 20 minutes, including initial cache generation.

The team’s advancements extend to enabling 20-second individual subnet searches, making hardware-aware FL deployments instantaneous and practical. By fundamentally transforming both supernet training and the subsequent search for optimal subnets, DeepFedNAS opens new avenues for intelligent, on-device applications in areas like mobile keyboard prediction and personalised healthcare. The complete source code and experimental scripts are publicly available, facilitating further research and development in this rapidly evolving field.

Scientists Method

Scientists developed DeepFedNAS, a novel two-phase framework to overcome bottlenecks in federated neural architecture search (FedNAS). The study addresses unguided supernet training and costly post-training subnet discovery, critical limitations in current privacy-preserving Federated Learning (FL) systems. Researchers engineered a principled, multi-objective fitness function, F(A), which synthesises mathematical network design with architectural heuristics to evaluate network architectures effectively. This function operationalises structural guidelines, such as depth uniformity and channel monotonicity, as penalty terms, enabling holistic single-objective optimisation within a federated, weight-sharing context.

To optimise shared supernet weights, the team pioneered Federated Pareto Optimal Supernet Training, leveraging a pre-computed Pareto-optimal cache of high-fitness architectures as an intelligent curriculum. This approach intelligently guides the training process, selecting architectures that balance multiple objectives and improve overall performance. The system delivers a significant improvement over random sampling methods commonly used in baseline frameworks, reducing noisy gradient updates and enhancing the performance of higher-performing subnets. Subsequently, the Predictor-Free Search Method eliminates the need for costly accuracy surrogates by directly utilising the fitness function as a zero-cost proxy for accuracy.

Experiments employed a re-engineered supernet architecture to facilitate this streamlined process, achieving on-demand subnet discovery in mere seconds. DeepFedNAS achieves state-of-the-art accuracy, demonstrating up to a 1.21% absolute improvement on the CIFAR-100 dataset. Furthermore, the research showcases superior parameter and communication efficiency alongside a substantial ~61x speedup in the total post-training search pipeline time. By reducing the pipeline from over 20 hours to approximately 20 minutes, including initial cache generation, and enabling 20-second individual subnet searches, DeepFedNAS makes hardware-aware FL deployments instantaneous and practical. The computations in this work were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no0.2022-06725.

DeepFedNAS boosts accuracy and speeds FL pipelines by

Scientists achieved a substantial breakthrough in Federated Learning (FL) with the development of DeepFedNAS, a novel framework designed to automate model design while preserving data privacy. Experiments revealed that DeepFedNAS delivers state-of-the-art accuracy, with up to a 1.21% absolute improvement on the CIFAR-100 dataset, demonstrating a significant advancement in performance. The team measured a remarkable ~61x speedup in the total post-training pipeline time, reducing the process from over 20 hours to approximately 20 minutes, including initial cache generation. This acceleration makes hardware-aware FL deployments instantaneous and practical for real-world applications.

Researchers engineered a re-engineered supernet and implemented Federated Pareto Optimal Supernet Training, leveraging a pre-computed Pareto-optimal cache to intelligently optimize shared supernet weights. Data shows that this approach creates a curriculum for training, resulting in superior model performance. Subsequently, the Predictor-Free Method eliminates the need for costly accuracy surrogates by directly utilizing the fitness function as a zero-cost proxy for accuracy, enabling on-demand subnet discovery in a mere 20 seconds. Tests prove that this streamlined process drastically reduces computational overhead and accelerates the deployment of federated learning models.

The study meticulously evaluated performance across diverse datasets, CIFAR-10, CIFAR-100, and CINIC-10, and varying MACs targets, consistently finding superior architectures, particularly on the more complex CIFAR-100 dataset. Measurements confirm that DeepFedNAS achieved 94.16% ±0.18 accuracy on CIFAR-10 with a 0.45-0.95 billion MACs budget, surpassing the baseline SuperFedNAS by 0.69%. Furthermore, the framework’s ability to discover high-performing subnets while adhering to explicit parameter and latency budgets was demonstrated, opening possibilities for resource-constrained devices. Scientists analyzed the framework’s robustness under varying FL conditions, including data heterogeneity and sparse client participation, using a Dirichlet distribution with α values of 100, 1, and 0.1. Results demonstrate that DeepFedNAS maintains a significant accuracy advantage, even in highly heterogeneous settings (α = 0.1), showcasing its resilience and adaptability. The supernet search space comparison revealed a minimum of 7.55M MACs and 0.13M parameters, a substantial reduction compared to SuperFedNAS’s 458.97M MACs and 10.40M parameters, highlighting the efficiency of the new framework.

DeepFedNAS accelerates federated architecture search efficiently by leveraging

Scientists have developed DeepFedNAS, a novel framework that significantly advances federated neural architecture search by tackling inefficiencies in supernet training and the high costs associated with post-training subnet discovery. This research introduces a principled, multi-objective fitness function combined with a re-engineered supernet, enabling two key innovations: Federated Pareto Optimal Supernet Training and a Predictor-Free Search Method. The former utilises a pre-computed set of high-performing architectures as a learning curriculum, resulting in more robust models, while the latter leverages this optimised supernet to use the fitness function as a direct, cost-effective proxy for subnet accuracy, allowing for rapid architectural searches in approximately 20 seconds. The findings demonstrate state-of-the-art accuracy, with improvements of up to 1.21% on the CIFAR-100 dataset, alongside enhanced parameter and communication efficiency.

Crucially, DeepFedNAS achieves a substantial 61x speedup in the total post-training search pipeline, reducing it from over 20 hours to just 20 minutes including initial cache generation, a considerable improvement for practical applications. The authors acknowledge that the performance gains are currently demonstrated within the specific supernet regime they have defined, potentially limiting generalisability to drastically different network structures. Future research could explore extending the Pareto path approach to broader architectural spaces and investigating the framework’s performance in more complex federated learning scenarios with greater data heterogeneity.