Opto-electronic Convolutional Neural Networks Achieve Twice the Accuracy Via Direct Kernel Optimization

Integrating optics and electronics promises faster, more energy-efficient vision systems, but designing these complex networks typically requires extensive computation and vast resources. Ali Almuallem, Harshana Weligampola, and colleagues from Purdue University and Samsung Research America, including Abhiram Gnanasambandam, Wei Xu, and Dilshan Godaliyadda, now present a new design strategy that significantly reduces computational demands while boosting performance. Their method involves initially training a conventional electronic convolutional neural network, then translating the initial layer into an optical component using a technique called direct kernel optimization. This two-stage approach dramatically lowers both computational cost and memory requirements, improves training stability, and, in tests of monocular depth estimation, achieves double the accuracy of traditional end-to-end training methods under the same conditions.

Metasurface Optimization for Hybrid Neural Networks

This research introduces a novel approach, Direct Optimization (DO), to train hybrid optical-electronic neural networks (HOENs) more efficiently. HOENs combine the speed and energy efficiency of optical computing with the flexibility of electronic processing. However, traditional training requires computationally expensive end-to-end optimization of both optical and electronic components. The key challenge lies in the slow and resource-intensive nature of this process. The authors propose a method that directly optimizes the metasurface while keeping the electronic network fixed, significantly reducing the computational burden.

They leverage a differentiable framework to enable gradient-based optimization of the metasurface parameters. DO achieves comparable or better performance than full end-to-end optimization with a substantial reduction in training time and computational resources, and scales effectively to larger and more complex HOENs. The authors demonstrate the effectiveness of DO on a challenging monocular depth estimation task using the KITTI dataset, preserving the accuracy of a purely electronic network while leveraging the benefits of optical computing. By simplifying the training process, DO makes HOENs more accessible and potentially unlocks their full potential for a wide range of applications.

Metasurface Optimization for Efficient Neural Networks

Scientists have developed a two-stage strategy for designing opto-electronic convolutional neural networks, achieving significant improvements in both accuracy and computational efficiency. This work addresses the challenge of integrating optical front-ends with electronic back-ends for fast and energy-efficient vision systems. The team’s approach first trains a standard electronic convolutional neural network, then realizes the front-end optically using a metasurface array, optimized directly for its first convolutional layer. This direct kernel optimization dramatically reduces the number of trainable parameters, from over 400 million in an end-to-end approach to just 403,000 for the optical layer, and a total of 14.

84 million for the entire network. Experiments demonstrate that this method requires only 100 milliseconds for one forward and backward pass, compared to 1. 1 milliseconds for computational training and 418 milliseconds for end-to-end optimization. The reduction in computational burden allows for faster training and reduced resource demands. The team validated the effectiveness of their approach through monocular depth estimation, achieving twice the accuracy of end-to-end training under the same time and resource constraints.

Learned metasurface kernels closely match those of a pre-trained model, as demonstrated by quantitative evaluations across all kernels, which show a high degree of correspondence. Further quantitative results on the KITTI dataset demonstrate superior performance, with AbsRel of 0. 199, SqRel of 1. 674, and RMSE of 6. 996 meters. The team completed the entire process, including training the network and optimizing the metasurface kernels, in under 12 hours on a single GPU.

Two-Stage Optoelectronic Neural Network Design Achieves Gains

This research presents a novel two-stage framework for designing opto-electronic convolutional neural networks, addressing the significant computational challenges of end-to-end training in hybrid systems. The team successfully demonstrated that by first training a standard electronic network and then optimizing a metasurface to replicate the kernels of its initial convolutional layer, substantial reductions in training time and computational cost can be achieved. This approach bypasses the need to simultaneously tune millions of optical parameters. The method’s effectiveness was validated through monocular depth estimation, where the two-stage design achieved twice the accuracy of traditional end-to-end training under the same constraints. Importantly, this work extends beyond tasks with simple outputs, proving viable for dense prediction problems requiring spatially resolved outputs, such as depth estimation. The researchers acknowledge that the framework is readily adaptable to other dense prediction tasks like semantic segmentation and surface normal estimation, paving the way for scalable hybrid vision systems that effectively combine electronic and optical computing.