Scientists have successfully used a nanowire network (NWN) device to perform online dynamical learning, specifically the MNIST handwritten digit classification task. The task, which had not previously been implemented on an NWN device, involves converting digit images into 1-D temporal voltage pulse streams and delivering them to an electrode. The network’s response is read from other electrode channels and classification is performed externally. The weights are learned from the dynamical features and updated after each digit sample using an online iterative algorithm based on recursive least squares. The online method outperformed the batch method in terms of accuracy and speed.
Learning Handwritten Digit Classification
This task involves classifying handwritten digits from 0 to 9, and it has been implemented on a nanowire network (NWN) device. The digit images are converted into 1-D temporal voltage pulse streams and delivered to an electrode. The network’s response is read out from other electrode channels and classification is performed in an external fully-connected output layer. The weights are learned from the dynamical features and updated after each digit sample using an online iterative algorithm based on recursive least squares (RLS).
Dynamical Feature Generation
The NWN device nonlinearly maps the input signals into a higher-dimensional space. Rich and diverse dynamical features are embedded into the channel readouts from the spatially distributed electrodes. These dynamical features, as well as their intra-class diversity, can be harnessed to perform online classification of the MNIST digits.
Online Learning and Classification Accuracy
The MNIST handwritten digit classification results using the online method showed that accuracies increase with the number of readout channels, demonstrating the non-linearity embedded by the network in the readout data. The online method outperforms the batch method, achieving a higher classification accuracy. The online classifier requires only a single epoch of 50,000 training samples, compared to 100 training epochs for the batch method. The accuracy of the online classifier becomes comparable to that of the batch classifier when active error correction is not used in the RLS algorithm.
Mutual Information and Learning Progress
Mutual information (MI) is an information-theoretic metric that can help uncover the inherent information content within a system and provide a means to assess learning progress during training. The coincidence of the saturation in MI with the peak in the mean of the magnitude of the change in the weight matrix between 102 − 103 samples demonstrates learning is associated with information dynamics.
Sequence Memory Task and Time-Dependent Information Processing
An RC framework with online learning is used to demonstrate the capacity of NWNs to recall a target digit in a temporal digit sequence constructed from the MNIST database. The sequence memory task involves a semi-repetitive sequence of 8 handwritten digits delivered consecutively into the network. Using a sliding memory window, the earliest (first) digit is reconstructed from the memory features embedded in the conductance readout of subsequent digits. The network conductance time series and readout voltages for one of the digit sequence samples show that NWNs retain the memory of previous inputs.
“The first task we performed to test online dynamical learning is the MNIST handwritten digit classification task, which has not previously been experimentally implemented on an NWN device (but has been implemented in NWN simulations).”
Authors of the study
“For each class, the readout voltages from each channel (columns 3–7, blue) are distinctly different from the corresponding input voltages and exhibit diverse characteristics across the readout channels. This demonstrates that the NWN nonlinearly maps the input signals into a higher-dimensional space.”
Authors of the study
For the same number of channels, however, the online method outperforms the batch method. In addition to achieving a higher classification accuracy, the online classifier W requires only a single epoch of 50,000 training samples, compared to 100 training epochs for the batch method using 500 mini-batches of 100 samples and a learning rate η = 0.1.
Authors of the study
“The rise and fall of the learning rate profile can be interpreted in terms of maximal dynamical information being extracted by the network.”
Authors of the study
“The NWN is driven to different internal states as different digits are delivered to the network in sequence. While the dynamics corresponding to digits from the same class show some similar characteristics in the I − V phase space (e.g., digit ‘1’), generally, they exhibit distinctive characteristics due to their sequence position.”
Authors of the study
“This indicates that the repeat digits produce memory traces that are not completely forgotten before each repetition (i.e., nano-filaments in memristive junctions do not completely decay). On average, the linear reconstructor is able to recall these digits better than the non-repeat digits.”
Authors of the study
Quick Summary
The article discusses an experiment where a nanowire network (NWN) device was used to perform online dynamical learning, specifically the classification of handwritten digits from the MNIST database. The NWN device was able to learn from the dynamical features of the data, updating its weights after each digit sample using an online iterative algorithm, and demonstrated the ability to recall a target digit in a temporal digit sequence, indicating the presence of memory in the system.
- The article discusses a test of online dynamical learning using the MNIST handwritten digit classification task, which was implemented on a Nanowire Network (NWN) device.
- The NWN device was used to convert MNIST digit images into 1-D temporal voltage pulse streams, which were then delivered to one electrode. The network’s response was read out from other electrode channels and classification was performed externally.
- The weights were learned from the dynamical features and updated after each digit sample using an online iterative algorithm based on recursive least squares (RLS).
- The online method outperformed the batch method in terms of classification accuracy and required only a single epoch of 50,000 training samples, compared to 100 training epochs for the batch method.
- The article also discusses the use of mutual information (MI) to assess learning progress during training. The MI values for each channel were calculated by averaging the values across the 784 pixel positions.
- The article concludes with a discussion of a sequence memory task, where the NWN device was used to recall a target digit in a temporal digit sequence. The best reconstruction results were achieved for digits with relatively simple structures.