Implicit Function Theorem Reveals Data Encoding in Neural Networks with Weights

Researchers are increasingly exploring how neural network weights can be treated as a data modality, a field known as Weight Space Learning, with promising applications for data representation and transfer. Tianming Qiu, Christos Sonis, and Hao Shen, all from the Technical University of Munich and fortiss, have now applied the Implicit Function Theorem to rigorously map data to its corresponding weight representation within Implicit Neural Representations (INRs). This work addresses a crucial gap in understanding how data semantics are encoded into network weights, providing a theoretical framework to analyse this process and demonstrating competitive performance on 2D and 3D classification tasks using a hypernetwork approach. These findings offer a valuable new perspective for investigating the information contained within the often-opaque landscape of neural network weights.

They analyse a framework that maps instance-specific embeddings to INR weights via a shared hypernetwork, achieving performance competitive with existing baselines on downstream classification tasks.

Weight space analysis using implicit neural representations offers a powerful approach to understanding model behavior

Scientists are increasingly viewing network weights as a structured data modality, capable of encoding semantics of training data, building on concepts from Bayesian neural networks and recent Weight Space Learning research. Using INR weights directly for downstream tasks, such as classification, can be challenging, as optimised INRs can vary significantly depending on their initialisations, even for the same signal?
Permutation-equivariant classifiers, such as DWSNet and NFN, address these parameter symmetry issues by ensuring invariance to weight reordering. Similarly, Transformer-based architectures can produce permutation-equivariant representations of weights. More recently, MWT introduces equivariant INR classifiers with end-to-end supervision, achieving strong performance in a setting where labels are integrated into the representation.

While these approaches make progress in handling permutation-induced symmetry, a more fundamental challenge remains unresolved: optimising neural networks, even for the same data samples, cannot be guaranteed to converge to the same set of weights. This reflects a fundamental non-uniqueness of over-parameterised INR weights arising from the optimisation process itself, beyond permutation symmetry.

In practice, this often leads symmetry-aware methods to depend on more intensive sampling of equivalent weight configurations, or increased model capacity to achieve invariance. Functa formalises INR weights as data, with a shared base network capturing common semantics and per-sample modulation vectors encoding variations.

These low-dimensional modulation vectors are effective representations for downstream tasks. Schürholt et al. further learn hyper-representations via self-supervised training on the weights of INR model zoos, yielding embeddings that generalise across tasks. Similarly, inr2vec maps INR weights into latent embeddings for classification and generation, showing that meaningful semantic structure is preserved in weight space representations.

ProbeGen optimises latent embeddings to generate probes for classifying trained INRs. These works indicate a growing interest and challenge in directly analysing neural network weights and discovering compact latent representations that capture their underlying semantics. Early work on implicit neural representations, such as SIREN, explores the use of hypernetworks, where network weights are generated from latent embeddings conditioned on inputs.

Spurek et al. use a hypernetwork to generate 3D point clouds. DeepSDF introduces an auto-decoder architecture that learns instance-specific latent representations. SRNs combine jointly trainable latent embeddings with a hypernetwork to reconstruct 3D scenes and objects.

More recently, D’OH proposed a decoder-only hypernetwork formulation, where instance embeddings are directly optimised from random initialisations, without imposing structural assumptions on the latent space. A growing body of work adopts HyperINR formulations to improve generalisation and reconstruction quality across instances and tasks?

Instead of learning low-dimensional representations from the weights of individually pretrained INRs, a generic HyperINR framework deploys a hypernetwork to map a learnable low-dimensional latent space to the weight space of INR main networks. Let P ⊂ Rp be an open set of p-dimensional coordinates and W ⊂ Rd be the set of weights of INRs.

An INR main network takes sampled coordinates p ∈ P as input and generates the corresponding pixel value or Unsigned Distance Function value in Rc. Specifically, for a set of sampled coordinates {pi}n i=1 ⊂ P, any data example X:= {xi}n i=1 ∈ X ⊂ Rn×c can be represented as a sampled mapping from coordinate pi to data value xi.

Here, an INR main network is defined as the mapping f: W × P → Rc, (w, p) 7→ f(w, p). The weights wi for each data sample are generated by a hypernetwork, which maps a learnable latent embedding zj ∈ Z ⊂ Rl to the weights wi ∈ W of the main network, i.e., φ: V × Z → W, (v, zj) 7→ φ(v, zj), where V ⊂ Rk denotes the space of hypernetwork weights, shared across all data examples.

For a given set of data samples {Xj}t j=1, a common training strategy is to find an optimal set of weights of the hypernetwork v∗ ∈ V and a set of latent embeddings {z∗j}t j=1 ⊂ Z, such that the corresponding INRs exactly represent the data samples, i.e., for all i = 1, . . . , n and j = 1, . . . t, the following equation holds true f(φ(v∗, z∗j), pi) = xj,pi, with Xj:= {xj,i}n i=1 ∈ X. The classic residual loss for assessing the reconstruction of the INR against a given data sample can be defined as l: V × Z × X → R, l(v, z, X) := 1/2 Σi=1n f(φ(v, z), pi) − X 2F = 1/2n Σi=1n f(φ(v, z), pi) − xpi 22, where xpi denotes the value of sample X at coordinate pi, and ∥·∥F denotes the Frobenius norm of matrices.

Assuming that the size of the INRs is sufficiently large to ensure exact reconstruction in the whole data manifold X, there is a hypernetwork weight v∗ ∈ V, which ensures exact reconstruction of any sample X ∈ X. Consequently, for any pair (z, X) ∈ Z × X, the gradient of l vanishes at (v∗, z, X). With a fixed v∗ for the hypernetwork, the function lv∗: Z × X → R, (z, X) 7→ l(v∗, z, X) is defined.

Let w:= φ(v∗, z), and the pixel-wise residual by εi:= f(φ(v∗, z), pi) − xpi. The research team analysed a framework that maps instance-specific embeddings to Implicit Neural Representation (INR) weights via a shared hypernetwork, achieving performance competitive with existing baselines on both 2D and 3D classification tasks.

Experiments revealed that training embeddings form well-separated clusters, and test samples fall into the neighbourhoods of their respective categories when the hypernetwork is fixed, demonstrating semantic preservation in the latent space. The team measured the effectiveness of their HyperINR framework and learned latent embeddings through classifications on five datasets: MNIST, FashionMNIST, ModelNet40, ShapeNet10, and ScanNet10.

Using ShapeNet10, researchers visualised embeddings from chair, sofa, and table categories, applying Principal Component Analysis (PCA) to avoid distortions. Results demonstrate that the training embeddings form distinct clusters, and test samples align with their categories, indicating semantic information is preserved within the latent embeddings.

PCA projections of the hypernetwork-generated weights also exhibited similarly distinct clusters, reinforcing semantic preservation in both latent and weight spaces. Classification tests using a simple MLP classifier on the five datasets yielded top-1 accuracy scores. On FashionMNIST, the method achieved 86.82 ±0.98% using latent embeddings (z) and 86.50 ±1.37% using generated weights (w).

The work achieves state-of-the-art performance on FashionMNIST, ModelNet40, and ShapeNet10, while remaining competitive on MNIST and ScanNet10. Specifically, on MNIST, the method attained 97.89 ±0.29% accuracy with latent embeddings and 97.65 ±0.23% with weights. Data shows that the latent embeddings provide a compact and effective representation of the weights, suggesting the high-dimensional weight space is constrained to a low-dimensional structured manifold.

The team reports all results as “mean ± standard error” over five different random seeds, with detailed seed selection and experimental variability provided in an appendix. Their research introduces HyperINR, a model that maps low-dimensional latent embeddings into the weight space of an Implicit Neural Representation.

This approach allows semantic information from data to be reflected in the network’s weights, demonstrated through competitive performance on 2D and 3D classification tasks. The learned latent embeddings exhibit natural clustering, suggesting effective retention of original data semantics. Notably, the model achieves this performance with a relatively lightweight architecture, employing a hypernetwork containing approximately one million parameters.

The authors acknowledge that the current work focuses on relatively simple datasets and that further investigation is needed to assess the scalability of the approach to more complex scenarios. Future research could explore the potential of this framework for other applications within weight space learning and investigate the broader implications for understanding neural network behaviour.