Hypernetworks
This work explores hypernetworks: an approach of using a one network, also known as a hypernetwork, to generate the weights for another network.
Hypernetworks provide an abstraction that is similar to what is found in nature: the relationship between a genotype - the hypernetwork - and a phenotype - the main network. Though they are also reminiscent of HyperNEAT in evolution, our hypernetworks are trained end-to-end with backpropagation and thus are usually faster. The focus of this work is to make hypernetworks useful for deep convolutional networks and long recurrent networks, where hypernetworks can be viewed as relaxed form of weight-sharing across layers.
Our main result is that hypernetworks can generate non-shared weights for LSTM and achieve near state-of-the-art results on a variety of sequence modelling tasks including character-level language modelling, handwriting generation and neural machine translation, challenging the weight-sharing paradigm for recurrent networks. Our results also show that hypernetworks applied to convolutional networks still achieve respectable results for image recognition tasks compared to state-of-the-art baseline models while requiring fewer learnable parameters. 27 Sep 2016
It should be noted that this concept is entirely disparate from the HyperNetworks introduced by Ha et al in 2016, which work by modifying or generating the weights of the model, while our Hypernetworks apply a single small neural network (either a linear layer or multi-layer perceptron) at multiple points within the larger network, modifying the hidden states.
この概念は、2016年にHaらが紹介したHyperNetworksがモデルの重みを修正または生成することで機能するのに対し、我々のHypernetworksは、大きなネットワーク内の複数のポイントで単一の小さなニューラルネットワーク（線形層または多層パーセプトロン）を適用し、隠れた状態を修正することとは全く異なることに留意すべきです。