Hebbian learning with gradients: Hebbian convolutional neural networks with modern deep learning frameworks

Deep learning networks generally use non-biological learning methods. By\ncontrast, networks based on more biologically plausible learning, such as\nHebbian learning, show comparatively poor performance and difficulties of\nimplementation. Here we show that Hebbian learning in hierarchical,\nconvolutional neural networks can be implemented almost trivially with modern\ndeep learning frameworks, by using specific losses whose gradients produce\nexactly the desired Hebbian updates. We provide expressions whose gradients\nexactly implement a plain Hebbian rule (dw ~= xy), Grossberg's instar rule (dw\n~= y(x-w)), and Oja's rule (dw ~= y(x-yw)). As an application, we build Hebbian\nconvolutional multi-layer networks for object recognition. We observe that\nhigher layers of such networks tend to learn large, simple features (Gabor-like\nfilters and blobs), explaining the previously reported decrease in decoding\nperformance over successive layers. To combat this tendency, we introduce\ninterventions (denser activations with sparse plasticity, pruning of\nconnections between layers) which result in sparser learned features, massively\nincrease performance, and allow information to increase over successive layers.\nWe hypothesize that more advanced techniques (dynamic stimuli, trace learning,\nfeedback connections, etc.), together with the massive computational boost\noffered by modern deep learning frameworks, could greatly improve the\nperformance and biological relevance of multi-layer Hebbian networks.\n