Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies

Circuits of biological neurons, such as in the functional parts of the brain\ncan be modeled as networks of coupled oscillators. Inspired by the ability of\nthese systems to express a rich set of outputs while keeping (gradients of)\nstate variables bounded, we propose a novel architecture for recurrent neural\nnetworks. Our proposed RNN is based on a time-discretization of a system of\nsecond-order ordinary differential equations, modeling networks of controlled\nnonlinear oscillators. We prove precise bounds on the gradients of the hidden\nstates, leading to the mitigation of the exploding and vanishing gradient\nproblem for this RNN. Experiments show that the proposed RNN is comparable in\nperformance to the state of the art on a variety of benchmarks, demonstrating\nthe potential of this architecture to provide stable and accurate RNNs for\nprocessing complex sequential data.\n