The effectiveness of recurrent neural networks can be largely influenced by\ntheir ability to store into their dynamical memory information extracted from\ninput sequences at different frequencies and timescales. Such a feature can be\nintroduced into a neural architecture by an appropriate modularization of the\ndynamic memory. In this paper we propose a novel incrementally trained\nrecurrent architecture targeting explicitly multi-scale learning. First, we\nshow how to extend the architecture of a simple RNN by separating its hidden\nstate into different modules, each subsampling the network hidden activations\nat different frequencies. Then, we discuss a training algorithm where new\nmodules are iteratively added to the model to learn progressively longer\ndependencies. Each new module works at a slower frequency than the previous\nones and it is initialized to encode the subsampled sequence of hidden\nactivations. Experimental results on synthetic and real-world datasets on\nspeech recognition and handwritten characters show that the modular\narchitecture and the incremental training algorithm improve the ability of\nrecurrent neural networks to capture long-term dependencies.\n