We introduce an extended Backpropagation Through Time (eBPTT) learning algorithm for SegmentedMemory Recurrent Neural Networks. The algorithm was compared to an extension of the Real-Time Recurrent Learning algorithm (eRTRL) for these kind of networks. Using the information latching problem as benchmark task, the algorithms’ ability to cope with the learning of long-term dependencies was tested. eRTRL was generally better able to cope with the latching of information over longer periods of time. On the other hand, eBPTT guaranteed a better generalisation when training was successful. Further, due to its computational complexity, eRTRL becomes impractical with increasing network size, making eBPTT the only viable choice in these cases.
Paper
Full text
Extension of Backpropagation through Time for Segmented-memory Recurrent Neural Networks
Semantic Scholar · Computer Science · 2012
Abstract
We introduce an extended Backpropagation Through Time (eBPTT) learning algorithm for SegmentedMemory Recurrent Neural Networks. The algorithm was compared to an extension of the Real-Time Recurrent Learning algorithm (eRTRL) for these kind of networks. Using the information latching problem as benchmark task, the algorithms’ ability to cope with the learning of long-term dependencies was tested. eRTRL was generally better able to cope with the latching of information over longer periods of time. On the other hand, eBPTT guaranteed a better generalisation when training was successful. Further, due to its computational complexity, eRTRL becomes impractical with increasing network size, making eBPTT the only viable choice in these cases.