Titel: Learning long-term dependencies in segmented-memory recurrent neural networks with backpropagation of error
Autor/-in: Glüge, Stefan
Böck, Ronald
Palm, Günther
Wendemuth, Andreas
Erschienen in: Neurocomputing
Band(Heft): 141
Seiten: 54
Seiten bis: 64
Verlag / Hrsg. Institution: Elsevier
Erscheinungsdatum: 2014
Lizenz (gemäss Verlagsvertrag): Lizenz gemäss Verlagsvertrag
Art der Begutachtung: Peer review (Publikation)
Sprache: Englisch
Fachgebiet (DDC): 004: Informatik
Zusammenfassung: In general, recurrent neural networks have difficulties in learning long-term dependencies. The segmented-memory recurrent neural network (SMRNN) architecture together with the extended real-time recurrent learning (eRTRL) algorithm was proposed to circumvent this problem. Due to its computational complexity eRTRL becomes impractical with increasing network size. Therefore, we introduce the less complex extended backpropagation through time (eBPTT) for SMRNN together with a layer-local unsupervised pre-training procedure. A comparison on the information latching problem showed that eRTRL is better able to handle the latching of information over longer periods of time, even though eBPTT guaranteed a better generalisation when training was successful. Further, pre-training significantly improved the ability to learn long-term dependencies with eBPTT. Therefore, the proposed eBPTT algorithm is suited for tasks that require big networks where eRTRL is impractical. The pre-training procedure itself is independent of the supervised learning algorithm and can improve learning in SMRNN in general.
Departement: Life Sciences und Facility Management
Organisationseinheit: Institut für Angewandte Simulation (IAS)
Publikationstyp: Beitrag in wissenschaftlicher Zeitschrift
DOI: 10.1016/j.neucom.2013.11.043
ISSN: 0925-2312
1872-8286
URI: https://digitalcollection.zhaw.ch/handle/11475/15300
Enthalten in den Sammlungen:Publikationen Life Sciences und Facility Management

Dateien zu dieser Ressource:
Es gibt keine Dateien zu dieser Ressource.


Alle Ressourcen in diesem Repository sind urheberrechtlich geschützt, soweit nicht anderweitig angezeigt.