A hybrid training algorithm for recurrent neural network using particle swarm optimization-based preprocessing and temporal error aggregation

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Recurrent neural network has been widely used as auto-regressive model for time series. The most commonly used training method for recurrent neural network is back propagation. However, recurrent neural networks trained with back propagation can get trapped at local minima and saddle points. In these cases, auto-regressive models cannot effectively model time series patterns. In order to address these problems, we propose a hybrid recurrent neural network training algorithm that consists of two phases: exploration and exploitation. Exploration phase uses synchronous particle swarm optimization to search for parameter settings with high activation score and low error. The results of exploration phase are trained with proposed enhanced back propagation, an improved algorithm over traditional back propagation that aggregates temporal errors across timestamps, in exploitation phase. We evaluate our proposed methods using four real-world datasets. Our proposed algorithm, applied to both regularized and adaptive momentum back propagation, increases convergence speed by 10% to 20% and reduces testing mean square error(MSE) at convergence by 5% to 30%. Using particle swarm optimization and activation list in exploration phase, the hybrid training algorithm reduces testing MSEs by more than 30% at convergence compared with traditional back propagation.

Original languageEnglish (US)
Title of host publicationProceeding - 17th IEEE International Conference on Data Mining Workshops, ICDMW 2017
EditorsRaju Gottumukkala, George Karypis, Vijay Raghavan, Xindong Wu, Lucio Miele, Srinivas Aluru, Xia Ning, Guozhu Dong
PublisherIEEE Computer Society
Pages812-817
Number of pages6
ISBN (Electronic)9781538614808
DOIs
StatePublished - Dec 15 2017
Event17th IEEE International Conference on Data Mining Workshops, ICDMW 2017 - New Orleans, United States
Duration: Nov 18 2017Nov 21 2017

Publication series

NameIEEE International Conference on Data Mining Workshops, ICDMW
Volume2017-November
ISSN (Print)2375-9232
ISSN (Electronic)2375-9259

Other

Other17th IEEE International Conference on Data Mining Workshops, ICDMW 2017
CountryUnited States
CityNew Orleans
Period11/18/1711/21/17

Keywords

  • Particle swarm optimization
  • Recurrent neural networks

ASJC Scopus subject areas

  • Computer Science Applications
  • Software

Fingerprint Dive into the research topics of 'A hybrid training algorithm for recurrent neural network using particle swarm optimization-based preprocessing and temporal error aggregation'. Together they form a unique fingerprint.

  • Cite this

    Kang, Q., Liao, W. K., Agrawal, A., & Choudhary, A. (2017). A hybrid training algorithm for recurrent neural network using particle swarm optimization-based preprocessing and temporal error aggregation. In R. Gottumukkala, G. Karypis, V. Raghavan, X. Wu, L. Miele, S. Aluru, X. Ning, & G. Dong (Eds.), Proceeding - 17th IEEE International Conference on Data Mining Workshops, ICDMW 2017 (pp. 812-817). (IEEE International Conference on Data Mining Workshops, ICDMW; Vol. 2017-November). IEEE Computer Society. https://doi.org/10.1109/ICDMW.2017.112