Autoclip: Adaptive gradient clipping for source separation networks

Prem Seetharaman, Gordon Wichern, Bryan Pardo, Jonathan Le Roux

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Clipping the gradient is a known approach to improving gradient descent, but requires hand selection of a clipping threshold hyperparameter. We present AutoClip, a simple method for automatically and adaptively choosing a gradient clipping threshold, based on the history of gradient norms observed during training. Experimental results show that applying Au-toClip results in improved generalization performance for audio source separation networks. Observation of the training dynamics of a separation network trained with and without AutoClip show that AutoClip guides optimization into smoother parts of the loss landscape. AutoClip is very simple to implement and can be integrated readily into a variety of applications across multiple domains.

Original languageEnglish (US)
Title of host publicationProceedings of the 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing, MLSP 2020
PublisherIEEE Computer Society
ISBN (Electronic)9781728166629
DOIs
StatePublished - Sep 2020
Event30th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2020 - Virtual, Espoo, Finland
Duration: Sep 21 2020Sep 24 2020

Publication series

NameIEEE International Workshop on Machine Learning for Signal Processing, MLSP
Volume2020-September
ISSN (Print)2161-0363
ISSN (Electronic)2161-0371

Conference

Conference30th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2020
Country/TerritoryFinland
CityVirtual, Espoo
Period9/21/209/24/20

Keywords

  • Audio source separation
  • Computer audition
  • Deep learning
  • Optimization

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Signal Processing

Fingerprint

Dive into the research topics of 'Autoclip: Adaptive gradient clipping for source separation networks'. Together they form a unique fingerprint.

Cite this