Exact learning dynamics of deep linear networks with prior knowledge

Clémentine C. J Dominé, Lukas Braun, James E. Fitzgerald, Andrew M. Saxe*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

Abstract

Learning in deep neural networks is known to depend critically on the knowledge embedded in the initial network weights. However, few theoretical results have precisely linked prior knowledge to learning dynamics. Here we derive exact solutions to the dynamics of learning with rich prior knowledge in deep linear networks by generalising Fukumizu’s matrix Riccati solution (Fukumizu 1998 Gen 1 1E-03). We obtain explicit expressions for the evolving network function, hidden representational similarity, and neural tangent kernel over training for a broad class of initialisations and tasks. The expressions reveal a class of task-independent initialisations that radically alter learning dynamics from slow non-linear dynamics to fast exponential trajectories while converging to a global optimum with identical representational similarity, dissociating learning trajectories from the structure of initial internal representations. We characterise how network weights dynamically align with task structure, rigorously justifying why previous solutions successfully described learning from small initial weights without incorporating their fine-scale structure. Finally, we discuss the implications of these findings for continual learning, reversal learning and learning of structured knowledge. Taken together, our results provide a mathematical toolkit for understanding the impact of prior knowledge on deep learning.

Original languageEnglish (US)
Article number114004
JournalJournal of Statistical Mechanics: Theory and Experiment
Volume2023
Issue number11
DOIs
StatePublished - Nov 1 2023

Keywords

  • deep learning
  • learning theory
  • machine learning

ASJC Scopus subject areas

  • Statistical and Nonlinear Physics
  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Fingerprint

Dive into the research topics of 'Exact learning dynamics of deep linear networks with prior knowledge'. Together they form a unique fingerprint.

Cite this