Global convergence of policy gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time

Weichen Wang, Jiequn Han, Zhuoran Yang, Zhaoran Wang

Research output: Contribution to journalArticlepeer-review

Abstract

Reinforcement learning is a powerful tool to learn the optimal policy of possibly multiple agents by interacting with the environment. As the number of agents grow to be very large, the system can be approximated by a mean-field problem. Therefore, it has motivated new research directions for mean-field control (MFC) and mean-field game (MFG). In this paper, we study the policy gradient method for the linear-quadratic mean-field control and game, where we assume each agent has identical linear state transitions and quadratic cost functions. While most of the recent works on policy gradient for MFC and MFG are based on discrete-time models, we focus on the continuous-time models where some analyzing techniques can be interesting to the readers. For both MFC and MFG, we provide policy gradient update and show that it converges to the optimal solution at a linear rate, which is verified by a synthetic simulation. For MFG, we also provide sufficient conditions for the existence and uniqueness of the Nash equilibrium.

Original languageEnglish (US)
JournalUnknown Journal
StatePublished - Aug 16 2020

Keywords

  • Continuous linear dynamics
  • Mean-field control/game
  • Policy gradient
  • Reinforcement learning

ASJC Scopus subject areas

  • General

Fingerprint Dive into the research topics of 'Global convergence of policy gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time'. Together they form a unique fingerprint.

Cite this