Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning

Lingxiao Wang; Zhuoran Yang; Zhaoran Wang

Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning

Lingxiao Wang^*, Zhuoran Yang, Zhaoran Wang

^*Corresponding author for this work

Industrial Engineering and Management Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

5 Scopus citations

Abstract

Multi-Agent reinforcement learning (MARL) achieves significant empirical successes. However, MARL suffers from the curse of many agents. In this paper, we exploit the symmetry of agents in MARL. In the most generic form, we study a mean-field MARL problem. Such a mean-field MARL is defined on mean-field states, which are distributions that are supported on continuous space. Based on the mean embedding of the distributions, we propose MF-FQI algorithm, which solves the mean-field MARL and establishes a non-Asymptotic analysis for MF-FQI algorithm. We highlight that MF-FQI algorithm enjoys a "blessing of many agents" property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.

Original language	English (US)
Title of host publication	37th International Conference on Machine Learning, ICML 2020
Editors	Hal Daume, Aarti Singh
Publisher	International Machine Learning Society (IMLS)
Pages	10034-10045
Number of pages	12
ISBN (Electronic)	9781713821120
State	Published - 2020
Event	37th International Conference on Machine Learning, ICML 2020 - Virtual, Online Duration: Jul 13 2020 → Jul 18 2020

Publication series

Name	37th International Conference on Machine Learning, ICML 2020
Volume	PartF168147-13

Conference

Conference	37th International Conference on Machine Learning, ICML 2020
City	Virtual, Online
Period	7/13/20 → 7/18/20

ASJC Scopus subject areas

Computational Theory and Mathematics
Human-Computer Interaction
Software

Cite this

Wang, L., Yang, Z., & Wang, Z. (2020). Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning. In H. Daume, & A. Singh (Eds.), 37th International Conference on Machine Learning, ICML 2020 (pp. 10034-10045). (37th International Conference on Machine Learning, ICML 2020; Vol. PartF168147-13). International Machine Learning Society (IMLS).

Wang, Lingxiao ; Yang, Zhuoran ; Wang, Zhaoran. / Breaking the curse of many agents : Provable mean embedding q-iteration for mean-field reinforcement learning. 37th International Conference on Machine Learning, ICML 2020. editor / Hal Daume ; Aarti Singh. International Machine Learning Society (IMLS), 2020. pp. 10034-10045 (37th International Conference on Machine Learning, ICML 2020).

@inproceedings{203ed7d353a04d84bdac2f678c956ba9,

title = "Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning",

abstract = "Multi-Agent reinforcement learning (MARL) achieves significant empirical successes. However, MARL suffers from the curse of many agents. In this paper, we exploit the symmetry of agents in MARL. In the most generic form, we study a mean-field MARL problem. Such a mean-field MARL is defined on mean-field states, which are distributions that are supported on continuous space. Based on the mean embedding of the distributions, we propose MF-FQI algorithm, which solves the mean-field MARL and establishes a non-Asymptotic analysis for MF-FQI algorithm. We highlight that MF-FQI algorithm enjoys a {"}blessing of many agents{"} property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.",

author = "Lingxiao Wang and Zhuoran Yang and Zhaoran Wang",

note = "Publisher Copyright: {\textcopyright} 2020 by the Authors.; 37th International Conference on Machine Learning, ICML 2020 ; Conference date: 13-07-2020 Through 18-07-2020",

year = "2020",

language = "English (US)",

series = "37th International Conference on Machine Learning, ICML 2020",

publisher = "International Machine Learning Society (IMLS)",

pages = "10034--10045",

editor = "Hal Daume and Aarti Singh",

booktitle = "37th International Conference on Machine Learning, ICML 2020",

}

Wang, L, Yang, Z & Wang, Z 2020, Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning. in H Daume & A Singh (eds), 37th International Conference on Machine Learning, ICML 2020. 37th International Conference on Machine Learning, ICML 2020, vol. PartF168147-13, International Machine Learning Society (IMLS), pp. 10034-10045, 37th International Conference on Machine Learning, ICML 2020, Virtual, Online, 7/13/20.

Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning. / Wang, Lingxiao; Yang, Zhuoran; Wang, Zhaoran.
37th International Conference on Machine Learning, ICML 2020. ed. / Hal Daume; Aarti Singh. International Machine Learning Society (IMLS), 2020. p. 10034-10045 (37th International Conference on Machine Learning, ICML 2020; Vol. PartF168147-13).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Breaking the curse of many agents

T2 - 37th International Conference on Machine Learning, ICML 2020

AU - Wang, Lingxiao

AU - Yang, Zhuoran

AU - Wang, Zhaoran

PY - 2020

Y1 - 2020

N2 - Multi-Agent reinforcement learning (MARL) achieves significant empirical successes. However, MARL suffers from the curse of many agents. In this paper, we exploit the symmetry of agents in MARL. In the most generic form, we study a mean-field MARL problem. Such a mean-field MARL is defined on mean-field states, which are distributions that are supported on continuous space. Based on the mean embedding of the distributions, we propose MF-FQI algorithm, which solves the mean-field MARL and establishes a non-Asymptotic analysis for MF-FQI algorithm. We highlight that MF-FQI algorithm enjoys a "blessing of many agents" property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.

AB - Multi-Agent reinforcement learning (MARL) achieves significant empirical successes. However, MARL suffers from the curse of many agents. In this paper, we exploit the symmetry of agents in MARL. In the most generic form, we study a mean-field MARL problem. Such a mean-field MARL is defined on mean-field states, which are distributions that are supported on continuous space. Based on the mean embedding of the distributions, we propose MF-FQI algorithm, which solves the mean-field MARL and establishes a non-Asymptotic analysis for MF-FQI algorithm. We highlight that MF-FQI algorithm enjoys a "blessing of many agents" property in the sense that a larger number of observed agents improves the performance of MF-FQI algorithm.

UR - http://www.scopus.com/inward/record.url?scp=85105316928&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85105316928&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85105316928

T3 - 37th International Conference on Machine Learning, ICML 2020

SP - 10034

EP - 10045

BT - 37th International Conference on Machine Learning, ICML 2020

A2 - Daume, Hal

A2 - Singh, Aarti

PB - International Machine Learning Society (IMLS)

Y2 - 13 July 2020 through 18 July 2020

ER -

Breaking the curse of many agents: Provable mean embedding q-iteration for mean-field reinforcement learning

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this