Fully decentralized multi-agent reinforcement learning with networked agents

Kaiqing Zhang; Zhuoran Yang; Han Liu; Tong Zhang; Tamer Başar

Fully decentralized multi-agent reinforcement learning with networked agents

Kaiqing Zhang^*, Zhuoran Yang, Han Liu, Tong Zhang, Tamer Başar

^*Corresponding author for this work

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

101 Scopus citations

Abstract

We consider the fully decentralized multi-agent reinforcement learning (MARL) problem, where the agents are connected via a time-varying and possibly sparse communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network. To maximize the globally averaged return over the network, we propose two fully decentralized actor-critic algorithms, which are applicable to large-scale MARL problems in an online fashion. Convergence guarantees are provided when the value functions are approximated within the class of linear functions. Our work appears to be the first theoretical study of fully decentralized MARL algorithms for networked agents that use function approximation.

Original language	English (US)
Title of host publication	35th International Conference on Machine Learning, ICML 2018
Editors	Andreas Krause, Jennifer Dy
Publisher	International Machine Learning Society (IMLS)
Pages	9340-9371
Number of pages	32
ISBN (Electronic)	9781510867963
State	Published - 2018
Event	35th International Conference on Machine Learning, ICML 2018 - Stockholm, Sweden Duration: Jul 10 2018 → Jul 15 2018

Publication series

Name	35th International Conference on Machine Learning, ICML 2018
Volume	13

Other

Other	35th International Conference on Machine Learning, ICML 2018
Country/Territory	Sweden
City	Stockholm
Period	7/10/18 → 7/15/18

Funding

The work of K.Z. and T.B. was supported in part by the US Army Research Laboratory (ARL) Cooperative Agreement W911NF-17-2-0196, and in part by the US Army Research Office (ARO) Grant W911NF-16-1-0485. The research of H.L. was supported by NSF CAREER Award DMS1454377, NSF IIS1408910, NSF IIS1332109. This material is based upon work supported by the National Science Foundation under grant no. 1740762 "Collaborative Research: TRIPODS Institute for Optimization and Learning". We would also like to thank all the anonymous reviewers for their helpful suggestions and supportive comments.

ASJC Scopus subject areas

Computational Theory and Mathematics
Human-Computer Interaction
Software

Cite this

@inproceedings{3feff5db21b34ab28b36ef648d08263e,

title = "Fully decentralized multi-agent reinforcement learning with networked agents",

abstract = "We consider the fully decentralized multi-agent reinforcement learning (MARL) problem, where the agents are connected via a time-varying and possibly sparse communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network. To maximize the globally averaged return over the network, we propose two fully decentralized actor-critic algorithms, which are applicable to large-scale MARL problems in an online fashion. Convergence guarantees are provided when the value functions are approximated within the class of linear functions. Our work appears to be the first theoretical study of fully decentralized MARL algorithms for networked agents that use function approximation.",

author = "Kaiqing Zhang and Zhuoran Yang and Han Liu and Tong Zhang and Tamer Ba{\c s}ar",

note = "Publisher Copyright: {\textcopyright} Copyright 2018 by the author(s). All rights reserved.; 35th International Conference on Machine Learning, ICML 2018 ; Conference date: 10-07-2018 Through 15-07-2018",

year = "2018",

language = "English (US)",

series = "35th International Conference on Machine Learning, ICML 2018",

publisher = "International Machine Learning Society (IMLS)",

pages = "9340--9371",

editor = "Andreas Krause and Jennifer Dy",

booktitle = "35th International Conference on Machine Learning, ICML 2018",

}

Zhang, K, Yang, Z, Liu, H, Zhang, T & Başar, T 2018, Fully decentralized multi-agent reinforcement learning with networked agents. in A Krause & J Dy (eds), 35th International Conference on Machine Learning, ICML 2018. 35th International Conference on Machine Learning, ICML 2018, vol. 13, International Machine Learning Society (IMLS), pp. 9340-9371, 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, 7/10/18.

Fully decentralized multi-agent reinforcement learning with networked agents. / Zhang, Kaiqing; Yang, Zhuoran; Liu, Han et al.
35th International Conference on Machine Learning, ICML 2018. ed. / Andreas Krause; Jennifer Dy. International Machine Learning Society (IMLS), 2018. p. 9340-9371 (35th International Conference on Machine Learning, ICML 2018; Vol. 13).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Fully decentralized multi-agent reinforcement learning with networked agents

AU - Zhang, Kaiqing

AU - Yang, Zhuoran

AU - Liu, Han

AU - Zhang, Tong

AU - Başar, Tamer

PY - 2018

Y1 - 2018

N2 - We consider the fully decentralized multi-agent reinforcement learning (MARL) problem, where the agents are connected via a time-varying and possibly sparse communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network. To maximize the globally averaged return over the network, we propose two fully decentralized actor-critic algorithms, which are applicable to large-scale MARL problems in an online fashion. Convergence guarantees are provided when the value functions are approximated within the class of linear functions. Our work appears to be the first theoretical study of fully decentralized MARL algorithms for networked agents that use function approximation.

AB - We consider the fully decentralized multi-agent reinforcement learning (MARL) problem, where the agents are connected via a time-varying and possibly sparse communication network. Specifically, we assume that the reward functions of the agents might correspond to different tasks, and are only known to the corresponding agent. Moreover, each agent makes individual decisions based on both the information observed locally and the messages received from its neighbors over the network. To maximize the globally averaged return over the network, we propose two fully decentralized actor-critic algorithms, which are applicable to large-scale MARL problems in an online fashion. Convergence guarantees are provided when the value functions are approximated within the class of linear functions. Our work appears to be the first theoretical study of fully decentralized MARL algorithms for networked agents that use function approximation.

UR - http://www.scopus.com/inward/record.url?scp=85057269259&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85057269259&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85057269259

T3 - 35th International Conference on Machine Learning, ICML 2018

SP - 9340

EP - 9371

BT - 35th International Conference on Machine Learning, ICML 2018

A2 - Krause, Andreas

A2 - Dy, Jennifer

PB - International Machine Learning Society (IMLS)

T2 - 35th International Conference on Machine Learning, ICML 2018

Y2 - 10 July 2018 through 15 July 2018

ER -

Fully decentralized multi-agent reinforcement learning with networked agents

Abstract

Publication series

Other

Funding

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this