Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach

Yingjie Fei; Zhuoran Yang; Zhaoran Wang

Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach

Yingjie Fei^*, Zhuoran Yang, Zhaoran Wang

^*Corresponding author for this work

Industrial Engineering and Management Sciences

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

31 Scopus citations

Abstract

We study function approximation for episodic reinforcement learning with entropic risk measure. We first propose an algorithm with linear function approximation. Compared to existing algorithms, which suffer from improper regularization and regression biases, this algorithm features debiasing transformations in backward induction and regression procedures. We further propose an algorithm with general function approximation, which is shown to perform implicit debiasing transformations. We prove that both algorithms achieve a sublinear regret and demonstrate a tradeoff between generality and efficiency. Our analysis provides a unified framework for function approximation in risk-sensitive reinforcement learning, which leads to the first sub-linear regret bounds in the setting.

Original language	English (US)
Title of host publication	Proceedings of the 38th International Conference on Machine Learning, ICML 2021
Publisher	ML Research Press
Pages	3198-3207
Number of pages	10
ISBN (Electronic)	9781713845065
State	Published - 2021
Event	38th International Conference on Machine Learning, ICML 2021 - Virtual, Online Duration: Jul 18 2021 → Jul 24 2021

Publication series

Name	Proceedings of Machine Learning Research
Volume	139
ISSN (Electronic)	2640-3498

Conference

Conference	38th International Conference on Machine Learning, ICML 2021
City	Virtual, Online
Period	7/18/21 → 7/24/21

Funding

We thank the reviewers for their constructive feedback. Z. Yang acknowledges Simons Institute (Theory of Reinforcement Learning). Z. Wang acknowledges National Science Foundation (Awards 2048075, 2008827, 2015568, 1934931), Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two Sigma for their supports.

ASJC Scopus subject areas

Artificial Intelligence
Software
Control and Systems Engineering
Statistics and Probability

Cite this

@inproceedings{151f31bd6d4c45c5b0ab9f76dca040a5,

title = "Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach",

abstract = "We study function approximation for episodic reinforcement learning with entropic risk measure. We first propose an algorithm with linear function approximation. Compared to existing algorithms, which suffer from improper regularization and regression biases, this algorithm features debiasing transformations in backward induction and regression procedures. We further propose an algorithm with general function approximation, which is shown to perform implicit debiasing transformations. We prove that both algorithms achieve a sublinear regret and demonstrate a tradeoff between generality and efficiency. Our analysis provides a unified framework for function approximation in risk-sensitive reinforcement learning, which leads to the first sub-linear regret bounds in the setting.",

author = "Yingjie Fei and Zhuoran Yang and Zhaoran Wang",

note = "Funding Information: We thank the reviewers for their constructive feedback. Z. Yang acknowledges Simons Institute (Theory of Reinforcement Learning). Z. Wang acknowledges National Science Foundation (Awards 2048075, 2008827, 2015568, 1934931), Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two Sigma for their supports. Publisher Copyright: Copyright {\textcopyright} 2021 by the author(s); 38th International Conference on Machine Learning, ICML 2021 ; Conference date: 18-07-2021 Through 24-07-2021",

year = "2021",

language = "English (US)",

series = "Proceedings of Machine Learning Research",

publisher = "ML Research Press",

pages = "3198--3207",

booktitle = "Proceedings of the 38th International Conference on Machine Learning, ICML 2021",

}

Fei, Y, Yang, Z & Wang, Z 2021, Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach. in Proceedings of the 38th International Conference on Machine Learning, ICML 2021. Proceedings of Machine Learning Research, vol. 139, ML Research Press, pp. 3198-3207, 38th International Conference on Machine Learning, ICML 2021, Virtual, Online, 7/18/21.

Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach. / Fei, Yingjie; Yang, Zhuoran; Wang, Zhaoran.
Proceedings of the 38th International Conference on Machine Learning, ICML 2021. ML Research Press, 2021. p. 3198-3207 (Proceedings of Machine Learning Research; Vol. 139).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Risk-Sensitive Reinforcement Learning with Function Approximation

T2 - 38th International Conference on Machine Learning, ICML 2021

AU - Fei, Yingjie

AU - Yang, Zhuoran

AU - Wang, Zhaoran

N1 - Funding Information: We thank the reviewers for their constructive feedback. Z. Yang acknowledges Simons Institute (Theory of Reinforcement Learning). Z. Wang acknowledges National Science Foundation (Awards 2048075, 2008827, 2015568, 1934931), Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two Sigma for their supports. Publisher Copyright: Copyright © 2021 by the author(s)

PY - 2021

Y1 - 2021

N2 - We study function approximation for episodic reinforcement learning with entropic risk measure. We first propose an algorithm with linear function approximation. Compared to existing algorithms, which suffer from improper regularization and regression biases, this algorithm features debiasing transformations in backward induction and regression procedures. We further propose an algorithm with general function approximation, which is shown to perform implicit debiasing transformations. We prove that both algorithms achieve a sublinear regret and demonstrate a tradeoff between generality and efficiency. Our analysis provides a unified framework for function approximation in risk-sensitive reinforcement learning, which leads to the first sub-linear regret bounds in the setting.

AB - We study function approximation for episodic reinforcement learning with entropic risk measure. We first propose an algorithm with linear function approximation. Compared to existing algorithms, which suffer from improper regularization and regression biases, this algorithm features debiasing transformations in backward induction and regression procedures. We further propose an algorithm with general function approximation, which is shown to perform implicit debiasing transformations. We prove that both algorithms achieve a sublinear regret and demonstrate a tradeoff between generality and efficiency. Our analysis provides a unified framework for function approximation in risk-sensitive reinforcement learning, which leads to the first sub-linear regret bounds in the setting.

UR - http://www.scopus.com/inward/record.url?scp=85161302153&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85161302153&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85161302153

T3 - Proceedings of Machine Learning Research

SP - 3198

EP - 3207

BT - Proceedings of the 38th International Conference on Machine Learning, ICML 2021

PB - ML Research Press

Y2 - 18 July 2021 through 24 July 2021

ER -

Risk-Sensitive Reinforcement Learning with Function Approximation: A Debiasing Approach

Abstract

Publication series

Conference

Funding

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this