Markov decision processes: A tool for sequential decision making under uncertainty

Oguzhan Alagoz*, Heather Hsu, Andrew J. Schaefer, Mark S. Roberts

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

137 Scopus citations


We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment problem under uncertainty. Markov decision processes generalize standard Markov models in that a decision process is embedded in the model and multiple decisions are made over time. Furthermore, they have significant advantages over standard decision analysis. We compare MDPs to standard Markov-based simulation models by solving the problem of the optimal timing of living-donor liver transplantation using both methods. Both models result in the same optimal transplantation policy and the same total life expectancies for the same patient and living donor. The computation time for solving the MDP model is significantly smaller than that for solving the Markov model. We briefly describe the growing literature of MDPs applied to medical decisions.

Original languageEnglish (US)
Pages (from-to)474-483
Number of pages10
JournalMedical Decision Making
Issue number4
StatePublished - Jul 2010
Externally publishedYes


  • Markov decision processes
  • Markov processes
  • decision analysis

ASJC Scopus subject areas

  • Health Policy


Dive into the research topics of 'Markov decision processes: A tool for sequential decision making under uncertainty'. Together they form a unique fingerprint.

Cite this