Detecting bias due to input modelling in computer simulation

Lucy E. Morgan*, Barry L Nelson, Andrew C. Titman, David J. Worthington

*Corresponding author for this work

Research output: Contribution to journalArticle

Abstract

This is the first paper to approach the problem of bias in the output of a stochastic simulation due to using input distributions whose parameters were estimated from real-world data. We consider, in particular, the bias in simulation-based estimators of the expected value (long-run average) of the real-world system performance; this bias will be present even if one employs unbiased estimators of the input distribution parameters due to the (typically) nonlinear relationship between these parameters and the output response. To date this bias has been assumed to be negligible because it decreases rapidly as the quantity of real-world input data increases. While true asymptotically, this property does not imply that the bias is actually small when, as is always the case, data are finite. We present a delta-method approach to bias estimation that evaluates the nonlinearity of the expected-value performance surface as a function of the input-model parameters. Since this response surface is unknown, we propose an innovative experimental design to fit a response-surface model that facilitates a test for detecting a bias of a relevant size with specified power. We evaluate the method using controlled experiments, and demonstrate it through a realistic case study concerning a healthcare call centre.

Original languageEnglish (US)
Pages (from-to)869-881
Number of pages13
JournalEuropean Journal of Operational Research
Volume279
Issue number3
DOIs
StatePublished - Dec 16 2019

Fingerprint

Computer Simulation
Computer simulation
Modeling
Response Surface
Design of experiments
Expected Value
Delta Method
Call Centres
Evaluate
Output
Unbiased estimator
Stochastic Simulation
Long-run
Experimental design
Healthcare
Experiments
System Performance
Nonlinearity
Estimator
Imply

Keywords

  • Bias
  • Input modelling
  • Simulation
  • Uncertainty

ASJC Scopus subject areas

  • Computer Science(all)
  • Modeling and Simulation
  • Management Science and Operations Research
  • Information Systems and Management

Cite this

Morgan, Lucy E. ; Nelson, Barry L ; Titman, Andrew C. ; Worthington, David J. / Detecting bias due to input modelling in computer simulation. In: European Journal of Operational Research. 2019 ; Vol. 279, No. 3. pp. 869-881.
@article{c331094963d74ea8a752632d1f1dfa8e,
title = "Detecting bias due to input modelling in computer simulation",
abstract = "This is the first paper to approach the problem of bias in the output of a stochastic simulation due to using input distributions whose parameters were estimated from real-world data. We consider, in particular, the bias in simulation-based estimators of the expected value (long-run average) of the real-world system performance; this bias will be present even if one employs unbiased estimators of the input distribution parameters due to the (typically) nonlinear relationship between these parameters and the output response. To date this bias has been assumed to be negligible because it decreases rapidly as the quantity of real-world input data increases. While true asymptotically, this property does not imply that the bias is actually small when, as is always the case, data are finite. We present a delta-method approach to bias estimation that evaluates the nonlinearity of the expected-value performance surface as a function of the input-model parameters. Since this response surface is unknown, we propose an innovative experimental design to fit a response-surface model that facilitates a test for detecting a bias of a relevant size with specified power. We evaluate the method using controlled experiments, and demonstrate it through a realistic case study concerning a healthcare call centre.",
keywords = "Bias, Input modelling, Simulation, Uncertainty",
author = "Morgan, {Lucy E.} and Nelson, {Barry L} and Titman, {Andrew C.} and Worthington, {David J.}",
year = "2019",
month = "12",
day = "16",
doi = "10.1016/j.ejor.2019.06.003",
language = "English (US)",
volume = "279",
pages = "869--881",
journal = "European Journal of Operational Research",
issn = "0377-2217",
publisher = "Elsevier",
number = "3",

}

Detecting bias due to input modelling in computer simulation. / Morgan, Lucy E.; Nelson, Barry L; Titman, Andrew C.; Worthington, David J.

In: European Journal of Operational Research, Vol. 279, No. 3, 16.12.2019, p. 869-881.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Detecting bias due to input modelling in computer simulation

AU - Morgan, Lucy E.

AU - Nelson, Barry L

AU - Titman, Andrew C.

AU - Worthington, David J.

PY - 2019/12/16

Y1 - 2019/12/16

N2 - This is the first paper to approach the problem of bias in the output of a stochastic simulation due to using input distributions whose parameters were estimated from real-world data. We consider, in particular, the bias in simulation-based estimators of the expected value (long-run average) of the real-world system performance; this bias will be present even if one employs unbiased estimators of the input distribution parameters due to the (typically) nonlinear relationship between these parameters and the output response. To date this bias has been assumed to be negligible because it decreases rapidly as the quantity of real-world input data increases. While true asymptotically, this property does not imply that the bias is actually small when, as is always the case, data are finite. We present a delta-method approach to bias estimation that evaluates the nonlinearity of the expected-value performance surface as a function of the input-model parameters. Since this response surface is unknown, we propose an innovative experimental design to fit a response-surface model that facilitates a test for detecting a bias of a relevant size with specified power. We evaluate the method using controlled experiments, and demonstrate it through a realistic case study concerning a healthcare call centre.

AB - This is the first paper to approach the problem of bias in the output of a stochastic simulation due to using input distributions whose parameters were estimated from real-world data. We consider, in particular, the bias in simulation-based estimators of the expected value (long-run average) of the real-world system performance; this bias will be present even if one employs unbiased estimators of the input distribution parameters due to the (typically) nonlinear relationship between these parameters and the output response. To date this bias has been assumed to be negligible because it decreases rapidly as the quantity of real-world input data increases. While true asymptotically, this property does not imply that the bias is actually small when, as is always the case, data are finite. We present a delta-method approach to bias estimation that evaluates the nonlinearity of the expected-value performance surface as a function of the input-model parameters. Since this response surface is unknown, we propose an innovative experimental design to fit a response-surface model that facilitates a test for detecting a bias of a relevant size with specified power. We evaluate the method using controlled experiments, and demonstrate it through a realistic case study concerning a healthcare call centre.

KW - Bias

KW - Input modelling

KW - Simulation

KW - Uncertainty

UR - http://www.scopus.com/inward/record.url?scp=85067557438&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067557438&partnerID=8YFLogxK

U2 - 10.1016/j.ejor.2019.06.003

DO - 10.1016/j.ejor.2019.06.003

M3 - Article

AN - SCOPUS:85067557438

VL - 279

SP - 869

EP - 881

JO - European Journal of Operational Research

JF - European Journal of Operational Research

SN - 0377-2217

IS - 3

ER -