MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis

Alex Kale; Sarah Lee; Terrance Goan; Elizabeth Tipton; Jessica Hullman

doi:10.1145/3544548.3580869

MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis

Alex Kale, Sarah Lee, Terrance Goan, Elizabeth Tipton, Jessica Hullman

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

5 Scopus citations

Abstract

Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality and commensurability of evidence casts doubt on results from meta-analysis, yet existing software tools for meta-analysis do not provide an explicit software representation of these concerns. We present MetaExplorer, a prototype system for meta-analysis that we developed using iterative design with meta-analysis experts to provide a guided process for eliciting assessments of uncertainty and reasoning about how to incorporate them during statistical inference. Our qualitative evaluation of MetaExplorer with experienced meta-analysts shows that imposing a structured workflow both elevates the perceived importance of epistemic concerns and presents opportunities for tools to engage users in dialogue around goals and standards for evidence aggregation.

Original language	English (US)
Title of host publication	CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher	Association for Computing Machinery
ISBN (Electronic)	9781450394215
DOIs	https://doi.org/10.1145/3544548.3580869
State	Published - Apr 19 2023
Event	2023 CHI Conference on Human Factors in Computing Systems, CHI 2023 - Hamburg, Germany Duration: Apr 23 2023 → Apr 28 2023

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	2023 CHI Conference on Human Factors in Computing Systems, CHI 2023
Country/Territory	Germany
City	Hamburg
Period	4/23/23 → 4/28/23

Funding

This work was funded by US Navy STTR Contract N6833517-C-0410 in partnership with Stottler Henke Assoc, and NSF award #1749266.

Keywords

Meta-analysis
epistemic uncertainty
literature review

ASJC Scopus subject areas

Software
Human-Computer Interaction
Computer Graphics and Computer-Aided Design

Access to Document

10.1145/3544548.3580869

Cite this

Kale, A., Lee, S., Goan, T., Tipton, E., & Hullman, J. (2023). MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis. In CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems Article 465 (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/3544548.3580869

@inproceedings{fb0a307e481447d5b77491fee1d6a3a6,

title = "MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis",

abstract = "Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality and commensurability of evidence casts doubt on results from meta-analysis, yet existing software tools for meta-analysis do not provide an explicit software representation of these concerns. We present MetaExplorer, a prototype system for meta-analysis that we developed using iterative design with meta-analysis experts to provide a guided process for eliciting assessments of uncertainty and reasoning about how to incorporate them during statistical inference. Our qualitative evaluation of MetaExplorer with experienced meta-analysts shows that imposing a structured workflow both elevates the perceived importance of epistemic concerns and presents opportunities for tools to engage users in dialogue around goals and standards for evidence aggregation.",

keywords = "Meta-analysis, epistemic uncertainty, literature review",

author = "Alex Kale and Sarah Lee and Terrance Goan and Elizabeth Tipton and Jessica Hullman",

note = "Publisher Copyright: {\textcopyright} 2023 ACM.; 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023 ; Conference date: 23-04-2023 Through 28-04-2023",

year = "2023",

month = apr,

day = "19",

doi = "10.1145/3544548.3580869",

language = "English (US)",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

booktitle = "CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems",

}

Kale, A, Lee, S, Goan, T, Tipton, E & Hullman, J 2023, MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis. in CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems., 465, Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023, Hamburg, Germany, 4/23/23. https://doi.org/10.1145/3544548.3580869

MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis. / Kale, Alex; Lee, Sarah; Goan, Terrance et al.
CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2023. 465 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - MetaExplorer

T2 - 2023 CHI Conference on Human Factors in Computing Systems, CHI 2023

AU - Kale, Alex

AU - Lee, Sarah

AU - Goan, Terrance

AU - Tipton, Elizabeth

AU - Hullman, Jessica

PY - 2023/4/19

Y1 - 2023/4/19

N2 - Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality and commensurability of evidence casts doubt on results from meta-analysis, yet existing software tools for meta-analysis do not provide an explicit software representation of these concerns. We present MetaExplorer, a prototype system for meta-analysis that we developed using iterative design with meta-analysis experts to provide a guided process for eliciting assessments of uncertainty and reasoning about how to incorporate them during statistical inference. Our qualitative evaluation of MetaExplorer with experienced meta-analysts shows that imposing a structured workflow both elevates the perceived importance of epistemic concerns and presents opportunities for tools to engage users in dialogue around goals and standards for evidence aggregation.

AB - Scientists often use meta-analysis to characterize the impact of an intervention on some outcome of interest across a body of literature. However, threats to the utility and validity of meta-analytic estimates arise when scientists average over potentially important variations in context like different research designs. Uncertainty about quality and commensurability of evidence casts doubt on results from meta-analysis, yet existing software tools for meta-analysis do not provide an explicit software representation of these concerns. We present MetaExplorer, a prototype system for meta-analysis that we developed using iterative design with meta-analysis experts to provide a guided process for eliciting assessments of uncertainty and reasoning about how to incorporate them during statistical inference. Our qualitative evaluation of MetaExplorer with experienced meta-analysts shows that imposing a structured workflow both elevates the perceived importance of epistemic concerns and presents opportunities for tools to engage users in dialogue around goals and standards for evidence aggregation.

KW - Meta-analysis

KW - epistemic uncertainty

KW - literature review

UR - https://www.scopus.com/pages/publications/85160023199

UR - https://www.scopus.com/inward/citedby.url?scp=85160023199&partnerID=8YFLogxK

U2 - 10.1145/3544548.3580869

DO - 10.1145/3544548.3580869

M3 - Conference contribution

AN - SCOPUS:85160023199

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - CHI 2023 - Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

PB - Association for Computing Machinery

Y2 - 23 April 2023 through 28 April 2023

ER -

MetaExplorer: Facilitating Reasoning with Epistemic Uncertainty in Meta-analysis

Abstract

Publication series

Conference

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this