Abstract
A core tradition of HCI lies in the experimental evaluation of the effects of techniques and interfaces to determine if they are useful for achieving their purpose. However, our individual analyses tend to stand alone, and study results rarely accrue in more precise estimates via meta-analysis: in a literature search, we found only 56 meta-analyses in HCI in the ACM Digital Library, 3 of which were published at CHI (often called the top HCI venue). Yet meta-analysis is the gold standard for demonstrating robust quantitative knowledge. We treat this as a user-centered design problem: the failure to accrue quantitative knowledge is not the users' (i.e. researchers') failure, but a failure to consider those users' needs when designing statistical practice. Using simulation, we compare hypothetical publication worlds following existing frequentist against Bayesian practice. We show that Bayesian analysis yields more precise effects with each new study, facilitating knowledge accrual without traditional meta-analyses. Bayesian practices also allow more principled conclusions from small-n studies of novel techniques. These advantages make Bayesian practices a likely better fit for the culture and incentives of the field. Instead of admonishing ourselves to spend resources on larger studies, we propose using tools that more appropriately analyze small studies and encourage knowledge accrual from one study to the next. We also believe Bayesian methods can be adopted from the bottom up without the need for new incentives for replication or meta-analysis. These techniques offer the potential for a more user-(i.e. researcher-) centered approach to statistical analysis in HCI.
Original language | English (US) |
---|---|
Title of host publication | CHI 2016 - Proceedings, 34th Annual CHI Conference on Human Factors in Computing Systems |
Publisher | Association for Computing Machinery |
Pages | 4521-4532 |
Number of pages | 12 |
ISBN (Electronic) | 9781450333627 |
DOIs | |
State | Published - May 7 2016 |
Event | 34th Annual Conference on Human Factors in Computing Systems, CHI 2016 - San Jose, United States Duration: May 7 2016 → May 12 2016 |
Publication series
Name | Conference on Human Factors in Computing Systems - Proceedings |
---|
Other
Other | 34th Annual Conference on Human Factors in Computing Systems, CHI 2016 |
---|---|
Country/Territory | United States |
City | San Jose |
Period | 5/7/16 → 5/12/16 |
Funding
Matthew Kay was supported by the Intel Science and Technology Center for Pervasive Computing (ISTC-PC). Gregory Nelson was supported by the National Science Foundation (NSF) Graduate Research Fellowship Program under Grant No. DGE-1256082 and under NSF grant SCH-1344613. Eric Hekler's work was supported, in part, by a grant from the Robert Wood Johnson Foundation (PI: Hekler, 71995).
Keywords
- Bayesian statistics
- Effect size
- Estimation
- Meta-analysis
- Replication
- Small studies
ASJC Scopus subject areas
- Software
- Human-Computer Interaction
- Computer Graphics and Computer-Aided Design