Many analysts, one data set: Making transparent how variations in analytic choices affect results

R. Silberzahn; E. L. Uhlmann; D. P. Martin; P. Anselmi; F. Aust; E. Awtrey; Bahník; F. Bai; C. Bannard; E. Bonnier; R. Carlsson; F. Cheung; G. Christensen; R. Clay; M. A. Craig; A. Dalla Rosa; L. Dam; M. H. Evans; I. Flores Cervantes; N. Fong; M. Gamez-Djokic; A. Glenz; S. Gordon-Mckeon; T. J. Heaton; K. Hederos; M. Heene; A. J. Hofelich Mohr; F. Högden; K. Hui; M. Johannesson; J. Kalodimos; E. Kaszubowski; D. M. Kennedy; R. Lei; T. A. Lindsay; S. Liverani; C. R. Madan; D. Molden; E. Molleman; R. D. Morey; L. B. Mulder; B. R. Nijstad; N. G. Pope; B. Pope; J. M. Prenoveau; F. Rink; E. Robusto; H. Roderique; A. Sandberg; E. Schlüter; F. D. Schönbrodt; M. F. Sherman; S. A. Sommer; K. Sotak; S. Spain; C. Spörlein; T. Stafford; L. Stefanutti; S. Tauber; J. Ullrich; M. Vianello; E. J. Wagenmakers; M. Witkowiak; S. Yoon; B. A. Nosek

doi:10.1177/2515245917747646

Many analysts, one data set: Making transparent how variations in analytic choices affect results

R. Silberzahn^*, E. L. Uhlmann^*, D. P. Martin^*, P. Anselmi, F. Aust, E. Awtrey, Bahník, F. Bai, C. Bannard, E. Bonnier, R. Carlsson, F. Cheung, G. Christensen, R. Clay, M. A. Craig, A. Dalla Rosa, L. Dam, M. H. Evans, I. Flores Cervantes, N. FongM. Gamez-Djokic, A. Glenz, S. Gordon-Mckeon, T. J. Heaton, K. Hederos, M. Heene, A. J. Hofelich Mohr, F. Högden, K. Hui, M. Johannesson, J. Kalodimos, E. Kaszubowski, D. M. Kennedy, R. Lei, T. A. Lindsay, S. Liverani, C. R. Madan, D. Molden, E. Molleman, R. D. Morey, L. B. Mulder, B. R. Nijstad, N. G. Pope, B. Pope, J. M. Prenoveau, F. Rink, E. Robusto, H. Roderique, A. Sandberg, E. Schlüter, F. D. Schönbrodt, M. F. Sherman, S. A. Sommer, K. Sotak, S. Spain, C. Spörlein, T. Stafford, L. Stefanutti, S. Tauber, J. Ullrich, M. Vianello, E. J. Wagenmakers, M. Witkowiak, S. Yoon, B. A. Nosek^*

^*Corresponding author for this work

Psychology

Research output: Contribution to journal › Article › peer-review

420 Scopus citations

Abstract

Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

Original language	English (US)
Pages (from-to)	337-356
Number of pages	20
Journal	Advances in Methods and Practices in Psychological Science
Volume	1
Issue number	3
DOIs	https://doi.org/10.1177/2515245917747646
State	Published - 2018

Funding

D. P. Martin was supported by the Institute of Education Sciences, U.S. Department of Education (Grant No. R305B090002). The contribution of R. D. Morey and E.-J. Wagenmakers was supported by a grant from the European Research Council (Grant No. 283876). M. Johannesson received funding from the Jan Wallander and Tom Hedelius Foundation (Grant No. P2015-0001:1), as well as from the Swedish Foundation for Humanities and Social Sciences (Grant No. NHS14-1719:1). S. Liverani was supported by a Leverhulme Trust Early Career Fellowship (Grant No. ECF-2011-576). C. R. Madan was supported by a Canadian Graduate Scholarship, Doctoral-level, from the Natural Sciences and Engineering Research Council of Canada (Grant No. CGSD2-426287-2012). T. Stafford was supported by a Leverhulme Trust Research Project Grant (Grant No. RPG2013-326).

Keywords

Crowdsourcing science
Data analysis
Open data
Open materials
Scientific transparency

ASJC Scopus subject areas

General Psychology

Access to Document

10.1177/2515245917747646

Cite this

Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Rosa, A. D., Dam, L., Evans, M. H., Cervantes, I. F., ... Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337-356. https://doi.org/10.1177/2515245917747646

@article{252f0b6c2d434572b41809cbbf1d92e6,

title = "Many analysts, one data set: Making transparent how variations in analytic choices affect results",

abstract = "Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts{\textquoteright} prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.",

keywords = "Crowdsourcing science, Data analysis, Open data, Open materials, Scientific transparency",

author = "R. Silberzahn and Uhlmann, {E. L.} and Martin, {D. P.} and P. Anselmi and F. Aust and E. Awtrey and Bahn{\'i}k and F. Bai and C. Bannard and E. Bonnier and R. Carlsson and F. Cheung and G. Christensen and R. Clay and Craig, {M. A.} and Rosa, {A. Dalla} and L. Dam and Evans, {M. H.} and Cervantes, {I. Flores} and N. Fong and M. Gamez-Djokic and A. Glenz and S. Gordon-Mckeon and Heaton, {T. J.} and K. Hederos and M. Heene and {Hofelich Mohr}, {A. J.} and F. H{\"o}gden and K. Hui and M. Johannesson and J. Kalodimos and E. Kaszubowski and Kennedy, {D. M.} and R. Lei and Lindsay, {T. A.} and S. Liverani and Madan, {C. R.} and D. Molden and E. Molleman and Morey, {R. D.} and Mulder, {L. B.} and Nijstad, {B. R.} and Pope, {N. G.} and B. Pope and Prenoveau, {J. M.} and F. Rink and E. Robusto and H. Roderique and A. Sandberg and E. Schl{\"u}ter and Sch{\"o}nbrodt, {F. D.} and Sherman, {M. F.} and Sommer, {S. A.} and K. Sotak and S. Spain and C. Sp{\"o}rlein and T. Stafford and L. Stefanutti and S. Tauber and J. Ullrich and M. Vianello and Wagenmakers, {E. J.} and M. Witkowiak and S. Yoon and Nosek, {B. A.}",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2018.",

year = "2018",

doi = "10.1177/2515245917747646",

language = "English (US)",

volume = "1",

pages = "337--356",

journal = "Advances in Methods and Practices in Psychological Science",

issn = "2515-2459",

publisher = "SAGE Publications Inc.",

number = "3",

}

Silberzahn, R, Uhlmann, EL, Martin, DP, Anselmi, P, Aust, F, Awtrey, E, Bahník, Bai, F, Bannard, C, Bonnier, E, Carlsson, R, Cheung, F, Christensen, G, Clay, R, Craig, MA, Rosa, AD, Dam, L, Evans, MH, Cervantes, IF, Fong, N, Gamez-Djokic, M, Glenz, A, Gordon-Mckeon, S, Heaton, TJ, Hederos, K, Heene, M, Hofelich Mohr, AJ, Högden, F, Hui, K, Johannesson, M, Kalodimos, J, Kaszubowski, E, Kennedy, DM, Lei, R, Lindsay, TA, Liverani, S, Madan, CR, Molden, D, Molleman, E, Morey, RD, Mulder, LB, Nijstad, BR, Pope, NG, Pope, B, Prenoveau, JM, Rink, F, Robusto, E, Roderique, H, Sandberg, A, Schlüter, E, Schönbrodt, FD, Sherman, MF, Sommer, SA, Sotak, K, Spain, S, Spörlein, C, Stafford, T, Stefanutti, L, Tauber, S, Ullrich, J, Vianello, M, Wagenmakers, EJ, Witkowiak, M, Yoon, S & Nosek, BA 2018, 'Many analysts, one data set: Making transparent how variations in analytic choices affect results', Advances in Methods and Practices in Psychological Science, vol. 1, no. 3, pp. 337-356. https://doi.org/10.1177/2515245917747646

TY - JOUR

T1 - Many analysts, one data set

T2 - Making transparent how variations in analytic choices affect results

AU - Silberzahn, R.

AU - Uhlmann, E. L.

AU - Martin, D. P.

AU - Anselmi, P.

AU - Aust, F.

AU - Awtrey, E.

AU - Bahník,

AU - Bai, F.

AU - Bannard, C.

AU - Bonnier, E.

AU - Carlsson, R.

AU - Cheung, F.

AU - Christensen, G.

AU - Clay, R.

AU - Craig, M. A.

AU - Rosa, A. Dalla

AU - Dam, L.

AU - Evans, M. H.

AU - Cervantes, I. Flores

AU - Fong, N.

AU - Gamez-Djokic, M.

AU - Glenz, A.

AU - Gordon-Mckeon, S.

AU - Heaton, T. J.

AU - Hederos, K.

AU - Heene, M.

AU - Hofelich Mohr, A. J.

AU - Högden, F.

AU - Hui, K.

AU - Johannesson, M.

AU - Kalodimos, J.

AU - Kaszubowski, E.

AU - Kennedy, D. M.

AU - Lei, R.

AU - Lindsay, T. A.

AU - Liverani, S.

AU - Madan, C. R.

AU - Molden, D.

AU - Molleman, E.

AU - Morey, R. D.

AU - Mulder, L. B.

AU - Nijstad, B. R.

AU - Pope, N. G.

AU - Pope, B.

AU - Prenoveau, J. M.

AU - Rink, F.

AU - Robusto, E.

AU - Roderique, H.

AU - Sandberg, A.

AU - Schlüter, E.

AU - Schönbrodt, F. D.

AU - Sherman, M. F.

AU - Sommer, S. A.

AU - Sotak, K.

AU - Spain, S.

AU - Spörlein, C.

AU - Stafford, T.

AU - Stefanutti, L.

AU - Tauber, S.

AU - Ullrich, J.

AU - Vianello, M.

AU - Wagenmakers, E. J.

AU - Witkowiak, M.

AU - Yoon, S.

AU - Nosek, B. A.

PY - 2018

Y1 - 2018

N2 - Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

AB - Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.

KW - Crowdsourcing science

KW - Data analysis

KW - Open data

KW - Open materials

KW - Scientific transparency

UR - http://www.scopus.com/inward/record.url?scp=85116837831&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85116837831&partnerID=8YFLogxK

U2 - 10.1177/2515245917747646

DO - 10.1177/2515245917747646

M3 - Article

AN - SCOPUS:85116837831

SN - 2515-2459

VL - 1

SP - 337

EP - 356

JO - Advances in Methods and Practices in Psychological Science

JF - Advances in Methods and Practices in Psychological Science

IS - 3

ER -

Many analysts, one data set: Making transparent how variations in analytic choices affect results

Abstract

Funding

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint