File Download
  Links for fulltext
     (May Require Subscription)
Supplementary

Article: Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

TitleMany Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results
Authors
Issue Date2018
PublisherSAGE Publications. The Journal's web site is located at http://journals.sagepub.com/home/amp
Citation
Advances in Methods and Practices in Psychological Science, 2018, p. 251524591774764 How to Cite?
AbstractTwenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.
Persistent Identifierhttp://hdl.handle.net/10722/258977
ISSN

 

DC FieldValueLanguage
dc.contributor.authorSilberzahn, R-
dc.contributor.authorUhlmann, EL-
dc.contributor.authorMartin, DP-
dc.contributor.authorAnselmi, P-
dc.contributor.authorAust, F-
dc.contributor.authorAwtrey, E-
dc.contributor.authorBahník, Š-
dc.contributor.authorBai, F-
dc.contributor.authorBannard, C-
dc.contributor.authorBonnier, E-
dc.contributor.authorCarlsson, R-
dc.contributor.authorCheung, CKF-
dc.contributor.authorChristensen, G-
dc.contributor.authorClay, R-
dc.contributor.authorCraig, MA-
dc.contributor.authorDalla Rosa, A-
dc.contributor.authorDam, L-
dc.contributor.authorEvans, MH-
dc.contributor.authorFlores Cervantes, I-
dc.contributor.authorFong, N-
dc.contributor.authorGamez-Djokic, M-
dc.contributor.authorGlenz, A-
dc.contributor.authorGordon-McKeon, S-
dc.contributor.authorHeaton, TJ-
dc.contributor.authorHederos, K-
dc.contributor.authorHeene, M-
dc.contributor.authorHofelich Mohr, AJ-
dc.contributor.authorHögden, F-
dc.contributor.authorHui, K-
dc.contributor.authorJohannesson, M-
dc.contributor.authorKalodimos, J-
dc.contributor.authorKaszubowski, E-
dc.contributor.authorKennedy, DM-
dc.contributor.authorLei, R-
dc.contributor.authorLindsay, TA-
dc.contributor.authorLiverani, S-
dc.contributor.authorMadan, CR-
dc.contributor.authorMolden, D-
dc.contributor.authorMolleman, E-
dc.contributor.authorMorey, RD-
dc.contributor.authorMulder, LB-
dc.contributor.authorNijstad, BR-
dc.contributor.authorPope, NG-
dc.contributor.authorPope, B-
dc.contributor.authorPrenoveau, JM-
dc.contributor.authorRink, F-
dc.contributor.authorRobusto, E-
dc.contributor.authorRoderique, H-
dc.contributor.authorSandberg, A-
dc.contributor.authorSchlüter, E-
dc.contributor.authorSchönbrodt, FD-
dc.contributor.authorSherman, MF-
dc.contributor.authorSommer, SA-
dc.contributor.authorSotak, K-
dc.contributor.authorSpain, S-
dc.contributor.authorSpörlein, C-
dc.contributor.authorStafford, T-
dc.contributor.authorStefanutti, L-
dc.contributor.authorTauber, S-
dc.contributor.authorUllrich, J-
dc.contributor.authorVianello, M-
dc.contributor.authorWagenmakers, EJ-
dc.contributor.authorWitkowiak, M-
dc.contributor.authorYoon, S-
dc.contributor.authorNosek, BA-
dc.date.accessioned2018-09-03T03:59:29Z-
dc.date.available2018-09-03T03:59:29Z-
dc.date.issued2018-
dc.identifier.citationAdvances in Methods and Practices in Psychological Science, 2018, p. 251524591774764-
dc.identifier.issn2515-2459-
dc.identifier.urihttp://hdl.handle.net/10722/258977-
dc.description.abstractTwenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions. Crowdsourcing data analysis, a strategy in which numerous research teams are recruited to simultaneously investigate the same research question, makes transparent how defensible, yet subjective, analytic choices influence research results.-
dc.languageeng-
dc.publisherSAGE Publications. The Journal's web site is located at http://journals.sagepub.com/home/amp-
dc.relation.ispartofAdvances in Methods and Practices in Psychological Science-
dc.rightsAdvances in Methods and Practices in Psychological Science. Copyright © SAGE Publications.-
dc.rightsThis work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.-
dc.titleMany Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results-
dc.typeArticle-
dc.identifier.emailCheung, CKF: felixckc@hku.hk-
dc.identifier.authorityCheung, CKF=rp02275-
dc.description.naturepostprint-
dc.identifier.doi10.1177/2515245917747646-
dc.identifier.hkuros289249-
dc.identifier.spage251524591774764-
dc.identifier.epage251524591774764-
dc.publisher.placeUnited Kingdom and United States-

Export via OAI-PMH Interface in XML Formats


OR


Export to Other Non-XML Formats