Digitala Vetenskapliga Arkivet

Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
The reliability of replications: a study in computational reproductions
Organization and Program Planning, German Institute for Adult Education, Leibniz Center of Lifelong Learning, Bonn, Germany.
School of Politics and International Studies, University of Leeds, Leeds, United Kingdom.
Geschwister Scholl Institute, LMU Munich, Munich, Germany.
Department of Sociology, University of Maryland, College Park, United States.
Show others and affiliations
2025 (English)In: Royal Society Open Science, E-ISSN 2054-5703, Vol. 12, no 3, article id 241038Article in journal (Refereed) Published
Abstract [en]

This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.

Place, publisher, year, edition, pages
Royal Society, 2025. Vol. 12, no 3, article id 241038
Keywords [en]
computational reproduction, reliability, replications, social and behavioural sciences
National Category
Psychology (Excluding Applied Psychology)
Identifiers
URN: urn:nbn:se:umu:diva-237136DOI: 10.1098/rsos.241038Scopus ID: 2-s2.0-105001207223OAI: oai:DiVA.org:umu-237136DiVA, id: diva2:1952452
Funder
German Research Foundation (DFG), 464546557Available from: 2025-04-15 Created: 2025-04-15 Last updated: 2025-04-29Bibliographically approved

Open Access in DiVA

fulltext(1074 kB)17 downloads
File information
File name FULLTEXT01.pdfFile size 1074 kBChecksum SHA-512
57eec97caa746571d8f7457fdc70cb7ff6fefe650e7c8b647c395769a74fd6b9bf580b1962a9bf132a77af2ee9a986da86170fd81edbd8f099eb137b3647c50a
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Search in DiVA

By author/editor
Bohman, AndreaHjerm, Mikael
By organisation
Department of Sociology
In the same journal
Royal Society Open Science
Psychology (Excluding Applied Psychology)

Search outside of DiVA

GoogleGoogle Scholar
Total: 25 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 216 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf