DiVA

diva-portal.org

Digitala Vetenskapliga Arkivet

Driftstörningar
Det förekommer återkommande driftstörningar på några av sök-portalerna på grund av hög belastning. Vi arbetar på att lösa problemet, ni kan tillfälligt mötas av ett felmeddelande.
EnglishSvenskaNorsk
Ändra sökning
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science
Univ Ottawa, Canada; Inst Replicat, Canada.
Univ Ottawa, Canada; Inst Replicat, Canada.
Univ Cambridge, England.
Univ Ottawa, Canada; Inst Replicat, Canada.
Visa övriga samt affilieringar
2026 (Engelska)Ingår i: Proceedings of the National Academy of Sciences of the United States of America, ISSN 0027-8424, E-ISSN 1091-6490, Vol. 123, nr 22, artikel-id e2524747123Artikel i tidskrift (Refereegranskat) Published
Abstract [en]

Large Language Models (LLMs) such as ChatGPT are transforming how scientists conduct and validate research, offering promise as tools to improve scientific reproducibility. However, computational reproducibility and error detection remain expensive and labor-intensive. We experimentally test how collaboration between researchers and LLM assistants influences the reproduction of quantitative social science findings across different levels of AI autonomy. We randomly assigned 288 researchers to 103 teams working under three conditions: human-only, AI-assisted (using ChatGPT as a collaborative tool), or AI-led (ChatGPT operating with minimal human oversight). Teams reproduced published results from leading social science journals, detected coding errors, and proposed robustness checks. Human-only and AI-assisted teams achieved comparable reproduction rates (94% vs. 91%) and performed similarly on most outcomes, except human-only teams identified significantly more major coding errors. Both substantially outperformed AI-led teams, which achieved only a 37% reproduction rate, detected fewer errors across all categories, proposed weaker robustness checks, and required more time. This autonomous approach, however, likely represents only a lower bound of AI capabilities. Despite rapid model advances, expert human judgment currently remains indispensable for reliable empirical verification. While AI assistance did not degrade most outcomes, it provided no measurable advantages and was associated with reduced detection of major errors. However, the 37% autonomous reproduction rate indicates that AI could provide value in settings where scale or cost constraints preclude human review of papers, even though general-purpose LLMs offer no immediate advantages for human-supervised verification.
Ort, förlag, år, upplaga, sidor
NATL ACAD SCIENCES , 2026. Vol. 123, nr 22, artikel-id e2524747123
Nyckelord [en]
AI; reproducibility; large language models
Nationell ämneskategori
Datorgrafik och datorseende
Identifikatorer
URN: urn:nbn:se:liu:diva-226926DOI: 10.1073/pnas.2524747123ISI: 001821087300001Scopus ID: 2-s2.0-105040657951OAI: oai:DiVA.org:liu-226926DiVA, id: diva2:2094598
Anmärkning

Funding Agencies|Coefficient Giving project "Benchmarking LLM agents on real-world tasks: Reproducibility"; Alfred P. Sloan Foundation Foundation [G-2023-22326]; University of Toronto; University of Ottawa; University of Cornell; University of Tilburg; Leverhulme Early Career Research Fellowship [ECF-2022-761]; Australian Research Council [DP200102935]; Ministero dell'Universita e della Ricerca

Tillgänglig från: 2026-08-24 Skapad: 2026-08-24 Senast uppdaterad: 2026-10-04

Open Access i DiVA

fulltext(996 kB)3 nedladdningar
Filinformation
Filnamn FULLTEXT01.pdfFilstorlek 996 kBChecksumma SHA-512
ee3d4fb87d42b66d92624abe5f5b916bf6b1229d7726b6963ad70ee10d5c6a5fd33e32513dada2caf516d5038fd9acfba27aa867b9096fa65fc6559abb933195
Typ fulltextMimetyp application/pdf

Övriga länkar

Förlagets fulltextScopus

Sök vidare i DiVA

Av författaren/redaktören
Batinovic, Lucija
Av organisationen
Avdelningen för funktionsnedsättning och samhälleFilosofiska fakulteten
I samma tidskrift
Proceedings of the National Academy of Sciences of the United States of America
Datorgrafik och datorseende

Sök vidare utanför DiVA

GoogleGoogle Scholar
Antalet nedladdningar är summan av nedladdningar för alla fulltexter. Det kan inkludera t.ex tidigare versioner som nu inte längre är tillgängliga.

doi
urn-nbn

Altmetricpoäng

doi
urn-nbn
Totalt: 44 träffar
RefereraExporteraLänk till posten
Permanent länk

Direktlänk
Referera
Referensformat
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annat format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annat språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
v. 2.49.0
|
WCAG
|
Om DiVA portal
|
Om DiVA-konsortiet