Digitala Vetenskapliga Arkivet

Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology
School of Agriculture Food and Ecosystem Sciences, University of Melbourne, Grattan Street, VIC, Parkville, Australia.
School of Historical and Philosophical Studies, University of Melbourne, Grattan Street, VIC, Parkville, Australia.
Department of Biology, Whitman College, 345 Boyer Ave, WA, Walla Walla, United States.
School of Biological, Earth & Environmental Sciences, University of New South Wales, NSW, Sydney, Australia.
Show others and affiliations
2025 (English)In: BMC Biology, E-ISSN 1741-7007, Vol. 23, no 1, article id 35Article in journal (Refereed) Published
Abstract [en]

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.

Place, publisher, year, edition, pages
BioMed Central (BMC), 2025. Vol. 23, no 1, article id 35
Keywords [en]
Analytical heterogeneity, Many-analyst, Metascience, Replication crisis, Reproducibility
National Category
Evolutionary Biology
Identifiers
URN: urn:nbn:se:umu:diva-236118DOI: 10.1186/s12915-024-02101-xISI: 001426577300001PubMedID: 39915771Scopus ID: 2-s2.0-85218290951OAI: oai:DiVA.org:umu-236118DiVA, id: diva2:1942984
Available from: 2025-03-07 Created: 2025-03-07 Last updated: 2025-04-24Bibliographically approved

Open Access in DiVA

fulltext(2676 kB)26 downloads
File information
File name FULLTEXT01.pdfFile size 2676 kBChecksum SHA-512
3b8ca0b5c1632f5060d1933c1a1b1539dbba77dcb0e90e59e4b64a988e68214cb5c87ab9fb68159e679c4e4501844878f6f3ddd7449b8d0f062e941e131c333d
Type fulltextMimetype application/pdf

Other links

Publisher's full textPubMedScopus

Search in DiVA

By author/editor
Malm, Lisa E.
By organisation
Department of Ecology and Environmental Sciences
In the same journal
BMC Biology
Evolutionary Biology

Search outside of DiVA

GoogleGoogle Scholar
Total: 26 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
pubmed
urn-nbn

Altmetric score

doi
pubmed
urn-nbn
Total: 404 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf