Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
P-hacking in academic research: a critical review of the job strain model and of the association between night work and breast cancer in women
Stockholm University, Faculty of Social Sciences, Department of Psychology.
2017 (English)Doctoral thesis, monograph (Other academic)
Abstract [en]

P-hacking can be described as a more or less deliberate, explorative approach to data analysis with a flexible/opportunistic search space and the reporting of primarily statistically significant findings. This leads to inflated type-1 error rates and to bias in reported estimates in the scientific literature.

This thesis aims to describe how p-hacking can be manifested in academic research and to illustrate how bias due to p-hacking is expected to affect the veracity of published findings using two specific examples from the literature. This thesis also argues that when evaluating published findings in the current academic environment, we should assume a priori that biases due to p-hacking and publication bias are present.

The thesis used Monte Carlo simulations and systematic reviews of the literature in two specific fields: the proposed associations between exposure to night work and breast cancer in women, and between job strain and coronary heart disease.

A general model and mathematical framework to predict expected bias from p-hacking was developed, and can be used for  a priori defined protected inferences of any published finding, under explicit assumptions of various levels of p-hacking. The model indicated a close to 100% chance of demonstrating a false positive association in larger studies, but also showed that even minimal p-hacking results in substantial bias in estimates.

The literature review identified large flexibility in the analytical process, allowing for the final model to be picked from a large pool of available models, with an implied search space of thousands of estimates. Some of the specific observations made here could be used to argue evidence for high risk of p-hacking and publication bias in the reviewed literature:

  • None of the 17 reviewed studies on job strain and coronary heart disease reported the proper estimate of the job strain interaction (chapter 6) and our analysis showed that the proper estimate would not have been statistically significant in any of the studies (chapter 7).
  • One study described a data driven approach with an implied search space of at least 502 models, where adjusting for confounding did not reduce the strength of the association, as would be expected, but instead increased its strength so it fell above the threshold for statistical significance (chapter 5).
  • One study was based on a speculative and marginally significant estimate after arbitrarily restricting the analysis to a subgroup, when estimates on the full group were available and indicated a non-significant association (chapter 5).
  • Statistical power analyses on research into night work and breast cancer indicated that statistically significant findings were over-represented in the literature (p≈.001) suggesting the presence of bias from p-hacking or selective publishing of significant findings (chapter 5).

The findings also suggest that previously reported estimates in meta-analyses was likely to represent prevailing bias in the two fields reviewed here. A bias-adjusted meta-analysis on the job strain model and coronary heart disease with a total of 462,220 subjects and 6,836 CHD events indicated no support for the job strain interaction (RR=1.00; 95% CI: 0.88--1.14). In addition, it did not show an increased risk due to high job demand (RR=1.03; 95% CI: 0.97--1.11) but it did confirm previously reported risks due to low job control (RR=1.11; 95% CI: 1.03--1.20).

Place, publisher, year, edition, pages
Stockholm: Department of Psychology, Stockholm University , 2017. , p. 197
National Category
Psychology (excluding Applied Psychology)
Research subject
Psychology
Identifiers
URN: urn:nbn:se:su:diva-141136ISBN: 978-91-7649-799-9 (print)ISBN: 978-91-7649-800-2 (electronic)OAI: oai:DiVA.org:su-141136DiVA, id: diva2:1086273
Public defence
2017-05-22, David Magnussonsalen (U31), Frescati Hagväg 8, Stockholm, 13:00 (English)
Opponent
Supervisors
Available from: 2017-04-26 Created: 2017-03-31 Last updated: 2017-06-01Bibliographically approved

Open Access in DiVA

P-hacking in academic research(8327 kB)411 downloads
File information
File name FULLTEXT01.pdfFile size 8327 kBChecksum SHA-512
5b69d9f60f6beef0d2f50ef1ba84b69c16b65d2282470940b8eb790d570bd095e678adfbefafd6772f303cbc916341055b3b8fd2a6a77b61d80d7f43f3bf0e2f
Type fulltextMimetype application/pdf
Errata P-hacking in academic research(187 kB)21 downloads
File information
File name ERRATA01.pdfFile size 187 kBChecksum SHA-512
b79a3b4c1ba51d01500c58ecd271f7f3e584f34cbcc03aacaf8564310f1005af3fc6070e587e52a9c9afd7ba7e59c0467087602bd56124bae8bbab14eca47e40
Type errataMimetype application/pdf

Search in DiVA

By author/editor
Ingre, Michael
By organisation
Department of Psychology
Psychology (excluding Applied Psychology)

Search outside of DiVA

GoogleGoogle Scholar
Total: 411 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 14224 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf