idw – Informationsdienst Wissenschaft
Nachrichten, Termine, Experten
Nachrichten, Termine, Experten
A large-scale, multi-analyst study on the objectivity of statistical analyses has now been published. Jan Landwehr, Professor of Marketing at Goethe University Frankfurt, says the findings highlight the critical role of collaboration and exchange across diverse research teams.
FRANKFURT. A new study published in Nature, “Estimating the Analytic Robustness of Social and Behavioural Sciences,” finds that scientific conclusions can shift dramatically depending on who conducts the analysis. The results come from a large-scale international collaboration led by Balázs Aczél and Barnabás Szászi (Eötvös Loránd University and Corvinus University), conducted as part of the Systematizing Confidence in Open Research and Evidence (SCORE) program. A team of 457 independent analysts from institutions around the world conducted 504 re-analyses of data from 100 previously published studies across the social and behavioral sciences. All analysts received the same dataset and the same key research question but were given freedom in how to conduct the analysis based on their informed judgment.
Over the past decade, the social and behavioral sciences have undergone substantial reforms aimed at making research more transparent, rigorous, and reliable. Preregistration, registered reports, replication studies, and checks of analytical reproducibility all seek to reduce the prevalence of chance findings and biased results. One important question, however, has received relatively little attention: to what extent do research findings depend on the specific way in which data are analyzed?
In standard scientific practice, a dataset is typically analyzed by a single researcher or research team, and the resulting publication presents the outcome of one particular analytical pathway. While peer review assesses methodological acceptability, it rarely reveals what results might have emerged under alternative, yet equally defensible, statistical decisions.
Yet empirical research involves numerous decision points: how data are cleaned, how variables are defined, which statistical models or software are used, and how results are interpreted. Together, these choices constitute what is known as analytic variability – the flexibility that can fundamentally influence final conclusions.
Key Findings
The study now published in Nature shows substantial variation in the outcomes of independent analyses of the same question using the same data across 100 studies. Although most re-analyses broadly supported the main claims of the original studies, effect sizes, statistical estimates, and levels of uncertainty often differed meaningfully. All analysts reached the same conclusion as the original authors in about one third of cases.
Importantly, these discrepancies were not due to a lack of expertise. Experienced researchers with strong statistical backgrounds were just as likely to arrive at divergent results as others. At the same time, observational studies proved less robust than experimental ones, suggesting that more complex data structures allow greater analytical flexibility – and thus greater uncertainty.
Prof. Dr. Jan Landwehr of Goethe University Frankfurt, who was involved in the study as an analyst, explains the findings: “Just as major decisions should not rest on a single study, they should not depend on a single data analysis either. Only when different, well-founded analytical approaches converge on a consistent pattern can a result be considered truly robust. In that sense, our study is also a call for stronger collaboration across research teams and for more intensive scientific exchange.”
Prof. Dr. Jan Landwehr, Chair of Market and Consumer Psychology, Goethe University Frankfurt. Tel. +49 (0)69 798-34631; landwehr@wiwi.uni-frankfurt.de
The study, published in Nature, is available here:
https://www.nature.com/articles/s41586-025-09844-9
Criteria of this press release:
Journalists
Economics / business administration
transregional, national
Research results
English
You can combine search terms with and, or and/or not, e.g. Philo not logy.
You can use brackets to separate combinations from each other, e.g. (Philo not logy) or (Psycho and logy).
Coherent groups of words will be located as complete phrases if you put them into quotation marks, e.g. “Federal Republic of Germany”.
You can also use the advanced search without entering search terms. It will then follow the criteria you have selected (e.g. country or subject area).
If you have not selected any criteria in a given category, the entire category will be searched (e.g. all subject areas or all countries).
