:: Why most published research findings are false.
Apptech email (remove spam text)
The claim that 'most published research findings are false',
while demonstrably true, is so utterly contrary to intuition
and to what we think we know about research methods,
statistical analysis and more as to be rejected out of hand
This sounds like an April fools hoax or similar - it's not.
Most published research findings are false.
This subject, and the report that it is based on, is of
crucial importance to any engineers or scientists or others
who are interested in understanding how accurate or
believable the results of well conducted and apparently well
conducted research may be and how much they can be depended
on to reflect actual reality. It can be and has been shown
that the majority of claims and research results in even top
class peer reviewed journals are in fact incorrect.
In 2005 a seminal analysis with the title "Why most
published research findings are false" was published in
PLoS medicine ( PLoS Medicine (2:e272) by an
epidemiologist John P. A. Ioannidis (Department of Hygiene
and Epidemiology, University of Ioannina School of Medicine,
Ioannina, Greece, and Institute for Clinical Research and
Health Policy Studies, Department of Medicine, Tufts-New
England Medical Center, Tufts University School of Medicine,
Boston, Massachusetts). This paper (published as an "essay"
was essentially well received by the scientific community
and (AFAIAA) no major attempts have been made to refute it's
claims. I am aware of some subsequent papers which appear to
take exception to its completeness in some areas but what I
have seen seems more an attempt to join the band-wagon than
to destroy it.
I'll write a brief summary here.
This is necessarily a generalisation and to some extend
overdone - better that you get the point and read the report
than "feel safe".
The intent of the following points is well supported by the
original analysis and AFAIK no major claims have been made
to refute them. If you do research, read research papers,
depend on research results etc then you really want to look
into these results. They apply especially in the medical
research field for reasons commented on in ref 1 below,
would probably be equally or more true in softer* or soft*
science areas (cognitive, psychology, theological,
biological general) and still highly applicable if possibly
less severe in the hard science areas.
[[My rough definitions: Soft - human mind, behavioural,
mental etc. Softer biological and living systems. Hard:
Depend on core 'laws of physics'.]][[No denigration
intended - just trying to scope applicability]].
- Based both on actual analysis of results AND studies of
how results are arrived at MOST published research results
- Small studies are more liable to be false.
- Even small studies with excellent statistical support are
liable to be false.
- When many studies are done in a field the chances of false
results being produced grows until it becomes almost certain
that every major hypothesis is covered by reports claiming
- Journals tend to accept papers "going against the flow"
only when they make large and grand contrary claims.
- Better results are obtained by very large studies, by many
coordinated but independent studies of the same basic
premise and using the same premises and approaches.
- Studies which study another researcher's hypothesis are
more liable to be correct than those which study the
researchers own hypothesis.
- All the factors and more that one may suggest may cause
inaccuracies do, and more. Attributions of the effect of
perceived bias on results tends to often enough prove true.
Corollaries from the original report:
Corollary 1: The smaller the studies conducted in a
scientific field, the less likely the research findings are
to be true.
Corollary 2: The smaller the effect sizes in a scientific
field, the less likely the research findings are to be true.
Corollary 3: The greater the number and the lesser the
selection of tested relationships in a scientific field, the
less likely the research findings are to be true.
Corollary 4: The greater the flexibility in designs,
definitions, outcomes, and analytical modes in a scientific
field, the less likely the research findings are to be true.
Corollary 5: The greater the financial and other interests
and prejudices in a scientific field, the less likely the
research findings are to be true.
Corollary 6: The hotter a scientific field (with more
scientific teams involved), the less likely the research
findings are to be true.
1. While the paper (ref 2 below) is not too too complex
mathematically and is moderately easy to read, it's not as
clear to the layman as it could be. A good starting
commentary can be seen at
*** READ THIS COMMENT FIRST ***
2. The original paper is (gratifyingly) available for
free online under a creative commons licence.
Two versions at
Ioannidis has previously identified statistical and
problems based on high-throughput techniques such as
microarrays that can
lead to gene-disease predictions being no better than chance
(see the Dec.
20, 2004, issue of The Scientist). He has also followed the
fate of research
findings to quantify their falsification rate, demonstrating
example, that five of the six most cited epidemiological
studies since 1990
have already been refuted (JAMA, 294:218-28, 2005).
4. Supporting comments
"He has done systematic looks at the published literature
and empirically shown us what we know deep inside our
hearts," said Muin Khoury, director of the National Office
of Public Health Genomics at the U.S. Centers for Disease
Control and Prevention. "We need to pay more attention to
the replication of published scientific results."
5. PLoS medicine
PLoS Medicine is a peer-reviewed, international, open-access
journal publishing important original research and analysis
relevant to human health.
6. Various on this result
Wall Street Journal Sep 14, 2007
7. OK blogs thereon
... Sometimes it's OK for results to be wrong ...
... more & bigger studies is better ... [[But he
already says that ]]
9. Gargoyle sez
See also: www.piclist.com/techref/index.htm?key=why+most+published
You must be a member of the
piclist mailing list
(not only a www.piclist.com member) to post to the