Searching \ for '[TECH]:: Why most published research findings are' in subject line. ()
Make payments with PayPal - it's fast, free and secure! Help us get a faster server
FAQ page: www.piclist.com/techref/index.htm?key=why+most+published
Search entire site for: ':: Why most published research findings are'.

Exact match. Not showing close matches.
PICList Thread
'[TECH]:: Why most published research findings are '
2008\07\20@225105 by Apptech

face
flavicon
face
The claim that 'most published research findings are false',
while demonstrably true, is so utterly contrary to intuition
and to what we think we know about research methods,
statistical analysis and more as to be rejected out of hand
by many.

This sounds like an April fools hoax or similar - it's not.

           Most published research findings are false.

This subject, and the report that it is based on, is of
crucial importance to any engineers or scientists or others
who are interested in understanding how accurate or
believable the results of well conducted and apparently well
conducted research may be and how much they can be depended
on to reflect actual reality. It can be and has been shown
that the majority of claims and research results in even top
class peer reviewed journals are in fact incorrect.

In 2005 a seminal analysis with the title "Why most
published research findings are false" was published in
PLoS medicine ( PLoS Medicine (2[8]:e272)  by an
epidemiologist John P. A. Ioannidis (Department of Hygiene
and Epidemiology, University of Ioannina School of Medicine,
Ioannina, Greece, and Institute for Clinical Research and
Health Policy Studies, Department of Medicine, Tufts-New
England Medical Center, Tufts University School of Medicine,
Boston, Massachusetts). This paper (published as an "essay"
was essentially well received by the scientific community
and (AFAIAA) no major attempts have been made to refute it's
claims. I am aware of some subsequent papers which appear to
take exception to its completeness in some areas but what I
have seen seems more an attempt to join the band-wagon than
to destroy it.
I'll write a brief summary here.
This is necessarily a generalisation and to some extend
overdone - better that you get the point and read the report
than "feel safe".

The intent of the following points is well supported by the
original analysis and AFAIK no major claims have been made
to refute them. If you do research, read research papers,
depend on research results etc then you really want to look
into these results. They apply especially in the medical
research field for reasons commented on in ref 1 below,
would probably be equally or more true in softer* or soft*
science areas (cognitive, psychology, theological,
biological general) and still highly applicable if possibly
less severe in the hard science areas.
[[My rough definitions: Soft - human mind, behavioural,
mental etc. Softer biological and living systems. Hard:
Depend on core 'laws of physics'.]][[No denigration
intended - just trying to scope applicability]].

- Based both on actual analysis of results AND studies of
how results are arrived at MOST published  research results
are false.
- Small studies are more liable to be false.
- Even small studies with excellent statistical support are
liable to be false.
- When many studies are done in a field the chances of false
results being produced grows until it becomes almost certain
that every major hypothesis is covered by reports claiming
it's truth.
- Journals tend to accept papers "going against the flow"
only when they make large and grand contrary claims.
- Better results are obtained by very large studies, by many
coordinated but independent studies of the same basic
premise and using the same premises and approaches.
- Studies which study another researcher's hypothesis are
more liable to be correct than those which study the
researchers own hypothesis.
- All the factors and more that one may suggest may cause
inaccuracies do, and more. Attributions of the effect of
perceived bias on results tends to often enough prove true.

Corollaries from the original report:

Corollary 1: The smaller the studies conducted in a
scientific field, the less likely the research findings are
to be true.
Corollary 2: The smaller the effect sizes in a scientific
field, the less likely the research findings are to be true.
Corollary 3: The greater the number and the lesser the
selection of tested relationships in a scientific field, the
less likely the research findings are to be true.
Corollary 4: The greater the flexibility in designs,
definitions, outcomes, and analytical modes in a scientific
field, the less likely the research findings are to be true.
Corollary 5: The greater the financial and other interests
and prejudices in a scientific field, the less likely the
research findings are to be true.
Corollary 6: The hotter a scientific field (with more
scientific teams involved), the less likely the research
findings are to be true.



1.    While the paper (ref 2 below) is not too too complex
mathematically and is moderately easy to read, it's not as
clear to the layman as it could be. A good starting
commentary can be seen at

*** READ THIS COMMENT FIRST ***

   http://www.marginalrevolution.com/marginalrevolution/2005/09/why_most_publis.html

2.    The original paper is (gratifyingly) available for
free online under a creative commons licence.
Two versions at

   Better formatting:
   http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020124&ct=1

       http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1182327

3.
http://sci.tech-archive.net/Archive/sci.bio.evolution/2005-09/msg00376.html

Ioannidis has previously identified statistical and
experimental design
problems based on high-throughput techniques such as
microarrays that can
lead to gene-disease predictions being no better than chance
(see the Dec.
20, 2004, issue of The Scientist). He has also followed the
fate of research
findings to quantify their falsification rate, demonstrating
recently, for
example, that five of the six most cited epidemiological
studies since 1990
have already been refuted (JAMA, 294:218-28, 2005).

4. Supporting comments

   http://online.wsj.com/public/article_print/SB118972683557627104.html

Quotes -

"He has done systematic looks at the published literature
and empirically shown us what we know deep inside our
hearts," said Muin Khoury, director of the National Office
of Public Health Genomics at the U.S. Centers for Disease
Control and Prevention. "We need to pay more attention to
the replication of published scientific results."

Quoting Ioannidis

5.    PLoS medicine
PLoS Medicine is a peer-reviewed, international, open-access
journal publishing important original research and analysis
relevant to human health.

       http://medicine.plosjournals.org/perlserv/?request=index-html&issn=1549-1676

6. Various on this result

Wall Street Journal Sep 14, 2007
   http://online.wsj.com/public/article_print/SB118972683557627104.html


7. OK blogs thereon

  http://universalacid.blogspot.com/2005/08/why-most-published-research-findings.html



8.    Wannabees
       ... Sometimes it's OK for results to be wrong ...
       ... more & bigger studies is better ... [[But he
already says that ]]

   http://www.eurekalert.org/pub_releases/2007-02/plos-imp022207.php

9.         Gargoyle sez

       http://www.google.co.nz/search?num=100&hl=en&newwindow=1&safe=off&q=%22JOHN+IOaNNiDIS%22+%22why+most+published+research+findings+are+false%22&meta=




       Russell McMahon


2008\07\21@025321 by cdb

flavicon
face
On Mon, 21 Jul 2008 14:50:15 +1200, Apptech wrote:

:: Corollary 4: The greater the flexibility in designs,
:: definitions, outcomes, and analytical modes in a scientific
:: field, the less likely the research findings are to be true.

well this seems plain logical to me. If you accept that there is no
such thing as absolute certainty, (Heidelberg was it?) then you can
only opt for ' that which seems most likely at the time given the
information available ' . Empirically this would also be seen to be
true, when one considers all the scientific 'facts' that have been
found, only for 8 months later someone else to discover they're
probably wrong, and in all that time a gazillion people around the
world have declared themselves, fitter/better etc.

:: Corollary 5: The greater the financial and other interests
:: and prejudices in a scientific field, the less likely the
:: research findings are to be true.

I think this comes about due to the way funding is provided,
especially for Universities.  Much research is done off the back of
other, so you only need, and error in the original research for it to
be continued through.

As 'them' in charge are more likely to allocate funds only for those
projects that are liklet to succeed or get big headlines, thereby
amassing more money thorugh government grants, I wouldn't be
surprised.

In fact, despite those who will leap in, I don't deride people who
think they may or have discovered things such as Perpetual Motion.

1. it depends on  the definition of PM,
2. every now and then some bright spark changes some of the laws of
maths and physics, so there is no guarantee that one day someone will
a 'have an Einstein' moment, and discover the apparently immutable
laws of physics aren't.

I'd be interest in seeing that paper about TMA's, until recently I
provided equipment and software for researches who use that method.

Colin

--
cdb, spam_OUTcolinTakeThisOuTspambtech-online.co.uk on 21/07/2008

Web presence: http://www.btech-online.co.uk  

Hosted by:  http://www.1and1.co.uk/?k_id=7988359







2008\07\21@032410 by Apptech

face
flavicon
face
> :: Corollary 4: The greater the flexibility in designs,
> :: definitions, outcomes, and analytical modes in a
> scientific
> :: field, the less likely the research findings are to be
> true.


> ...If you accept that there is no such thing as absolute
> certainty, (Heidelberg was it?)

If you pressed me, I'd say no. But who can tell for sure?

I think the point he is making is that increased freedoms in
asking the actual question leads to a greater chance that
the answers are wrong. ie where the method of study and the
definitions and accepted results that define success and
failure are tightly constrained the answers are more likely
to be correct than when the opposite is the case and each
researcher is free to establish their own guidelines.

In one of the other papers I cited (one of the two wannabees
I think) someone suggests that society accepts being fed
incorrect results if the subject is important enough (or was
that unimportant enough?). AFAIR it was 'important enough'
which seems the opposite of what I'd expect. In skimming I
may have missed his point.



       Russell

2008\07\21@051155 by Yair Mahalalel

flavicon
face
Contrary to intuition? We are dealing with ventures into the
qualitatively unknown, of the sort that often are also previously
unimaginable, and you never know when.

And you're applying to it the most flexible and inventive device we've
ever seen, able to generate dozens of hypotheses and ad-hoc models a
minute, and immediately rely and use any of them, if only to save its
life, or its research grants.

And you find it unintuitive they're mostly wrong?

In my line of work there are several methods used to minimize that
effect -

- Data analysis has to quantify all the sources of noise, background
 events and all other possible causes of measurement inaccuracy,
 including how they might combine forces to generate artifacts
 masquerading as new findings.

- The measurement data itself is often inaccessible to the researchers
 until the last moment of the analysis. It is kept in a "locked box"
 until all the software and procedures are in place and only after they
 have been satisfactorily demonstrated on simulation and control data,
 it is run on the real data, to prevent people from optimizing the code and
 trying to "improve their accuracy" after seeing potentially biasing
 results.

- In order to claim a discovery, the researchers must show that the
 chances that the outcome was a results of a statistical fluke is less
 that one in ten million.

- In large experiments the results usually spend another year or so
 undergoing internal scrutiny by other experiment members which are not
 part of the original team. This is usually the harshest step, where
 every impossible scenario is checked. Nobody likes wrong results
 published under his name.

- Peer reviews are another step. No one would like to see you fail more
 than the competition.

- Results of a single experiment are never accepted, no matter how
 convincing they look. That is why on the LHC ring there are two large
 experiments, identical in their requirements, and as different as
 possible in their technology and implementation. And still this is
 considered a sad state of affairs. Originally there was supposed to be
 the SSC with its different energy range, location (and therefore
 mentality), etc.

And we expect to be wrong. What is often perceived as the conservatism
of the scientific establishment has its basis in the exact opposite.
There wouldn't be any point to this job if not to break current theories
and establish new, truer ones. But the human mind being what it is (with
the only thing more susceptible to self-deception being groups of human
minds), exhilarating as any result might be it takes years from initial
publication until it is accepted (and the prizes granted), and only
then, after it has been throughly cross-checked, we might expect to be
wrong only half of the times.

Cheers,
Yair.

On Mon, Jul 21, 2008 at 02:50:15PM +1200, Apptech wrote:
{Quote hidden}

> --

2008\07\21@064822 by Yair Mahalalel

flavicon
face
On Mon, Jul 21, 2008 at 07:20:48PM +1200, Apptech wrote:
>
> In one of the other papers I cited (one of the two wannabees
> I think) someone suggests that society accepts being fed
> incorrect results if the subject is important enough (or was
> that unimportant enough?). AFAIR it was 'important enough'
> which seems the opposite of what I'd expect. In skimming I
> may have missed his point.

It appears to me that indeed society's tolerance to bovine solid waste
is strongly and positively correlated with the importance of the
subject. It is clearly visible even here on the list, where people are
quick to correct others' mistakes when it comes to e.g. component
parameters, but when presented with evidence that the troposphere might
be on the brink of thermal runaway, you'd be hard pressed to find
anyone who will bother to investigate in which direction it might
spiral.

I'd even say that rational discussion is deferred on practically all
topics which are very important - energy, pollution, economic and
social disparity, race and gender inequality, the exponential nature of
technological progress, radical life extension, ownership laws of
information (i.e. subsets of the natural numbers), etc. etc. On most of
these it is almost impossible to even find raw facts which are of decent
quality.

The reason is probably that even though democracy and liberalism have
been quite successful in the past couple of centuries, the basic
instincts of the common man are still those of a slave. We expect to be
lied to when at war, we expect to be lied to when interested parties are
involved (compare medical journals to mathematical, for example), etc.
Often we only discover something to be important by first noticing
the hand waving and verbal acrobatics. Often our first protest isn't in
defence of honesty - we resent when being lied to _needlessly_.

(Admittedly the middle east might be a vantage point that introduces
some bias in itself...)

Cheers,
Yair.

2008\07\21@065039 by Apptech

face
flavicon
face
> Contrary to intuition? ...

> And you find it unintuitive they're mostly wrong?

You don't know me overly well, do you ? :-).
(I have no problem with that, I'm about half a world away).

>> ... while demonstrably true, is so utterly contrary to
>> intuition
>> ... as to be rejected out of hand by many.

The wording was chosen with more care than may be obvious.
I'm trying to slip through mental filters here, not just on
list but also for one chosen BCCee. And in that case there
are several aspects being addressed at once. Important ones
that I want to try and get the subject's "foot in the door"
and not rejected out of hand without thought.

BUT the list was my main audience. It is very easy for
people to throw up their hands in disgust and walk away from
such grandiose and sweeping claims, so a certain amount of
preparation is required before just springing the subject on
the unwary. If I'd just provided a subject line and a web
link it would have taken me much less effort. Hopefully this
way maximises the audience.

Lessons learned from this may/should serve people a
lifetime.

____________

I did / do like your list of checks and balances.

A shame they are not (or, alas, cannot) be applied in a few
other 'sciences' and areas of endeavour that I won't even
name here for risk of starting a conflagration.


       Russell






2008\07\21@065042 by Apptech

face
flavicon
face
{Quote hidden}

It's the Higgs Bo ...
Er.
Um.
Maybe not.
In fact, definitely not.
Just another Boojum.
Oh well ...


       Russell
       (For the Higgs was a Boojum you see*).

__________

* For the perplexed

   http://en.wikipedia.org/wiki/Snark_(Lewis_Carroll)

And, perhaps, " ... threatened its life with a railway share
..." is explained here

       http://en.wikipedia.org/wiki/Railway_Mania







2008\07\21@172145 by piclist

flavicon
face
On Mon, 21 Jul 2008, cdb wrote:
> 1. it depends on  the definition of PM,
> 2. every now and then some bright spark changes some of the laws of
> maths and physics, so there is no guarantee that one day someone will
> a 'have an Einstein' moment, and discover the apparently immutable
> laws of physics aren't.

I am not suporised most papers are wrong.  Thats the nature of science,
throwing theories out there and letting them fight to the death. :-)

As far as overturning the laws of physics, for a long time now we have
just been adding more details on to the laws, not overturning them.

The laws of orbital motion are still good enough for Nasa to use in
plotting spacecraft.  Einstien didn't blow the old laws of gravity away..
the results of both theories are nearly identical.  And when the next one
comes along (quantum gravity maybe?) I fully expect it will give results
nearly exactly with Newton as well, just differing at the far end of the
decimal point.

Now.. it's all still just observation and theory, so yes, someone COULD
come up with a theory that turns something major upside down.  But looking
at the progression of science, it seems pretty unlikely.

PS. I would be thrilled and estatic to read that some major physical law
just got proven wrong, as would most every researcher.  I just don't see
it happening.  Kind of like my feelings on winning the lottery.  Would be
nice, but the chances are too low to waste money buying a ticket on. :-)

--
Ian Smith
http://www.ian.org

2008\07\22@205437 by Cedric Chang

flavicon
face
> ___________
>
> I did / do like your list of checks and balances.
>
> A shame they are not (or, alas, cannot) be applied in a few
> other 'sciences' and areas of endeavour that I won't even
> name here for risk of starting a conflagration.
>
>
>         Russell



Maybe we could mosey on over to the [OT] tag and discuss 'science  
areas' that are truly bogus, such as psychology and psychiatric  
medicine.
cc




2008\07\23@174047 by Tony Smith

flavicon
face
> > A shame they are not (or, alas, cannot) be applied in a few other
> > 'sciences' and areas of endeavour that I won't even name
> here for risk
> > of starting a conflagration.
> >
> >
> >         Russell
>
>
> Maybe we could mosey on over to the [OT] tag and discuss
> 'science areas' that are truly bogus, such as psychology and
> psychiatric medicine.
> cc


Is that you, Tom Cruise?

Tony

2008\07\24@093411 by Matthew Rhys-Roberts

flavicon
face
For further interest, an in relation to the more recent PIClist thread
regarding mobile phones and cancer, please see also
http://www.powerwatch.org.uk/science/bias.asp

Matt

Apptech wrote:
> The claim that 'most published research findings are false',
> while demonstrably true, is so utterly contrary to intuition
> and to what we think we know about research methods,
> statistical analysis and more as to be rejected out of hand
> by many.
>  
[snip]

2008\07\24@195159 by Cedric Chang

flavicon
face
I read this paper and the authors make some pretty amazing leaps of  
logic.  This does not mean they are wrong and I believe they are  
probably right ; still it makes me realize what a cat's bag of  
confusion most information packets are these days.  Probably for the  
past 10,000 years as well.
cc

{Quote hidden}

More... (looser matching)
- Last day of these posts
- In 2008 , 2009 only
- Today
- New search...