The study of psychology is facing a crisis. The Research Excellence Framework (the Ref) has led to a research culture which is holding back attempts to stabilize psychology in particular, and science in general. The Ref encourages universities to push for groundbreaking innovative, and exciting research in the form of 4* papers, but it does not reward the efforts of those who replicate studies.
The point of replicating a study is to test whether a statistically significant result will appear again if the experiment is repeated. Of course, a similar result may not appear – casting into questions the validity of the results from the first experiment.
Last year, the Open Science Collaboration attempted to replicate 100 studies from highly ranked psychological journalists. While 97% of the original studies had a statistically significant result, just 36% of the replications had the same outcome. Equally worrying: when an effect did appear, it was often much smaller than previously thought.
Recent data calls into question some widely influential findings in psychological science. These problems are not confined to psychology however – many findings published in scientific literature may actually be false.
Science is supposed to be self-correcting and reproducible is a cornerstone of the scientific method. Yet, we simply aren't invested in replicating findings. We all want to be good researchers and understand more about how the world works. So why are we so reluctant to check our conclusions are valid?
Because no incentive is provided by the system we carry out our research in. In the UK, the Ref ranks the published works of researchers according to their originality (how innovative is the research? ), significance (does it have practical or commercial importance? ), and rigour (is the research technically right? ). Outputs are then awarded one to four stars. 4* papers are considered world-leading. The cumulative total of 3* and 4* papers determines research funding allocation and has a knock-on effect on institutional position in league tables(排名表) and therefore attractiveness to students. Obviously, the more publications, the better.
Worrying, many academics admit to engaging in at least one questionable research practice in order to achieve publication. Examples of this include: coming up with a theory after data is collected, stopping collecting data when an effect appears in case it disappears later, or only reporting the significant effects from collected data. Others simply fabricate data-Dutch psychologist Diederik Stapel shockingly falsified data from more than 50 studies.
The Ref completely harms our efforts to produce a reliable body of knowledge. Why? The focus on originality – publications exploring new areas of research using new paradigms, and avoiding testing well-established theories – is the exact opposite of what science needs to be doing to solve the troubling replication crisis. According to Ref standards, replicating an already published piece of work is simply uninteresting.
With the next Ref submission just four years away, many researchers are effectively faced with a choice: be a good scientist, or be a successful academic who gets funding and a promotion.
