Research change is behaviour change

The last ten years have seen more and more interest in issues of research rigour and transparency – catalysed by what many described as a 'reproducibility crisis'. From early provocations by a small number of voices to systematic, empirical attempts to estimate the extent of the issue, these discussions are now mainstream. This is reflected in the recent Science Innovation and Technology Committee inquiry into reproducibility and research integrity, and the emphasis placed on 'People, Culture and Environment' in the plans for Research Excellence Framework 2029.

The conversation has moved from 'Is there a problem?,' through 'What is the scale of the problem?', to 'What can we do to improve things?'. This is perhaps the most exciting part. The level of innovation happening across the sector is remarkable. Not all of these innovations will persist, of course, or indeed even work as intended. That's in the nature of innovation. But some certainly will persist, and will work. So what are these innovations, and what can we do to maximise their potential to change things for the better?

A marker for excellence

One innovation that has gained a lot of prominence has been the Registered Reports format for journal articles. First described by psychologist Chris Chambers in an article in Cortex over ten years ago, it is now offered at over 350 journals (including all British Psychological Society journals). When authors submit a Registered Report to a journal they do so in two stages. In Stage 1, they submit their study protocol, effectively the introduction and methods section of what will become the final article. This is peer-reviewed in the usual way before the research begins, on the basis of the importance of the question and the rigour of the methods that will be used to answer it.

If the article passes peer review the journal offers in-principle acceptance, meaning they will publish the full paper once data collection and analysis is complete, irrespective of the nature of the findings. The Stage 2 submission, which is effectively the full article, is also reviewed, but reviewers cannot reject the article at this stage if, for example, the results are unexpected or seen as less 'interesting' than they might have been (but see Box). Changes from the original study plans are permitted if they are agreed with the editor as the need for them emerges. The format is intended to correct current incentives for p-hacking, and publishing eye-catching, positive results, and disincentives for publishing null results.

There is emerging evidence that Registered Reports are higher quality, and less subject to publication bias. However, uptake in absolute terms remains modest – although the growth in participating journals has been rapid, and covered a broad range of disciplines, it remains only a small fraction of the number of scholarly journals that exist. Moreover, the number of Registered Reports published in each journal is only a (generally small) proportion of the total number of articles published. Given the clear benefits of the format – now empirically demonstrated, why is this?

Part of the reason is, perhaps, again down to incentives. At present, authors elect to use the Registered Reports model. This creates a barrier for most busy academics (whilst the net workload involved in a Registered Report is probably not that different to a conventional article, more of it is at the beginning, which can feel like more work). For Registered Reports to really take off, we need to create incentives for their uptake. Exercises like REF 2029 could help with this; for example, if panels are given guidance that the Registered Reports format can be taken as a marker of excellence.

Control in communities

What else can we do? And, indeed, what else are we doing? The Peer Community In (PCI) initiative publishes peer reviews of preprints. The overarching is to create specific communities of researchers reviewing and recommending, for free, unpublished preprints in their field. A PCI community exists for Registered Reports (PCI-RR), and a number of journal have committed to publishing Registered Reports that receive positive reviews via PCI-RR. This inverts the typical process whereby work is submitted to a journal first, which then controls the process, and places control in the hands of communities of researchers – authors and reviewers.

Another approach is to link funder and journal peer review processes. The questions that a journal asks of a Stage 1 Registered Report – Is the question important? Is the methodology robust? – are not that different to those that a funder asks of a grant application. So is it possible to link the two review processes, so that at the end a researcher has funding for their work and a commitment from a journal to publish their results? Cancer Research UK and 12 journals from PLOS, Springer-Nature and Wiley are collaboration to pilot this Registered Reports Funding Partnership model.

It may be that these initiatives don't work, although early signs are promising – more and more journals are opting in to the PCI-RR framework, and opt in rates for the Cancer Research UK partnership are around 50 per cent. But we need to be thinking about how to create incentives to drive uptake of these new initiatives – using the tools and knowledge available to psychologists, such as the COM-B model for behaviour change, and the Centre for Open Science Theory of Change.

One reason why psychology has been at the forefront of discussions around research rigour and transparency is that research is conducted by people, who (however well they are trained) will bring their own cognitive biases to their work and will respond to incentives. In other words, the mechanisms that drive the problems that have been identified over the last 20 years are behavioural, so of course psychology would have something to say about that. We should apply the same expertise to ensuring the proposed solutions work.

See also Dr Pamela Jacobsen on 'Registered reports: what are they, and why should journal authors consider them?'

Do Registered Reports really make a difference?

What evidence is there that Registered Reports do actually ensure the publication of results that might otherwise be suppressed? Gilad Feldman and colleagues conducted a replication of an influential study in judgment and decision-making, eventually published in the International Review of Social Psychology. Unusually, in this case we have direct evidence that it would have struggled to be published as a regular article.

The article proceeded via the PCI-RR route. After getting a positive Stage 2 recommendation from PCI RR, Feldman and colleagues submitted it to JEP: General. At the time, JEP: General was a 'PCI RR-interested' journal – these are journals that monitor PCI-RR recommendations but make no commitment to accept them without further review (unlike 'PCI RR-friendly' journals). 

JEP: General reviewed it again (as is their right) but rejected it specifically and explicitly because of the mixed/null results. It's an example of how RRs and PCI-RR prevent good research from being corrupted or suppressed by publication bias – you can read it all from Feldman's point of view.

Marcus Munafò is a Professor of Biological Psychology and MRC Investigator, and Associate Pro Vice-Chancellor - Research Culture, at the University of Bristol. [email protected]