'We need less research, better research, and research done for the right reasons…'

We often don't stop to think about the system we work within, and whether it works (until it doesn't).

Over 30 years ago, writing in the BMJ, the late (and genuinely great) statistician Doug Altman said that 'We need less research, better research, and research done for the right reasons'. He was talking about medical research, but the challenge could be levelled at many more disciplines. Fast-forward to the present day, and I would say it remains a fair comment. In fact, the situation may even have changed for the worse.

There are ever-more journals, and even these are being swamped by a growing number of manuscripts (often generated by paper mills). The situation is so bad that journals (and increasingly even funders) are finding it increasingly difficult to find enough reviewers.

Let's examine the three parts of that quote in reverse order.

The 'right reasons'

What does 'research done for the right reasons' mean? There is a tension here – academic freedom means that we are free to pursue research that we personally consider to be important or interesting. 'Curiosity-led research' – that is an extraordinary privilege. But universities are also – in my view, at least! – meant to be in service of society. We are no longer the independent scholars of the 19th century. We are salaried employees of publicly funded organisations.

So we have a responsibility to keep one eye on whether our research will actually be valuable. That value can take many forms, of course – from advancing fundamental knowledge through to having a real-world impact. But just because we personally think something is interesting doesn't itself constitute a 'right reason'. If we cast a dispassionate eye over much of the research we see published, I'm sure we could all point to studies that do not add much to the sum of human knowledge.

As Editor-in-Chief of the journal Nicotine & Tobacco Research, I would receive perfectly well-conducted studies that showed, for example, an association between smoking and lung cancer or heart disease. This happened more than once! We don't need those studies – we know the answer already! What are these studies adding? The problem, of course, is that publishing is the currency of academia. But publishing merely for the sake of publishing is not doing research for the right reasons.

'Better research'

Even if we have identified a research question that, when answered, will genuinely add to the sum of human knowledge, we need to conduct a well-designed study to be able to answer that question confidently. That seems obvious. But in my view, many studies simply cannot answer the question they are ostensibly designed to answer.

What do I mean by that? Most of us still use Frequentist statistics and calculate p-values. And we are taught that we can't interpret null results because, in a Frequentist framework, you can't prove the null. That's true, but we can design our studies to exclude anything other than a trivial effect. We can specify a minimum effect size of interest – theoretically, biologically or clinically – and ensure our study sample size is large enough to detect it.

If we do this, and our confidence interval excludes the minimum effect size of interest, we can draw conclusions from our null results. We can say: 'Well, there might be an effect too small for this study to detect, but even if there is, it will be too small to be of interest'. We can answer our question – whichever way the results go! But as Kate Button showed in 2013, most studies are too small and underpowered to do this.

'Less research'

All of this makes it easy to address the last part of Altman's quote. We need to do 'less research' because we're too busy to waste our precious time! The sector has changed enormously in the last decade or two, and the pressures this has created are well known. Identifying genuinely important research questions and conducting a smaller number of well-conducted (e.g., larger!) studies is surely the strategy we should be adopting!

Dorothy Bishop has written about how difficult we find it to stop doing research – and in particular to stop pursuing blind alleys even after the writing has been on the wall for a long time. She describes the presentation of a convincingly null result at a scientific meeting. But the audience, instead of applauding the honest and straightforward finding that there is no link between the two, began offering explanations for why the result was null. Different approaches to sampling, measurement, statistical analysis and so on could all perhaps 'rescue' the underlying hypothesis.

As a researcher, I do understand this. We really want to discover things! We feel empathy for researchers who have turned up blank. But offering endless auxiliary hypotheses and invoking hidden moderators doesn't help anyone if the avenue is indeed a dead end. Sometimes, no further research is needed, and we should focus our efforts elsewhere.

As Dorothy says in her blog post (and asked in the session): 'What evidence would we need to convince us that there is really no association between musicality and handedness?' This particular literature began in 1922. Surely, if there was a real link, we'd have robust evidence for now. We perhaps fail to realise the opportunity cost of this research – the time that could be spent on other avenues – because as long as we can keep publishing our careers will flourish. But this is enormously wasteful if we are not just in the business of publishing for the sake of it, but actually interested in generating knowledge.

Unfortunately, there are plenty of examples of literature that have flourished and continued long after the writing was on the wall. My personal favourite example – one that I published on myself – is that of the relationship between variation in the serotonin transporter gene and anxiety- and depression-related traits. The first article to report on this, in 1996, was conducted in a few hundred individuals. Subsequent studies suggested a mechanism by reporting evidence that this variation was also associated with the response of the amygdala to threat.

The story became richer. Other studies suggested that this genetic variation could explain individual differences in response to early life adversity in terms of the subsequent risk of depression. It was a compelling story made up of many linked elements. I even taught it myself in individual differences lectures. But this whole edifice rested on the underlying finding – that this genetic variation was linked to individual differences in anxiety and depression (or the risk of corresponding clinical outcomes). Take that away, and the rest comes falling down.

And as Scott Alexander (a psychiatrist and blogger) writes: 'ALL. OF. THIS. IS. LIES.' Strong words. But it is now very clear – from large-scale studies of that specific genetic variant through to the genome-wide association studies of anxiety and depression that have included data on around one million (not a few hundred) participants – that there is no meaningful link between the serotonin transporter gene and these traits. The first finding was a false positive. So were the studies that looked at the amygdala and gene x environment interactions.

That's fine, of course. We will inevitably generate false positives in our research. The question is: Why didn't science self-correct? Why didn't we stop publishing on this topic? Depressingly, if you search for '5-HTTLPR' in PubMed (the abbreviation for the gene in question), you will still find publications emerging, albeit far fewer and in far more obscure journals. As early as 2005, there was a direction replication study three orders of magnitude larger than the original that was a convincing null result. But the bandwagon rolled on regardless. (Disclosure: I was an author on that 2005 paper, and stopped including the serotonin transporter gene in my individual differences lecture at that point – I moved it to my talks on reproducibility!)

All of these studies were conducted in good faith and were largely well done (although they were far too small to reliably detect effects of common genetic variants, based on what we now know). The problem is that there was no incentive to stop doing this kind of research. For a while, funders continue to fund it (which means the reviewers and panel members – our peers – liked it). Journals (ditto) continued to publish it.

The British Medical Journal – some time ago – banned the phrase 'Further research is needed' from the conclusion of manuscripts because they felt that it was a trivial thing to say. Surely further research will always be needed. But perhaps we should encourage the use of the phrase 'Further research is not needed' in those cases where it's clear that we've reached a dead end. We could save ourselves a lot of time and effort – and free up our capacity to pursue more fruitful avenues of enquiry – if we could be better at admitting when what seemed like a good idea has run its course.

You are 'the system'

So, the sheer volume of research being produced is causing problems. Many of us will have experienced the feeling of overwhelm created by how many peer review requests are landing in our inboxes. But it's a problem of our own collective making. Like hamsters on a treadmill, we're running to stand still. But we can get off if we want to – by doing less research, better research, and research done for the right reasons.

But who has the power to initiate that change?

Many of us can feel powerless in the face of an academic system and incentive structure that we feel simultaneously locked into but unable to change. Yes, we might want to do less – but better – research. But will that undermine our career chances? This might feel particularly true for early-career researchers

The first point I would make is that there is no system 'out there' that does things to us. We are the system. Academics – us – are the people who review papers and grants, sit on editorial boards and funding panels, making hiring and promotion decisions. So the ability to change the decision sits with us – collectively.

The second is that I think there are a lot of mistaken beliefs out there. I have seen plenty of early career researchers flourish by being focused and boundaried, working in a sustainable way, and doing high-quality work that allows them to demonstrate their skills and talent without burning themselves out. (Kate Button, whom I mentioned above, is one example of this – now a senior lecturer at the University of Bath.)

Third, and finally, there is a vocational flavour to an academic career that is valuable but needs to be handled carefully. Remember, work doesn't love you back. But also, when we reach the end of our career, we won't look back fondly on the volume of papers we published. We'll remember one or two highlights – key findings that contributed.

Let's all try to bear that in mind when we decide to focus our efforts and work in a way that means we still love our work.

Marcus Munafò is Deputy Vice-Chancellor and Provost at the University of Bath

Image: Dorothy Bishop has written about how difficult we find it to stop doing research – and in particular to stop pursuing blind alleys even after the writing has been on the wall for a long time.