The National Health and Medical Research Council (NHMRC) funds most of the medical and health research carried out in Australia and therefore pays the salaries of most of our medical and health researchers.
Recently, a decrease in the level of NHMRC funding has meant that fewer grant applications are being funded and this, together with an assessment system that allows for bias and the opportunity to influence outcomes of requests, is leading to demoralized and exhausted medical research. the workforce, according to Professor Tony Blakely of the University of Melbourne’s Melbourne School of Population and Global Health.
In the article below, originally published on The Conversation, Blakely discusses how the NHMRC could improve the way it makes grants to support our health and medical research workforce and improve the value that their work brings offers the Australian community.
Tony Blakely writes:
Most health research in Australia is funded by the National Health and Medical Research Council (NHMRC), which distributes around $800 million each year through competitive grant programs. An additional $650 million a year is funded through the Medical Research Future Fund, but this focuses more on large-scale “missions” than on researcher-initiated projects.
Ten years ago, around 20% of NHMRC funding applications were successful. Now, only 10-15% are approved.
Over the same ten-year period, NHMRC funding has remained stable while prices and population have increased. In inflation-adjusted and per capita terms, available NHMRC funding has fallen by 30%.
As a growing number of researchers compete for real funding from the NHMRC, research is at risk of becoming ‘a big economy’. To fix this, we need to spend more on research, and we need to spend it smarter.
Funding needs to be increased
To keep pace with other countries and to keep health research a viable career, Australia first needs to increase the total amount of research funding.
Between 2008 and 2010, Australia matched the OECD average of investing 2.2% of GDP in research and development. More recently, Australia’s spending has fallen to 1.8%, while the OECD average has risen to 2.7%.
When only one in ten applications is funded, there is a large element of chance in who is successful.
Think of it this way: apps are ranked in order of best to worst and then funded in order of top to bottom. If the ranking of a successful application is within five percentage points of the funding cut-off, it might have been lost if the assessment process was run again, because the process is always somewhat subjective and never will produce exactly the same results twice.
Thus, 5% of applications are “lucky” to get funding. When only 10% of applications receive funding, that means half of the applications were lucky. But if there’s more money to hand out and 20% of applicants are funded, the lucky 5% are only a quarter of selected applicants.
This is a simplistic explanation, but you can see that the lower the percentage of grants funded, the more of a lottery it is.
This growing element of “luck” is demoralizing to Australia’s research workforce, leading to academic burnout and brain drain.
The “application-centric” model.
As well as increasing total funding, we need to look at how the NHMRC allocates these valuable funds.
In the past five years, the NHMRC has moved to a system called ‘application-focused’ funding. Five (approximately) assessors are selected for each grant and asked to score the applications independently.
There are usually no panels for discussing and rating applications, which is what used to happen.
The advantages of application-focused evaluation include (hopefully) getting the best experts on a particular grant to evaluate it, and a less logistically difficult task for the NHMRC (convening panels is hard work and takes a lot of time).
Disadvantages of application-focused assessment
However, application-focused assessment has disadvantages.
First, reviewers’ reviews are not subject to any scrutiny. In a panel system, differences of opinion and errors can be managed through discussion.
Second, many appraisers will work in a “grey area.” If you are an expert in the area of a proposal and do not already work with applicants, you are likely to be competing with them for funding. This can lead to unconscious bias or even deliberate manipulation of scores.
And thirdly, there is simple “noise”. Imagine that each score given by an evaluator is made up of two components: the “true score” an application would receive in an unobservable gold standard evaluation, plus or minus some “noise” or random error. This noise is probably half or more of the actual variation between raters’ scores.
So how do we reduce the influence of both rater bias and simple “noise”?
First, raters’ scores must be “standardized” or “normalized.” This means rescaling the scores of all raters to have the same mean (standardization) or the same mean and standard deviation (normalization).
This is a no-brainer. You can use a fairly simple Excel model (I have) to show that this would reduce the noise substantially.
Second, the NHMRC could use other statistical tools to reduce both bias and noise.
One method would be to take the average ranking of applications in five methods:
- with the raw scores (ie as done now)
- with standardized scores
- with standardized scores
- by dropping the lowest score for each application
- by downloading the highest score for each application.
The last two “down-score” methods aim to remove the influence of potentially biased raters.
Applications that make the cut-off range in all methods are funded. Those who are always below the threshold are not funded.
Applications that pass some tests but fail others could be sent for further scrutiny, or the NHMRC could judge them by their average rank across the five methods.
This proposal will not solve the problem with the total amount of funding available, but it would make the system fairer and less open to gaming.
A fairer system
Researchers know that any funding system contains an element of chance. A study by Australian researchers found that they would be happy with a funding system that, if run twice in parallel, would see at least 75% of grants funded in both rounds.
I strongly suspect (and have modelled) that the current NHMRC system is achieving well below this 75% repeatability target.
Further improvements to the NHMRC system are possible and needed. Evaluators could provide feedback, as well as scores, to applicants. Better training of assessors would also help. And the most important interdisciplinary scholarship should be evaluated by panels.
No funding system will be perfect. And when financing rates are low, these imperfections stand out more. But right now, we’re not making the system as robust as we can or protecting ourselves enough from wayward scores that fly under the radar.
See Croakey’s archive of medical research stories here