Opinion: The journey to responsible and experimental research funding

There is a global drive for more responsible and inclusive cultures of research assessment. This often goes hand in hand with a shift towards more systematic experimentation with modes and methods of research funding.

On 23 June 2023, members of CoARA (the Coalition for Advancing Research Assessment) gathered for its General Assembly in the science community’s equivalent of a virtual public square. It was the first such meeting since the coalition was constituted six months earlier, and the diversity of participants reflected how fast it has grown–-with a tally of organisational signatories now standing at 600, and rising by dozens each week.

In my work with research funders around the world through the Research on Research Institute (RoRI), I see two big changes underway. CoARA is the most energetic manifestation of the first: the drive for more responsible and inclusive cultures of research assessment. This often goes hand in hand with the second: a shift towards more systematic experimentation with modes and methods of research funding.

Both shifts are motivated by similar concerns about tensions in conventional systems of peer review, allocation and evaluation. The perception and evidence that these are broken is mounting, as is the cost in terms of wasted resources, untapped ideas and potential, and lost or frustrated talent.

CoARA builds on more than a decade of grassroots efforts to tackle these problems. DORA (The Declaration on Research Assessment) recently marked its 10th anniversary with a series of workshops around the world. At a national level, we’ve seen a sharper focus on these agendas in light of ongoing or proposed reforms to assessment frameworks in Australia, Czech Republic, Italy, New Zealand, Sweden and the UK. China is also getting in on the act, with a renewed drive to break the influence of the “four-onlys” — shorthand for a reliance on “only papers, only titles, only education background, and only awards” to determine the funding and promotion of researchers.  

A new report on The Future of Research Evaluation by the InterAcademy Partnership, Global Young Academy and International Science Council, provides a snapshot of the reforms now visible across funding and assessment systems worldwide and concludes that “The imperative to rethink the way in which research individuals, institutions and outputs are evaluated is clear and urgent.”

The emergence of CoARA reflects an evolution of these debates towards implementing, sharing and scaling solutions. It goes beyond debates over measurement to encompass questions of how to create a healthy work culture, how to promote research integrity, how to move from closed to open scholarship, and how to embed principles of equality, diversity and inclusion across the research community.

Similar concerns lie behind the recent wave of experiments with novel modes of evaluation and allocation, such as partial randomisation and lottery mechanisms. Funders in the vanguard of this movement – such as the Volkswagen Foundation, Swiss National Science Foundation, and the Health Research Council of New Zealand – are now inspiring others, including the European Research Council, to adopt more critical and creative approaches, and to test, trial and scale alternative methods and modes of review, evaluation and allocation.

A new study by the consultancy Technopolis for UK Research and Innovation (UKRI) reviews 38 potential interventions by research funders “looking to optimise and innovate in award-making processes”, and summarises where there is reasonable evidence of effectiveness, and where – in the majority of cases – evidence gaps remain.

It’s the need to fill these and related gaps in our understanding of research systems and decision-making that is propelling the growth in meta-research, meta-science, and the science of science. This can only become more crucial as the ways in which research is funded, conducted, communicated and evaluated continue to evolve. New approaches to open peer review, next generation metrics and applications of AI and machine learning are just some of the opportunities and dilemmas that lie ahead.

Realising more of the potential of the US$ 2.5 trillion being invested annually in R&D worldwide will require […] more responsible assessment, richer experimentation and more sophisticated research on research.

Realising more of the potential of the US$ 2.5 trillion being invested annually in R&D worldwide will require all of these moves in concert – more responsible assessment, richer experimentation and more sophisticated research on research.

By turning the tools of research back on itself, we can create fairer, more inclusive and robust funding and assessment systems, better equipped to meet the challenges of the day. Recent moves by the ERC to reform its own assessment processes are welcome steps towards ensuring the excellence of how it funds, as well as what and who.

James Wilsdon is director of the Research on Research Institute and professor of research policy at University College London.