Changes in the evaluation of proposals for ERC grants
It is part of the ERC’s core business to carefully select the best and most creative researchers from all fields in its grant competitions. In December 2022, the Scientific Council of the ERC announced plans for changes to the evaluation forms and processes that are used to decide which research proposals to fund. Excellence remains the sole selection criterion.
The Scientific Council continuously assesses the evaluation processes of the ERC and monitors the interests of the research community. In July 2021, the ERC formally endorsed the San Francisco Declaration on Research Assessment (DORA), and the use of Journal Impact Factors is not allowed in our evaluations. More recently, there has been more broadly expressed unease, especially among the younger community, with the way in which researchers are assessed, culminating in the Agreement on Reforming Research Assessment, which we have also signed.
We share the concerns that current practices often use inappropriate metrics and narrow methods to assess the quality, performance and impact of research and researchers. We set up a task force to look into the way the ERC currently assesses researchers and research proposals on the one hand, and at the concerns of the community and proposals on how to deal with them on the other. This included studying the solutions developed by other funders and institutions, but also soliciting direct input from a range of interested parties, including representatives of younger researchers’ organisations.
Following the suggestions from the task force, the Scientific Council agreed on changes that will be implemented in the 2024 Work Programme.
The Scientific Council reiterated the philosophy of the ERC that excellence is the sole criterion for the selection of proposals to be funded. We also established formally a principle that has already been used in practice, namely that the major weight of the evaluation is put on the project proposal, with the applicants being evaluated primarily on their excellence and likely ability to carry out the proposed work successfully.
The proposals will continue to be judged on their originality, creativity and ground-breaking ideas. For the evaluation of the researchers, we wanted to allow applicants to provide - and evaluation panels to see - a more holistic and fuller account of their research careers and contributions.
The proposals will continue to be judged on their originality, creativity and ground-breaking ideas. For the evaluation of the researchers, we wanted to allow applicants to provide - and evaluation panels to see - a more holistic and fuller account of their research careers and contributions.
We were aware of the need to ensure that early career researchers, researchers working in less prominent fields, or those from less well-known institutions would not be disadvantaged by any changes. The changes should support diversity and avoid favouring any particular types of research or research outputs over others. The career stage and personal context will continue to be taken into account to create a fair evaluation.
The scientific outputs will be limited to ten, and each one can be accompanied by a brief narrative on how the output has advanced knowledge in the field. This is a further turn-away from quantitative assessment towards content, and the narrative part should allow evaluators to understand the significance of the work, without necessarily having to read each item in detail.
Narrative elements in the track record may also provide a more comprehensive view of a researcher's career, contributions, and potential, and they can highlight important aspects of a researcher's work that may not be reflected by traditional metrics. In this edition of the ERC Magazine, ERC grantee Anja Leist shares her experience with narrative CVs as an applicant and as a reviewer, and considers the advantages and challenges of this approach.
The review panels are the backbone of the ERC’s proposal evaluation, and the panel members will be the ones in charge of implementing the changes. We are preparing guidance for our evaluators and for applicants to ensure that the purpose of the changes is fully understood.
The Scientific Council will monitor the effects of the changes and, if necessary, refine the procedure in the future in response to feedback from the applicants, evaluation panels and the scientific community..
There is always more than one way to achieve any aim, and we have invited James Wilsdon, Director of the Research on Research Institute (RoRI), to share his views on research assessment reform in this issue of the ERC Magazine.
Maria Leptin
President of the ERC's Scientific Council
Changes to the ERC’s evaluation processes are outlined in the recent press release accompanying the new Work Programme and in the 2024 work programme itself.
