Responsible metricsLibrary+ Research update - August 2021
At Maastricht University Library, we are committed to the responsible use of indicators.
We believe that quantitative indicators should support, not replace, qualitative evaluations. Analyses should be based on a careful selection of indicators and methods tailored to the particular goals of the assessment. Questions, not indicators, should drive research assessment; in other words: measuring what you want to know rather than measuring what you can measure.
Assessment and metrics
Along with developing the Recognition & Rewards programme of Maastricht University, various other developments are ongoing regarding research evaluation and assessment: signage of the San Francisco Declaration on Research Assessment (DORA), Leiden Manifesto, and the VSNU’s Strategy Evaluation Protocol 2021-2027 (SEP, PDF).
What all these initiatives have in common is:
- Greater flexibility in the aspects on assessing both scientists and research units;
- Use a narrative supported by bibliometric data instead of focusing on excellence (and thus competition) and metrics like the Journal Impact Factor (JIF) or the H-index.
You may wonder why these initiatives advise against the use of the JIF and the H-index. We will explain this briefly in this post. Don’t hesitate to contact the library’s Research Intelligence Team for more information via firstname.lastname@example.org or use the contact form below.
JIF and h-index
The Journal Impact Factor (JIF) was initially created as a tool to help librarians identify journals to purchase, not as a measure of the scientific quality of research in an article.
With that in mind, it is critical to understand that the JIF has several well-documented deficiencies as a tool for research assessment. These limitations include:
- Citation distributions within journals are highly skewed;
- The properties of the JIF are field-specific and therefore not comparable between fields;
- It is a composite of multiple, highly diverse article types, including primary research papers and reviews;
- JIFs can be manipulated (or “gamed”) by the editorial policy;
- Data used to calculate the JIFs are neither transparent nor openly available to the public.
Our colleagues from CWTS Leiden have nicely depicted the disadvantages of the H-index in the infographic below:
CWTS, Halt the h-index. Download the original PDF.
Category normalized citation index (CNCI)
Category normalised citation indexes (CNCIs) are article-based and not journal-based and can help overcome some of the biases inherent in the H-index.
In the CNCI, the number of citations is normalised against the average expected number of citations for publications of the same age, type and field of research (with 1.00 indicating that the actual number of citations equals the expected number of citations).
These indexes can be computed on the level of an individual publication (e.g., to support the narrative for a case study) as well as on the level of a research unit (e.g. to get an impression of scholarly impact over time or of the relative amount of publications that belong to the top-1% in their subject area in terms of citations).
Contact the author or a subject specialist
Use this webform for all your questions and remarks. The author of this post or a subject specialist will handle your message and respond or contact you within 1 business day.
Research Support Updates
Keep up-to-date with Research Support services and topics. Join our support mailing lists and receive updates of published news & events.