Which indicators to address in an assessment context
Which indicators to use and which aspects of one’s academic profile to highlight depends on the assessment context, one’s discipline (typical outputs and activities), one’s personal situation, or on the specifics of the research project for which a grant is applied. Although we consider all the indicators in this overview responsible, narrative CVs can differ in the indicators that are allowed. One has to pay close attention to these restrictions, and it is the sole responsibility of the scholar that the indicators used match with the requirements for a specific type of narrative CV.
The academic profile as the foundation for indicators: UM’s Current Research Information System (CRIS): Pure
Pure offers opportunities to register many research-related outputs, activities, awards, etc.
The following sub-categories are available under ‘activities’ in Pure:
- Talk or presentation – at conference
- Editorial activity, journal
- Organizing or contriuting to an event
- Peer review (publication)
- PhD (co-)supervision / assessment committee – internal promotion
- Membership of committee
- Consultancy / Advise external parties
- Membership of network
- Membership of board
- External collaboration in research projects
- Membership of council
- Editorial activity, book
- Attending an event
- Teaching to External parties / At external institution
- Performance, Talk or Presentation – not at conference
- Inaugural speech
- Farewell speech
- PhD assessment committee – external promotion
- PhD (co-)supervision – external promotion
- Advise to PR / Marketing internal
More information on Pure: https://library.maastrichtuniversity.nl/pure-support/
The main advantage of using Pure to register your accomplishments – from an assessment perspective – is that you can refer others to it without addressing individual entries. The Pure portal not only shows the kind and number of objects and activities registered under your name but also information on:
- the intended public of an item (academic / professional / popular)
- with whom you collaborate (internal and external)
- the status of a publication (refereed/non-refereed, OA/not OA)
- the number of times a publication has been cited in the Web of Science, or downloaded from Pure, and
- if a research output has generated online attention which can be further investigated through the Altmetric and PlumX links presented next to an output.
This additional information enables others to get a much quicker and more complete view of one’s academic profile, provided that Pure is used to its full.
For researcher themselves, the additional information in Pure enables one to look for publications with citations, downloads, or media attention that could be worthwhile to investigate further. Publications with high numbers of citations or attention do not necessarily reflect main contributions. However, the (kind of) sources, channels, institutes and people behind these numbers might provide additional context for a publication or to show how your research has been picked up in the academic realm or by other societal stakeholders or target groups.
Another advantage of Pure is that entries are validated by third persons (faculty Pure key-users) making Pure a ‘trusted’ source.
What to do with the information gathered from Pure and on my research outputs
‘Specify’ does not mean that one has to address these ‘specifies’, but they are examples of what one could specify. Which indicators to use and which aspects of one’s academic profile to highlight depends on the assessment context, one’s discipline (typical outputs and activities), one’s personal situation, or on the specifics of the research project for which a grant is applied.
Specify the intended audience(s) of your output(s) or activities
Specifying the intended audience for your output (educational or research related) may illustrate or indicate your (main) academic or societal focus as a researcher. As societal outreach and societal influence of research have become important elements in evaluations or assessments, specifying the intended audiences as an amount or percentage of your complete output per audience category, might serve as evidence of your focus or focusses.
In Pure, the intended audience (academic, professional, and popular) is specified for publications, news/media, and activities. The societal target groups or stakeholders are of course not limited to academics, professionals or the general public but may also encompass the commercial, the (non) governmental and the industrial sector. The industrial sector can be analysed by using InCites.
If one uses non-traditional communications channels to address different groups or stakeholders (frequently) one might mention these outreach activities as well, because they not only take a lot of time but also are also often undervalued. The same applies to activities such as organising events, media appearances and the like.
Specify the audience(s) who picked-up your research
The analyses on who is citing your research and where, including the demographic and geographic information gathered with Altmetric Explorer (link back) should provide you with good picture of who is interested in your research. However, describing the reasons why your research has been cited (citation context) or mentioned (sentiment), might cost more time.
Because societal relevance and the (re-)use of research results has gained such as an importance in research assessments, you may pay special attention to:
- citations or mentions in policy documents, patents, clinical guidelines, or systematic reviews
- mentions by corporate and non-profit organisations, patient associations, politicians, practitioners and professionals, or well-known institutes.
Specify the PIDs for your outputs
When listing research outputs it is often mandatory that you specify the persistent digital identifier (PID) for the output because it facilitates the discoverability and makes your research, (more) FAIR. For the researcher, PIDs are also the quickest way to search for a set of specific publications, datasets or software.
Records in Pure are enhanced with persistent identifiers like DOIs or ISBNs. If you need these you can copy them manually from the Pure-portal, or you can contact your faculty Pure- key-user for an Excel file. Preferably amended with direct links to other sources such as Altmetric Explorer, which you can use to provide a quicker access to the source. If persistent identifiers are missing in Pure or in your own records, you can use other bibliographical databases to see if they provide an identifier for your published output.
Another tool to discover the PID of a book (chapter), dataset, or article is Crossref, or Crossref’s free DOI lookup. If Crossref fails to turn up a DOI, go to the main page of that journal on the publisher website, copy the URL for that main page, and paste that URL following “Retrieved from” in place of the DOI as the final element in the citation. For example; Fink, C. K. (2013). The Cultivation of Virtue in Buddhist Ethics. Journal of Buddhist Ethics, 20. Retrieved from http://blogs.dickinson.edu/buddhistethics.
If there is no DOI discoverable for a print book, the location and publisher remain in their usual place at the end of the citation. For an e-book with no discoverable DOI, the final element in the citation may be a retrieval statement (e.g. ‘Retrieved from http://www.qualitynonfiction.com’) rather than either a DOI or a location and publisher.
There are more PIDs than DOIs and ISBNs. We most commonly see two varieties: those for objects (publications, data, software, such as URNs, DOIs, ARKs, Handle) and those for people (researchers, authors, contributors, such as ORCIDs, ISNIs). Many repositories will assign a PID of the former type when an object is deposited. If a platform does not issue DOIs, or has not yet issued a DOI for an object, use their ID’s to point to the places where they are published, such as PubMed IDs, ISBNs, e-ISBNs, Handles, arXiv IDs, ADS IDs, SSRN IDs, RePEC IDs, URNs or the ClinicalTrials.gov record number.
When citing software make sure to use the DOI that refers to the correct version.
 ISNI is the ISO certified global standard number for identifying the millions of contributors to creative works and those active in their distribution, including researchers, inventors, writers, artists, visual creators, performers, producers, publishers, aggregators, and more.
Identify ‘special’ research outputs or activities
Research outputs can be ‘special’ for many reasons. For example because many other researchers or institutes are using your research software, instruments, tests or methods. The alignment of your research outputs or activities to societal needs might be another reason. The reuse of data and research software from others can also be worth mentioning., or that the publication was the result of a replication study or a study to reproduce research results. Outputs that should be recognized and rewarded on an equal footing as innovative or impactful research (Hong Kong Principles).
Other outputs recognized under the Hong Kong principles are reviews and systematic reviews because they synthesize existing research or those that report negative results that normally take more effort to get them published. In line with the Hong Kong principles, one could also specify if the research was pre-registered or if reporting guidelines – to increase the transparency and reproducibility of your research- are used. Important values in the transition to Open Science and under the HKPs, as well as the use of data-management plans, and all the efforts to make data and research software as open as possible.
When addressing educational outputs like textbooks, presentations, etc. developed for your own institution or external third parties, also mention the efforts to enhance the accessibility of the outputs – not only for vision-impaired students, but also for students tired of screen reading or watching videos in crowded situations. Enhancing the accessibility makes it more inclusive.
Academic impact of journal articles
Academic impact of preprints
Academic impact of reports
Assessing the impact of books/chapters
Assessing the impact of datasets
Assessing the impact of dissertations
Assessing the impact of research software
Normalized citation indicators