Determining cognitive distance between publication portfolios of evaluators and evaluees in research evaluation: An exploration of informetric methods

Date: 9 February 2018

Venue: UAntwerp Campus Middelheim, A.143 - Auditorium Jan Fabre - Middelheimlaan 1 - 2020 Antwerpen (route: UAntwerpen, Campus Middelheim)

Time: 4:00 PM

PhD candidate: Jakaria Rahman

Principal investigator: Prof Tim Engels, Dr Raf Guns

Short description: PhD defence Jakaria Rahman (ECOOM) - Faculty of Social Sciences



Abstract

When an expert panel evaluates research groups in a discipline specific research evaluation, it is an open question how one can determine the extent to which the panel members are in a position to evaluate the research groups. The expertise of the panel members should be well-matched with the research groups to ensure the quality and trustworthiness of the evaluation. Panel members who are credible experts in the field are most likely to provide valuable, relevant recommendations and suggestions that should lead to improved research quality. Due to absence of methods to determine the cognitive distance between evaluators and evaluees, this doctoral research leads to the development of informetric methods for expert panel composition. This contributes to the literature by proposing six informetric approaches to measure the match between evaluators and evaluees in a discipline specific research evaluation using their publications as a representation of their expertise.

The study proposes three methods, namely the use of barycenters, of similarity-adapted publication vector and of weighted cosine similarity. This takes two levels of aggregation: similarity between Web of Science subject categories and between journals.  This considers the publication portfolio of the involved researchers to reflect the position of the research groups and panel members in cognitive space and, hence, to determine cognitive distance. Expressed in general terms, this measures cognitive distance between research groups and panel members based on how often they have published in the same or similar journals. Two levels of aggregation and three methods lead to the development of six approaches of the expert panel composition for the research evaluation exercises.

The propose approaches give the opportunity to assess the composition of the panel in terms of cognitive distance if one or more panel members are replaced and compare the relative contribution of each potential panel member to the panel fit as a whole, by observing the changes to the distance between the panel’s and the groups’. In addition, the approaches allow the panel composition authority to see in advance about the panel’s fit to the research groups that are going to be evaluated. Therefore, the concerned authority will have the opportunity to replace outliers among the panel members to make the panel fit well with the research groups to be evaluated.



Contact email: jakaria.rahman@uantwerpen.be

Link: https://www.ecoom.be/nl/