In collaboration with Maarten Goossens (Training and Education Sciences, and co-founder of Comproved, UAntwerp), Karla Groen (E-campus, UAntwerp) and Philip Lambrechts (Department of Education, UAntwerp)
Lecturers often use criteria to assess student performance. Such assessment methods help them to gauge student learning in an objective manner. However, using a set of criteria to assess students isn't always self-evident. What if you're unsure of the exact meaning of the description of a criterion? And what if you estimate, based on years of experience, that a certain paper deserves about 12/20, but when you add up the marks for all the criteria you arrive at a total score of 15/20? Do you stick to the criteria and award a 15, or do you make some adjustments here and there so that the final score is 12? There are several caveats when it comes to the development and use of criteria. For example, it can be difficult to define the competence(s) to be measured. And can all aspects of a competence like 'cooperation' really be captured in a description? The pairwise comparison method provides opportunities to overcome these difficulties.
Pairwise comparison
In the pairwise comparison method, multiple assessors always choose between two objects. In education, this means that several teachers compare pairs of tasks performed by different students and indicate which of the two is better when it comes to a certain competence. On the basis of the choices made, a reliable ranking can then be made from best to worst. This method deliberately calls on people's expertise as assessors, without forcing them to look through a specific lens (e.g. a criteria list).
Potential use in education
In education, pairwise comparison is used to assess products from different contexts, including mathematical reasoning tasks, writing and design products, presentations, etc. An extra advantage of pairwise comparison is the potential openness of the tasks. After all, when developing assignments, you no longer have to ask yourself: 'How can this question be marked?' It doesn't matter whether someone's argumentation is right or wrong, because you can simply indicate which of the two is the strongest.
You can also use pairwise comparison as a teaching method in your teaching practice. For instance, you can present two examples and encourage your students to look for reasons why one is better than the other. This allows students to develop their own quality criteria in a simple way.
If you want to use pairwise comparison as an assessment method, you need support. One of the tools providing such support is Comproved.
Comproved: pros and cons
The assessment is completely anonymous, which is why students perceive Comproved as a safe environment.
Since comparisons are done holistically and each comparison usually takes no more than three minutes, the total time it takes for one assessor to compare twenty papers is about ninety minutes.
Like any tool, Comproved also has its limitations. It's less suitable for lengthy papers, and a minimum of four assessors is recommended.
You can find a practical example in the full Teaching Tip, which you can access by clicking the button below. It includes useful links to more information on the use of Comproved and more general sources on comparative assessment.