Challenges to Informed Peer Review Matching Algorithms

Document Type

Article

Journal/Book Title/Conference

Journal of Engineering Education

Volume

99

Issue

4

Publisher

American Society for Engineering Education

Publication Date

2010

First Page

397

Last Page

408

Abstract

BACKGROUND
Peer review is a beneficial pedagogical tool. Despite the abundance of data
instructors often have about their students, most peer review matching is by
simple random assignment. In fall 2008, a study was conducted to investigate
the impact of an informed algorithmic assignment method, called Un-weighted
Overall Need (UON), in a course involving Model-Eliciting Activities
(MEAs). The algorithm showed no statistically significant impact on the
MEA Final Response scores. A study was then conducted to examine the
assumptions underlying the algorithm.

PURPOSE (HYPOTHESIS)
This research addressed the question: To what extent do the assumptions used
in making informed peer review matches (using the Un-weighted Overall Need
algorithim) for the peer review of solutions to Model-Eliciting Activities decay?

DESIGN/METHOD
An expert rater evaluated the solutions of 147 teams’ responses to a particular
implementation of MEAs in a first-year engineering course at a large mid-west
research university. The evaluation was then used to analyze the UON
algorithm’s assumptions when compared to a randomly assigned control group.

RESULTS
Weak correlation was found in the five UON algorithm’s assumptions: students
complete assigned work, teaching assistants can grade MEAs accurately,
accurate feedback in peer review is perceived by the reviewed team as being
more helpful than inaccurate feedback, teaching assistant scores on the first
draft of an MEA can be used to accurately predict where teams will need
assistance on their second draft, and the error a peer review has in evaluating a
sample MEA solution is an accurate indicator of the error they will have while
subsequently evaluating a real team’s MEA solution.

CONCLUSIONS
Conducting informed peer review matching requires significant alignment
between evaluators and experts to minimize deviations from the algorithm’s
designed purpose.

Comments

Originally published by the American Society for Engineering Education (ASEE). Publisher's PDF available through remote link.

Link to Full Text

Share

COinS