Challenges to Informed Peer Review Matching Algorithms
Journal of Engineering Education
American Society for Engineering Education
Peer review is a beneficial pedagogical tool. Despite the abundance of data
instructors often have about their students, most peer review matching is by
simple random assignment. In fall 2008, a study was conducted to investigate
the impact of an informed algorithmic assignment method, called Un-weighted
Overall Need (UON), in a course involving Model-Eliciting Activities
(MEAs). The algorithm showed no statistically significant impact on the
MEA Final Response scores. A study was then conducted to examine the
assumptions underlying the algorithm.
This research addressed the question: To what extent do the assumptions used
in making informed peer review matches (using the Un-weighted Overall Need
algorithim) for the peer review of solutions to Model-Eliciting Activities decay?
An expert rater evaluated the solutions of 147 teams’ responses to a particular
implementation of MEAs in a first-year engineering course at a large mid-west
research university. The evaluation was then used to analyze the UON
algorithm’s assumptions when compared to a randomly assigned control group.
Weak correlation was found in the five UON algorithm’s assumptions: students
complete assigned work, teaching assistants can grade MEAs accurately,
accurate feedback in peer review is perceived by the reviewed team as being
more helpful than inaccurate feedback, teaching assistant scores on the first
draft of an MEA can be used to accurately predict where teams will need
assistance on their second draft, and the error a peer review has in evaluating a
sample MEA solution is an accurate indicator of the error they will have while
subsequently evaluating a real team’s MEA solution.
Conducting informed peer review matching requires significant alignment
between evaluators and experts to minimize deviations from the algorithm’s
Verleger, M., Diefes-Dux, H., Ohland, M., Besterfield-Sacre, M., Brophy, S. (2010). “Challenges to Informed Peer Review Matching Algorithms”. Journal of Engineering Education, 99, 397-408.