In this post you wil learn:
- What Massive Peer Assessment is
- Its implications for student feedback in online courses.
- How it solves the 2-Sigma problem in pedagogy
SUMMARY (for those in a hurry): The Massive Peer Assessment techniques and features used in massive online courses (Moocs) can dramatically increase the volume of high-quality instructional feedback, allowing students of cohorts of any size to learn as fast and effectively as they would with a one-to-one tutor.
Imagine a university professor, sitting by himself in a cozy café. In front of him lie three dozen student projects. He looks at them and sighs: It will take him all weekend to read, comment, and grade them.
It is hard work, but someone has to do it.
Imagine a different situation now. What if the professor were not faced with three dozen projects, but three hundred?
How about three thousand?
The “Human-Computer Interaction” Mooc at Stanford/Coursera.
This was the problem faced by the staff at Coursera (an online course platform) and Stanford University during 2013’s Human-Computer Interaction Mooc, a Massive Open Online Course. They had to assess and provide feedback on homework submissions from over 3,000 students, who submitted sketches and prototypes of designs that required the human eye to evaluate.
They solved the problem by outsourcing the students’ assessments to other students. All of the students, after submitting their own assignments, received specific instructions and rubrics(rules) on how to evaluate their peers’ work. After demonstrating with a quiz that they understood these basic rules, they were entrusted with grading 5 of their peer’s papers. The process was facilitated by Coursera’s online peer assessment feature.
How reliable was this solution?
The accuracy of this method was surprisingly high. After all the 3000 papers were graded, the professors at Stanford also evaluated many of them. They found that the difference between the average grade given by the 5 assessing peers and the grade given by an experienced professor was of only 3%. The fact that each student’s paper was graded by 5 people helped in reaching such high accuracy: the grades coming from students who over-graded and those who under-graded sort of balanced out (an example of “the Wisdom of the Crowd” effect)
Just 3% different from professors…
Not perfect, but still amazing if you ask me, especially since it was one of the first times, if not the first time, that peer assessment had been used a such a massive scale.
Additionally, in later iterations of the course, the reliability of the grading was increased with a technique called calibration, which works like this: One of the papers given to each student for assessment had been previously graded by a professor, so the paper’s “true grade” was known. If the grade given by the assessing student was too far away from this “true grade”, then the assessment produced by this student was considered unreliable and then discarded.
Ever since its use in the Human-Computer Interaction course at Stanford – Coursera, Massive Peer Assessment (MPA) has been used in hundreds of Moocs in several platforms.
I believe that the possibilities of Massive Peer Assessment both online and offline , are, well, massive.
You can call it “crowdsourcing of feedback and grading”
Or you can call it “assessment by the students, for the students”
I call it a killer application that solves one of the most important problems in pedagogy.
To see why this is the case, let’s first talk about student feedback.
The general importance of feedback
As we saw with the example, Massive Peer Assessment makes feedback more timely and a lot more abundant. These are exciting news for several reasons:
- First, feedback is crucial for reaching learning objectives. In the now classic “The 2-Sigma problem” paper, the educational psychologist Benjamin B. Bloom showed that one-on-one-tutoring using effective techniques can help any student in a cohort raise to the top 2% of her class. Yes, any student.
- Second, feedback is precious for increasing a student’s motivation. In another classic paper and series of books about “Flow Theory”, psychologist Mihaly Csikszentmihalyi (pronounced Chick-Zen-Me-Hi), really )established that immediate feedback is crucial for a person to concentrate fully on a task, a state of enjoyment also called being in the zone. So, constant feedback helps students keep engaged in their projects (additionally to providing the theoretical and technical corrections needed to keep the student on the right path)
- On the flipside, lack of feedback is dangerous. Research has found that ignoring a person’s efforts is almost as bad for motivation as destroying their work.
The current state of student feedback in many learning institutions
A lot of academic work done by students is sorely lacking feedback, which is unfortunate, yet understandable: It is indeed impossible for most educators to provide personalized, constant feedback for dozens or even hundreds of students; there just isnn’t enough time. In my personal experience, an arguably absurd phenomenon observed when I was a student was not receiving feedback after major educational milestones such as final exams and final projects: I often received just a grade, some quick notes at the margin, and hop! I was ready for the next academic year. Whatever the context, I think you will agree, dear reader, that in the majority of learning institutions in our day feedback for students is very scarce.
This is why I believe there are very juicy, low-hanging fruits in terms of student achievement and motivation and that Massive Peer Assessment has now made those fruits accessible.
Indeed, I believe educators should look into these techniques to obtain the kind of abundant, quality feedback that the people at Stanford did and thereby create learning experiences that approach ‘One-on-one tutoring’ in terms of achievement and motivation.
Benefits Beyond the higher quality feedback.
MPA might be precious for the student receiving it, but the same is true for the students generating it.
Indeed, assessing someone else’s work and explaining things to others are first-class learning experiences. Massive Peer assessment democratizes such experiences.
Coming soon:
Part 2: Why assessing peers is a world-class learning experience
In the meanwhile, you can read more about Coursera and Stanford’s use of MPA here
Rodrigo Serrano 