As I mentioned, I'm using SWoRD in my writing class for econ majors. SWoRD is a site that not only facilitates peer review, it allows for student grades to actually be determined by their classmates' reviews. For each assignment, the instructor creates both open-ended comment prompts and a numeric rubric (the SWoRD template requires a 1 to 7 scale, though you can sort of get around that by skipping some of the numbers). Students submit their papers to SWoRD and once the deadline has passed, papers are assigned to peer reviewers (minimum of three, maximum of six; the creators of SWoRD strongly recommend at least five reviews if the scores will be used for grading). Everything is anonymous, as each student creates a pseudonym within the system (you just have to make sure students don't put their names in the text of their files!). I can either assign specific reviewers or have the system automatically assign them randomly. After the reviews are completed, the authors have the opportunity to 'back evaluate' the open-ended comments, indicating how helpful the comments were, or weren't; this is done before the authors see the numeric scores assigned by reviewers so the back evaluation is based purely on the open-ended comments.
One of the coolest things about the SWoRD system is how it calculates grades. Students receive a grade both for reviewing and for writing. The reviewing grades are based half on 'consistency', which takes into account things like if a student just gives all high scores or all low scores, or scores that are really different from the other reviewers of the same papers, and half on the back evaluation 'helpfulness' scores. The writing grades are based on the numeric rubric scores from the reviewers but adjusted for the consistency of the reviewers so, for example, if a paper has four reviewers who give high scores and one reviewer who gives low scores, the low scores from that one reviewer will be given less weight. The instructor can also adjust how much weight is given to the reviewing and the writing scores for each assignment.
[Update: I was mistaken about the grading - see my follow-up post for clarification]
Part of the reason I agreed to do this pilot is that I have always had students do peer review for this course anyway. So I already have many of the comment prompts and rubrics created (though they need some revising for SWoRD) and the fixed costs of getting things set up in the system seemed like they would be rather low while the benefits are potentially huge. In particular, in the past, I had students simply swap papers with one other classmate, in class, and there has always been huge variance in the quality of feedback that students give/receive and I always felt bad for the students that did not get very good feedback. With SWoRD, each student gets feedback from several classmates so even if the comments from one or two are not that great, the combination should mean that they get something useful. I was also hoping that with the grading system, I might be able to focus much more on just giving students comments and not have to worry about grading as much. Next time, I'll talk about how that's working out...
Follow-up posts
Can SWoRD really replace instructor grading?
More about SWoRD reviewing
SWoRD follow-up
Also related: Other peer reviewing tools
One of the coolest things about the SWoRD system is how it calculates grades. Students receive a grade both for reviewing and for writing. The reviewing grades are based half on 'consistency', which takes into account things like if a student just gives all high scores or all low scores, or scores that are really different from the other reviewers of the same papers, and half on the back evaluation 'helpfulness' scores. The writing grades are based on the numeric rubric scores from the reviewers but adjusted for the consistency of the reviewers so, for example, if a paper has four reviewers who give high scores and one reviewer who gives low scores, the low scores from that one reviewer will be given less weight. The instructor can also adjust how much weight is given to the reviewing and the writing scores for each assignment.
[Update: I was mistaken about the grading - see my follow-up post for clarification]
Part of the reason I agreed to do this pilot is that I have always had students do peer review for this course anyway. So I already have many of the comment prompts and rubrics created (though they need some revising for SWoRD) and the fixed costs of getting things set up in the system seemed like they would be rather low while the benefits are potentially huge. In particular, in the past, I had students simply swap papers with one other classmate, in class, and there has always been huge variance in the quality of feedback that students give/receive and I always felt bad for the students that did not get very good feedback. With SWoRD, each student gets feedback from several classmates so even if the comments from one or two are not that great, the combination should mean that they get something useful. I was also hoping that with the grading system, I might be able to focus much more on just giving students comments and not have to worry about grading as much. Next time, I'll talk about how that's working out...
Follow-up posts
Can SWoRD really replace instructor grading?
More about SWoRD reviewing
SWoRD follow-up
Also related: Other peer reviewing tools
Thanks for the description. I've been playing around with peer review and it has proven to be a pain. I'll certainly check out SWoRD.
ReplyDeleteFYI, here are a couple other peer review methods I've played around with, but I think SWoRD sounds better than these.
No blind: Have students trade papers and sit in class and talk about the papers. Many times they just give compliments and talk about something else. I think it's hard to give good critical feedback face-to-face. I haven't had this peer review count towards the grade.
Single blind: I have students submit their papers to a common course folder with their last name as the file name. I assign everyone a random "Peer Review ID" and give each reviewer the names of three papers to review. They submit their reviews to me, I remove their identifying info, and pass them along to the authors.
Single blind, with peer review grading: Same as above, but I have the authors rank their peer reviews for helpfulness. Those reviewers ranked most helpful receive extra credit (and as everyone is reviewing multiple papers, there are multiple chances to be ranked "most helpful").
One question I've run into when having the peer reviews graded, what do you do if a student drops the class? For instance, I have some students who peer reviewed a student's paper who has since dropped the class. They now won't have a chance to earn extra credit for helpful comments on his paper. At the same time, there are a couple people in the class who were supposed to receive comments from the dropper, but now have less feedback.
The challenges of doing something "different"!
Thanks for sharing your approach Aaron! SWoRD would definitely help automate some of what you are already doing (there's also a peer reviewing tool within Turnitin if your campus uses that). I haven't had too many issues with students dropping but I also don't usually assign the first peer-reviewed assignment until a few weeks into the term. I also rotate the reviewers for each assignment so if someone were to withdraw (as long as it weren't in the middle of an assignment), I could just not include them in the rotation. I'll be posting more about SWoRD over the next couple weeks...
ReplyDelete