Skip to main content

Peer review with SWoRD

As I mentioned, I'm using SWoRD in my writing class for econ majors. SWoRD is a site that not only facilitates peer review, it allows for student grades to actually be determined by their classmates' reviews. For each assignment, the instructor creates both open-ended comment prompts and a numeric rubric (the SWoRD template requires a 1 to 7 scale, though you can sort of get around that by skipping some of the numbers). Students submit their papers to SWoRD and once the deadline has passed, papers are assigned to peer reviewers (minimum of three, maximum of six; the creators of SWoRD strongly recommend at least five reviews if the scores will be used for grading). Everything is anonymous, as each student creates a pseudonym within the system (you just have to make sure students don't put their names in the text of their files!). I can either assign specific reviewers or have the system automatically assign them randomly. After the reviews are completed, the authors have the opportunity to 'back evaluate' the open-ended comments, indicating how helpful the comments were, or weren't; this is done before the authors see the numeric scores assigned by reviewers so the back evaluation is based purely on the open-ended comments.

One of the coolest things about the SWoRD system is how it calculates grades. Students receive a grade both for reviewing and for writing. The reviewing grades are based half on 'consistency', which takes into account things like if a student just gives all high scores or all low scores, or scores that are really different from the other reviewers of the same papers, and half on the back evaluation 'helpfulness' scores. The writing grades are based on the numeric rubric scores from the reviewers but adjusted for the consistency of the reviewers so, for example, if a paper has four reviewers who give high scores and one reviewer who gives low scores, the low scores from that one reviewer will be given less weight. The instructor can also adjust how much weight is given to the reviewing and the writing scores for each assignment.
[Update: I was mistaken about the grading - see my follow-up post for clarification]

Part of the reason I agreed to do this pilot is that I have always had students do peer review for this course anyway. So I already have many of the comment prompts and rubrics created (though they need some revising for SWoRD) and the fixed costs of getting things set up in the system seemed like they would be rather low while the benefits are potentially huge. In particular, in the past, I had students simply swap papers with one other classmate, in class, and there has always been huge variance in the quality of feedback that students give/receive and I always felt bad for the students that did not get very good feedback. With SWoRD, each student gets feedback from several classmates so even if the comments from one or two are not that great, the combination should mean that they get something useful. I was also hoping that with the grading system, I might be able to focus much more on just giving students comments and not have to worry about grading as much. Next time, I'll talk about how that's working out...

Follow-up posts
Can SWoRD really replace instructor grading?
More about SWoRD reviewing
SWoRD follow-up

Also related: Other peer reviewing tools

Comments

  1. Thanks for the description. I've been playing around with peer review and it has proven to be a pain. I'll certainly check out SWoRD.

    FYI, here are a couple other peer review methods I've played around with, but I think SWoRD sounds better than these.

    No blind: Have students trade papers and sit in class and talk about the papers. Many times they just give compliments and talk about something else. I think it's hard to give good critical feedback face-to-face. I haven't had this peer review count towards the grade.

    Single blind: I have students submit their papers to a common course folder with their last name as the file name. I assign everyone a random "Peer Review ID" and give each reviewer the names of three papers to review. They submit their reviews to me, I remove their identifying info, and pass them along to the authors.

    Single blind, with peer review grading: Same as above, but I have the authors rank their peer reviews for helpfulness. Those reviewers ranked most helpful receive extra credit (and as everyone is reviewing multiple papers, there are multiple chances to be ranked "most helpful").

    One question I've run into when having the peer reviews graded, what do you do if a student drops the class? For instance, I have some students who peer reviewed a student's paper who has since dropped the class. They now won't have a chance to earn extra credit for helpful comments on his paper. At the same time, there are a couple people in the class who were supposed to receive comments from the dropper, but now have less feedback.

    The challenges of doing something "different"!

    ReplyDelete
  2. Thanks for sharing your approach Aaron! SWoRD would definitely help automate some of what you are already doing (there's also a peer reviewing tool within Turnitin if your campus uses that). I haven't had too many issues with students dropping but I also don't usually assign the first peer-reviewed assignment until a few weeks into the term. I also rotate the reviewers for each assignment so if someone were to withdraw (as long as it weren't in the middle of an assignment), I could just not include them in the rotation. I'll be posting more about SWoRD over the next couple weeks...

    ReplyDelete

Post a Comment

Comments that contribute to the discussion are always welcome! Please note that spammy comments whose only purpose seems to be to direct traffic to a commercial site will be deleted.

Popular posts from this blog

What are the costs?

I came across an interesting discussion about a 19-year-old intern who was fired from The Gazette in Colorado Springs for plagiarism. There appears to be some controversy over the fact that the editor publicly named the girl in a letter to readers (explaining and apologizing for the plagiarism), with some people saying that doing so was unduly harsh because this incident will now follow her for the rest of her career. I was intrigued by this discussion for two reasons - one, it seems pretty clear to me that this was not a case of ignorance (as I have often encountered with my own students who have no idea how to paraphrase or cite correctly) and two, putting aside the offense itself, I have often struggled with how to handle situations where there are long-term repercussions for a student, repercussions that lead the overall costs to be far higher than might seem warranted for the specific situation. As an example of the latter issue, I have occasionally taught seniors who need to p

What was your high school economics experience like?

As I mentioned in my last post , I am asking my Econ for Teachers students to reflect on their reading by responding to discussion prompts. It occurred to me that it wouldn't be a bad idea for me to share my thoughts on those issues here and see if anyone wants to chime in. For this week, the students were asked to read the California and national content standards , an article by Mark Schug and others about why social science teachers dread teaching economics and how to overcome the dread, an article by William Walstad on the importance of economics for understanding the world around us and making better personal decisions (with some evidence on the dismal state of economic literacy in this country), and another article by Walstad on the status of economic education in high schools (full citations below). The reflection prompt asks the students to then answer the following questions: What was your high school econ experience like? What do you remember most from that class? How do

When is an exam "too hard"?

By now, you may have heard about the biology professor at Louisiana State (Baton Rouge) who was removed from teaching an intro course where "more than 90 percent of the students... were failing or had dropped the class." The majority of the comments on the Inside Higher Ed story about it are supportive of the professor, particularly given that it seems like the administration did not even talk to her about the situation before acting. I tend to fall in the "there's got to be more to the story so I'll reserve judgment" camp but the story definitely struck a nerve with me, partly because I recently spent 30 minutes "debating" with a student about whether the last midterm was "too hard" and the whole conversation was super-frustrating. To give some background: I give three midterms and a cumulative final, plus have clicker points and Aplia assignments that make up about 20% of the final grade. I do not curve individual exams but will cu