Do Coursera staff really monitor the discussion boards? I am increasingly thinking that they do not; if they do, that is even worse, since no response of any kind has been provided for the people who have been complaining for weeks about abusive feedback. I'm not just talking about bland, unhelpful, vague, or inaccurate feedback, but instead about abusive language and mean-spiritedness of the worst kind. Here are a couple of examples people have complained about at the discussion board:
I believe you are either awfully young, typing for a parent who has no time to do it herself, or simply have received an inferior education. My guess is you lived in one of the Carolina’s where you neither spoke nor wrote a high quality of English. YOU CAN CHANGE THAT, if you work hard at it. If there were a zero to give, that would be your grade.
Well, I just have to say this. What the fuck? You completely force your arguments as if you were trying to fit a square into a circle. (Might this be another homosexual sign to you?)Then there are the one-word comments:
Ug.Of all the mean-spirited feedback I have seen reported, I would say the most bizarre and strangely cruel is this one:
No.
Terrible.
One, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen, fourteen, fifteen, sixteen, seventeen, eighteen, nineteen, twenty, twenty one, twenty two, twenty three, twenty four, twenty five, twenty six, twenty seven, twenty eight, twenty nine, thirty.Yes, this is because our comments are supposed to be 30 words long. The software does not police this (hence the abundant one-word and two-word comments: "good!" or "liked it!") - but the idea that someone would deliberately put in a comment like this to meet the word count shows that there are some serious problems with the feedback culture in the class. Even if it is just a tiny percentage of the feedback overall, Coursera is going to have to find a way to do something about this; you cannot mandate participation in a peer feedback system as a requirement for the grade/certificate while allowing this kind of thing to go on unchecked and unattended. We are "graded" on our participation, and that participation grade consists of completing the peer feedback assignment. Someone who submits a comment that reads "One, two, three, four, five, ..." gets a full participation grade. Of course, a GREAT solution would be simply to get rid of the grading entirely - but I don't think Coursera has any intention of doing that.
By far the biggest problem, though, is vague and/or inaccurate feedback… and that's a much harder problem to solve. It's much like the problem with the poor quality of the essays overall; yes, there are inappropriate essays (blank essays, essays only a few words long, plagiarized essays, even spam essays) that need to be flagged - but the larger problem is the bewildering number of essays that are of such poor quality that it gets very discouraging to spend time on them. Without some kind of additional instructional component to the class, I am just not convinced that this often unreliable and/or unhelpful anonymous peer feedback can really help people to improve their writing.
Of course, to get a sense of what is going on overall, Coursera would need to ask us how things are going - for Week 4 (most recent week completed), it appears that 2500 people turned in essays (compared to Week 1, when apparently 5000 people turned in essays), while there are maybe a hundred or so people (just a guess) who participate at the discussion boards. So, without gathering feedback from us week by week about our experience (self-assessment of our own writing, self-assessment of our improvement in writing, feedback about the feedback we are receiving, etc.), there's really no way to know what's going on overall. Yet Coursera is collecting no feedback of any kind, except for the chaotic comments at the discussion board and the grades assigned by the peer reviewers.
One discussion board thread proposes: "Peer Grading Exposed as Milgram Experiment." I have to admit, that made me laugh. But it's not a happy laugh. I really hope Coursera does something about this. Since they added a flagging system for the discussion board, maybe they will eventually do that for the inappropriate essays and feedback, too. It would also really be nice for there to some kind of communication about all this, as opposed to the outdated and stale content that currently appears on the homepage Announcements (the last announcement was made on August 14, ten days ago). I participate pretty regularly at the discussion boards (although, admittedly, less than I used to; it's not the most fun place to spend time), and it's been weeks since I saw a discussion board comment that came from a Coursera staff person or a member of the course's instructional staff. Is it a Milgram Experiment... or, shudder, Lord of the Flies...?
"Of course, a GREAT solution would be simply to get rid of the grading entirely..."
ReplyDeleteYes, please!
As far as I can tell the peer feedback system as well as the feedback from and to staff is broken. Interestingly if you Google Eric Rabkin, a common student complaint at University of Michigan is that his grading system is obscure to unfair. While I tend to feel that student reviews on the internet need to be taken with a truckload of salt, this type of comment crops up in almost every review. A significant part of the peer review debacle may fit on his shoulders rather than the Coursera model.
ReplyDeleteHaving been a member of an internet writing group, I am not surprised at the terrible quality of the feedback. My experience with any sort of internet feedback is platitudes, vagueness, or vitriol. At least in the peer review if one person says something nasty all their friends don't jump on board to agree--not that we don't see that on the discussion board. Without the check of direct social interaction, the internet seems to bring out the worst in people.
Even for the students who are trying their best to provide feedback, they are woefully unprepared. It is obvious from the quality of many of the essays that only a small percentage of the students have sufficient knowledge to provide quality feedback in either content or form. While my last two essays received good marks, (whatever that means) and almost no useful feedback, my Alice essay met the feedback of those trying but with insufficient skills. One student went on in a long apologetic statement/rant that it didn't enrich his/her reading so she was giving it a one and that she was sure I would get better all in strained and awkward English. I have functioned in a second language, and I know how difficult it is, but I was honestly shocked that both the non native speakers that week reached for the lowest grade when they met the language barrier. I would never have felt comfortable responding in Russian to a native speaker with that sort of response. I realize there has been more than a touch of inappropriate derogatory comments toward second language learners on the boards, and I expect I met the response to that in the feedback I received.
I agree with you on the quality of the essays. Only a very few of the essays I have reviewed were truly college work. It is hard to write thoughtful comments on an essay that resembles a semi-drunken midnight email to a friend or a list of facts on the back of a cereal box. I, of course, didn't put that in the feedback, but it didn't stop me from thinking it.
I'm not sure eliminating the grading would solve the problem. In the internet fiction world, there are no grades and the feedback is in general useless to occasionally nasty. Perhaps the hard fact is that scaling literature courses for a MOCC is not going to be feasible. We can all read the books and watch the videos, but that is not a literature class. Literature requires an active discussion and a writing component that enhances the reading and improves the students' writing, not that makes everyone furious.
Thanks for your comments, Celine and Natasha! About the grading, I was thinking that for the people who are motivated by the grade (and I cannot think of any other way to explain the plagiarism or the one-two-three-four comment), if all of a sudden there were no more grade, they would either leave the class OR do the sensible thing, which is to participate in the weeks that interest them and ignore the rest. If the only reason they are participating every single week is because of the grade, then maybe getting rid of the grade (which is clearly meaningless anyway) might eliminate at least that part of the bad "noise" in the class.
ReplyDeleteRe: the essays, I've been fortunate I guess you could call it in that the poor-quality essays I've read (and that's at least half of the essays I've seen) have had so many problems with grammar and sentence structure that I can just focus my comments there, which is quick and easy for me to do, and something I can do accurately. Is that feedback useful to the recipients? I would sincerely like to know. There was a thread at the discussion board about "how much time do you spend on feedback?" where people were talking about spending 20 minutes or more on each essay. Eegad. I'm not sure how people manage to spend that much time on them - I have had two really thought-provoking essays out of the 34 that I've read, a group of perfunctory essays not of real interest one way or the other, and a whole heap of error-ridden essays. Now, a lot of college students really struggle with writing, so I'm not going to say they are not college-level... but I also get the feeling that there is just not a critical mass of people with good writing skills in the class to tip the balance and make this odd system work, either on the essay-writing side or on the feedback side.
In my own classes, the feedback works GREAT, but that is because it is predicated on a real sense of community and person-to-person commitment. I use a Ning, which is such a fun social place to interact, and college students are, by and large, a pretty social bunch of people - they really thrive on this new way to interact with their classmates, which is so different from their other classes, either from their classroom-based classes or from their other online classes too (most of the online instructors rely on the course management system which is a dreadful way to interact, very klunky and impersonal, not fun in any way, shape or form). Also, I don't ask my students to comment on grammar and sentence structure (many of them are not really qualified to do that); instead, they are responding to each other's creative writing, so they are commenting on things like plot, character development, etc., along with webpage design and presentation, all things that they can feel confident about - no vitriolic arguments, no condescension. The biggest problem I have is getting people to be SPECIFIC in their comments, so I've prepared some examples of commenting that is lively and specific, but it's based on very personal engagement and a product totally different from in this class. I'm really not even sure how I could go about cultivating good essay writing or good essay feedback. But something surely needs to happen for this Coursera experiment to succeed. I wonder if we will see any changes in the next six weeks.
I think the real sense of community is the key, and Coursera is not structured that way. Maybe students could be in circles of ten or twenty to create a community within the large framework. They would only give feedback on each other, but this still requires students being at a similar level, not something that is occurring in this class. The essay format by its very nature is also less amenable to amateur feedback. With fiction a suitably motivated student can at least comment on plot or characters without special knowledge. Essay writing is a narrow and specific skill set requiring some training to comprehend. For example I can comment of my preference for one actor or another, but I cannot comment on their stagecraft. I know nothing of the theater beyond I liked that. Certainly we're seeing many of those sort of responses in the feedback.
ReplyDeleteWhat a great comparison re: stagecraft, Natasha - I'm imagining now a musical composition class where maybe we are supposed to compose a new minuet every week or something like that. No matter how much I might like listening to minuets, I certainly would not have the slightest idea how to compose one myself, and I would not be able to give feedback to someone else about their minuet, except to say "That sounded good!" or "I didn't really like that very much."
DeleteLaura, I've been genuinely dismayed by what I've seen on the forum regarding the peer reviews received. I'd already posted about the whole peer review process and why it is not working and cannot work as it currently is.
ReplyDeleteThere is language being used, things being said, that would simply not be tolerated in a classroom. Some of what's occurring within this course could get a student expelled from a traditional college but, thanks to anonymity, nothing is being done. The flag system is utterly useless, as far as I can tell.
So I am frustrated for my fellow-students and I was further frustrated because I didn't want to write more about the peer-review process. You've done it for me and I do want to write another post. If not today then tomorrow and I'll be linking to this and some other things as well.
As a change of pace, my next post will be "in praise of" because I need to shift my energy into what I like for at least a little bit.
Satia, I totally know what you mean about the negativity. Someone accused me at the discussion board yesterday of having a vendetta against Coursera. That is not true at all, but I just haven't seen much good going on. If I did, I would be writing about it. But I'm excited to write my Poe story and add it to my portfolio - and I'll get to do that tomorrow. Whoo-hoo!
Delete