Teaching Notes: Trying Peer Review

 So--I attempted my first peer reviews in my script analysis class today. I've taught the course for years, but this is the first time I've carved out time (by cutting out lots of assignments) for peer responses.

I'd love to claim I was acting in Best Pedagogical Interests, but really I instituted this as a response to losing TA support for the classes this semester. It's all me, facing a digital pile of 50 analyses that historically have taken 20 minutes per paper to do. I don't have that time or energy now, so I wanted to farm out some of the low-level responding to students.

Of course, I do want the process also to make students think more carefully about their writing and the plays themselves. 

How'd it go? Well, I haven't graded their response sheets yet, but I saw some good and some meh. 

The meh: lots of people didn't show up. There's always a deficit between enrollment in the class and those who turn in an analysis assignment. But today I saw also a deficit between those who turned in an assignment and those who showed up for the peer review.

The good: I successfully impressed on them the requirement to have brought in a printed-out copy of the assignment. I even got them to agree last Friday that anyone not having a printout copy could not participate in the peer review. Sure enough, everyone showed up with a printed out copy.

The meh: There was a good bit of blowing off. Lots of people finished so quickly that I gather they didn't do much actual reading and responding. This is apparently a common reaction in peer reviews: "great work!" and nothing else. One student asked if she could get some additional peer review, as her peer had not (by their own admission) read the play she had written about. 

The fix: I think I'm going to borrow a bit of ungrading tactics here and let people choose which grade level they want to do for the peer review. Perhaps I'll do something like the following:

0/25--You choose to skip the review process altogether

C (15/25)--You do a self-review based on a Q&A checklist I provide.

A (25/25)--You commit to reading all plays people have written about, and you commit to being in class to do a more in-depth response to their paper.

The fix, part II: I need to scaffold peer responding a bit more. I think a lot of the finish-quickly-say-it's-good factor has to do with the fact that people don't know how to respond. 

The fix, part III: perhaps I'll anonymize peer responding? It could be that some folk are just very nervous about responding to another person in person. It might be easier for them to respond to someone anonymously. I'm not exactly sure how to make that happen?

We'll see on Friday (with the revised analysis drafts) how yay or meh the process is at nipping in the bud some common pitfalls.

Clearing Out the Crap

 Whew! This is the second night in a row that I've nearly forgotten to blog here. Usually this means that I've been writing in some other venue. 

I read somewhere once that every writer has about a million words of crap in them that they have to get through before they start writing good stuff. I write this blog in part to clear out my own backlog of crap-words. 

Most days this blog helps get the gears of writing and thinking going. On some days--like yesterday and today--my work in other areas of my life has obliged me to write already. I wrote a sermon yesterday. I wrote several backgrounder bits for my PhD seminar today. My writing brain thus tells me I've done my exercises for the day.

But no--discipline demands that I write here as well.

I find myself reclaiming and re-preaching the gospel I once evangelized about constantly: writing is a mode of thinking. My students come to me mainly with the misconception that writing consists of the one-time report you make about what's already in your head. So much of the writing they've been expected to do has them regurgitate information they've been taught. Writing really does function as a report in those cases. Do you know the respiration cycle? Explain it to me in a paragraph. Such writing has its place.

But I find students run into a wall when they attempt that kind of writing for classes like mine that require them to create, structure, and communicate an argument or an analysis. They sit down to write what they assume already exists in their head--another report--and find instead they don't know what they think about it. The mental report they want simply to copy onto the Google Doc or Word Doc? It doesn't exist. 

These students (and, to be fair, many students already know or sort-of know this already) miss the idea that writing functions as a way of finding out what we think. It's a tool for thinking, not reporting. 

I had drifted away from this truth in my teaching. Sure, I would tell students they need to take lots of notes, do a lot of pre-writing, before attempting an analysis. But few of them actually seemed to take that advice. Students lead busy lives. They don't necessarily see the benefits of doing writing as thinking. After all, writing takes energy and time. It's exhausting, like calisthenics. And, as in calisthenics, writing seems so much harder for students who have avoided it. Their writerly muscles lack the strength and stamina for long runs of productive writing. 

I recognize that, in my classes now, I'm asking them to start doing pushups when they really dislike pushups. 

Nevertheless, I persist. Why? Generative AI. Among my many gripes--some curmudgeonly, some more well founded, I fixate currently on the fact that Generative AI/LLMs sell themselves as an end-run around the hard work of writing. Why do all those pushups, lift all those weights, train in all those HIIT cardio rounds, when you can just ask Gemini or Claude to produce the report you'd eventually have gotten to? But the end-run around the work of writing also misses the benefits of thinking. As sci-fi author Ted Chiang argues, having AI write for you is like bringing a forklift to the gym. Sure, the weights get lifted, and--hurrah--you feel no aches, but your muscles don't develop. You miss the point of a gym in the first place. 

I've explained this to students. Some get it. Some nod and seem to at least consider it. And others, I know, will simply decide that they would prefer not to do that exercise, thank you very much. 

Anyway. No great writing or thinking tonight, I fear. But at least there's some crap cleared out for future thinking.

Class Prep: Lots of Low-Stakes Writing

 So, after finally letting go of the need to absorb any more play scripts--it really is just procrastination at this stage--I sat down and reformatted my script analysis course (both sections). I've made the class both shorter and longer. I took out some of the play readings; there are now 11 instead of the usual 13. And I took out one of the four analysis papers. With this additional breathing room, I inserted at least a day between each play I assign. My thinking is that we'll discuss the play one day and then, on the following day, we'll use that script to unpack/explore some facet of script analysis. 

I've also both cut down and vastly expanded the amount of writing students do. A class I taught this last semester turned out to be one of my favorite courses I've ever taught. I structured the class around lots of low-stakes writing and responses. Students brought in "sparks" for the day's material. A spark could be a quote that stood out, a substantive reaction, a question--just about anything so long as it demonstrated a real engagement with the material. Occasionally, students were required to go beyond a spark and bring in a "rabbit hole report"--a bit of research about something the material for the day inspired them to look into. A few students (those contracting for an A) had to do at least one "lagniappe report"--essentially a longer, deeper rabbit hole report on some bit of extra material not formally assigned. 

What made all this work so well, I think, was the work I put in compiling each class day a list of some of the standout sparks comments/questions. I'd begin each next class by sharing these. I could see students light up when they recognized that I was reading their contribution. I think this practice gave students a sense that their reactions mattered, that the sparks weren't mere busywork. 

It was, of course, a lot of work on my part.

What I was doing, in the lingo of writing instruction, was creating a lot of low-stakes writing assignments. Students would get either "Satisfactory" or "Unsatisfactory" on these assignments. If they were Unsatisfactory, I'd let them know what the problem was and give them the chance to revise. They had to be in class to turn them in, and I accepted handwritten ones provided I could understand the handwriting. Nearly all of them were handwritten, which frankly cut down on any suspicions on my part that they were AI-generated. I also gave students a few minutes at the top of class to do these exercises. If the didn't read the material, they'd get a U and the chance to revise. (Revisions were more work, though, so as to dissuade them from just backlogging U's.)

What if, I thought to myself, I made my script analysis class more like that class? Every day would have some kind of low-stakes writing--some they could bring to class prewritten, some they'd write in class. And I could--would have to--again do the hard work of grading every day and compiling responses to share the next day. I'd also need to get students into the habit of bringing something to write with and write on every day. 

I'm still thinking through exactly how to realize this in a script analysis class. A lot of this class is going to be experimental--on my part particularly. And it's going to be a lot of work--though I may be able to grade again mostly on S/U status. 

But: I'm also thinking I may need to let go of contract/ungrading in favor of the simpler (less pedagogically responsible) numerical grading system...

And then, aside from that, I really need to get my grad class sorted! Lots of article-reading hours tomorrow. 

The Simulcraprum

 Timothy Burke, one of my favorite scholars to follow, posted a doozy of a Substack today. There he looks ahead at the future of AI in pedagogical settings. It's not a sunny vision:

My worries are many, but one of the most prepossessing concerns I have, perhaps outweighing the dark political clouds on the horizon, are the consequences of the unconstrained, unmediated, unconsidered release of generative AI tools and components into our informational and expressive environment. Whatever happens to our governments and institutions in the episodic to-and-fro of the next decade may turn out to be less consequential by far than the conjuncture of our informational and cultural infrastructures with generative AI.

Burke explains that his concern isn't so much with students' pernicious use of AI to cheat. That problem, he notes, is not new, nor is it entirely the fault of students themselves. "The use of writing as a proof that a student did the reading has always been a baleful mistake," he argues. "The idea that a large quantity of writing would by itself secure mastery of writing expression equally so. At the university level, combine that with introductory courses that have hundreds of students listening to lectures and then meeting with teaching assistants and you have a recipe that has always encouraged cheating, that has always been bedeviled by forms of indetectable dishonesty."

Ouch--but accurate. I assign a lot of writing in my script analysis class, and it bears rethinking why I do so. I try not to have my writing assignments perform did-you-read-this police functions. (That's what short daily quizzes are for--policing reading and policing attendance--a practice I'm uneasily resigned to for now.) I'd hoped to move my script analysis class more fully into the realm of writing as process this next semester. By splitting my one section (with 40 students on average) into two, I hoped to qualify for writing-intensive designation. With that designation and the assistance it brings, I could craft a course in which students write to understand and then revise that writing in order to communicate effectively. I'm currently in a pretty dark place in terms of my optimism regarding those goals this semester. I have fifty-two students in the two classes and--for the first time teaching this class in over a decade--no TAs. I don't see a way for me alone to teach them both right now, not at least without radically reneging on my original plans.

Student reading and student writing seem to have gotten worse--drastically so--in the last few years post COVID. And, pace Burke's arguments, AI really has changed the game. Have some students always cheated and BS'd their way through essays? Sure. But AI does more than just make this halfassery easier to accomplish and harder to detect (though it does both those things). A student copy-and-pasting from some website at least realizes on some level that they're doing something wrong, literally taking someone else's work and presenting it as their own. Most of the plagiarism I see in student papers now is different. Sure, sometimes a student has made use of one of the many, many AI essay-writing products advertising themselves to students as harmless, even essential helps. ("My prof wanted me to write three whole pages!" sales-students in these ads gripe. "Who has time for that? That's why I let [PRODUCT] do it for me...") And sure, "just say you used Grammarly" is a common bit of get-out-of-jail-free advice circulating online for students.

But increasingly, I fear that students really are just using Grammarly or some program like it to check their writing, only to find that the program is AI-helping ("AIding"?) them by replacing words or phrases with "better" ones. The result is often a ship-of-Theseus trick, where the new ship consists of lots of adverbs and pseudo-profound statements about how we humans need to hold onto our dreams or somesuch. Like I heard from a colleague: "I used to get terribly written papers with some good ideas. Now I get perfectly written papers with terrible ideas." When I flag such AI-ded essays as violations of my course policies, I think some students really are honestly confused. They don't know what good writing looks like. They don't know what their own writing feels like. They can't tell why the AI version, while usually grammar-perfect--reads like queasy mush. And from their perspective, they did nothing wrong. They wrote something and, as I (used to) tell them to, checked and proofread their writing with a program that purports to do just that. But AI's version of "improvement" tends to mean "make the essay sound smart [lots of big words and adverbs] and uplifting [unctuous praise for the play/playwright, grand talk about generic, humanistic themes]." Students just learning how to write, just learning what real criticism and analysis are, can't tell the difference between quality and crap. And AI makes it all the less likely they will ever be able to do so.

My concern here echoes Burke's big worries about society at large:

[T]he indiscriminate vomiting of generative AI into everything we read and view, every tool we use, every device in our homes, every technological infrastructure we operate or own, means at best an unproductive estrangement, a new mediating layer that no one, expert or otherwise, can really understand or control. A kind of techno-tinnitus, a buzzing hum of interference or diffusion. When things break, when things don’t quite do what we want them to do, when we don’t get what we’re owed from what we’ve created or done, there won’t be anything to do about it. When we’re described, evaluated, measured, assessed in all the ways that are already balefully mindless when they’re done by actual human beings that we actually can see, we’ll suddenly become even less recognizable, even less true to our realities, and there won’t be anything to be done about it. You’ll complain about the AI processes in the black box to another black box AI and there will be no one anywhere who actually knows why it gave the results that it did, no expert Delphic oracle who can take it apart and get it back on track. At worst, it means that everything that is translocal to our material surrounds will be untrustworthy and unknowable. Not even an interesting fiction, just a kind of informational drift, a noise so pervasive that the entirety of the signal gets lost. 

It'd be one thing if the infonoise (cAIcophony? I'll stop now) were reliable. But LLMs like ChatGPT specialize not in truth but in bullshit--words spewed out with utter confidence and no regard for their truth value. It's like Poe's Law but applied to all digital representations of reality itself: can't tell if true or BS. Students using AI now often, I'm convinced, simply don't know that they're indulging in BS, producing it, absorbing it, starving themselves on a diet of predigested slop.

Increasingly, it looks like infopoop is going to be what we get from AI. Or, as Burke puts it:

Generative AI is being used so heedlessly, so much like a silicon equivalent of the Human Centipede, gulping down its own shit as it hungrily demands more and more and more text for its training models, that it is going to end up spewing informational diarrhea forever all over the entire infrastructure of knowledge production. It is going to pollute all forms of many-to-many communication, all forms of mass media. When we hit that point, it will be impossible to cleanse it all out again. Everything we know will become a Superfund toxic waste site, full of forever hallucinations and distortions.

Now there's an image: AI as (in)Human Centipede. BS recycling BS recycling BS, crowding out other exchanges, eating up available energy and attention. "We’re probably only a few years away from mandatory brown-outs in ordinary homes," predicts Burke, "because the AI needs the power." 

What did Baudrillard say about the ultimate stage of the simulacrum? The representation replaces reality? Only now the eclipsing representation is crap. Welcome to the simulcraprum. 

Sigh.