You asked, and Bwog delivered.

There are some constants of Columbia life. The water fountains will be broken, Ferris pizza will taste like salty cardboard, and Katrina Armstrong will send a page-long email to the student body that somehow manages to say absolutely nothing. 

As the leader of a university, a college president has a duty to communicate with their students and receive student feedback. Interim President Armstrong takes this duty perhaps a little too seriously. Last semester, Armstrong sent out 22 emails over 20 weeks; that’s more than an email a week. She is on track for a similar average this semester. Some of her messages contained information about happenings on campus, such as the stepping down of the Mailman School Dean or the formation of yet another committee (whether or not these committees are useful is another question entirely). But as far as I can tell, most of these emails aren’t about anything. The Morningside Campus access emails could all be summarized by: “No, we still aren’t opening the gates.”

We’ve all received countless emails proclaiming Armstrong’s commitment to Columbia’s values. These emails are so painfully generic that many students have joked that Interim President Armstrong uses ChatGPT to write these emails, including a writer for the Federalist. But what if these jokes are based on reality? As a news organization, it is Bwog’s duty to investigate all happenings on campus, even rumors. In the spirit of journalism, I have endeavored to determine whether or not Armstrong uses generative AI to send out campus emails. 

Let me start with the bad news first: there is no surefire way to determine whether a piece of writing is AI-generated. I know. What’s the point of this article, you ask? Even though I can’t definitively say whether or not Interim President Armstrong uses generative AI, there were some interesting findings. Do with this information what you will. 

I began by compiling some basic statistics. AI-generated writing may have more regular patterns than human writing, so I looked for similar word and paragraph counts. The word counts ranged from 283 to 1071, and emails had four to nine paragraphs. These statistics did not point me in either direction, as word counts and the number of paragraphs can easily be changed. I was only looking for unusual uniformity. I noted that this semester’s five emails (not including the short statement on class disruptions) each had four paragraphs. I was slightly suspicious, but further investigation was required. 

Online AI detectors were my next stop. Paywalls limited my options, so I used two free platforms that could analyze a corpus as large as Armstrong’s emails: Content Detector AI and Myessaywriter.ai. As a STEM major, I felt obligated to use the scientific method. I devised a (semi) controlled experiment. In each platform, I input Armstrong’s emails, a human writing sample I wrote, and AI-generated emails.

Both AI detectors concluded that my writing sample was likely not AI-generated and that the AI writing samples I used were not human-written. While not a definitive metric of accuracy, these results lend a little bit of credibility to the results. 

The moment of truth: Content Detector AI and Myessaywriter concluded Armstrong’s emails were somewhat likely to have been AI-generated; their numbers clocked in at a 70% chance and 50% chance, respectively. These figures were less definitive than most of the controls. Still, Content Detector AI concluded that Armstrong’s emails were more likely to have been AI than some AI-generated emails I inputted.

Does this mean Katrina Armstrong is unequivocally an unrepentant ChatGPT user? Unfortunately, no. Most AI detectors work by measuring perplexity and burstiness (yes, this is the official term). While these algorithms sometimes work, as they did in my controls, they cannot detect AI with 100% accuracy. 

Perplexity is the unpredictability of word choice. AI algorithms will measure how often they can predict the next word. To explain the potential problems of this model, we’ll use an example. Take the beginning of the sentence, “I swam in the…” The words ocean, pool, or lake would have low perplexity scores––these are all common endings to this sentence. The word swamp might have a more middling score; you don’t hear about people swimming in swamps very often, but it’s possible. Words like bookshelf or pants would have extremely high perplexity scores. While grammatically correct, they don’t make logical sense and are highly unlikely to end that sentence. In other words, this algorithm will penalize the use of common words and sentences, which are used by humans and generative AI models alike. 

Burstiness measures the variation in sentence length and structure. AI writing tends to be more formulaic than human writing. The problem? Illinois State University’s article on why AI detectors don’t work puts it best: “Some human writers are monotonous.”  These results suggest that Armstrong could be using AI, but they may indicate something more damning: Armstrong is simply a boring writer. In fairness to our Interim President, she’s not writing about the most riveting topics. I don’t know if I could make the fifth email about campus climate and “historic uncertainty” terribly interesting, either. There’s simply not much to say beyond generic platitudes. 

Although the AI detectors may not always be accurate, my investigations were not fruitless. For the AI-generated control emails, I asked ChatGPT, Claude, and Gemini to generate five emails that sounded like they were from a Columbia University President to its student body. The ChatGPT emails were nearly indistinguishable from many of Armstrong’s emails. 

Let’s play a little guessing game. I’ll give you three paragraphs, and you guess if it’s ChatGPT or Armstrong: 

1. “At a time when public discourse can often feel polarized, I encourage all of us to engage in dialogue with respect, openness, and a willingness to listen. As members of the Columbia community, we must strive to understand perspectives different from our own and to approach difficult conversations with intellectual curiosity rather than division.” 
2. “Real progress requires collective action, and I invite each of you to contribute your ideas and perspectives as we move forward. In the coming weeks, we will host a series of forums where students, faculty, and staff can engage in discussion about these efforts and explore additional ways we can work together to strengthen our community. Let us use this moment to reaffirm our commitment to these values and to each other, recognizing that the richness of our community lies in its diversity of thought, experience, and aspiration.”
3. “At the heart of our work are our students, our patients, our faculty, and our research. Each of these groups are fundamental not only to the future of our University but also to the advancement of knowledge and society as a whole. Our commitment to their success remains resolute. We understand that this moment may prompt concern, but it is in times such as these that our strength shines brightest.”

Here are the answers: ChatGPT, ChatGPT, and Armstrong.

If you got these answers right, I salute you. Even I had to go back and check which ones were which when I was editing this article. Again, the similarity doesn’t prove anything. But it certainly shows that ChatGPT can produce writing nearly identical to Armstrong’s without examples of her writing.

So it all boils down to one question: do we really think Katrina Armstrong is writing these multi-paragraph emails every week on top of her other presidential duties (I’ve never been entirely sure what it is she does—she must do something)? My instinct is no. And given that the university’s Generative AI policies only condemn AI use in academic work and encourage its use as a learning tool, Armstrong wouldn’t be breaking any rules if she were to use AI-generated writing. I also find it equally likely some underpaid employee is pumping these emails out, possibly with the help of ChatGPT. Whether it’s ChatGPT, an intern, monotonous writing, or some combination of all three, the basic reality does not change. All these emails add to my day are a few extra seconds spent decluttering my inbox.

ChatGPT via Bwog Archives