If you’ve been getting cozy with the idea of your friendly AI chatbot, ChatGPT, whipping up your final thesis paper, it’s time for a reality check. While AI-generated writing has wowed the masses, researchers at the University of Kansas have just upped the ante by developing a method to identify AI-generated academic science writing with over 99% accuracy.
The research, published in the scientific journal Cell Reports, explains how it came to be.
“The need to discriminate human writing from AI is now both critical and urgent, particularly in domains like higher education and academic writing, where AI had not been a significant threat or contributor to authorship,” the researchers wrote.
'Black Mirror' Creator Says ChatGPT Episode Script Was 'Shit'
While OpenAI's generative AI chatbot has taken the internet by storm, “Black Mirror” creator Charlie Brooker is not impressed with ChatGPT. “I’ve toyed around with ChatGPT a bit," Brooker told Empire Magazine, saying that he told the chatbot to generate an episode of his dystopian sci-fi series. Instead of generating a new episode, Brooker said ChatGPT summarized old episodes of “Black Mirror” into an amalgamation as if it were the script for a new episode. "It comes up with something that, at f...
Past attempts to detect AI-generated content have met with varying degrees of success. OpenAI’s own efforts, for instance, haven’t been reliably effective. But these detection tools have traditionally focused on a broad spectrum of writing styles, rather than honing in on the specific tone and purpose that characterizes academic science writing.
The team's method, conversely, was fine-tuned for academic writing. By comparing 64 human-written perspectives and 128 ChatGPT-generated articles on the same research topics, they identified key markers of AI writing.
For example, ChatGPT’s penchant for using the words "others" and "researchers" over words like "however," "but," and "although" proved to be a dead giveaway. Also, humans’ predilection for complex paragraph structures, varied sentence lengths, and shifting word counts was another thumbprint missing in the AI writing.
When put to the test, the model managed to identify AI-generated articles from human ones with a 100% accuracy rate and distinguished AI-written paragraphs within human articles with a 92% rate. In similar tests, the team's model even outshone an existing AI text detector on the market.
According to an article sent by Cell to Techxplore, the next step is to push the boundaries of the model’s applicability, testing it on more extensive datasets and different types of academic science writing. As ChatGPT and its ilk become more sophisticated, it remains to be seen if the model will maintain its effectiveness.
Senators Question Meta CEO Mark Zuckerberg Over LLaMA AI Model “Leak”
Mark Zuckerberg, Meta CEO, is in the crosshairs of two U.S senators. In a letter today, Sens. Richard Blumenthal (D-CT), chair of the Senate’s Subcommittee on Privacy, Technology, & the Law, and Josh Hawley (R-MO), ranking member, raised concerns about the recent leak of Meta's groundbreaking large language model, LLaMA. The Senators expressed their concerns over the "potential for its misuse in spam, fraud, malware, privacy violations, harassment, and other wrongdoing and harms." The two politi...
Moreover, users could potentially fine-tune open-source Large Language Models (LLMs) and manipulate the language to evade this and other detection methods. It would be yet another technological arms race: one between “jailbreakers” seeking to create the most advanced AI tools, and researchers aiming to develop the best detectors for AI-generated content.
But is this tool available for educators to use on student papers?
Desaire says the model is not quite there yet. However, their methods can be easily replicated for different applications.
In the meantime, remember: Before you instruct ChatGPT to “write my thesis,” be aware that your AI pal might be more predictable than you think, so it's better to Do Your Own Research (papers). As for ChatGPT, it might be time to go back to AI school for some lessons in nuance and variety.
