I ran my own 2019 essays through a detector and half came back as AI. Is that normal?

Not surprised at all. Formal, tidy, structured writing is what these tools seem to score as AI. I was flagged on a client piece that was the most heavily edited thing I ever wrote. Clean writing gets punished.

Good experiment. I would try your most formal essay in a second detector and see if the results match.

It is a useful anecdote, and I would present it as one, not as a measurement. Twelve essays and one tool tells you that this tool, on this kind of text, can misfire. It does not tell you a rate. To get closer to a rate you would want many documents, several tools, and a record of what text length and format you used.

Still, as a demonstration for colleagues, "here are my own essays from before the technology existed" is vivid and honest. Just state the limits yourself before someone else does.

This is what I fear every term. My writing is careful and formal because I am writing in a language I learned later, and I check every sentence. I would be so worried to see a score like that on my own essay. Thank you for doing the experiment, even if it is small.

Everything I have seen says the longer the text, the more reliable the tool. So the explanation is simple: the detector had a few paragraphs of each essay and not enough to go on. Paste a whole essay and it will be accurate.

@Dan Okafor I would not rely on the rule that longer is always more reliable, and in this case the essays were pasted whole. Priya said she pasted each one in full, so snippet length is not the explanation. Longer text can give a detector more to work with, but formal, predictable writing can still score high however long it is.

For the department meeting I would add one point of procedure. The question is not whether detectors are sometimes wrong. Everyone concedes that. The question is what our policy says a score may be used for. Your experiment is a concrete illustration of why a score alone cannot be the basis of an allegation, and illustrations persuade committees more than reports do.

Bring the data in a table: tool, date, text length, result, and the fact that these predate the technology.

One technical aside. If any of the essays were pasted as a whole and still flagged, that is a stronger demonstration than short snippets. Keep the original text files and the tool's output, with the date. Someone will ask you to rerun it and the results may differ slightly. That is not a reason to hide the experiment, just a reason to record it.

Not typical for a well tested detector, but it can happen with some free tools, and your result is a fair reason to be sceptical of a score used alone. Formal, formulaic writing and shorter passages are known trouble spots, and no tool's error rate transfers to your text without testing.

For a sense of the range, Turnitin's AI writing detection FAQ says it aims to keep its false positive rate under 1% for documents with more than 20% AI writing, and that it tests on over 700,000 academic papers written before ChatGPT was released. It also warns that false positives can occur in text with little structural variation, text that repeats itself, or text paraphrased without new ideas, and that in documents of only a few hundred words the prediction is mostly all or nothing because there is a single segment. That is a vendor describing its own testing, so read it as a stated target, not a guarantee, and it applies only to that product.

The independent evidence points the other way for some tools. The Stanford study GPT detectors are biased against non-native English writers tested seven detectors and found they misclassified more than half of the TOEFL essays, which are real human writing, as AI generated (average false positive rate 61.22%), while being near perfect on US eighth grade essays. The authors link this to low perplexity, a measure of how predictable the text is, and note their sample was small and the detectors were older.

So your result fits a pattern more than it contradicts the vendors. Repeat it on several tools, record text length and format, and present it as a demonstration, as Hana suggests. Our thread on how accurate text detectors really are has more on testing.

@Maya Lindqvist very helpful, thank you. I will rerun the twelve on two or three more tools, record length and format, and present it as a small demonstration with the limits stated up front, not as a rate. The vendor's own note about formulaic text helps my case.