What does an AI detector's confidence score actually mean, 80% of what?
It depends on the tool, which is the problem. A percentage can mean a probability for the whole document, a share of text flagged, or a confidence bucket. GPTZero documents its score as a class probability, read as how often the detector is right on similar documents. That describes the tool's average behaviour, not who wrote your text.
Cite
Global 100 Forum, "What does an AI detector's confidence score actually mean, 80% of what?", https://global100.org/forum/q/what-does-an-ai-detectors-confidence-score-actually-mean-80-of-what/, accessed 2026-11-12.I work in ML, and I keep seeing people argue about what "80% AI" means, usually with a lot of certainty and no definition. I have tried to pin it down for the detectors I can find documentation for and I am still not sure.
There seem to be at least three readings. (a) The tool thinks there is an 80% chance the whole document is AI-written. (b) 80% of the text, by sentences or tokens, was judged to be AI-written. (c) The model is 80% confident in whatever label it gave, which could be "human". These are very different claims, and a person reading a screenshot will not know which one they are looking at.
I pasted the same paragraph into two tools. One returned "80% AI". The other returned a "mostly human" label with a confidence of about 80%. I would like to know whether either number is calibrated, meaning that of all the texts scored 80% roughly 80% really are AI, or whether it is just the raw output of a classifier. Has anyone found a vendor that states this plainly?
Write something first.
Give people something to work with: at least 30 words on what happened and what you tried.
That is too long. Keep it under 6,000 characters.
Write the question as the title, 15 to 140 characters, no links.
Pick a category.
Add a name (2 to 40 characters, no links).
That email address does not look right.
That was quick. Read the thread, then try again.
The form expired. Reload the page and post again.
Something went wrong with the form. Reload and try again.
Please complete the check and post again.
Limit reached for now. Try again later.
This thread is closed to new replies.
This thread no longer accepts replies.
Something went wrong with the form. Reload and try again.
Post a reply or question first (name and email), then this browser can vote, edit and accept answers.
You cannot vote on your own post.
Only the author (within 30 days) or the forum team can do that.
That email already has an account. Use your sign-in link instead.
New accounts are paused for the moment. Try again later.
That was already posted.
6 answers
- Accepted answerMaya LindqvistStaffEdits the AI text detection and academic integrity sections ·
It depends on the tool, which is the problem. A percentage can mean a probability for the whole document, a share of text flagged, or a confidence bucket. GPTZero documents its score as a class probability, read as how often the detector is right on similar documents. That describes the tool's average behaviour, not who wrote your text.
GPTZero's FAQ is one of the few vendor pages that spells this out. Its API returns a document classification of human only, mixed or AI only, with a probability for each class. It says the probability for the predicted class "can be interpreted as the chance that the detector is correct," so "90% means that 90% of the time on similar documents our detector is correct." It adds a separate confidence category (high, medium, low) tuned so that, on its own test data, a high-confidence result classifies 99.1% of human articles as human and 98.4% of AI articles as AI. Those are the vendor's figures on its own data, and I cannot verify them. It also says the highlighted sentences "should not be solely used to indicate that an essay contains AI."
So, mapping to your three readings: that is closest to (c), a per-document class probability, not (b), the share of the text. Your second tool's "mostly human, 80%" is probably in the same family. I do not know how that particular tool defines it, and I would check its documentation rather than assume.
On calibration, there is a deeper limit. Sadasivan and colleagues show that the best possible detector's performance is bounded by how far apart the distributions of human and AI text are, so the task gets harder as language models improve. Their experiments also found that recursive paraphrasing can significantly reduce detection rates. A score is a model output, not a measurement of authorship. See how accurate AI text detectors really are.
Sources:
- Priya N.Member ·
From the TA seat: instructors read it as reading (b). "80% AI" becomes "80% of this essay was written by a machine" in about two seconds, and then it is in an email to a student. I have never seen a tool's help page get quoted in one of those emails, so the definition does not travel with the number.
- Sam WhitlockMember ·
Even a perfectly calibrated score does not answer the question the reader cares about, which is "how likely is it that this student used AI". That depends on the base rate: how many of the texts in the pool are AI to begin with. The same score means something different in a class where almost nobody uses AI than in one where most people do. Without that prior the number is half of a calculation.
- Dan OkaforMember ·
Pretty sure "80% AI" means 80% of the sentences were written by AI. That is how I explain it to parents, anyway. It seems like the obvious way to read it.