Munim Technologies

Research/AI

Does published AI essay feedback quote students' words exactly?

Across public datasets of AI-written essay feedback, most quotes keep the student's words once formatting is allowed for, but a few percent silently change them, short quotes most of all, and the rate varies more than fiftyfold between models.

Oct 26, 2026 · 6 min read

When feedback quotes a student's sentence, the quote should be what the student wrote. If the tool quietly fixes a spelling or a verb form while copying, the student never sees their own mistake. We checked this in published datasets of AI-written essay feedback. In 50,742 quotes that GLM-4.7 wrote about 8,946 IELTS essays, only 37.7% are character-for-character copies, but most of the gap is formatting: 95.7% keep the student's words, and 2.8% of all quotes change words without marking the change. On a second dataset covering six models, 1.18% of quotes change the student's words, with single models ranging from 0.22% to 13.2%.

95.7%

of GLM-4.7 quotes keep the student's words, after formatting

2.8%

change words with no mark that anything changed

0.22–13.2%

words-changed rate across six models on FeedType

The question

Our earlier preregistered studies measured quoting in feedback that small local models generated themselves. This project asks the question of feedback that has already been published as datasets, which is closer to what students actually receive. Each study's plan, with its predictions, was committed before any quote was checked, and the results files record a hash of the plan.

What we did

A script finds every passage the feedback puts in double quotation marks. It sets aside quotes of the exam question and anything introduced as a suggested rewrite, then checks each remaining quote of four or more words against the essay. A quote is exact, the same words (only punctuation, spacing or capitals differ), changed words, or not found. We publish only counts, never essay or feedback text.

What we found

GLM-4.7 feedback quotes on IELTS essays, by match class

Primary set of 50,742 matched quotes in the feedback text. Our preregistered prediction of at least 95% exact copies did not hold. Source: results.json (evaluation.primary_categories)

We had predicted at least 95% exact copies; the rate was 37.7%. A follow-up, planned before it ran, traced the gap. Nearly half of all quotes (47.5%) differ only in punctuation or quote marks at their two ends. Once Markdown emphasis and "..." are allowed for, 95.7% of quotes keep the student's words. Of the 4.3% that change words, about a third show the change in bold, which leaves 2.8% of all quotes changing the student's words with nothing to show it.

Short quotes are the problem. We predicted that long quotes would drift more. The opposite was true: quotes of four to six words change the student's words most often, and each word in a long quote is also less likely to change. Essays scored band 7.0 or above also got more changed quotes than essays at 5.5 or below, the reverse of what we predicted.

Quotes that change the student's words, by quote length

GLM-4.7 feedback on IELTS essays, after formatting is allowed for. 22,413 quotes fall in the 4–6-word bin. Source: length.json (by_length.changed_rate)

Six models on a second dataset. FeedType holds feedback from six models under three prompts on university essays. Here all three predictions held, the first audit to replicate rather than reverse. Across 27,462 primary quotes, 1.18% change the student's words, and short quotes again change words more than long ones (2.29% against 0.68%). The pooled rate hides a wide spread. LearnLM wrote most of the quotes and almost never changed words; Mistral-24B, a larger model than two of the others, changed words in 13.2% of its quotes. Model size was not what separated them.

Quotes that change the student's words, by model (FeedType)

Primary-set quotes: Mistral-24B 500, Qwen2.5-70B 632, LLaMA-3.1 8B 2,466, Gemini 2 Flash 1,614, Mistral-7B 933, LearnLM 21,317. Source: feedtype.json (by_model.word_changed_rate)

Later audits widened the picture. When a small open model wrote feedback on matched pairs of school essays, excerpts from English learners' essays changed their words in 15.8% of cases against 10.0% for matched non-learners. In programming hints from a real course, the model named students' own variables almost perfectly, but 21% of the code snippets it showed appear nowhere in the student's file or the assignment, although it had been told not to suggest code.

Verdicts on all 44 preregistered predictions

Across audits 1 to 8b, as of the latest release. Source: predictions.json (tally)

What it means

Most AI feedback quotes are faithful, so the problem is not that models invent what students wrote. It is that a small, steady share of quotes, concentrated in short phrases where a single altered word matters most, show the student a corrected version of their own sentence. For anyone choosing a feedback tool, the model matters more than the aggregate suggests: rates differed more than fiftyfold on the same essays. Checking a sample of quotes against the essay takes a few minutes, and we publish a free offline checker for exactly that.

More research