A rubric changes the AI grade
What happens when an AI grader receives more than a generic instruction to assign a score? A Turkish history study compared seven models and five prompting approaches against an instructor's ratings of 24 university students' answers. The same answers produced different levels of alignment as prompts changed. For example, the correlation between ChatGPT o3 and instructor scores rose from .56 with basic instructions to .76 with weighted criteria. But stronger correlation did not remove statistically significant score differences for that model. Researchers also observed fluent, lengthy answers receiving generous treatment despite incomplete content. The useful lesson is about assessment design: explicit criteria can improve some comparisons, while matching a ranking remains different from awarding the right score.
Why it matters for leaders
Require a local, teacher-scored validation set before any grading pilot. Compare absolute score errors and consequential pass/fail disagreements, including terse but correct answers. Keep a teacher responsible for the final grade and give students an understandable route to challenge it. Recheck performance whenever the model or rubric changes.
The evidence
Exploratory analysis of 72 responses to three questions. Statistical comparisons use 24 student composite scores, not 2,520 independent students or scoring events. The human reference was one instructor.
Who was studied
University history answers in Turkish; indirect evidence for K-12 assessment.
What the evidence cannot tell us
Single language, course and small sample; model versions age quickly. Nonsignificant differences do not establish grading equivalence or fairness.
Read the original sources
Summary and leadership analysis from the Week ending 2026-10-02 brief. The edition’s selection notes explain source access and evidence limitations.