A better tutor still needs students
A model that solves mathematics well may still explain it poorly. Sherpa trains a tutoring model through repeated conversations with simulated learners that require different instructional approaches. Across those artificial learner types, reported performance improved by an average of 20.5 percentage points; a separate tutoring benchmark also improved. In a human evaluation, 96 high school teachers judged paired responses, preferring Sherpa in 79.6% of non-tied comparisons. These are encouraging results about training and perceived teaching quality, not measured learning gains for children. Some other benchmark skills, including locating mistakes, declined after training. The findings make a case for evaluating several teaching behaviors together before treating an adaptive tutor as classroom-ready.
Why it matters for leaders
Ask tutoring vendors to separate simulated learning, expert ratings and actual pupil outcomes in their evidence. Include error diagnosis, answer leakage and independent transfer in a local evaluation. A persuasive explanation should earn further testing, not automatically satisfy the purchase criterion.
The evidence
Teachers supplied 960 judgments on 320 response pairs. The human task compared responses; it did not follow pupils through instruction or retention tests.
Who was studied
Artificial mathematics learners and adult teacher raters; no pupil learning trial.
What the evidence cannot tell us
Simulated preferences are deliberately engineered. Teacher preference cannot establish student learning, and improvement on a combined score can conceal weaker component skills.
Read the original sources
Summary and leadership analysis from the Week ending 2026-10-09 brief. The edition’s selection notes explain source access and evidence limitations.