Look inside the tutoring conversation
A wrong final answer can hide several successful intermediate steps. This study turns mathematical tasks inside tutoring conversations into standalone questions, then combines those responses with ordinary question records to estimate difficulty and student ability. Researchers used Eedi data from 588 students with human tutors and Khan Academy data from 136 students with an AI tutor. Across both datasets, the combined model improved prediction of difficulty and response correctness. On Eedi, difficulty prediction’s mean squared error fell 7.7% against the strongest baseline. The contribution is richer measurement of existing interactions, rather than evidence that deploying this model will improve learning or produce fair high-stakes assessments.
Why it matters for leaders
Request examples of the intermediate evidence behind a tutoring dashboard’s mastery estimate. Give teachers a correction route and compare predictions with independent work. Keep experimental conversation-derived scores out of consequential placement decisions until accuracy and subgroup performance are established locally.
The evidence
Five-fold cross-validation separated problems or students by prediction task. Human review found 85.5% of rewritten subtasks complete and mathematically faithful.
Who was studied
Mathematics platform records; pupil demographics and grade-level detail unavailable.
What the evidence cannot tell us
Difficulty labels were model estimates. Demographics were unavailable, and source datasets have access restrictions. Automated rewriting and scoring can introduce errors or bias.
Read the original sources
Summary and leadership analysis from the Week ending 2026-10-16 brief. The edition’s selection notes explain source access and evidence limitations.