Confidence is not critical thinking
Can a chatbot become a thinking partner rather than an answer supplier? A semester-long pilot followed 14 undergraduates in an applied statistics course that asked them to think independently, consult ChatGPT, then explain how their thinking changed. Students reported greater statistical confidence and stronger AI literacy in eight of nine dimensions. Yet the standardized critical-thinking measures showed no statistically significant improvement. Analysis of their interactions also found that students commonly sought validation rather than sustained dialogue. This distinction matters: the positive statistics results measured perceived mastery, not an independent achievement test. The pilot suggests that a thoughtful instructional protocol can coexist with shallow AI use; it does not establish either a learning benefit or an absence of benefit.
Why it matters for leaders
Ask the curriculum team to give each AI pilot two separate measures: students' confidence and an unaided reasoning task. Require a first attempt, a critique of the AI response, and an explanation of the revision. Review a sample of actual conversations before expanding a program described as Socratic tutoring.
The evidence
The preprint combines before-and-after questionnaires, Watson-Glaser reasoning measures, observations and 442 student prompts. There was no comparison class. The statistics confidence instrument developed for this study showed post-test ceiling effects.
Who was studied
Undergraduate applied statistics, spring 2025; indirect evidence for K-12.
What the evidence cannot tell us
Small, self-selected university sample; no causal estimate. A nonsignificant result is not proof that critical thinking cannot improve.
Read the original sources
Summary and leadership analysis from the Week ending 2026-10-02 brief. The edition’s selection notes explain source access and evidence limitations.