A randomized study in 18 Tennessee middle schools suggests that simply placing an AI tutor within students’ reach does not guarantee that they will use it to improve their learning. Over two years, 96% of participating students tried Khan Academy’s Khanmigo at least once, but the median student used it on only about one-third of the days when it was available.
The findings come from “One Click Away,” a National Bureau of Economic Research working paper by economists Philip Oreopoulos and Nina Low. Students used Khan Academy during daily math intervention sessions, with Khanmigo designed to coach them rather than provide direct answers. Compared with students receiving conventional remedial instruction, those with access to the AI tutor achieved modest math gains, similar to the results expected from Khan Academy practice without AI.
Most students used the tutor infrequently, and their conversations were rarely focused on mathematics. Khan Academy CEO Sal Khan said the results highlighted the importance of motivation, a point the organization had already been making publicly.
The article argues that optional AI assistance can encourage students to choose tools that provide answers instead of tutors that ask them to reason through a problem. Requiring students to use an AI system may improve participation, but it does not necessarily produce effort or meaningful interaction. Khan Academy chief learning officer Kristen DiCerbo has described some student exchanges with AI tutors as more passive than educators would like.
Khan Academy has responded by integrating Khanmigo directly into practice problems rather than leaving students to initiate conversations with it. The change reflects the view that an AI tutor must be embedded in learning activities and that opportunities to avoid productive struggle should be limited.
The proposed alternative is to make the conversation with AI part of the assessment. In a separate study involving 21 12th-grade students at Collège du Léman International School in Switzerland, students examined examples of strong and weak AI interactions, helped create a rubric for effective use and then submitted complete ChatGPT transcripts from a media analysis project.
Students were assessed on the precision of their questions, how critically they responded to AI suggestions and whether they retained ownership of the thinking. After the four-week study, 18 students said their approach to AI had changed, while 10 reported paying more attention to their own input and how they could build on it during the conversation. The number who felt completely confident obtaining useful responses from AI rose from two to 11.
The researchers also tracked whether greater confidence was accompanied by a better understanding of AI’s limitations. Five students became substantially more confident while also recognizing more risks. Four became less confident in their ability to judge AI accuracy, which the researchers interpreted as a potentially healthy adjustment. Two students became much more confident while developing a dismissive attitude toward the risks.
The study was a small pilot without a control group, and much of its evidence was self-reported. Its central premise, however, is that students are more likely to make an effort when they know their interaction will be reviewed and judged against clear criteria.
The same idea was tested through role-playing exercises in which an AI represented a difficult person whom a student had to persuade, teach or negotiate with. The author of the article found these exchanges more engaging than conventional Socratic tutoring, largely because a professor would evaluate the conversations. The knowledge that the interaction would be judged increased the incentive to think carefully.
In classroom trials of a platform called AI Friction Labs, students used AI role-plays designed to make their reasoning visible. In a calculus class at Cheshire Academy in Connecticut, 21 students acted as teachers while the AI played a classmate struggling to understand tangent lines. Students had to explain the concept in accessible language rather than relying on technical terminology. Five students repeated the exercise and improved their scores by an average of about 34 points.
At San Domenico School in California, an AI playing an experienced ecologist challenged students with repeated “why” and “how” questions. One student said the exercise pushed them beyond memorizing terms and toward understanding the mechanisms involved.
The approach also generated resistance. Some students found the questioning repetitive, overwhelming or similar to being interrogated. These reactions point to the need for carefully calibrated difficulty, adequate preparation and teacher supervision. The article also acknowledges that the classroom examples were beta tests of a company owned by the author, not controlled research.
The proposed model places the teacher at the center. AI can produce an initial evaluation of a conversation against specified criteria and highlight moments where a student appeared to succeed or struggle. The teacher then reviews the exchange, corrects the AI’s assessment and decides what feedback the student should receive.
Under this approach, AI is not primarily a tutor waiting for a student to request help. It becomes part of an assessment in which students must demonstrate their thinking through an adaptive conversation. A transcript can show where they became stuck, how they responded to challenges and whether they recovered without surrendering ownership of the work.
The article concludes that schools do not need to use AI in every classroom. But when educators want AI to support thinking, access alone is unlikely to be enough. Designing activities in which students’ reasoning is visible—and evaluating how they use the system—may offer a stronger incentive for genuine engagement than leaving an optional tutor at the edge of the screen.
Sources: EdSurge.