Five Recent AI Tutoring Studies - Arjun Panickssery
Last week some results were released from a 6-week study using AI tutors in Nigeria. Below I summarize the results of that and four other recent studies about AI tutoring (the dates reflect when the study was conducted rather than when papers were published):
Summer 2024 — 15–16-year olds in Nigeria They had 800 students total. The treatment group studied with GPT-based Microsoft Copilot twice weekly for six weeks, studying English. They were just provided an initial prompt to start chatting—teachers had a minimal “orchestra conductor” role—but they achieved “the equivalent of two years of typical learning in just six weeks.”
Spring 2024 — K-12 Title I schools in the South They had 1,800 K-12 students in a low-income school district and gave human tutors to both the treatment and control group, though in the treatment group the tutors had access to the “Tutor CoPilot” button designed by the researchers to provide hints, similar problems, worked examples, etc. In only 29% of treatment sessions did the tutor use the button. An “exit ticket” problem was solved by 66% of treatment versus 62% of control students.
Spring 2024 — 16–18-year-olds in Italy They split 76 students (85% girls) from an Italian technical institute (a high school not aimed at university) into two groups for their ESL class: the treatment group had their weekly homework assignments supported by an interactive tutoring session using GPT-4. I don’t see the raw scores printed but the effect sizes reported as Cohen’s d are small and not significant.
