#235 Building an AI tutor with Google DeepMind – the sequel – with Bibi Groot (Eedi’s Chief Impact Officer)

Craig Barton
Apple PodcastsSpotifyGoogle PodcastsAnchorPodbeanStitcherPocketCasts

Episode Overview

Bibi Groot, Eedi’s Chief Impact Officer, returns to unpack the second Eedi and Google DeepMind AI tutor trial: 1,179 students, four arms, ten weeks. She and Craig discuss where AI tutoring matched expert human tutors, the null result on Star Maths, an unexpected gender effect, tutor wellbeing, safety data and what comes next.

Talking points

  • A recap of the first Google DeepMind trial: 165 students over seven weeks, tutors accepting 74.4% of AI drafts unedited, and the supervised AI matching expert human tutors in the lesson, with signs of stronger near transfer
  • The design of the second trial: 1,179 students aged 12 to 15 in ten UK state schools, each randomised for the whole term to static fluency practice, an expert human tutor, an AI tutor with question and misconception context, or an AI tutor with personalised student history, with 58 tutors reviewing every AI message before it reached a student
  • Why the personalised-context arm was not a full test of student history: the difficulty of deciding how to present student data to the model, and a latency-related bug that stopped session history from carrying over
  • The in-lesson results: any form of tutoring beat static content on correcting the original mistake and on the next topic, with AI and human tutors on a par, and the gap narrowing on the interim quiz
  • The null effect on Star Maths, and the breadth-versus-depth explanation: students on static practice got through around 132 questions, those in the AI arms around 99, and those with human tutors around 82
  • A gender interaction nobody predicted: tutoring appeared to help girls but not boys, with teachers reporting that many boys experienced being pulled into a dialogue as a punishment for getting a question wrong
  • Why lower-prior-attaining students gained least from human tutors, Craig’s curse-of-knowledge hypothesis, and the finding that the AI ended with a check-for-understanding question far more often than the human tutors did
  • What tutors’ daily diaries revealed: the AI was rated helpful and time-saving, but tutors felt less enthusiastic on days spent supervising it, which has informed the move from human-in-the-loop to human-on-the-loop
  • Safety and accuracy: tutors approved 94.7% of AI drafts unchanged, and the full conversation analysis found seven confirmed maths errors from the AI and seven from human tutors
  • Bibi responds to five common critiques of AI tutoring, then previews the year-long trial with over 15,000 students in 26 UK schools and new trials in Saint Vincent and the Grenadines, India and Honduras

Links from the show

https://youtu.be/AQUQShCms_o

New stuff I have been working on:

  1. My Tips for Teachers Guides to… series
  2. My updated mrbartonmnaths website

My usual plugs

Leave a Reply