The Stanford AI Lab Blog

tl;dr. With the rise of large online computer science courses, there is an abundance of high-quality content. At the same time, the sheer size of these courses makes high-quality feedback to student work more and more difficult. Talk to any educator, and they will tell you how instrumental instructor feedback is to a student’s learning process. Unfortunately, giving personalized feedback isn’t cheap: for a large online coding course, this could take months of labor. Today, large online courses either don’t offer feedback at all or take shortcuts that sacrifice the quality of the feedback given.

Several computational approaches have been proposed to automatically produce personalized feedback, but each falls short: they either require too much upfront work by instructors or are limited to very simple assignments. A scalable algorithm for feedback to student code that works for university-level content remains to be seen. Until now, that is. In a recent paper, we proposed a new AI system based on meta-learning that trains a neural network to ingest student code and output feedback. Given a new assignment, this AI system can quickly adapt with little instructor work. On a dataset of student solutions to Stanford’s CS106A exams, we found the AI system to match human instructors in feedback quality.  

To test the approach in a real-world setting, we deployed the AI system at Code in Place 2021, a large online computer science course spun out of Stanford with over 12,000 students, to provide feedback to an end-of-course diagnostic assessment. The students’ reception to the feedback was overwhelmingly positive: across 16,000 pieces of feedback given, students agreed with the AI feedback 97.9% of the time, compared to 96.7% agreement to feedback provided by human instructors. This is, to the best of our knowledge, the first successful deployment of machine learning based feedback to open-ended student work.

Leave a Comment