Mastery Challenges

The Problem

We’ve all been there. You learn a concept, feel like you’ve got it, apply it and then weeks later can’t even remember seeing it.

Students and teachers deal with this all the time, especially with math. Imagine you’re a 5th grader learning how to add and subtract decimals, you study hard for a quiz, and then weeks later find that you don’t remember anything about it. One of the ways teachers in the classroom have learned to combat this is to apply a pedagogical pattern known as spiral review. The general idea is that concepts previously learned are later reinforced to help learners commit those concepts to long-term memory.

In Khan Academy, we realized we had developed a gap in our pedagogical approach. Teachers kept providing us feedback that they could only rely so much on our platform as a supplement to their teaching since it didn’t incorporate ways to help students really understand concepts they previously experienced.

The forgetting curve and how consistent review helps for long-term knowledge retention

Process & solution

The idea of launching a spiral review feature came at a delicate time for Khan Academy. We were in the midst of deprecating an outdated, but beloved feature known as Missions that had a small, but vocal user base. Through interviewing many of these users, we understood that one of the main reasons they loved Missions was because of its ability for students to review material they had previously seen before. This form of space repetition, while not perfect, was a huge help for teachers using our platform. We also interviewed teachers who had once tried Khan Academy and abandoned it. Many of these users indicated that the primary reason they were less inclined to rely on Khan Academy was due to their class being unable to consistently review content from earlier subjects. Both of these teachers had concerns that their students wouldn’t be able to retain what they were learning on Khan Academy without consistent review.

The elimination of spiraling…is the primary reason our schools switched from Khan…. Khan is just much less effective for our needs this way.

District Leader

…this year my honors students only saw a 5% gain over their prior year ranking. I suspect this decrease is largely attributable to recent changes in KA. Specifically, lack of Spiral Review…

Teacher

Through our research we identified 3 goals:

1. Boost learner knowledge retention over time, and increase teacher confidence in our Course Mastery system by introducing a spiraling feature that was personalized to the learner
2. Give learners a less-punitive place to review and level up their skills. Currently, there are only a few ways to show you’ve mastered a skill, and those are quite time-consuming for learners!
3. Encourage review of previously mastered skills. Our system previously had no mechanism to review skills students had already achieved mastery with.

Having the green light on this feature set the stage for a few explorations. Our first approach involved incorporating “flashback” questions in new quizzes and unit tests that learners would take on the platform. After a few iterations on this concept, cross-functional discussions and interviews with teachers, we realized this approach wouldn’t work for a few key reasons. To start, having these flashback questions interspersed within existing quizzes and unit tests meant that teachers lacked visibility into the black box of our personalized learning platform. As a result, the skills and specific questions students addressed wouldn’t be visible to a teacher so they’d have no way to intervene and help reinforce concepts. Teachers and our pedagogy working group also felt that having these review questions tacked on to existing quizzes and unit tests wouldn’t allow for sufficient review to help with knowledge retention. On top of that, having questions embedded in other quizzes and unit tests meant increased complexity for our engineering team as opposed to having a standalone review feature, especially when it came down to showing this data on teacher reports. My team and I ended up taking this feedback and iterating on the design of a standalone module for spiral review called Mastery Challenges.

Our first approach at incorporating spiral review questions, which was unsuccessful

After a student spends some time working on a course, they would “unlock” a 6-question Mastery Challenge that spirals skills they have previously worked on. In order to do this, a learner would have to work on at least 3 skills in the course, 1 of which that they are ready to prove mastery on. The skills would be selected by our underlying system based on a student’s level of proficiency and the amount of time that’s elapsed since they last worked on that skill. Once a Mastery Challenge is unlocked, the student would have 12 hours to complete it before it resets. Having a time element on a Mastery Challenge ensures that the reviewed skills are always relevant (and adds a little gamification to the mix).On the teacher side of the experience, this item would show up as a unique entry on a student’s activity log. Teachers would be able to see the specific skills a student worked on and whether or not they leveled up in mastery. This aspect addressed the specific user need of providing visibility into what a student reviewed.

Paper prototype of the shipped experience

Default Mastery Challenge state

Paused Mastery Challenge state

Completed Mastery Challenge state

Report data showing Mastery Challenge skills

Outcome & next steps

In all honesty, this feature release has been one of the highlights of my career. After launching this beta to 40+ teachers and their students, our NPS Score on the Spiral Review feature amongst teachers was 95.24%!

We announced this beta in a webinar to teachers, and of the 77% who took a post-webinar survey, 100% responded with “Launch it!”

There are so many standards to cover in 4th grade math, that as a teacher I am forced to jump from one topic to another all year. This is a great tool to reinforce previous topics. Students also gain confidence and strengthen their math skills.

— Robert

I am a real believer in learning that it’s not “one and done”. I see kids who do well so they think they are “experts” and once they have been away from a subject for awhile they have forgotten part or all of the concept. This will continue to challenge them and ensure they really are masters.

— Marcy

Overall, this feature has been a huge success. The next step for us as a team will be to continue monitoring data over time to measure efficacy results and see if knowledge retention increases on the platform.