The idea of launching a spiral review feature came at a delicate time for Khan Academy. We were in the midst of deprecating an outdated, but beloved feature known as Missions that had a small, but vocal user base. Through interviewing many of these users, we understood that one of the main reasons they loved Missions was because of its ability for students to review material they had previously seen before. This form of space repetition, while not perfect, was a huge help for teachers using our platform. We also interviewed teachers who had once tried Khan Academy and abandoned it. Many of these users indicated that the primary reason they were less inclined to rely on Khan Academy was due to their class being unable to consistently review content from earlier subjects. Both of these teachers had concerns that their students wouldn’t be able to retain what they were learning on Khan Academy without consistent review.
The elimination of spiraling…is the primary reason our schools switched from Khan…. Khan is just much less effective for our needs this way.
…this year my honors students only saw a 5% gain over their prior year ranking. I suspect this decrease is largely attributable to recent changes in KA. Specifically, lack of Spiral Review…
Through our research we identified 3 goals:
Having the green light on this feature set the stage for a few explorations. Our first approach involved incorporating “flashback” questions in new quizzes and unit tests that learners would take on the platform. After a few iterations on this concept, cross-functional discussions and interviews with teachers, we realized this approach wouldn’t work for a few key reasons. To start, having these flashback questions interspersed within existing quizzes and unit tests meant that teachers lacked visibility into the black box of our personalized learning platform. As a result, the skills and specific questions students addressed wouldn’t be visible to a teacher so they’d have no way to intervene and help reinforce concepts. Teachers and our pedagogy working group also felt that having these review questions tacked on to existing quizzes and unit tests wouldn’t allow for sufficient review to help with knowledge retention. On top of that, having questions embedded in other quizzes and unit tests meant increased complexity for our engineering team as opposed to having a standalone review feature, especially when it came down to showing this data on teacher reports. My team and I ended up taking this feedback and iterating on the design of a standalone module for spiral review called Mastery Challenges.
After a student spends some time working on a course, they would “unlock” a 6-question Mastery Challenge that spirals skills they have previously worked on. In order to do this, a learner would have to work on at least 3 skills in the course, 1 of which that they are ready to prove mastery on. The skills would be selected by our underlying system based on a student’s level of proficiency and the amount of time that’s elapsed since they last worked on that skill. Once a Mastery Challenge is unlocked, the student would have 12 hours to complete it before it resets. Having a time element on a Mastery Challenge ensures that the reviewed skills are always relevant (and adds a little gamification to the mix).On the teacher side of the experience, this item would show up as a unique entry on a student’s activity log. Teachers would be able to see the specific skills a student worked on and whether or not they leveled up in mastery. This aspect addressed the specific user need of providing visibility into what a student reviewed.
A paper prototype of the shipped experience. This would mirror the experience of taking a quiz or a unit test on Khan Academy.
The default state of a Mastery Challenge and how it would appear to a learner when unlocked.
After starting a Mastery Challenge, a learner would have 12 hours to complete it.
After completing a Mastery Challenge, a learner would be able to see what skills it covered and how their mastery progress changed.
When a student completes a Mastery Challenge, both the student and teacher have visibility into report data covering specific skills they practiced.
In all honesty, this feature release has been one of the highlights of my career. After launching this beta to 40+ teachers and their students, our NPS Score on the Spiral Review feature amongst teachers was 95.24%!
We announced this beta in a webinar to teachers, and of the 77% who took a post-webinar survey, 100% responded with “Launch it!”
There are so many standards to cover in 4th grade math, that as a teacher I am forced to jump from one topic to another all year. This is a great tool to reinforce previous topics. Students also gain confidence and strengthen their math skills.
I am a real believer in learning that it's not "one and done". I see kids who do well so they think they are "experts" and once they have been away from a subject for awhile they have forgotten part or all of the concept. This will continue to challenge them and ensure they really are masters.
Overall, this feature has been a huge success. The next step for us as a team will be to continue monitoring data over time to measure efficacy results and see if knowledge retention increases on the platform.