Pre-emptive video feedback: what our UCL EDUCATE study found

In 2018 Tassomai ran a study with the UCL EDUCATE programme to test one idea: if our data shows that a student is likely to get a hard science question wrong, does offering them a short video before the question help? Students who chose to watch answered the question correctly 70.7% of the time, against 27.6% for a control group. About a week later, 52.3% of the video watchers still got it right, against 41.5% of the control group [1].

Tassomai’s founder Murray Morrison presented the findings at the Association for Science Education (ASE) Annual Conference in Birmingham on Wednesday 9th January 2019, and the full study was published in the peer-reviewed journal Research for All in 2021 [1]. This post sets out what we did, what we found and what the results can and cannot tell you. Our earlier posts covered the Evidence Applied award the project earned and the launch of the videos in 2019.

Tassomai is built on frequent, low-stakes quizzing. Testing is not only a way to measure learning: retrieving an answer also helps students remember it, an effect known as test-enhanced learning [2]. Dunlosky and colleagues reviewed ten common study techniques and rated only two of them as high utility: practice testing and distributed practice [3]. Spacing study over time is also the first recommendation of the US Institute of Education Sciences practice guide on organising instruction [4].

Without that kind of practice, memory fades quickly. Murre and Dros’s 2015 replication of Ebbinghaus’s classic experiment confirmed how fast unrehearsed material is forgotten [5].

Until then, Tassomai gave its feedback after a mistake. A student who picked a wrong answer saw the correct one straight away, and the question came back soon after with related questions around it. Feedback is one of the most powerful influences on learning, but its effect depends on whether it gives students information they can use on the task in front of them [6]. We wanted to know whether giving that information before the question, to the students most likely to need it, would do more.

We had held back from video for two reasons. Teachers had told us they worried video would make learning passive. And because most students use Tassomai on their phones, video could eat into their data allowance, which hits students from less wealthy homes hardest [1].

Our content team picked six ‘problem questions’ from popular GCSE science courses, each from a different topic and each among the hardest in the question bank by our own difficulty measure. For each one we filmed a short explainer video of one minute to 90 seconds, drawn on a blackboard, and added subtitles [1].

Any student whose most recent answer to one of those questions was wrong became a candidate. Candidates were randomly assigned to a test group or a control group [1]:

• The test group was offered the video at the start of their next quiz on that topic and could watch it or skip it. The choice mattered to us, both to measure appetite and because watching on a phone could cost students data.

• The control group used Tassomai as normal, with the usual feedback after each answer.

Both groups then met the problem question as the first question of the quiz: the CHECK question. Students who got it right saw the same question again, without a video, as the first question of a later quiz after a random embargo of two to seven days: the RETENTION question. In practice the gap was about a week for both groups (a median of 6.8 days for the test group and 7.3 days for the control group) [1].

Of 2,220 video offers, 36.2% were accepted. Of the students whose video loaded and started, 60.1% watched at least 80% of it, the threshold we used to count a student as having watched [1]. Both figures were well above what we had expected.

At the CHECK question, students who had watched the video were about 2.6 times as likely to answer correctly as the control group: 70.7% against 27.6% [1].

The researchers had expected the video group might do less well after a gap than students who had reached the right answer through Tassomai’s usual feedback alone. That is not what happened. About a week later, 52.3% of the video group answered the same question correctly again, against 41.5% of the control group [1]. The video helped students get the question right in the moment, and more of them still had it right a week on.

The video group in the results is the students who chose to watch. Candidates were assigned to the test or control group at random, and test students who skipped the video or watched less than 80% of it were left out of the results [1].

Three more points from the paper. Almost half (46.5%) of each group never reached the RETENTION question, because their course ended for the summer exams before it came round. The six problem questions were all hard ones, which lower-attaining students rarely reached, so the results say little about easier questions or about those students. And the results combine all six videos, so they can’t show which videos worked best [1].

The useful idea here is timing. A short, focused explanation offered just before a student meets a question they have already got wrong gave them a much better chance of getting it right, and more of them still got it right a week later.

• Use the errors you already have. Wrong answers in homework and quizzes tell you which students to target before the next attempt, not only after it.

• Keep explanations short and specific. Each of our videos covered one idea in one minute to 90 seconds.

• Offer help rather than force it. More than a third of students accepted the video, and most who started it watched nearly all of it.

• Check again later. A correct answer straight after an explanation is not the same as remembering it a week later, so come back to the question.

After the study we committed to a roll-out of 150 targeted videos for the following school year [1], which we announced in May 2019.

If you would like to see how Tassomai targets practice at each student’s gaps, book a demo with our team. Schools that like what they see can start a free three-week trial.

1. Morrison, M., Blake, C., Embleton-Smith, F., Gosiewski, J., & Zvesper, J. (2021). Pre-emptive intervention and its effect on student attainment and retention. Research for All, 5(1), 36-51. https://doi.org/10.14324/RFA.05.1.05

2. Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249-255. https://doi.org/10.1111/j.1467-9280.2006.01693.x

3. Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4-58. https://doi.org/10.1177/1529100612453266

4. Pashler, H., Bain, P. M., Bottge, B. A., Graesser, A., Koedinger, K., McDaniel, M., & Metcalfe, J. (2007). Organizing instruction and study to improve student learning (NCER 2007-2004). National Center for Education Research, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/20072004.pdf

5. Murre, J. M. J., & Dros, J. (2015). Replication and analysis of Ebbinghaus’ forgetting curve. PLOS ONE, 10(7), e0120644. https://doi.org/10.1371/journal.pone.0120644

6. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81-112. https://doi.org/10.3102/003465430298487