As schools all over Europe remain shuttered for the second time this winter because of the Covid pandemic, concerns are growing about the impact on children. In terms of well-being and mental health, these appear to be large, especially for adolescents and young adults. In terms of learning, our picture is still incomplete. There are very few studies from secondary or vocational education. One study of higher education found learning in college students holding up well, even in the face of demotivation and dissatisfaction brought about by being cloistered at home. Primary education has been the focus of several studies, and there the results have not been positive. Standardized tests administered after the first school closures of spring 2020 revealed substantial decrements in learning compared to previous years. But large variation in the results of individual schools suggest reasons for hope. Might there be recipes that prevent learning decrements? Here, I focus on one candidate: adaptive practice software and mathematics.

Learning losses in primary education

Four studies, from the Netherlands, US, Flanders and again the Netherlands, have now analyzed results from school closures in primary education. Two that focused on the Netherlands used nation-wide tests usually administered in June to determine students’ progress (so-called LVS tests). In June 2020, schools were just getting back on steam after the lockdown and a few weeks of teaching half-grade classes, so some skipped the tests. But other schools did administer them. Both studies found substantially lower result as compared to preceding years, with students performing, on average, three percentile points lower than they otherwise would have. These losses were not equally divided over students. Students whose parents had enjoyed less formal education were found to have about a 50% greater learning loss than other students. This is particularly worrisome. There were also large differences between schools: while some schools saw no losses, students in other schools suffered very large ones. The same was found in Flanders, where schools with many disadvantaged pupils were found to be more likely to show large decrements in learning.

Differences among schools

Differences between schools are cause for concern, but also reasons for hope. If some schools saw hardly any learning loss, might those schools have something to teach others? There were large differences in how schools approached teaching during the school closures. While in many schools distance education during the lockdowns was limited to emails detailing the work to be done at home, other schools organized substantial online teaching and contact between teachers and students. While we are not yet able to connect that to how much was learned, but it is already clear that schools are learning from their experiences and each other. In the case of the Netherlands, almost all Dutch schools have taught online at least part of the lockdowns.

Digital learning tools in math

Dutch primary schools hold another trump card. Many already worked extensively with educational software. For example, in many schools adaptive practice software plays a major role in math and language teaching. Such software was introduced a decade ago as extra practice on top of the normal teaching, but by now some of these programs have been restructured to be integrated in the normal flow of lessons and replace much of the paper-and-pencil exercising that happens in class. Such software can often also be accessed from home, which is a huge advantage during school closings.

Using data from one such provider, Snappet, I looked at how it was used in the first lockdown for math, and what the learning outcomes were. Snappet can be used for instruction, sample tasks, independent work and differentiation, for language and math. Each student typically works on his or her own, school-provided device (tablet or chromebook) on which the software is pre-installed. During school closures, elementary schools seem to have mostly handed out these devices to their students for home use: During the weeks of school closures, there were about the same number of users as the weeks before, with only a small dip (-15%) just after schools were unexpectedly ordered in March to close their doors (schools were told on a Thursday evening that they would not open on the Monday thereafter). In the weeks that followed, students used Snappet an average of 20% more than they had in other weeks.

And that had results. Students made not slower, but faster progress during the lockdown than their predecessors had made during the same period in the 2018-2019 school year. The figure below shows this surprising result. Each point represents a weekly average, by group and year, of math proficiency as estimated by Snappet (this estimate is used for calibrating exercise difficulty to the individual user). Grades 2 to 6 thus generate two lines of estimates for the 2018-19 and 2019-2020 school years, respectively. These are largely on top of one another for the first half of the year, but diverge in the weeks of school closings (where the orange points turn yellow): During those weeks, students in grades 2, 3 and 4 did better in Covid ‘19 year than in the preceding control year. I.e., they learned faster than they would have otherwise. These learning gains disappeared to some extent during the weeks when students returned to school (May and June, where the orange points move closer to blue again). Nevertheless, for groups 4 to 6 there was still a small gain at the end of the year.

In addition, the learning gains were found to be particularly large for students who were behind their classmates before the school closures. This is a very surprising result. In other studies, learning decrements were the same for strong and weaker students. Here, we found gains, and these were skewed towards weaker learners who caught up with their class mates.

In another aspect, the results do mirror those of preceding studies. In one study, students from disadvantaged backgrounds (not necessarily the ones with lower results beforehand) had larger learning decrements after the school closures. No such data on student background was available in the Snappet data, but schools could be categorized on their percentage of such students. While learning gains during the school closures were found for all schools, these gains were somewhat smaller for schools many disadvantaged students.

What do we learn from this?

These results are encouraging, but they also raise a number of questions. Were students only better at technical mathematical skills or did they also reach better conceptual understanding? Did the gains seen in the data translate into gains on standardized tests? Why did students who were behind benefit so strongly from the closures, and why do such students catch up to a lesser extent in normal years? Did Snappet also help prevent learning decrements in other areas such as spelling? What happened after schools reopened in May and June of 2020 that caused students’ gains to shrink again?

More research is needed to answer all these questions. For now, it seems that adaptive practice software like Snappet’s can help students learn at a high rate during school closings. That is reassuring knowledge at a moment that primary schools are shuttered again (or still) in many countries.

Know more?

A preprint describing the research in more detail can be found here.

Snappet’s data is not allowed to leave the company for privacy reasons. The research was therefore performed as follows: a research assistant and I wrote a script, which was then run by Snappet employees on data from 800 schools that had given permission for scientific research. This resulted in aggregated data that were then analyzed by me. I have no interest in Snappet, receive no research funds from them, and am free to publish about the research. An earlier version of the preprint was submitted to Snappet to check for factual inaccuracies (which indeed turned out to be there), but this did not result in changes in content.