Michael Hicks: Some surprising lessons from COVID learning loss – The Daily Reporter

Last month, my colleague Dagney Faulk and I published a study on COVID-related learning loss in Indiana schools (available at https://projects.cberdata.org). The results were surprising and largely positive – or, at least, more promising than I expected. The aim of this work was to better understand which factors contribute to learning loss. What we know so far has mostly been limited to simple descriptive statistics about changes in test scores. It’s a good start, but it can’t speak of correlation, let alone causation on learning loss. To do this, more math is needed.

In a perfect world, we would have detailed student data over time. In the absence of this, school-level data provide a fairly good basis for assessing the effects of COVID and school-level responses to the pandemic on learning loss. Our aim was to examine how children of different age groups passed the same standardized test before and after COVID. This approach, coupled with statistical modeling, dodges most of the well-known criticisms of standardized tests.

We looked at all public schools, grades 3-8 in 2019 and 2021, the great COVID disruption. During this period, the middle school saw pass rates on standardized math and English tests drop by more than 10%. Some schools have actually done better due to COVID, but the vast majority have not. A handful of schools have even seen a 50% drop in pass rates.

In the best-performing schools, most children failed math and English tests. At worst, about half of the classrooms would have no students who passed both tests. This is frightening data that potentially affects educational attainment and long-term economic growth. Just so no one arbitrarily dismisses the past 18 months, I’m willing to place a $1,000 bet that the learning loss of this age cohort will still be evident in the 2060 census. The only question is how much will this loss be and what compensating factors, such as resilience and courage, will replace the classroom competence of these students.

Our statistical models that test for learning loss allow us to measure each variable together. In this way, we control several differences at the same time. For example, in crude comparisons offered by the Indiana Department of Education last summer, African-American students suffered more learning losses, as did poor children. However, when we control for both race and poverty, the statistical significance of race disappears.

Another way to explain this is that two schools with different racial mixes but the same level of poverty experienced the same level of learning loss. Our study could not tell which aspect of poverty caused the learning loss, but there are many potential factors such as lack of broadband access for distance learning. This should be fertile ground for research for years to come.

Our second big finding is that schools that performed better on standardized tests in 2019 experienced greater learning loss from COVID. We believe this is due to specialized programs in higher performing schools that were not easily executed during COVID. There are many other plausible contributing factors, so it may take some time before causes are identified.

The rest of our findings were really “non-findings” in that most of the differences between schools that we could measure were not correlated with learning loss. Race and ethnicity did not play a role in learning loss, nor did the share of English learners. The type of school did not matter, whether it was elementary, intermediate or combined. The size of the school did not matter, nor did absenteeism throughout the year. There was some evidence that declining enrollment increased learning loss, but this is a small effect.

The big surprise was that the combination of teaching – in-person, online or hybrid – had no effect on learning loss. This differs from the raw numbers shared by the Department of Education, but again without controlling for other factors, these comparisons offer no useful interpretation. I think the explanation for this result is quite simple.

Hoosier schools, like many others across the country, have struggled with scheduling and education decisions throughout the 2020-2021 school year. There is no doubt that each superintendent and school board has struggled to juggle several different priorities such as health, learning and enrollment. But, ultimately, most decisions could be reduced to two trade-offs: learning loss due to online learning or learning loss due to quarantine and isolation. Here’s how it works.

Suppose schools that have chosen to go fully online minimize the spread of disease in the school, but maximize learning loss from online learning. Alternatively, schools that accepted the COVID risk and went entirely in-person minimized learning loss from online learning. However, in doing so, they would have suffered more learning losses due to quarantine and isolation of students and staff. Either way, there is a risk of learning loss. The trade-off was not between health and learning, but between two different types of learning loss.

This is where statistical modeling of this type is so badly needed to understand how these policies have affected learning. If, on average, Indiana schools misjudged this trade-off and spent too much time in online instruction, or too much time in person, that would show up in our statistical model. But if, on average, schools were effectively balancing instructional metrics throughout the year, no specific type of instructional form would be correlated with learning loss.

Of the dozens of statistical tests we performed, none indicated a correlation between learning loss and the teaching setting. This is a total and complete rejection of the scientific hypothesis that there is a correlation between these modes of instruction – in-person, online or hybrid – and learning loss in Indiana.

This is an important finding for schools, policy makers and taxpayers. The COVID pandemic has been a difficult time for schools. While I believe the Holcomb administration provided clear and consistent guidance, the CDC’s federal communications could hardly have been more confusing. Federal failures have helped fuel mistrust and frustration that have surely made educational decisions very difficult for school boards and superintendents.

There are undoubtedly many lessons to be learned from COVID, and some schools have done better than others. But with the data and analysis available on learning loss, schools in Indiana seem to have done about as well as they can. This should give the rest of us great confidence that they will tackle the problem of learning loss with the same good judgment.

Michael J. Hicks is director of the Center for Business and Economic Research and the George and Frances Ball Distinguished Professor of Economics at Ball State University’s Miller College of Business. His column appears in Indiana newspapers.