Research vs policy in online learning

For those of us who have been in the K-12 digital learning field for a while, it’s especially useful to get out of the bubble and hear what others think about online and blended learning. Reports from the Brooking Institution are especially valuable because Brookings is centrist, non-partisan, and evidence-based. Although the recent study Online schooling: Who is harmed and who is helped? reviewed literature mostly about post-secondary education, it makes a particularly valuable distinction in the K-12 realm that should be required reading for people who believe that research should drive all policy.

The Brookings report discusses a study of 8th grade students in Maine and Vermont taking online algebra. The students attend schools that don’t offer algebra, so the alternative is for them to take a general 8th grade math course. Students in this alternative, face-to-face, general math course serve as the control group. The study found that “students taking the course online did substantially better on assessments of algebra knowledge at the end of eighth grade, scoring 0.4 standard deviations higher than students in the control group…The treated students were also twice as likely to complete advanced math courses in high school, completing at least Algebra II by tenth grade (26 percent in the control group vs. 51 percent in the treatment group).”

That finding is not very surprising. But the Brookings report goes on to make a distinction that is worth highlighting because it demonstrates the type of thinking that can hold back policy progress:

“Note that this study tested a mixed treatment: exposure to Algebra I in eighth grade and enrollment in an online course, relative to exposure to general math in eighth grade in a face-to-face course. We can’t tell which of these aspects of the treatment is producing the effect we observe. Treated students may have learned even more had they learned algebra in a face-to-face course, rather than online. From a scientific perspective, the findings are therefore a bit unsatisfying: we can’t separate these two channels of the treatment’s effect. From a policy perspective, however, the findings are quite satisfying: online math courses can provide a productive learning experience for academically proficient adolescents in eighth grade who otherwise would not have access in that grade to that content.”(Emphasis added.)

That paragraph states in a fairly wonkish way what perhaps should be obvious—but too often is not. Online courses often provide the student the only option to take a course because the student’s school does not offer that course. Therefore, the question “is it better than a face-to-face course” is immaterial. The result may be “unsatisfying” to researchers, but it certainly is satisfying to the students who take the course and educators who offer it, because there is no face-to-face course available to that student. If she wants to take Algebra, it’s an online course or nothing. This is the situation that course choice policies attempt to address.

The Brookings report then looks at a second study that finds that students in online credit recovery courses in Chicago public schools did substantially worse in terms of a final test, and earning credits, than students in a face-to-face credit recovery course. This finding raises the questions of whether and how the online credit recovery course could be improved, and suggests that they must be improved so that students are not receiving an inferior mode of instruction when a better one is available. This is the right application of the “is it as good or better” research question.

Sometimes the “it is better than f2f?” question makes sense, and sometimes it doesn’t. Policymakers and advocates need to understand this distinction. Most people in the bubble get it. But perhaps because of churn in education policy circles, there is a need to continue making this case.