This is post two of four about the SRI study about Khan Academy; post one can be found here. We are often asked: do online and blended learning work? As we have explained several times (e.g. in Keeping Pace 2011, pp 40-49), the question is more difficult than it might appear for at least two reasons.

First, defining “what works” is not easy. If “working” is defined as providing new opportunities to students, then the answer is undoubtedly yes. If it is defined as improving student outcomes in some cases, then again the answer is yes, as with students taking online Advanced Placement courses and in some programs exceeding national averages for AP exam scores. But if “working” is defined as raising student achievement across large scales, based on randomized control trials, the answer is that the research is limited and inconclusive.

The second reason that answering the “does it work” question is hard is because so many types of online and blended learning exist. The result is that comprehensive research is difficult and expensive, and applies to only a very small percentage of overall online and blended learning implementations.

The recent report from SRI looking at Khan Academy use in schools supports and amplifies the above points:

“SRI conducted an implementation study rather than an evaluation of Khan Academy’s impact on learning, given that implementation of Khan Academy varied across sites, Khan Academy tools and resources evolved over time, and schools, Khan Academy, and researchers collaborated regularly. Ongoing communications and tight relationships between the participating teachers, students, and school leaders with Khan Academy and SRI International resulted in both a more useful tool for teachers and students, and a better understanding of the various ways Khan Academy can be used in school settings to promote student learning.

…teachers and students were using Khan Academy tools and resources in considerably different ways across the nine study sites, and some of the sites also changed the ways they used it during the course of the two-year study. For these reasons, it was methodologically unsound to conduct a rigorous evaluation of Khan Academy’s impact on learning during the study period, including any use of randomized control trials, which would have required Khan Academy tools and resources to remain unchanged during the study and for teachers and students to use Khan Academy the same way. Moreover, at all but one of the sites, Khan Academy was principally used as a supplementary tool—not as the core primary curriculum—so the effects of Khan Academy cannot be separated from those contributed by other elements of the math curriculum.” (emphasis added)

While the above text may sound like the study did not yield useful results because it did not come to any conclusion about Khan’s impact on learning outcomes, the study does in fact contain practical information. We will review these findings in the next blog post.