Discussion about this post

User's avatar
Zac Hill's avatar

It feels like what a lot of people like about AlphaSchool is less that it’s proactively enacting some amazing set of outcomes, and more that it *lacks* a great deal of the self-imposed downsides of many other schools.

Matt Bateman's avatar

Alpha insider here. Going to confine my comments to the “2x learning is bogus” part of this critique.

As noted in footnote 1, my view is that there’s a more statistically precise way of saying this, a way that (a) virtually no one understands and (b) *is in fact very impressive*. A 1.4 to 1.5 sigma effect size, which is what I typically get when I run the numbers, is very large. So the worst-case scenario for 2x learning is that it’s just an incorrect description of what it is in fact an impressive effect.

But how imprecise is “2x learning” exactly? Putting aside the standard deviation for a moment (I’ll return to it), and using my 6yo as an example, the following things are true: she scored a 179 RIT on the fall 2025 math MAP. She scored a 222 RIT on the spring 2026 math MAP. The median student scoring a 179 on the fall test gains 15 points over the year. My daughter gained 43 points.

Moving beyond what’s indisputably true: I don’t think my daughter would have had 43 points of gains in a typical school, or even another good Montessori school. I think it would have been much closer to 15. (If you believe something like this, almost all the other discussion is irrelevant, and it answers the question as to why Alpha is beloved to many parents.)

Alpha markets that to me (in my capacity as a parent) as 2.9x learning. That seems... fine to me? It’s not what I would publish for data wonks, but it gives an impression that is correct: she outperformed, by a lot.

Does that mean that she learned 3 years worth of material? Not necessarily. I don’t think Alpha has ever said that. It’s presented to parents as a testing target. *Grade levels gains are reported separately and clearly parents.* (Incidentally, in fact, she did, almost. She went from mid-K material to 3rd grade material. But this doesn’t always line up with the RIT multiples.)

Returning to the shape of the data. So the median yearlong score improvement is 15 points for a 179 math MAP. That the median is 15 doesn’t tell you anything about how likely it is that a student scores a 43; if the curve is relatively flat, it could be about as common as any other result. We need to know the standard deviation to estimate the effect size (e.g. (gain - median_gain) / sd). I can basically look that up (inferring from some other data that MAP has published). It’s 8.5. (43 - 15) / 8.5 == 3.3. 3.3 is an insane effect.

There are all sorts of reasons one might want to tweak or dispute that, or use slightly different measures, but you’re going to end up in the same ballpark. It’s going to be a big effect, because it is one.

Does this happen every single time you compute the results of 2x (or 3x or...) learning? Not exactly in this way. The SD isn’t always the same across ages and subjects. But this *is* the basic shape of the effect. As far as I can tell, having run this exercise across many contexts: it never turns out that the range is such that a 2x effect is basically nothing to write home about.

What about outlier cases, where you get, like, 20x learning? Again, this happened to my 6yo. Her reading last year went from 201 to 222 fall to winter. MAP exepcted her to gain 1 point; she gained 21. Is that “21x learning”?

Well, I mean... what do we make of the fact that her score might have gone up by 1 in a typical school? She’s a gifted reader; gifted readers are typically not challenged in most Kindergartens, and the Alpha platform absolutely challenged her every single day. It is entirely plausible to me that if I enrolled her in either a decent public school or a decent Montessori elementary program (two of my benchmarks for Montessorium, the Alpha Montessori program she is enrolled in), she would not have been challenged.

To be clear, I wouldn’t put 21x learning on a billboard. But I don’t think it’s fair to say that the numbers are meaningless. (If you run the same math as I just did with math with derived SDs from public MAP data, you get a 2.3 sigma effect. Not incidentally, you are basically breaking the models at this point, which are not designed to be informative about this kind of gain, from 99 point something to a higher 99 point something.)

As far as probably overhyped and/or imprecise marketing claims go, Alpha’s seems minimally objectionable. And the minimum objection is not that it’s *misleading*, not that there’s some sort of false promise embedded in it, but that the effects are quite real but confusingly described for people who want to understand it precisely. And they are confusing because the actual statistical truth is objectively hard to communicate; our marketing department has wisely opted to not communicate in sigmas.

25 more comments...

No posts

Ready for more?