
Jo Boaler is a professor of education at the Stanford Graduate School of Education, with an enormously influential body of work arguing that students learn math faster and more effectively through her “discovery”-based methods. Her work got Algebra removed from middle schools across the Bay Area.
It is some of the most incompetently or dishonestly conducted research I have seen in a decade as a journalist.
Take one example: A report she gave at the National Council of Teachers of Mathematics on the stunning success of her innovative new math curriculum at “Railside” (she did not disclose the name of the real school where the study took place). This was a poor, disadvantaged California school, where, she said, students adopting her curriculum rocketed ahead of students attending schools with traditional curricula.
When other researchers looked into her work — combing through every school in California to figure out which one “Railside” might be, so they could look at the performance data that Boaler had declined to share — they found that Boaler had compared the top two quartiles of students at “Railside” to the middle quartiles of students at the other schools; that “Railside” students were in fact dramatically underperforming students at the other schools on every single mathematical ability test conducted during the study period, except the one that Boaler highlighted in her presentation. And the one she did highlight was actually conducted on a population of students who weren’t even exposed to the innovative new curriculum.
They found that the “tests” Boaler used to evaluate whether students were succeeding generally:
Contained material two or three years below grade level.
Did not contain any significant Algebra 1 or Geometry material despite being for an Algebra 1 or Geometry class.
Had problems that were incorrectly graded.
Had no “predictive validity” for other measures of math performance like SAT scores.
There was simply no relationship between doing well on Boaler’s error-strewn test of basic math and having mastered the material that students were supposed to master. Furthermore, the paper claimed that Boaler’s tactics closed the mathematics performance gender gap, with girls scoring as well as boys, but performance on outside tests found the gender gap at “Railside” the same as everywhere else.
On a different occasion, Boaler claimed that a single four-week summer camp could give students several years of math performance gains. Her evidence, when people dug into it, was that she gave the same test at the start of the camp and at the end, and the students’ scores improved — but that, as other researchers pointed out, is probably just explained by the fact they had seen the exact same question only a few weeks earlier. These are cartoonishly bad standards for evidence.
I wish this were a critique specific to Jo Boaler, but it isn’t. Across the board, the state of education research is incredibly grim.
A recent analysis in Nature of 600 papers from 62 journals across the social and behavioral sciences looked at a simple measure of the health of a field: When a study is published, does the author share the data they used to reach their conclusions and the code for the analyses they ran? If they do, it is trivial for other researchers to directly check their work and reproduce their findings, as well as to run the same analysis on new data to check whether a result holds up in a new context.
Education, they found, was the field with the least data and code availability — the field where it was most common for the data and the analysis code to be unshared in the paper and unavailable if you asked for it. And even when it was available, it was never exactly correct.
Across all the social and behavioral sciences, they found 54% of published papers were precisely reproducible — you could repeat the author’s analysis precisely and get the same result. Among the education papers where they were eventually able to get enough data to check the results, they found zero results that were precisely reproducible — literally none of them.
Across all fields, about half of the results were “approximately” reproducible and nearly 75% of results were either precisely or approximately reproducible. So education results are markedly harder to check, and markedly less likely to be reproducible when checked.
When will the replication crisis fixes come for education?
“If we are to reverse the woeful performance of our students,” the authors of the article shredding Boaler’s “Railside” study observed, “It seems crucial that K-12 education research be subject to the same high standards as are the norm in medicine and the sciences. As a key step we believe that the analysis here shows the dangers of accepting the legitimacy of articles such as those mentioned above as long as the results cannot be independently studied and verified.”
That was in 2012, though, and nothing changed. Boaler claimed that she could not respond without clarifying what school “Railside” was — though many of the critiques had nothing to do with that. She instead went to the media, saying that she was experiencing “harassment and persecution” and characterized the reporting on the flaws in the “Railside” study as “bullying.” “They are stopping me from being able to do my job,” she told Inside Higher Ed. Her retort was the same when, 12 years later, a 100-page-long report detailed 52 occasions where she misrepresented citations in her work.
Over the last two decades, it has become clear that social science work from the second half of the 20th century often does not hold up at all. The statistical methods used simply weren’t good enough to support the conclusions reached. P-hacking — or running tons of analyses until you find one that is interesting to report on — was widespread. The rate of outright fraud was much higher than many had assumed.
In response, many disciplines changed how they practiced science. They started requiring scientists to share their code and data, which makes it much more difficult to commit fraud or to introduce honest errors into the literature. They started encouraging, and in some cases requiring, “preregistration” — announcing the analysis you plan to do before you do it, to cut down on p-hacking. Various efforts got underway to reward replication and combat publication bias (the tendency to not share results if they’re not interesting), and people developed new statistical toolsets to detect publication bias and understand the overall state of the literature.
The field of education did none of this. More than half of well-regarded psychology journals have adopted transparency and reporting requirements. Virtually all well-regarded economics journals recommend or require full sharing of the data and code that went into a paper. In education, only a small minority of journals even recommend code and data sharing, and none require it.
The fact that it’s normal to report on a school “confidentially,” without naming it, makes journals reluctant to require data sharing, and researchers almost never want to go to the extra effort to share their work if it’s not required or strongly encouraged.
It’s easy to say as an outsider to the field, but I think the idea of reporting on a school’s results “confidentially” just needs to go. Individual student results should be confidential, of course, but key information about school performance should be, and already is, public.
The norm that you can conceal at which school you conducted an intervention makes life easier for people who are committing fraud: no one can easily catch the fraud by calling up the school to ask if the data is accurate, or look at how it compares to other, publicly available data about the performance of the same students.
But the problems go much deeper than that. Studies that are comprehensive and well-designed enough to find meaningful results are generally large and expensive. If every researcher is trying to prove the merits of their own quixotic curriculum by convincing one school at a time to enroll in a pilot and try it, we’ll get what we currently have, which is a huge number of fairly low-quality studies of individual one-off interventions — none of which constitute convincing evidence because they simply don’t have enough statistical power.
Instead, it would be much better to have fewer, much higher-quality studies which look at the rollout of a new policy across a district or across many districts, conducted and analyzed according to the (much higher) standards for careful work from disciplines like economics.
But every researcher’s individual incentives run in the exact opposite direction — as long as the journals will continue publishing underpowered work.
Bring in the economists
I would be remiss to write about the ills plaguing education research without acknowledging that, as far as I can tell, a significant share of the problem is that PhDs and PhD students in education are worse at research than PhDs and PhD students in other fields. It’s well known that the GRE scores of would-be graduate students in education evaluation and research are lower than the GRE scores of would-be graduate students in several other fields, and they are especially weak in quantitative reasoning.
So it’s possible that in order to fix anything else, you simply need more competent and qualified people doing research in education. But as long as the field of education has low standards, it won’t attract very many researchers who are capable enough to instead work in a more respected field.
I have a solution here, though it won’t make me many friends: We need to convince more economists to do work on education, using modern econometrics and adhering to the standards of their own field, which has adopted norms to combat fraud, p-hacking, data concealment, and grossly underpowered studies. There are some; while Stanford brought us Jo Boaler, it also has applied economist Thomas Dee, whose education policy work I find valuable and interesting (and which, tellingly, cuts across ideological lines in its findings).
There would probably be more if there were more incentive to work in this space, both in the form of tantalizing data sets to analyze and districts eager to partner with qualified researchers to design their curriculum rollouts.
And I have another solution that will make me even fewer friends. A population that’s much, much easier to study at scale — and that is now large enough for all the statistical significance you’ll need — is homeschoolers. Every day, I see homeschool parents asking “which math and ELA curriculum should I use?” and adoption appears to often go off “what’s recommended in the comments on Facebook.”
There is potential for some very large-scale experiments where parents are encouraged to adopt different curricula (say, by offering different ones to them for free), which could be enormously useful in understanding which are the best of the publicly available programs. I am aware of private-sector efforts to research exactly this question — the academics should hop on board!
But really, the thing that fixed other social science fields when they became aware of the sorry state of their research was not “one weird trick,” whether it was preregistration or new data partnerships. It was a sensibility among everyone in the field that they could not allow this to keep happening, that the rate of fraud and error constituted an existential crisis for their discipline.
As long as no such sensibility exists in education, any good research that does happen will appear alongside the research of the likes of Jo Boaler, and any effort to criticize her for misrepresenting her results will be interpreted as either pointless unkindness or at best ordinary academic disagreement, which doesn’t require anyone to change anything.
Really, to fix education, education researchers would need to agree that it’s broken.
Recommended reading:
Is Mississippi cooking the books?
The most important story in K-12 education is that a handful of states — and not the ones most expect — have figured out the tricky business of teaching kids to read. Mississippi, Louisiana, Tennessee, and Alabama have each bucked nation…
"Some kids are just dumb" is not good education policy
The fact that ability is not equally distributed doesn’t actually tell us how much effort we can and should put into teaching kids with lower learning capacities.




A few corrective points to make here.
A few examples like Boaler or Hattie aside, most academic researchers are not writing curriculum. Those are by and large written by curriculum publishers and ed tech companies. I don’t think it’s right to claim that ed researchers are all pursuing their own quixotic curriculum.
School confidentiality isn’t just a “norm” it’s a requirement. IRB will reject studies that don’t obscure the schools. Journals won’t publish papers that don’t get IRB approval from the researcher’s institution. Journals often also have their own requirements to obscure school data.
Additionally, schools have to agree to let you do research there. You can’t just walk in and declare yourself to be conducting research. The schools where research takes place also want anonymity and often require it in order for research to take place.
Depending on the type of study, for example one that would require observing classrooms as part of evaluating a program, you might also have to get parents’ permission for each kid studied. Again, they often want anonymity.
In general, ethics tells us you can’t conduct research on people without their consent. While ed research may not be as intrusive or potentially directly harmful as, say, a clinical trial, you can harm children by subjecting them to a treatment that hinders their learning. Now, this should work to stop people like boaler but, really, what she’s doing is using a publisher to shortcut the research process. More on that in a sec.
It’s not the job of academic researchers to implement and enforce accountability laws, that should be done by the state education departments. If I’m evaluating whether a math curriculum is effective, the question of school accountability should not be addressed anyway, right? Because if the kids’ failure to achieve is the fault of the curriculum, it doesn’t make sense to then tell everyone in your paper that the school is bad. It’s not what you’re studying in the first place.
What all this has done is create a two-track system of “evidence”. On one side there’s ed research just kind of doing its own thing and nobody’s really looking at it for much guidance about effective teaching and learning. It’s slow, and like a lot of academic research, is often published for other researchers. It’s actually a low-status move to publish about educational practice. Far better to develop models and theories for other academicians, at least from the perspective of an academic career.
The other track is publishers commissioning their own proprietary research that seemingly always shows their curriculum or product is effective. iReady’s Curriculum Associates publisher is getting a lot of flack about this right now. But they don’t have to deal with any of the above ethical or institutional constraints so they can be more nimble, put consent for research in the terms of service or in the contracts they sign with schools, and then do whatever analysis they want with the data to puff up their efficacy. It’s research as sales.
None of this is defending the academic education side of things, but it’s important to report on the whole mess of education research, report on it accurately, and apply critiques consistently.
How is this NOT a colossal scandal?!?
"Among the education papers where they were eventually able to get enough data to check the results, they found zero results that were precisely reproducible — literally none of them."