A few examples like Boaler or Hattie aside, most academic researchers are not writing curriculum. Those are by and large written by curriculum publishers and ed tech companies. I don’t think it’s right to claim that ed researchers are all pursuing their own quixotic curriculum.
School confidentiality isn’t just a “norm” it’s a requirement. IRB will reject studies that don’t obscure the schools. Journals won’t publish papers that don’t get IRB approval from the researcher’s institution. Journals often also have their own requirements to obscure school data.
Additionally, schools have to agree to let you do research there. You can’t just walk in and declare yourself to be conducting research. The schools where research takes place also want anonymity and often require it in order for research to take place.
Depending on the type of study, for example one that would require observing classrooms as part of evaluating a program, you might also have to get parents’ permission for each kid studied. Again, they often want anonymity.
In general, ethics tells us you can’t conduct research on people without their consent. While ed research may not be as intrusive or potentially directly harmful as, say, a clinical trial, you can harm children by subjecting them to a treatment that hinders their learning. Now, this should work to stop people like boaler but, really, what she’s doing is using a publisher to shortcut the research process. More on that in a sec.
It’s not the job of academic researchers to implement and enforce accountability laws, that should be done by the state education departments. If I’m evaluating whether a math curriculum is effective, the question of school accountability should not be addressed anyway, right? Because if the kids’ failure to achieve is the fault of the curriculum, it doesn’t make sense to then tell everyone in your paper that the school is bad. It’s not what you’re studying in the first place.
What all this has done is create a two-track system of “evidence”. On one side there’s ed research just kind of doing its own thing and nobody’s really looking at it for much guidance about effective teaching and learning. It’s slow, and like a lot of academic research, is often published for other researchers. It’s actually a low-status move to publish about educational practice. Far better to develop models and theories for other academicians, at least from the perspective of an academic career.
The other track is publishers commissioning their own proprietary research that seemingly always shows their curriculum or product is effective. iReady’s Curriculum Associates publisher is getting a lot of flack about this right now. But they don’t have to deal with any of the above ethical or institutional constraints so they can be more nimble, put consent for research in the terms of service or in the contracts they sign with schools, and then do whatever analysis they want with the data to puff up their efficacy. It’s research as sales.
None of this is defending the academic education side of things, but it’s important to report on the whole mess of education research, report on it accurately, and apply critiques consistently.
The IRB standard that you cannot identify the school where you conducted research is absurd. It's wrong. It is not closely connected to preventing any specific harm, and it enables a ton of harms. If it were not the preexisting standard, no one would invent it, and it's not a similar requirement in other fields - you aim at anonymity of individuals, not of the institutions that are supposed to serve them! I do understand that researchers can't unilaterally change it, but that's just a question of "who is going to have to change their behavior for us to get good education research".
I agree wholeheartedly that the commercial research side of things is atrocious. That's true in every field. It's not good, but I think if the academic research were good it wouldn't be a huge problem - people ignore low-quality commercial studies if there's reliable research to turn to instead.
I have wondered how some research projects got past the IRB. A high poverty school district in NC (see I am not naming them out of habit--okay, it was Durham) had a foundation that wanted to fund a literacy project but they required a proposal. They asked me to help them write the proposal. A Harvard education group was doing a research project in the district and they wanted to include schools that were not part of that project. They had so much turnover in their literacy department that no one knew which schools were part of that research project, so they called the PI. They were intimidated by talking to the PI of the research, so they asked me to sit in on a call. The guy tried to intimidate me (but that is hard to do). He couldn't tell us which schools. He said he first had one set of schools. He had randomly selected kids in those schools to receive a book sent to their home. The kids in those schools that didn't get books were the control group. They were researching whether poor kids who get a book sent to their home would have greater reading gains than those who didn't. Then they discovered that some of the kids in the control group were siblings of kids in the treatment group and lived in the same house that got a book. So, they were now using other schools as control. They hadn't pinned them down so they wanted all other schools to not get any literacy interventions. Then, NC passed legislation called Read to Achieve, which called for kids who were not reading at grade level to continue to get reading instruction in the summer after 3rd grade and in 4th grade. The legislation specifically said "unless they were part of this research project." The project started in Durham but then grew to other districts. Just like the foundation funds that would have ruined the control group, continuing literacy support would ruin this research project. I wondered how this ever got an IRB.
There's a bit of a Chesterton's Fence thing going on here. Maybe the standards are absurd or wrong today but they didn't occur in a vacuum and exist for a variety of historical reasons that we might expect a journalist to look into. It also simply not true that institutions aren't anonymized in other fields. Medical research papers don't usually report all the participating hospitals for example. I have no idea if Northside Hospital in Atlanta was involved in the research that discovered the BRCA-2 gene's role in breast cancer. That work took place in dozens of hospitals worldwide and it's not clear to me why listing the participating hospitals in the published paper would improve that research. Likewise, I'm not sure that listing the schools in which research studies took place would make the research better. I think one important reason for this is to separate the hard work of program/curriculum/intervention evaluation from the hard work of school accountability. Doing both simultaneously would obscure the underlying reality, not illuminate it.
This post would give you details on why the current IRB system exists. The relevant section is:
"[In 1998], a Johns Hopkins doctor tested a new asthma treatment. A patient got sick and died. Fingers were pointed. Congress got involved. Grandstanding Congressmen competed to look Tough On Scientific Misconduct by yelling at Gary Ellis, head of the Office For Protection From Research Risks. They made it clear that he had to get tougher or get fired.
In order to look tough, he shut down every study at Johns Hopkins, a measure so severe it was called “the institutional death penalty”. Then he did the same thing (or various lesser penalties) at a dozen or so other leading research centers, often for trivial infractions. Duke got the axe because its IRB hadn’t properly documented whether a quorum of members was present at each meeting. Virginia University got the axe because, although it had asked patients for consents, it hadn’t asked the patient’s family members, and one family member complained that asking the patient for a family history was a violation of his privacy."
Not sure where you got the idea the medical research papers don’t report their participating hospitals, they absolutely do. Beyond that, medical studies manage to distribute their research over many facilities in multiple regions to account for regional bias, something education research rarely manages which makes it even more important to know in which schools the research takes place. There’s a high propensity for bias and confounding variables so we need to be able to dig into the data.
A school has to adopt some curriculum of some sort. It makes perfect sense, if there are a variety of different approaches, to keep track a d try to figure out what works. Obviously, Ted and Alice shouldn’t be outed, but if School A adopts Fuzzy Progressive Math and School B adopts Hardcore Drill Math, it makes sense to figure out what works, and it’s in the public interest to do so. It even benefits Ted and Alice!
“Research as sales” is so well-said. I appreciate your caveats, but it must be said: no one is working very hard to create transparency about what is used in schools, to enable better research. I have called for a national curriculum database to enable better research, and this plea has broadly fallen flat. By all means help me change that!
I'd like to see something bigger happen with the What Works Clearinghouse. The Federal level seems like the perfect place for a system of rigorous evaluation that states and districts could rely on, as opposed to seemingly gameable third parties like EdReports (you've been rightly throwing shade in their direction).
Academia is ill-equipped to do real research in some ways. IRBs are too strict in a way that inhibits real research. It’s going to be very interesting when China surpasses the US completely in biomedical research due to their effectively nonexistent privacy controls and IRB ethics standards - do we cede leadership in pharma trials etc entirely, or do we try to find a way to actually allow research while still having real ethical standards rather than bureaucratic pseudo-standards?
These requirements are often downstream from other higher-level requirements. The NIH, for example, has very strict rules about human subjects research and doesn't fund research that violates those rules. Other grant granting organizations require the same. We can wish it was different but nobody's going to bootstrap their own clinical trial or conduct a massive educational intervention without getting lots of money and support from some kind of grant. The ones with the money make the rules. It's easy for the Kelsey Pipers of the world to wish it were not so, but that doesn't make it any less of a reality. So long as the Feds, the Universities, the Journals, the school districts and the schools all agree that anonymity is an important ethical point, there's going to be anonymity. Complain all you want about university researchers but they're not the reason Jo Boaler anonymized the schools in her research. If you or Piper want it to be different, perhaps lobby a senator or someone who can change policy?
I agree entirely! Which is why I suspect the real change is going to be driven by China. Pharma research is already rapidly shifting over there, and decoupling in pharma will not be received well if it means American cancer patients are missing out on the most effective chemotherapeutics. So at some level I suspect our hand will be forced.
Kelsey is very Silicon Valley which comes with its own inherent inability to understand more regulated sectors (academia, medicine, government). I still like her and find her worth reading, it’s just an inherent blind spot for Silicon Valley types for some reason. Which is unfortunate for those of us who are more progress minded in these fields, since Silicon Valley types could actually be useful allies if they made an attempt to understand why universities might want privacy protections in research and where there might be opportunities for reform etc, instead of just writing all of it off as useless
I think the best thing about Kelsey's ed writing is that she has a positive-sum view of our education system even though she's very critical of the education system overall. I can't say the same about most reform-minded pundits, journalists, or advocates. I'll always click and usually comment on her education writing as a result.
Also the “publish 300 individual studies” academic is going to be a full professor and the person who takes 5 years to publish one super high quality large study is never going to get promoted past assistant professor. And this is fundamentally a problem of incentives - huge incentives to publish, very limited incentives to do a difficult research study that will take years to set up and do properly with the limited resources available to a typical academic faculty in these fields.
Sorry should have specified- modern Western academia is ill-equipped to do research involving human subjects at scale because bureaucratic hurdles are excessive. I don’t think we should go back to the 1920s where research participants don’t have rights. But Kelsey is right that entire schools being anonymous is a somewhat unrealistic standard, and I mean to say that IRB requirements sometimes go beyond anything a normal person would expect ethically and just create a proliferation of stupid paperwork.
It’s not that they’re infallible rules, but we shouldn’t pretend they can just be wiped away or that they served no purpose. Nor should we ignore the other institutional and funding constraints attached to anonymity requirements. I get that we want to pretend there are easy answers and one quick fix will somehow make ed research good, but the real word doesn’t work that way. You can’t just object to things and refuse to participate and expect those same institutions and journals and all that to publish your work or platform you at conferences. Beyond that, I suspect the larger problems Piper has with education research will not be solved by ending anonymity requirements anyway. While they may have helped in the case of Jo Boaler’s work, I’m not convinced that one neat trick is going to work here.
Well now at least I understand how Lucy Caulkins was able to advocate for teaching kids to read like someone who can’t read and be taken seriously for so long.
I have been evaluating grant-funded education programs for 30+ years. I used to work for a school system doing this and then started a small business to do this. I started seeing Lucy Caulkins being used decades ago. It was expensive so schools would write it into grants and then get trained, certified, etc. using the grant funds. I've observed hundreds of grant-funded training sessions. My staff and I could hardly believe that people fell for the snake oil. That was one we thought simply couldn't be real, but there are many many others. My staff would often say that we are in the wrong business. We should be selling snake oil to schools. We noticed the pattern that the schools like the stuff where the training is fun, it makes you feel good, and the theory is that you don't have to work very hard to get the expected results. And what are the expected results? There is training for that, too. Don't expect poor or minority kids to learn anything because they aren't cut out for it. So, when none of this stuff works, that is because nothing was going to work anyway. And you get to pay staff with grant funds, buy things, and pay me to evaluate it. We don't think anyone ever read our evaluation reports. And why can't poor kids learn? They hang their paintings too high and talk with a different speech register. The research is "qualitative."
The weird thing about Readers Workshop was that it really seems to be more work for teachers. In conventional reading instruction as I recall it you walk through letter combos and sounds week by week and have kids read appropriate texts aloud. Most of them will just get it. A few will struggle and need multiple cues. In Readers Workshop you just have the kids try to read, they will all struggle because you haven’t really taught them anything, and you have to spend an eternity offering cues that generally don’t encourage actual decoding of the text. Surely it’s harder?
A big part of what we would observe was that you were to make the atmosphere "feel" like a reading place. Teachers liked that. We didn't see hard work offering clues. These grants would almost always be coupled with the Poverty Training that lowers teachers expectations by teaching them why poor kids can't learn. Most of the grants we evaluated were targeting poor kids. (A lot of profiting from poverty going on.) So, you teach this way and no one learns but that is to be expected. It probably would be harder work, and no results, but if you expect no results there is no reason to work hard. Just make the kids feel good if you can.
I taught my neighbor how to read a few years ago. She was in 3rd grade and couldn't say her Ls. NC had adopted a phonics-based curriculum and teachers were taught that students should not move forward until they have mastered step 1, which is sounds of the ABCs. Her school had pulled her for a special program (which I have evaluated and it is snake oil--expensive snake oil) since Kindergarten. Her parents asked me to work with her.
There was nothing wrong with this girl. I was a math teacher so I had to just use common sense on how to teach her to read. It was hard work for both of us. She is a great reader now, and a straight A student in sixth grade. She learned pretty quickly. She had been pulled from core instruction for 3 years to get this special training where she would just say the sound of the letter that popped up on an iPad.
I think some kind of database like that is great! What's different is that in the published studies, you don't usually get the hospitals themselves listed and the goal of the study isn't to hold the hospital accountable for poor patient outcomes, it's to evaluate the medicine/treatment/procedure/whatever. Like, if Hospital A participates in a clinical trial and it fails to show efficacy, we don't point at that hospital and say it's bad. We say the treatment wasn't effective.
[edit: adding the next part as the comment was truncated and I didn't see the second half]
My point in the op wasn't to defend the practice so much as explain that it's far reaching and exists vertically throughout the education research system. Everyone from the feds on down to parents of the kids being studied has rules requiring consent, anonymity, and efforts to minimize harms. They exist and pinning all this on academic researchers is missing the larger systemic context. You may disagree that kids safety is at risk but that is, in fact, what the rules are there for. Go take a CITI training and you'll have hours of fun learning about protecting human subjects, even in non-medical research contexts.
I don’t think breaking anonymity is automatically going to be blaming the school. It’s just how Piper made that connection. If we’re talking about replication and researcher accountability, I see no problem. When you start to connect it to accountability and student performance, it feels like crossing the streams.
All that said, nothing I’ve written is an overall objection to Piper’s article. Ed research is a mess, probably not uniquely so but certainly a mess. The incentives are all screwed up and one of the most widely understood problems is that research (good or not) rarely makes it into schools and into the everyday practice of educators. You’d think this would drive researchers crazy but it kind of functions as a way to give them license to ignore schools entirely. Just do studies in labs or with undergrads and publish it!
Anyway, that’s a whole other rant but I think that’s also why I came away wanting Piper to dig deeper and get to more of the systemic issues behind low quality research.
"Among the education papers where they were eventually able to get enough data to check the results, they found zero results that were precisely reproducible — literally none of them."
Years ago I did a literature review on the quality of math education research. I found an article that had reviewed a sample of math education research articles that had been published in peer reviewed journals, and fewer than 10% of them had used valid methods. I wasn't surprised. When I worked for the research department of a large school system, our department head was the head of Division H of AERA. This is the Research division of American Education Research Association. Research papers would be sent to her for applying to present at the big conference. We would read and review them. Nearly all would be unbelievably bad. I was new to the education research field. I couldn't believe it then. Now, I expect things to be that bad.
Related: I’ve read a number of comments of yours in this thread, and it would be wonderful to read more full posts from you given your background and experience.
Because this happens all the time. Usually not this severely, which is why it's the example for this post. But the public hugely overestimates the rigor and reliability of research in most fields.
I think that the majority of people who are aware of the reproducibility crisis often go the other way though, and vastly *underestimate* the rigor and reliability of research in lots of fields. Kelsey says that social science research from the second half of the 20th century often doesn’t replicate - but the first half of the 20th century is worse at comparable questions (they often didn’t even attempt the questions that would need more data) and empirical research in the 19th century is just non-existent in most subjects. The physical and biomedical sciences went through the same transitions a few decades earlier, but they’re all improving, and even the best fields can’t help but have lots of individual papers that are just plain wrong.
Precise reproduction seems like a very high bar. Everything should be the same down to many decimal points. This is achievable and should be encouraged, but it's not trivial to achieve.
For example some statistical analyses involve numerical computations. If you use a slightly different version of the software, or even a different type of processor, the outputs may change slightly by like 0.00001 or something. This would violate precise reproducibility as far as I can tell based on how they define it in that Nature paper.
0 papers meeting this standard doesn't reflect well on the field though, especially if their statistics are relatively simple.
This is a pretty good point in general and I wonder whether reproducibility is pretty understated in fields that require relatively high precision calculations.
That said, I think (but am not confident) that most ed research does not fall into this category
I’ll start with the disclaimer that this is a fairly uninformed hot take, but I wonder if part of the problem is that the education Ph.D. serves two very different populations with very different purposes. One is students who will go on to academic careers and research. The other, probably much larger group, is teachers who are looking to advance into the ranks of administration. Seems like trying to serve both purposes would make each one worse.
Yes, this crossed my mind. In my state, an advanced degree comes with a baked in raise, and a doctorate commands the most. Rest assured, there's a lot of research done solely for credentialing.
But I also think the reasonable best case for the current education research paradigm is not that good.
1. Randomization in education is extremely hard.
2. Controls vs. treatment almost always have to share a ton of properties that I, at least, hate and consider a big part of the problem(s).
3. We (humanity) are quite bad at measuring many of the outcomes I think are important in a way that satisfied social science research.
So all of this *and* we should surrender to the need to look for different research paradigms and more generally be prepared to make good judgments about education on the basis of something other than research.
Economists will save us. Ironically, Jo Boaler was warmly embraced by the Freakonomics folk. It was three mathematicians who took the heat (with visits from the police, no less) for challenging the Railside study.
1. Economists can't change the biggest problem - it's extremely difficult to achieve true randomization when students are clustered into classes no randomly, which is pragmatically always going to be the case and which is often effectively mandated by racial representation regulations. You have an ebook textbook and a paper textbook and your null hypothesis is that students perform the same regardless of which they use. To test this you need to randomly assign students to condition. But students are already nonrandomly assigned to classes. You can flip a coin to decide which of two classes gets which textbook, but you've just broken randomization. You could do some sort of multi level modeling to compensate for it, but it's not ideal and it's The trouble is that there's often no practical way to divide students up for research that's remotely conducive to actual teaching; you end up telling teachers to use one lesson plan with one set of kids and a different plan with another, or different materials, etc. And then there's the parents; depending on context and situation, you have to get their permission, and if they get the impression that their kid is getting worse material or teaching than others.... It's not easy and I don't think just getting a different set of researchers is going to change. I invite you to try it yourself though.
2. You have to accept when research tells you things you aren't going to like, such as some Mississippi "miracle" research that is in preprint right now. You're not gonna like it!
Right and then midway into your textbook example, one of the participating schools pulls out of the study because some unrelated initiative means they’re adopting a whole new curriculum and won’t use either the eText or the textbook anymore. Ask me how I know!
We’re PhD students in Economics and Psychology at Stanford’s Graduate School of Education and we felt obligated to respond to this article. Much of what is written here echoes both our training and our own struggles with academia itself and science as a practice. We agree that we should be critical of preexisting claims and that we shouldn’t make strong arguments based on the results from a single article. Credibility and public trust in research should start with the methodology and researchers themselves, and it’s as important to use good methodology in the first place as it is important to call out bad science when it happens. We also agree that the existing standards of academia and research sometimes make it even more difficult for scientists to do high-quality, reproducible work. However, this article makes incredibly strong claims against the field of education at a time when public trust and investment in research is already low. We think the argument here is too generalized and lacks an understanding of the interdisciplinary nature of education as a field to begin with.
As other commenters have pointed out, education is a nebulous field with broad aims and purposes. Within the Graduate School of Education at Stanford, as at many schools of education, you will find researchers from many disciplines who vary widely in not just how they study education, but what they think the purpose of education schools should be. Education schools produce teachers and people who train teachers and people who write curriculum, but they also produce researchers who study children’s brain development and people who study education policy or the intersections of education with language or our political system or technology. Sometimes these researchers might look more like those who work in other disciplines. Some economists, for example, are trained in or work at schools of education but don’t exclusively publish in education journals. Tom Dee is a great example—his articles appear in economics, policy, and education journals. There are many other researchers (not just economists!) at Stanford, and other universities, who do the same. There are also researchers within other departments who study education but don’t publish in education journals at all. The Nature article cited here isn’t accounting for the many, many economists, sociologists, and psychologists who are doing excellent work in education and who are publishing outside of it. If we only look at work published in education journals, we are missing a large body of high-quality education research. This isn’t to say that we shouldn’t hold education research and journal quality to a high standard, but to generalize based on the evidence presented in this article misrepresents the complexity, interdisciplinarity, and true quality of education research.
Of course, bad work exists both within and outside of education and harms both students and public trust in science. This article is right that we should call it out and that we should criticize disciplines, journals, and individual researchers that handwave over quality issues. But low-quality work should not define the discipline. This article doesn’t spend enough time acknowledging that high-quality education research already exists and already meets the ideal standards outlined here. Someone who is not familiar with the world of education research might walk away from this article thinking that almost nobody is doing good science in education, and that researchers who are competent in quantitative methodology are working exclusively elsewhere. This is simply not true.
Furthermore, good science needs a good toolkit but these tools don’t have to come from one discipline. We think Drew Bailey puts this well. Economists can add a lot to the conversation, and should be part of it, but they can’t answer everything. Every researcher benefits when they can learn what scientists in other disciplines do well, and when their own methodology might not be the best tool for a given problem. Ironically, graduate schools of education are probably among the places best situated to do this—it is so rare in academia to get people from different disciplines in one room in conversation with each other. As students in a school of education, we do it every day. This article undermines the credibility of education research at a time when public trust and investment is already shaky and while we acknowledge that improving credibility can play a role in building that back, we urge caution in broadly generalizing.
I think that this comment is missing the point, or at least a large part of the point. In spite of the article's headline, Piper is clearly criticizing education research as done by professors of education in schools of education, and as published in education journals. She's not criticizing education research writ large.
If you doubt it, just remember her praise of Tom Dee. Or consider her contrast of education journals to the journals of other fields—which, of course, also publish research on education.
To be fair, Tom Dee is a professor of education in a school of education who publishes in education journals. He's also an economist. Education research writ large very much includes many people like him who do high-quality work within schools of education. I suppose that this article could have more carefully defined the parts of education research it (validly) aimed to criticize, including in its title.
I really enjoy your writing about education. Always sharp. I also agree that getting researchers from other fields interested in education would be valuable, but for very different reasons.
The truth is the quality of research in most fields is really poor, and declining.
The majority of papers in medical research that aren't RCTs are completely worthless, and for RCTs you have to ignore anything done in many large countries. Most economics papers - a field with a weirdly high reputation for rigor - use slight of hand to get to their causal claims. In my own, hard science field, most of the splashy papers have problems severe enough that I don't believe their conclusions.
But! While all these fields have their endemic problems, the key is *they are different problems.* The commonly accepted flaws of medical research will be obvious to an economist. The weak arguments in economics will stand out to a biologist. So not only do I think researching across disciplines is beneficial, I think peer review should include someone from a completely different field, as long as they have the basic technical chops for the subject. (A psychologist is just not going to be able to review a physics paper.)
“ the quality of research in most fields is really poor, and declining.”
I’ll grant “poor” - everything is far from perfect, and so everything counts as poor if you set the bar high enough. My thought is that it would be great to hit higher bars, but it’s valuable that we are at least hitting the bars we are hitting now.
But I want to dispute “declining” I can only understand this claim if you just mean there are more people doing research in every field every year, and the median is going down, right? Or do you actually mean to say that research published in the top five specialty journals in any given field is lower quality than the research published in the top five specialty journals in that field a decade or two ago?
Because it seems to me that in every field I’m aware of, things have basically always been generally improving over all, even though there are always little fads that make one pocket of research here or there problematic in new ways.
1. The quality of the median peer reviewed publication in fields I'm familiar with is declining. I'm pretty confident in this because the volume of papers has grown exponentially while the pace of discovery has slowed.
2. Possibly there is a core of load-bearing papers where quality is improving. But it is hard to know because heuristics like journal prestige or citations have at best a very thin relationship to reliability
3. It is really hard to fix these problems, because it is impossible to define a verifiable procedure that stops them and the incentives are stacked on the other side. If you solve p-hacking, you move immediately to the problem of null mispecification, which has the same end result. And adhering to sound practices is a constant battle. In my workplace, where I would say the standards are quite high, we are currently fighting over whether to release our negative results and whether we can ignore our false discovery estimates for things that look "compelling."
4. Completely agree that the general public mistrusts the wrong things about science. In particular the official recommendations of the US Gov were, until last year, reliably based on convincing, well-conducted research. The papers people find while doing "their own research" are almost always straight garbage.
Yeah, this is probably much more difficult in much bigger fields. In philosophy, where our big annual conferences (for some reason we have three!) tend to have something like 1,000-2,000 attendees, and there's only a couple dozen major generalist journals and a few specialty journals you have to pay attention to in various sub-fields, and there aren't many new journals every year, it's still possible to keep tabs on a lot of what's going on. It's also relevant that what matters is new ideas and arguments (basically none of which should be *trusted*, but should be engaged with), rather than verifiable empirical data that you might need to trust.
You knocked this out of the park, thank you! Also… I have been hopping up and down, trying to get people to support a national curriculum database, specifically to enable better research insights. You’d think this would attract lots of support and attention. You’d think wrong.
Also relevant that the 2012 critique referenced has only been cited 14 times. Which seems to be evidence either that there are some issues with that report or that education researchers are profoundly uncurious about this critique.
The homeschool idea is interesting, but seems to sacrifice any external validity in an effort to improve internal validity
A great article, and a case well argued. Another angle you might find interesting is what education researchers are currently devoting their attention to. It’s not just that much of the research is poorly conducted, but that it’s not even focused on effectiveness at all.
If you look at the AERA website, you will quickly realize there is very little attention being devoted to empirical studies of teaching and learning.
The 2026 Annual Meeting theme was "Unforgetting Histories and Imagining Futures: Constructing a New Vision for Education Research” to give you an idea.
1. Econs already do quite a bit of education research and have for at least 40 years. It’s certainly way better than a lot of the stuff highlighted here, but it hasn’t saved us yet.
2. The thing about ed research is that it’s very hard . As you almost point out, this is a poor match for an overall less smart and quantitative group. This is likely one reason Econs don’t do more of it. It’s just really hard to draw substantive conclusions.
3. There’s a lot of fraud. We should just say this. Based on what’s here, it’s very like that Jo Boaler’s research is fraud. The left (I consider myself a part of this group, at least loosely) needs to police itself better. Who’s doing more harm to poor kids of color than people who commit academic fraud to get advanced math classes removed from public schools? This is a serious secular sin.
I'm happy to see that this topic is being covered here at The Argument.
However, outside the progressive bubble, conservatives have known for many decades now that schools of education and education "research" are cesspools of incompetence and ideologically-driven insanity.
If progressives actually care about governing and earning the public trust, disestablishing frauds like Jo Boaler (who is actually completely typical for a professor of education, being notable only in attaining a greater degree of influence for her fraudulent snake oil than most of her low-IQ colleagues) would be one of many places to start.
And, if progressives actually want public schools to be broadly respected, effective public institutions, rather than asylums of ideological lunacy, reining in this sort of idiocy would also be an essential first step.
Anyone reading City Journal for the last couple of decades has known that schools of education need to be at the top of the list for defunding and liquidation...
I agree with so much of what you wrote here: fewer, larger, well-powered studies would be better than the status quo; lots of educational research is really bad; sometimes confidentiality concerns hinder reproducibility and allow questionable research practices to thrive. Probably my favorite thing about your post is the observation that it was really the "existential crisis" that made the replication crisis lessons sticky in some fields.
A non-trivial quibble though: while I agree economists are smart and do good work and are going to be part of the solution, the replication crisis was NOT centered in econ, and they will not solve this problem. They will help solve other problems (demanding causally informative designs, open code, more open data). But selecting reporting/p hacking are major problems in econ. This is not just because the replication crisis was centered in a different discipline (psychology), but also because the solutions that worked for psychology (including preregistration) are harder to adapt to the kinds of quasi-experimental designs that dominate in applied micro, and require a lot of data-dependent decisions. Tom Dee has a new paper about this. See:
(e.g.: IV, in contrast, has a large number of researcher degrees of freedom, tends to be difficult to publish with null results, and, as an observational research design, is not usually accompanied by a pre-analysis plan. IV estimates also have a tendency to be enormous (Card, 2001). A study of finance papers found that two-thirds were instances in which, according to theory, OLS is expected to be biased away from zero (Jiang, 2017). Among these, 80% had IV
estimates even larger than OLS. Even after winsorizing at 1%, the average IV estimate in
this category was nine times larger than OLS. A study in political economy found similarly:
IV estimates were 12.4 times larger than the OLS estimates from the same paper (Lal et al.,
A few corrective points to make here.
A few examples like Boaler or Hattie aside, most academic researchers are not writing curriculum. Those are by and large written by curriculum publishers and ed tech companies. I don’t think it’s right to claim that ed researchers are all pursuing their own quixotic curriculum.
School confidentiality isn’t just a “norm” it’s a requirement. IRB will reject studies that don’t obscure the schools. Journals won’t publish papers that don’t get IRB approval from the researcher’s institution. Journals often also have their own requirements to obscure school data.
Additionally, schools have to agree to let you do research there. You can’t just walk in and declare yourself to be conducting research. The schools where research takes place also want anonymity and often require it in order for research to take place.
Depending on the type of study, for example one that would require observing classrooms as part of evaluating a program, you might also have to get parents’ permission for each kid studied. Again, they often want anonymity.
In general, ethics tells us you can’t conduct research on people without their consent. While ed research may not be as intrusive or potentially directly harmful as, say, a clinical trial, you can harm children by subjecting them to a treatment that hinders their learning. Now, this should work to stop people like boaler but, really, what she’s doing is using a publisher to shortcut the research process. More on that in a sec.
It’s not the job of academic researchers to implement and enforce accountability laws, that should be done by the state education departments. If I’m evaluating whether a math curriculum is effective, the question of school accountability should not be addressed anyway, right? Because if the kids’ failure to achieve is the fault of the curriculum, it doesn’t make sense to then tell everyone in your paper that the school is bad. It’s not what you’re studying in the first place.
What all this has done is create a two-track system of “evidence”. On one side there’s ed research just kind of doing its own thing and nobody’s really looking at it for much guidance about effective teaching and learning. It’s slow, and like a lot of academic research, is often published for other researchers. It’s actually a low-status move to publish about educational practice. Far better to develop models and theories for other academicians, at least from the perspective of an academic career.
The other track is publishers commissioning their own proprietary research that seemingly always shows their curriculum or product is effective. iReady’s Curriculum Associates publisher is getting a lot of flack about this right now. But they don’t have to deal with any of the above ethical or institutional constraints so they can be more nimble, put consent for research in the terms of service or in the contracts they sign with schools, and then do whatever analysis they want with the data to puff up their efficacy. It’s research as sales.
None of this is defending the academic education side of things, but it’s important to report on the whole mess of education research, report on it accurately, and apply critiques consistently.
The IRB standard that you cannot identify the school where you conducted research is absurd. It's wrong. It is not closely connected to preventing any specific harm, and it enables a ton of harms. If it were not the preexisting standard, no one would invent it, and it's not a similar requirement in other fields - you aim at anonymity of individuals, not of the institutions that are supposed to serve them! I do understand that researchers can't unilaterally change it, but that's just a question of "who is going to have to change their behavior for us to get good education research".
I agree wholeheartedly that the commercial research side of things is atrocious. That's true in every field. It's not good, but I think if the academic research were good it wouldn't be a huge problem - people ignore low-quality commercial studies if there's reliable research to turn to instead.
“IRB” and “absurd” — two words that surely never have been used in the same sentence before ;)
I have wondered how some research projects got past the IRB. A high poverty school district in NC (see I am not naming them out of habit--okay, it was Durham) had a foundation that wanted to fund a literacy project but they required a proposal. They asked me to help them write the proposal. A Harvard education group was doing a research project in the district and they wanted to include schools that were not part of that project. They had so much turnover in their literacy department that no one knew which schools were part of that research project, so they called the PI. They were intimidated by talking to the PI of the research, so they asked me to sit in on a call. The guy tried to intimidate me (but that is hard to do). He couldn't tell us which schools. He said he first had one set of schools. He had randomly selected kids in those schools to receive a book sent to their home. The kids in those schools that didn't get books were the control group. They were researching whether poor kids who get a book sent to their home would have greater reading gains than those who didn't. Then they discovered that some of the kids in the control group were siblings of kids in the treatment group and lived in the same house that got a book. So, they were now using other schools as control. They hadn't pinned them down so they wanted all other schools to not get any literacy interventions. Then, NC passed legislation called Read to Achieve, which called for kids who were not reading at grade level to continue to get reading instruction in the summer after 3rd grade and in 4th grade. The legislation specifically said "unless they were part of this research project." The project started in Durham but then grew to other districts. Just like the foundation funds that would have ruined the control group, continuing literacy support would ruin this research project. I wondered how this ever got an IRB.
There's a bit of a Chesterton's Fence thing going on here. Maybe the standards are absurd or wrong today but they didn't occur in a vacuum and exist for a variety of historical reasons that we might expect a journalist to look into. It also simply not true that institutions aren't anonymized in other fields. Medical research papers don't usually report all the participating hospitals for example. I have no idea if Northside Hospital in Atlanta was involved in the research that discovered the BRCA-2 gene's role in breast cancer. That work took place in dozens of hospitals worldwide and it's not clear to me why listing the participating hospitals in the published paper would improve that research. Likewise, I'm not sure that listing the schools in which research studies took place would make the research better. I think one important reason for this is to separate the hard work of program/curriculum/intervention evaluation from the hard work of school accountability. Doing both simultaneously would obscure the underlying reality, not illuminate it.
This post would give you details on why the current IRB system exists. The relevant section is:
"[In 1998], a Johns Hopkins doctor tested a new asthma treatment. A patient got sick and died. Fingers were pointed. Congress got involved. Grandstanding Congressmen competed to look Tough On Scientific Misconduct by yelling at Gary Ellis, head of the Office For Protection From Research Risks. They made it clear that he had to get tougher or get fired.
In order to look tough, he shut down every study at Johns Hopkins, a measure so severe it was called “the institutional death penalty”. Then he did the same thing (or various lesser penalties) at a dozen or so other leading research centers, often for trivial infractions. Duke got the axe because its IRB hadn’t properly documented whether a quorum of members was present at each meeting. Virginia University got the axe because, although it had asked patients for consents, it hadn’t asked the patient’s family members, and one family member complained that asking the patient for a family history was a violation of his privacy."
https://www.astralcodexten.com/p/book-review-from-oversight-to-overkill
Not sure where you got the idea the medical research papers don’t report their participating hospitals, they absolutely do. Beyond that, medical studies manage to distribute their research over many facilities in multiple regions to account for regional bias, something education research rarely manages which makes it even more important to know in which schools the research takes place. There’s a high propensity for bias and confounding variables so we need to be able to dig into the data.
A school has to adopt some curriculum of some sort. It makes perfect sense, if there are a variety of different approaches, to keep track a d try to figure out what works. Obviously, Ted and Alice shouldn’t be outed, but if School A adopts Fuzzy Progressive Math and School B adopts Hardcore Drill Math, it makes sense to figure out what works, and it’s in the public interest to do so. It even benefits Ted and Alice!
I don’t think anyone is taking issue with informed consent.
“Research as sales” is so well-said. I appreciate your caveats, but it must be said: no one is working very hard to create transparency about what is used in schools, to enable better research. I have called for a national curriculum database to enable better research, and this plea has broadly fallen flat. By all means help me change that!
https://karenvaites.substack.com/p/a-call-for-a-national-curriculum?r=wsgsa&utm_medium=ios
I'd like to see something bigger happen with the What Works Clearinghouse. The Federal level seems like the perfect place for a system of rigorous evaluation that states and districts could rely on, as opposed to seemingly gameable third parties like EdReports (you've been rightly throwing shade in their direction).
Academia is ill-equipped to do real research in some ways. IRBs are too strict in a way that inhibits real research. It’s going to be very interesting when China surpasses the US completely in biomedical research due to their effectively nonexistent privacy controls and IRB ethics standards - do we cede leadership in pharma trials etc entirely, or do we try to find a way to actually allow research while still having real ethical standards rather than bureaucratic pseudo-standards?
These requirements are often downstream from other higher-level requirements. The NIH, for example, has very strict rules about human subjects research and doesn't fund research that violates those rules. Other grant granting organizations require the same. We can wish it was different but nobody's going to bootstrap their own clinical trial or conduct a massive educational intervention without getting lots of money and support from some kind of grant. The ones with the money make the rules. It's easy for the Kelsey Pipers of the world to wish it were not so, but that doesn't make it any less of a reality. So long as the Feds, the Universities, the Journals, the school districts and the schools all agree that anonymity is an important ethical point, there's going to be anonymity. Complain all you want about university researchers but they're not the reason Jo Boaler anonymized the schools in her research. If you or Piper want it to be different, perhaps lobby a senator or someone who can change policy?
I agree entirely! Which is why I suspect the real change is going to be driven by China. Pharma research is already rapidly shifting over there, and decoupling in pharma will not be received well if it means American cancer patients are missing out on the most effective chemotherapeutics. So at some level I suspect our hand will be forced.
Kelsey is very Silicon Valley which comes with its own inherent inability to understand more regulated sectors (academia, medicine, government). I still like her and find her worth reading, it’s just an inherent blind spot for Silicon Valley types for some reason. Which is unfortunate for those of us who are more progress minded in these fields, since Silicon Valley types could actually be useful allies if they made an attempt to understand why universities might want privacy protections in research and where there might be opportunities for reform etc, instead of just writing all of it off as useless
I think the best thing about Kelsey's ed writing is that she has a positive-sum view of our education system even though she's very critical of the education system overall. I can't say the same about most reform-minded pundits, journalists, or advocates. I'll always click and usually comment on her education writing as a result.
Almost all fundamental science research has been done by academics since... Forever.
Also the “publish 300 individual studies” academic is going to be a full professor and the person who takes 5 years to publish one super high quality large study is never going to get promoted past assistant professor. And this is fundamentally a problem of incentives - huge incentives to publish, very limited incentives to do a difficult research study that will take years to set up and do properly with the limited resources available to a typical academic faculty in these fields.
Sorry should have specified- modern Western academia is ill-equipped to do research involving human subjects at scale because bureaucratic hurdles are excessive. I don’t think we should go back to the 1920s where research participants don’t have rights. But Kelsey is right that entire schools being anonymous is a somewhat unrealistic standard, and I mean to say that IRB requirements sometimes go beyond anything a normal person would expect ethically and just create a proliferation of stupid paperwork.
You know, don’t you, that IRB rules weren’t brought down by Moses from Mount Ararat?
It’s not that they’re infallible rules, but we shouldn’t pretend they can just be wiped away or that they served no purpose. Nor should we ignore the other institutional and funding constraints attached to anonymity requirements. I get that we want to pretend there are easy answers and one quick fix will somehow make ed research good, but the real word doesn’t work that way. You can’t just object to things and refuse to participate and expect those same institutions and journals and all that to publish your work or platform you at conferences. Beyond that, I suspect the larger problems Piper has with education research will not be solved by ending anonymity requirements anyway. While they may have helped in the case of Jo Boaler’s work, I’m not convinced that one neat trick is going to work here.
Well now at least I understand how Lucy Caulkins was able to advocate for teaching kids to read like someone who can’t read and be taken seriously for so long.
I have been evaluating grant-funded education programs for 30+ years. I used to work for a school system doing this and then started a small business to do this. I started seeing Lucy Caulkins being used decades ago. It was expensive so schools would write it into grants and then get trained, certified, etc. using the grant funds. I've observed hundreds of grant-funded training sessions. My staff and I could hardly believe that people fell for the snake oil. That was one we thought simply couldn't be real, but there are many many others. My staff would often say that we are in the wrong business. We should be selling snake oil to schools. We noticed the pattern that the schools like the stuff where the training is fun, it makes you feel good, and the theory is that you don't have to work very hard to get the expected results. And what are the expected results? There is training for that, too. Don't expect poor or minority kids to learn anything because they aren't cut out for it. So, when none of this stuff works, that is because nothing was going to work anyway. And you get to pay staff with grant funds, buy things, and pay me to evaluate it. We don't think anyone ever read our evaluation reports. And why can't poor kids learn? They hang their paintings too high and talk with a different speech register. The research is "qualitative."
The weird thing about Readers Workshop was that it really seems to be more work for teachers. In conventional reading instruction as I recall it you walk through letter combos and sounds week by week and have kids read appropriate texts aloud. Most of them will just get it. A few will struggle and need multiple cues. In Readers Workshop you just have the kids try to read, they will all struggle because you haven’t really taught them anything, and you have to spend an eternity offering cues that generally don’t encourage actual decoding of the text. Surely it’s harder?
A big part of what we would observe was that you were to make the atmosphere "feel" like a reading place. Teachers liked that. We didn't see hard work offering clues. These grants would almost always be coupled with the Poverty Training that lowers teachers expectations by teaching them why poor kids can't learn. Most of the grants we evaluated were targeting poor kids. (A lot of profiting from poverty going on.) So, you teach this way and no one learns but that is to be expected. It probably would be harder work, and no results, but if you expect no results there is no reason to work hard. Just make the kids feel good if you can.
I taught my neighbor how to read a few years ago. She was in 3rd grade and couldn't say her Ls. NC had adopted a phonics-based curriculum and teachers were taught that students should not move forward until they have mastered step 1, which is sounds of the ABCs. Her school had pulled her for a special program (which I have evaluated and it is snake oil--expensive snake oil) since Kindergarten. Her parents asked me to work with her.
There was nothing wrong with this girl. I was a math teacher so I had to just use common sense on how to teach her to read. It was hard work for both of us. She is a great reader now, and a straight A student in sixth grade. She learned pretty quickly. She had been pulled from core instruction for 3 years to get this special training where she would just say the sound of the letter that popped up on an iPad.
I think some kind of database like that is great! What's different is that in the published studies, you don't usually get the hospitals themselves listed and the goal of the study isn't to hold the hospital accountable for poor patient outcomes, it's to evaluate the medicine/treatment/procedure/whatever. Like, if Hospital A participates in a clinical trial and it fails to show efficacy, we don't point at that hospital and say it's bad. We say the treatment wasn't effective.
[edit: adding the next part as the comment was truncated and I didn't see the second half]
My point in the op wasn't to defend the practice so much as explain that it's far reaching and exists vertically throughout the education research system. Everyone from the feds on down to parents of the kids being studied has rules requiring consent, anonymity, and efforts to minimize harms. They exist and pinning all this on academic researchers is missing the larger systemic context. You may disagree that kids safety is at risk but that is, in fact, what the rules are there for. Go take a CITI training and you'll have hours of fun learning about protecting human subjects, even in non-medical research contexts.
No worries. Tone is always weird on the internet.
I don’t think breaking anonymity is automatically going to be blaming the school. It’s just how Piper made that connection. If we’re talking about replication and researcher accountability, I see no problem. When you start to connect it to accountability and student performance, it feels like crossing the streams.
All that said, nothing I’ve written is an overall objection to Piper’s article. Ed research is a mess, probably not uniquely so but certainly a mess. The incentives are all screwed up and one of the most widely understood problems is that research (good or not) rarely makes it into schools and into the everyday practice of educators. You’d think this would drive researchers crazy but it kind of functions as a way to give them license to ignore schools entirely. Just do studies in labs or with undergrads and publish it!
Anyway, that’s a whole other rant but I think that’s also why I came away wanting Piper to dig deeper and get to more of the systemic issues behind low quality research.
How is this NOT a colossal scandal?!?
"Among the education papers where they were eventually able to get enough data to check the results, they found zero results that were precisely reproducible — literally none of them."
Years ago I did a literature review on the quality of math education research. I found an article that had reviewed a sample of math education research articles that had been published in peer reviewed journals, and fewer than 10% of them had used valid methods. I wasn't surprised. When I worked for the research department of a large school system, our department head was the head of Division H of AERA. This is the Research division of American Education Research Association. Research papers would be sent to her for applying to present at the big conference. We would read and review them. Nearly all would be unbelievably bad. I was new to the education research field. I couldn't believe it then. Now, I expect things to be that bad.
Related: I’ve read a number of comments of yours in this thread, and it would be wonderful to read more full posts from you given your background and experience.
Because this happens all the time. Usually not this severely, which is why it's the example for this post. But the public hugely overestimates the rigor and reliability of research in most fields.
I think that the majority of people who are aware of the reproducibility crisis often go the other way though, and vastly *underestimate* the rigor and reliability of research in lots of fields. Kelsey says that social science research from the second half of the 20th century often doesn’t replicate - but the first half of the 20th century is worse at comparable questions (they often didn’t even attempt the questions that would need more data) and empirical research in the 19th century is just non-existent in most subjects. The physical and biomedical sciences went through the same transitions a few decades earlier, but they’re all improving, and even the best fields can’t help but have lots of individual papers that are just plain wrong.
The public is actually deeply skeptical of the claims to authority of social 'scientists', and they're clearly right to be.
It's largely the intelligentsia who lap up the rubbish coming out of these academic journals as remotely credible.
Precise reproduction seems like a very high bar. Everything should be the same down to many decimal points. This is achievable and should be encouraged, but it's not trivial to achieve.
For example some statistical analyses involve numerical computations. If you use a slightly different version of the software, or even a different type of processor, the outputs may change slightly by like 0.00001 or something. This would violate precise reproducibility as far as I can tell based on how they define it in that Nature paper.
0 papers meeting this standard doesn't reflect well on the field though, especially if their statistics are relatively simple.
This is a pretty good point in general and I wonder whether reproducibility is pretty understated in fields that require relatively high precision calculations.
That said, I think (but am not confident) that most ed research does not fall into this category
I’ll start with the disclaimer that this is a fairly uninformed hot take, but I wonder if part of the problem is that the education Ph.D. serves two very different populations with very different purposes. One is students who will go on to academic careers and research. The other, probably much larger group, is teachers who are looking to advance into the ranks of administration. Seems like trying to serve both purposes would make each one worse.
This is my understanding as well: the majority of people getting PhDs in Education don't even want to do research, let alone rigorously find truth.
Yes, this crossed my mind. In my state, an advanced degree comes with a baked in raise, and a doctorate commands the most. Rest assured, there's a lot of research done solely for credentialing.
I have a friend who loves to yell about Jo Boaler and California math.
Personally I think no one has ever cared about education research and it shows. Everyone prefers their political agenda.
Did you talk to anyone in education research for this article? What did they say?
Yes, this does strike me as a gap in this particular story. But what would they say?
I mean, I can imagine a variety of things but that's why you ask.
I agree with every word of this.
But I also think the reasonable best case for the current education research paradigm is not that good.
1. Randomization in education is extremely hard.
2. Controls vs. treatment almost always have to share a ton of properties that I, at least, hate and consider a big part of the problem(s).
3. We (humanity) are quite bad at measuring many of the outcomes I think are important in a way that satisfied social science research.
So all of this *and* we should surrender to the need to look for different research paradigms and more generally be prepared to make good judgments about education on the basis of something other than research.
Economists will save us. Ironically, Jo Boaler was warmly embraced by the Freakonomics folk. It was three mathematicians who took the heat (with visits from the police, no less) for challenging the Railside study.
1. Economists can't change the biggest problem - it's extremely difficult to achieve true randomization when students are clustered into classes no randomly, which is pragmatically always going to be the case and which is often effectively mandated by racial representation regulations. You have an ebook textbook and a paper textbook and your null hypothesis is that students perform the same regardless of which they use. To test this you need to randomly assign students to condition. But students are already nonrandomly assigned to classes. You can flip a coin to decide which of two classes gets which textbook, but you've just broken randomization. You could do some sort of multi level modeling to compensate for it, but it's not ideal and it's The trouble is that there's often no practical way to divide students up for research that's remotely conducive to actual teaching; you end up telling teachers to use one lesson plan with one set of kids and a different plan with another, or different materials, etc. And then there's the parents; depending on context and situation, you have to get their permission, and if they get the impression that their kid is getting worse material or teaching than others.... It's not easy and I don't think just getting a different set of researchers is going to change. I invite you to try it yourself though.
2. You have to accept when research tells you things you aren't going to like, such as some Mississippi "miracle" research that is in preprint right now. You're not gonna like it!
Right and then midway into your textbook example, one of the participating schools pulls out of the study because some unrelated initiative means they’re adopting a whole new curriculum and won’t use either the eText or the textbook anymore. Ask me how I know!
We’re PhD students in Economics and Psychology at Stanford’s Graduate School of Education and we felt obligated to respond to this article. Much of what is written here echoes both our training and our own struggles with academia itself and science as a practice. We agree that we should be critical of preexisting claims and that we shouldn’t make strong arguments based on the results from a single article. Credibility and public trust in research should start with the methodology and researchers themselves, and it’s as important to use good methodology in the first place as it is important to call out bad science when it happens. We also agree that the existing standards of academia and research sometimes make it even more difficult for scientists to do high-quality, reproducible work. However, this article makes incredibly strong claims against the field of education at a time when public trust and investment in research is already low. We think the argument here is too generalized and lacks an understanding of the interdisciplinary nature of education as a field to begin with.
As other commenters have pointed out, education is a nebulous field with broad aims and purposes. Within the Graduate School of Education at Stanford, as at many schools of education, you will find researchers from many disciplines who vary widely in not just how they study education, but what they think the purpose of education schools should be. Education schools produce teachers and people who train teachers and people who write curriculum, but they also produce researchers who study children’s brain development and people who study education policy or the intersections of education with language or our political system or technology. Sometimes these researchers might look more like those who work in other disciplines. Some economists, for example, are trained in or work at schools of education but don’t exclusively publish in education journals. Tom Dee is a great example—his articles appear in economics, policy, and education journals. There are many other researchers (not just economists!) at Stanford, and other universities, who do the same. There are also researchers within other departments who study education but don’t publish in education journals at all. The Nature article cited here isn’t accounting for the many, many economists, sociologists, and psychologists who are doing excellent work in education and who are publishing outside of it. If we only look at work published in education journals, we are missing a large body of high-quality education research. This isn’t to say that we shouldn’t hold education research and journal quality to a high standard, but to generalize based on the evidence presented in this article misrepresents the complexity, interdisciplinarity, and true quality of education research.
Of course, bad work exists both within and outside of education and harms both students and public trust in science. This article is right that we should call it out and that we should criticize disciplines, journals, and individual researchers that handwave over quality issues. But low-quality work should not define the discipline. This article doesn’t spend enough time acknowledging that high-quality education research already exists and already meets the ideal standards outlined here. Someone who is not familiar with the world of education research might walk away from this article thinking that almost nobody is doing good science in education, and that researchers who are competent in quantitative methodology are working exclusively elsewhere. This is simply not true.
Furthermore, good science needs a good toolkit but these tools don’t have to come from one discipline. We think Drew Bailey puts this well. Economists can add a lot to the conversation, and should be part of it, but they can’t answer everything. Every researcher benefits when they can learn what scientists in other disciplines do well, and when their own methodology might not be the best tool for a given problem. Ironically, graduate schools of education are probably among the places best situated to do this—it is so rare in academia to get people from different disciplines in one room in conversation with each other. As students in a school of education, we do it every day. This article undermines the credibility of education research at a time when public trust and investment is already shaky and while we acknowledge that improving credibility can play a role in building that back, we urge caution in broadly generalizing.
- Maya and Olivia
I think that this comment is missing the point, or at least a large part of the point. In spite of the article's headline, Piper is clearly criticizing education research as done by professors of education in schools of education, and as published in education journals. She's not criticizing education research writ large.
If you doubt it, just remember her praise of Tom Dee. Or consider her contrast of education journals to the journals of other fields—which, of course, also publish research on education.
To be fair, Tom Dee is a professor of education in a school of education who publishes in education journals. He's also an economist. Education research writ large very much includes many people like him who do high-quality work within schools of education. I suppose that this article could have more carefully defined the parts of education research it (validly) aimed to criticize, including in its title.
I really enjoy your writing about education. Always sharp. I also agree that getting researchers from other fields interested in education would be valuable, but for very different reasons.
The truth is the quality of research in most fields is really poor, and declining.
The majority of papers in medical research that aren't RCTs are completely worthless, and for RCTs you have to ignore anything done in many large countries. Most economics papers - a field with a weirdly high reputation for rigor - use slight of hand to get to their causal claims. In my own, hard science field, most of the splashy papers have problems severe enough that I don't believe their conclusions.
But! While all these fields have their endemic problems, the key is *they are different problems.* The commonly accepted flaws of medical research will be obvious to an economist. The weak arguments in economics will stand out to a biologist. So not only do I think researching across disciplines is beneficial, I think peer review should include someone from a completely different field, as long as they have the basic technical chops for the subject. (A psychologist is just not going to be able to review a physics paper.)
“ the quality of research in most fields is really poor, and declining.”
I’ll grant “poor” - everything is far from perfect, and so everything counts as poor if you set the bar high enough. My thought is that it would be great to hit higher bars, but it’s valuable that we are at least hitting the bars we are hitting now.
But I want to dispute “declining” I can only understand this claim if you just mean there are more people doing research in every field every year, and the median is going down, right? Or do you actually mean to say that research published in the top five specialty journals in any given field is lower quality than the research published in the top five specialty journals in that field a decade or two ago?
Because it seems to me that in every field I’m aware of, things have basically always been generally improving over all, even though there are always little fads that make one pocket of research here or there problematic in new ways.
It's a complicated topic. I think:
1. The quality of the median peer reviewed publication in fields I'm familiar with is declining. I'm pretty confident in this because the volume of papers has grown exponentially while the pace of discovery has slowed.
2. Possibly there is a core of load-bearing papers where quality is improving. But it is hard to know because heuristics like journal prestige or citations have at best a very thin relationship to reliability
3. It is really hard to fix these problems, because it is impossible to define a verifiable procedure that stops them and the incentives are stacked on the other side. If you solve p-hacking, you move immediately to the problem of null mispecification, which has the same end result. And adhering to sound practices is a constant battle. In my workplace, where I would say the standards are quite high, we are currently fighting over whether to release our negative results and whether we can ignore our false discovery estimates for things that look "compelling."
4. Completely agree that the general public mistrusts the wrong things about science. In particular the official recommendations of the US Gov were, until last year, reliably based on convincing, well-conducted research. The papers people find while doing "their own research" are almost always straight garbage.
Yeah, this is probably much more difficult in much bigger fields. In philosophy, where our big annual conferences (for some reason we have three!) tend to have something like 1,000-2,000 attendees, and there's only a couple dozen major generalist journals and a few specialty journals you have to pay attention to in various sub-fields, and there aren't many new journals every year, it's still possible to keep tabs on a lot of what's going on. It's also relevant that what matters is new ideas and arguments (basically none of which should be *trusted*, but should be engaged with), rather than verifiable empirical data that you might need to trust.
You knocked this out of the park, thank you! Also… I have been hopping up and down, trying to get people to support a national curriculum database, specifically to enable better research insights. You’d think this would attract lots of support and attention. You’d think wrong.
By all means help me change that:
https://karenvaites.substack.com/p/a-call-for-a-national-curriculum?r=wsgsa&utm_medium=ios
Also relevant that the 2012 critique referenced has only been cited 14 times. Which seems to be evidence either that there are some issues with that report or that education researchers are profoundly uncurious about this critique.
The homeschool idea is interesting, but seems to sacrifice any external validity in an effort to improve internal validity
The thing about this is that without internal validity, external validity is zero anyway
A great article, and a case well argued. Another angle you might find interesting is what education researchers are currently devoting their attention to. It’s not just that much of the research is poorly conducted, but that it’s not even focused on effectiveness at all.
If you look at the AERA website, you will quickly realize there is very little attention being devoted to empirical studies of teaching and learning.
The 2026 Annual Meeting theme was "Unforgetting Histories and Imagining Futures: Constructing a New Vision for Education Research” to give you an idea.
Couple thoughts.
1. Econs already do quite a bit of education research and have for at least 40 years. It’s certainly way better than a lot of the stuff highlighted here, but it hasn’t saved us yet.
2. The thing about ed research is that it’s very hard . As you almost point out, this is a poor match for an overall less smart and quantitative group. This is likely one reason Econs don’t do more of it. It’s just really hard to draw substantive conclusions.
3. There’s a lot of fraud. We should just say this. Based on what’s here, it’s very like that Jo Boaler’s research is fraud. The left (I consider myself a part of this group, at least loosely) needs to police itself better. Who’s doing more harm to poor kids of color than people who commit academic fraud to get advanced math classes removed from public schools? This is a serious secular sin.
I'm happy to see that this topic is being covered here at The Argument.
However, outside the progressive bubble, conservatives have known for many decades now that schools of education and education "research" are cesspools of incompetence and ideologically-driven insanity.
If progressives actually care about governing and earning the public trust, disestablishing frauds like Jo Boaler (who is actually completely typical for a professor of education, being notable only in attaining a greater degree of influence for her fraudulent snake oil than most of her low-IQ colleagues) would be one of many places to start.
And, if progressives actually want public schools to be broadly respected, effective public institutions, rather than asylums of ideological lunacy, reining in this sort of idiocy would also be an essential first step.
Anyone reading City Journal for the last couple of decades has known that schools of education need to be at the top of the list for defunding and liquidation...
https://www.city-journal.org/article/pedagogy-of-the-oppressor
https://www.city-journal.org/article/the-parents-revolt
https://www.city-journal.org/article/the-assault-on-childrens-psyches
I agree with so much of what you wrote here: fewer, larger, well-powered studies would be better than the status quo; lots of educational research is really bad; sometimes confidentiality concerns hinder reproducibility and allow questionable research practices to thrive. Probably my favorite thing about your post is the observation that it was really the "existential crisis" that made the replication crisis lessons sticky in some fields.
A non-trivial quibble though: while I agree economists are smart and do good work and are going to be part of the solution, the replication crisis was NOT centered in econ, and they will not solve this problem. They will help solve other problems (demanding causally informative designs, open code, more open data). But selecting reporting/p hacking are major problems in econ. This is not just because the replication crisis was centered in a different discipline (psychology), but also because the solutions that worked for psychology (including preregistration) are harder to adapt to the kinds of quasi-experimental designs that dominate in applied micro, and require a lot of data-dependent decisions. Tom Dee has a new paper about this. See:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6308160
(e.g.: IV, in contrast, has a large number of researcher degrees of freedom, tends to be difficult to publish with null results, and, as an observational research design, is not usually accompanied by a pre-analysis plan. IV estimates also have a tendency to be enormous (Card, 2001). A study of finance papers found that two-thirds were instances in which, according to theory, OLS is expected to be biased away from zero (Jiang, 2017). Among these, 80% had IV
estimates even larger than OLS. Even after winsorizing at 1%, the average IV estimate in
this category was nine times larger than OLS. A study in political economy found similarly:
IV estimates were 12.4 times larger than the OLS estimates from the same paper (Lal et al.,
2024).4)
and:
https://journals.sagepub.com/doi/full/10.1177/0193841X251326738?casa_token=9VbzWPFDvnUAAAAA%3AkF7TQHVEICMf-X9Rl82ikF7Ue_mJGQTS8EVBndXav7spOZnae_ghkOJsgIJYrEYJ0sQR49lJVfby