In 1998, a researcher rolled out an innovative approach to education. And in 2015, Methodology Twitter had a big fight about it.
Support us:
Links!
Thanks to Doctor Dreamchip for our lovely theme song!
In 1998, a researcher rolled out an innovative approach to education. And in 2015, Methodology Twitter had a big fight about it.
Support us:
Links!
Thanks to Doctor Dreamchip for our lovely theme song!
Michael: Give me two seconds, I need to close the windows. It's so fucking hot in here and I was like, “I'll try to do it with the windows open,” but there's shitloads of bird sounds.
Aubrey: Yeah, totally.
Michael: [shutting windows] No birds.
Aubrey: [laughs]
Michael: This is now a No Birds Podcast.
Aubrey: You have never sounded more like Jerry Stiller.
Michael: No birds.
Aubrey: No birds.
[laughter]
Michael: What do you got? What have you got for us?
Aubrey: Ah, hi, everybody and welcome to Maintenance Phase, the podcast that's crawling out onto the sidewalk after a nice long rain.
Michael: Oh, it's a worm. It's a worm opener.
Aubrey: All I had was worms.
[laughter]
Michael: You had two words, worm, wars. [laughs]
Aubrey: Sure, man. I'll take my inspiration from my morning walk the other day. [laughs]
Michael: I'm Michael Hobbes.
Aubrey: I'm Aubrey Gordon. If you would like to support the show, you can do that at patreon.com/maintenancephase. And if you want to just keep listening, just keep listening. We're so happy to have you.
Michael: You are here with us.
Aubrey: Today, we're talking about worms?
Michael: Yes. So, okay, I'm so excited because I like telling you stories and I like fucking with you. And so, I told you--
Aubrey: [laughs]
Michael: I told you nothing other than the name of this episode “Worm Wars.” This is the controversy that tore methodology Twitter asunder in 2015.
Aubrey: Ah.
Michael: And I want to tell you all about it.
Aubrey: Okay. Tell me.
Michael: The hardest part of this episode is going to be getting through the first five minutes.
Aubrey: Oh.
Michael: There's just some gross stuff like body horror that we have to get through.
Aubrey: Okay.
Michael: Are you familiar with intestinal worms?
Aubrey: Not really.
Michael: Okay.
Aubrey: So, I assume they're like parasites, yeah?
Michael: Yes.
Aubrey: Okay.
Michael: Somewhere around 2 billion people in the world have intestinal worms, which are exactly what they sound like. They are little tiny worms that live inside of your digestive system.
Aubrey: Okay.
Michael: Some intestinal worms have lifespans of one to two years, and they'll literally just attach themselves to the inside of your intestine and just be there for years and then they die. Most people have light to moderate infections, which don't really have any symptoms, but sometimes you'll get much more heavy infections. And so, this causes various symptoms that it usually happens through giving you anemia. It makes you tired, it gives you brain fog, you have to sleep all the time, it can stunt kids growth in the developing world, it can make them too skinny.
Aubrey: I have never had a worm infection, or parasite, or any of that kind of stuff, but I have been anemic before and boy, oh, boy, it doesn't seem it's going to be that rough and then it's pretty rough.
Michael: Dude, a friend of mine passed out in a Starbucks, because she got anemia.
Aubrey: You just can't get your energy up at all.
Michael: Yes. Okay, so, here's the really gross part. The way that worms spread, as you get these worms in your intestine lining and they lay eggs, they lay hundreds of thousands of little microscopic eggs. These eggs end up in your poo and then you poo out the eggs. And so, the reason why worm infections are endemic throughout the developing world is because oftentimes there isn't running water. In some areas people will use soil to wash their hands after they go to the bathroom that's grabs dry soil from the ground. Sometimes, they use poo as fertilizers in gardens, and then it ends up in their food, or they're walking around, and they have a cut on their foot, and there's a little tiny egg that ends up getting into the cut and then swims its way to their intestinal tract.
Aubrey: Sure, sure, sure.
Michael: It's just the poop human interface is how people end up getting worms and this rolling snowball, where the more worms there are in the soil, the more likely kids are to get it, the more likely kids are to spread it, etc. It used to actually be a really big problem in America.
Aubrey: Here is what I appreciate about this whole episode is, I didn't have to bring up anything related to poop.
Michael: I know.
Aubrey: You just fully picked a topic and you were like, “You know what, I could go for right now as little poop research.” [laughs]
Michael: The thing is, I saw a methodology fight and I was like, “Ah, it's going to drive me into the poop world, isn't it? I have to go inward.”
Aubrey: [laughs]
Michael: The protagonist for this episode is a guy named Michael Kremer, who is currently a Harvard professor. Before all of that he had spent some time in Kenya as a kid and he was really fascinated by like, “Why are people poor in this country? What are we doing to alleviate global poverty?” He eventually gets his degree in economics and so, he decides to do his graduate work in Kenya. He moves to Kenya, and he starts befriending people in the development sector there, he meets World Bank people, and NGO people, and charity people. What he finds is that the field of international development is in what other people will later call a statistical Dark Ages. This is like the early 1990s. The way that people do development, it's on the Live Aid model, where it's like, a celebrity does something, and there's this big fanfare, and then no one ever follows up afterwards.
Aubrey: Yeah.
Michael: And if there's any statistical rigor or studies being done, they're oftentimes before and after studies, so, they're like, “We gave everyone a goat and then a year later they were doing better.” It's like, “Well, was just everyone doing better in the country that year because it was like a really good harvest or something?” You're not comparing them against a control group,
Aubrey: Or, was there another policy intervention?
Michael: Right.
Aubrey: There's so many things that it could be.
Michael: Right.
Aubrey: Think about the last year in your own life and think about if someone from the outside tried to determine what was the cause of any changes in your [laughs] life that would be a fool's errand, right?
Michael: Right. And a lot of the projects are done in this common-sense way. This is oftentimes how you get bad development projects where people were like, “Oh, it's just common sense to. Give everyone a bicycle or whatever.”
Aubrey: No.
Michael: It's like, “Well, did you ask them first if they needed a bicycle?” If you give everybody in the village a bicycle is that going to change the dynamics of the village?
Aubrey: Yeah.
Michael: It just being done in this really unsophisticated way. Michael Kremer has a friend who works at a Dutch NGO and they're about to implement a program where they give kids in Kenya textbooks. At the time, there're 30 kids in every classroom and there's one textbook per classroom. So, again, common sense. Well, if everybody had a textbook this is obviously going to make their grades better and school performance, attendance, everything's going to get better if everybody has their own textbook. Obvious, right?
Aubrey: Mm-hmm.
Michael: Michael Kremer suggests to his friend, he's like, “No, no, no, don't do it before and after study. Instead of just giving this to seven schools, take 14 schools and give seven of them textbooks, and the other don't get them textbooks, and then you can actually compare the effect of the textbooks.” They do this, the project goes for a couple years, and lo and behold, there's essentially no effect at the textbooks.
Aubrey: Uh-huh.
Michael: After the textbooks don't work, there's this period of testing stuff. It's like well, that's weird that the textbooks don't work. Then they try chalkboards like a lot of the schools don't have chalkboard. Again, they split into two groups. It turns out that doesn't work. They then do a thing where they hire more teachers’ half of the schools. That doesn't work. So, they're going over these obvious “education interventions” and it turns out that education interventions are not working to improve education, They're like, “Hmm.”
One night, Michael Kremer is talking to a friend of his who works at the World Bank, and he explains the symptoms of worm infection to Michael Kremer, and he tells him that there are treatments available for worms. There are pills for basically every kind of species that's endemic in that part of Kenya and they're really cheap. Diagnosing the kids is actually really expensive, because you literally have to go through their poop and measure how many eggs there are. And so, it's such a pain to diagnose this and it's so easy to give the kids these medical treatments like they don't really have any major side effects, they're pretty cheap, they're already available in that part of Africa. He's like, “I wonder what would happen if we just gave all the kids these deworming tablets, because some of them are probably infected with worms and just don't know it.”
They do the same thing. They find a district with 35,000 students, 75 schools, and they do the same thing. They separate it into control groups and treatment groups. They roll out this mass deworming program where they just give every kid deworming pills, right?
Aubrey: Okay.
Michael: I'm going to send you an excerpt. This is from a book called Doing Good Better: Effective Altruism and How You Can Make a Difference by William MacAskill.
Aubrey: “Kremer did an experiment to see whether treating children for these intestinal worms had an impact on education. The results were striking. Absenteeism is a chronic problem in schools in Kenya and deworming reduced it by 25%. In fact, every child treated spent an extra two weeks in school and every $100 spent on the program provided a total of 10 years of additional school attendance among all students. Enabling a child to spend an extra day in school, therefore, cost just five cents. It wasn't merely that “deworming children” worked at getting children into school, it worked incredibly well.”
Michael: It worked incredibly well.
Aubrey: Fucking yeah.
Michael: Yeah.
Aubrey: If you have something uncomfortable to painful, to debilitating going on with your body, when you feel better, I bet, you do better in school.
Michael: Yeah. It's a very appealing story also because they've tried all of these educational interventions and it turns out that there's a zigzag solution that maybe it's not an education intervention, maybe it's there's something broader and bigger at play that is just invisible to people that are in the school and also development aid workers.
Aubrey: Yep, totally.
Michael: The main headline from this study is that they reduced absenteeism by 25%. That's the thing that ends up in William MacAskill’s book. That's the main thing it's in the abstract. But there's also some other effects. One of them is, there's health benefits. So, the kids who got the deworming were slightly taller, they're slightly heavier. It was actually nice to read a study where it's like, “The kids are fat.”
Aubrey: Yeah. [laughs]
Michael: Yay. [laughs] It's like just once.
Aubrey: What a win.
Michael: They also had lower rates of anemia. One of the really interesting effects of the study is, these spillover effects. Because worm infections are an infectious disease, one person gives it to another person, once you blast hundreds of kids in these schools with deworming pills, they then don't spread it to other people. What they find is that there's actually attendance effects in schools like three kilometers away and six kilometers away. There're all these other benefits that come along with basically just breaking the chain of infection and they also find long-term effects. This is from MacAskill’s book.
Aubrey: Mm-hmm.
Michael: “When Kremer's colleagues followed up with the children 10 years later, those who had been dewormed were working an extra 3.4 hours per week and earning an extra 20% of income compared to those who had not been dewormed. In fact, deworming was such a powerful program that it paid for itself through increased tax revenue.”
Aubrey: Wow.
Michael: So, it’s like you can then even like fast forward 10 years and you find that the dewormed kids are doing better.
Aubrey: Big, if true, big, if true. [laughs]
Michael: Aubrey, it’s the first 5 minutes of the episode, you know everything I tell you at this point is true.
Aubrey: Everything goes according to plan. The nice white guy from Harvard is definitely right and no worm is going to come from his-- [laughs]
Michael: Stop remembering the way that I tell you stories.
Aubrey: [laughs] Stop remembering the premise of our show. [laughs]
Michael: The premise of our show. Over the next decade basically, deworming becomes the hottest idea in international development. I was working in international development. At this time, I remember all of the deworming national plans that were coming out. It paralleled the rise of this effective altruism thinking, where in Michael Kremer's original paper, which was published in 2004, a lot of the paper is actually about the cost benefits of this. When it comes to something like hiring more teachers, where every single teacher costs a lot of money and there's payroll and there's all these other costs associated with these educational interventions. Whereas this one, the deworming pills cost 30 cents each. The benefit of this versus the cost of doing it is just unprecedented international development.
In 2007, Michael Kremer found an NGO called Deworm the World, which helps governments implement their national deworming plans. There's also this charity, GiveWell, which basically has this model of like, “We're going to look at all the evidence and we're just going to tell you that, if you have 100 bucks to give away, here's probably the most effective thing to give it to. If you want to help people, this is what you should do.” That's one of the few charities that is ranked by GiveWell is this Deworm the World initiative and other kinds of deworming charities. There was a deworming strategy by the WHO, the World Bank was really big on deworming. deworming was like the hotness. Kenya has national plan on the basis of this 2004 paper, India declares February 10th National Deworming Day and has a program to deform 140 million kids a year.
Aubrey: Whoa.
Michael: This is huge. According to the Gates Foundation at one point, they put out a paper calling this the largest public health program ever attempted.
Aubrey: Wow. Holy shit.
Michael: Huge full court like decade long full court press on deworming.
Aubrey: There's part of this that feels very like Ted Talky life hacky. This one simple trick-
Michael: Aubrey.
Aubrey: -fixes everything.
Michael: Aubrey.
Aubrey: Am I getting ahead of myself?
Michael: I love that you're at the point with the show where you're like, “Wait a minute, there's a good story here.”
Aubrey: [laughs]
Michael: It’s probably fake. [laughs]
Aubrey: I don't trust it. [laughs]
Michael: Okay, I'm about to send you a couple paragraphs from a Guardian article in 2015 that kicks off the worm wars.
Aubrey: Whoa.
Michael: This is one of the longest excerpts that we've ever read in the show, but I wanted to you to get the flavor of this.
Aubrey: Great. The headline says, “New research debunks merits of global warming programs.” I'd say, “Fucking called it.”
Michael: Twistedy twist.
Aubrey: “Deworming children once ranked by Nobel laureates as the fourth most effective intervention to solve the health problems of the whole world offers very little benefit despite the millions of dollars spent on it according to a reanalysis of the evidence. The World Bank and the UK Government are among the big funders of deworming programs that were claimed to improve not only children's health and growth, but also their educational attainment and ultimately, their country's economic prosperity. However, much of that optimism rests on one particular study carried out in Kenya in 1998 to 1999, which reported that “deworming improved attendance not only in schools, where children were treated and taught about avoiding worms, but also, in other schools, because of an apparent reduction in transmission.”
Researchers from the London School of Hygiene and Tropical Medicine have now reanalyzed the trial data and published two papers in the International Journal of Epidemiology. Researchers used methods common to health research rather than those used in economic research as the original paper had.” God dammit, Michael.
Michael: Good sign, good sign. The epidemiologist finally looked into it 10 years later. [laughs]
Aubrey: There had been no increased attendance in the schools, where children had not been treated as the original study said. That was down to calculation the errors and was wrong.
Michael: [laughs]
Aubrey: The researcher also found missing data and some evidence of bias in the finding that school attendance improved.
Michael: Shocking twist.
Aubrey: Who could have seen that coming.
Michael: What is your overall impression? What is the plain language paraphrase of what you've just read?
Aubrey: The fucking epidemiologists never looked at an epidemiology issue and it was studied by fucking economist.
Michael: [laughs]
Aubrey: What are we doing? Then, whole fucking nations built their interventions around what a fucking economist thought was good for public health.
Michael: At this point in 2015, I've seen various numbers going around. One of the numbers is that one a billion deworming pills are being distributed annually-
Aubrey: What?
Michael: -under programs like this.
Aubrey: I assume you're going to walk me through this.
Michael: Yes.
Aubrey: It sounds like what they have debunked here is the idea that school attendance did not improve in schools where kids were not treated.
Michael: That's a very good catch. Yes.
Aubrey: It's not the whole thing is garbage. It's just this idea of school attendance going up in other schools is not true. That's what I'm taking away from this paragraph.
Michael: That was a very good read that a lot of people did not notice at the time. [laughs] You just noticed now.
Aubrey: Oh, fun. So, now, we get misunderstanding.
Michael: This is a thing. We have to talk about what they actually did and what they actually found. As you know, in psychology and other sciences, there's now this thing called the replication crisis. Essentially, the idea is that there's all these canonical studies that have affected policy, and have affected public understandings, and they're basically rerunning them. In a lot of cases, they don't pan out. Things like the infamous like marshmallow test. What happened here is that people looked at this study from 2004 in Kenya being used to justify deworming 140 million children in India and were like, “Wait a minute, a lot is resting on this paper. Let's just look into the findings.” They didn't redo any studies. This wasn't a replication. What they did was they went to the original author, so, it was Michael Kremer and his coauthors on the original paper and they said, “Give us all your data. Give us the software that you ran, give us the original questionnaires you used in the school. Just give us everything and all we're going to do is just run it again.”
Aubrey: Okay.
Michael: To their great credit, Kremer and his coauthors give them everything.
Aubrey: Okay.
Michael: These reanalysis researchers spend months going over everything and basically published a reanalysis of this paper. The biggest thing that they find is there's data missing and there's data that's miscalculated. You know like spreadsheets have all these conditionalities, where one cell refers to another cell refers to another cell?
Aubrey: Sure, yeah.
Michael: They basically got one of these little codes wrong and that ends up cascading down to all of the other codes in this way that was invisible to the original researchers.
Aubrey: It's essentially the level of a transcription error.
Michael: Exactly. As a result of all of these errors, everything, recalculating the data, all of the health effects of deworming disappear.
Aubrey: What?
Michael: The anemia link is gone, the kids’ difference in heights gone, difference in weights gone-
Aubrey: Wow.
Michael: -and then essentially all these spillover effects. All the stuff like the indirect three kilometers, six kilometers, those effects, the six kilometer effects are gone and the three kilometer effects are much, much, much weaker.
Aubrey: Oh.
Michael: To me, the main thing that comes out of this statistical reanalysis-- This is something very well known in studies of school attendance in Sub-Saharan Africa and really throughout the developing world that if you rely on school registers, they're not very accurate. Teachers write kids down as being here when they're not really here. If you want to do a study that actually measures school attendance, you have to send people to the school to physically count the kids, which to their credit they did in this study. They sent people to like, “Okay, one, two, three, four, five in every school when they wanted to do it” and they did it at random times. If you want to get an actual, an accurate count of the kids.
Aubrey: True.
Michael: What they found in this reanalysis was that the deworming schools got way more visits from the counters, from the auditors, and the schools that got visited more had better attendance rates. It could be just the fact that someone was checking that's doing this and not actually deworming.
Aubrey: Yeah.
Michael: This is what turned me into the fucking joker on this.
Aubrey: [laughs]
Michael: Once there's all this statistical stuff going on and this extra scrutiny on the original paper, people also go back and just read the original paper. The first thing that jumps out at you is that this wasn't a deworming program. It was a deworming and education program. The schools that got the deworming pills also got classes about like, “Why it's important to wash your hands, better sanitation techniques, how to prepare food?” The kids and their parents in some cases were getting information along with the deworming pills that was related to not getting reinfected with worms.
Aubrey: Nah, but it's definitely those pills.
Michael: This is the thing is like, the researchers say that, “When you survey the kids about how important is it to wash your hands and stuff, there's actually no difference between the kids that got the education stuff and the kids that didn't.”
Aubrey: Wow.
Michael: As we talk about on this show all the fucking time, public education campaigns don't really work.
Aubrey: Yeah.
Michael: Knowing that washing your hands is good, it doesn't give you running water in your house.
Aubrey: Yeah.
Michael: The researchers are like, “Oh, well, it doesn't matter that we also give them education programs, because the education programs didn't work. So, we can scale up the pills without scaling up the education.” But the thing is, they didn't check the effect of the education on the school attendance. There's literally no way to separate those things. You're doing two things at once and then school attendance gets better.
Aubrey: I am just really breathing through feeling my blood pressure rising.
Michael: [laughs]
Aubrey: You don't have to be the world's greatest fucking scientist to figure out that if you give somebody deworming pills and also, a bunch of educational resources and support for their lives, you can't then say, everything that's changed is just because of those pills.
Michael: We've been through this so many times as a field. As a former development aid guy, this is what drives me nuts is that what you have over and over again, is you have a technical intervention. Something like a medical intervention, or better lighting in schools, or whatever, something that's easy to scale, right?
Aubrey: Yeah.
Michael: Then you also have all these social things that go around it like getting to know the kids, having them attend these educational seminars, having probably just like more adults in the school generally if they're participating in a study. And then, when it's time to upscale, we're like, “Oh, this is a proven development intervention. So, we're going to upscale the technical parts.”
Aubrey: Yeah.
Michael: But you're not going to upscale all the fucking social stuff that made the technical stuff work.
Aubrey: God dammit.
Michael: That's not the purpose of randomized controlled trial. The entire point of all of this was to be like, “We're going to do it like pharmaceuticals do it. We're going to do it in a way that allows us to say with certainty that this is the intervention that mattered.”
Aubrey: Yeah.
Michael: And then they do this one-on-one shit, where it's like, “Oh, well, we're going to do two things at once and then upscale, one of them.”
Aubrey: Yeah, totally and also, this does feel related to, because this is all happening at the site of education as a field, it also feels totally fucking wackadoo and deeply predictable to have a bunch of economists ruling and be like, “Teachers, you've been fucking it up. You're welcome. We're here. We figured it out.”
Michael: I know. [laughs] Okay, are you are you ready to explode?
Aubrey: Oh, no. I guess. [laughs]
Michael: The other thing that people start bringing up about the original paper is that, that number, it reduced absenteeism by 25% is a little bit misleading.
Aubrey: Okay.
Michael: If you imagine a school with a 98% attendance rate and then you switch the vending machines from Coke to Pepsi, and then they have a 99% attendance rate, you could say that they reduced absenteeism by 50%.
Aubrey: Ah, God damn it.
Michael: That sounds really impressive. But what you've done is, you've gone from 98% to 99%, which is not that impressive.
Aubrey: No, this is the fucking diet marketing of like, “You're twice as likely to lose weight on our diet than on this other diet.”
Michael: Exactly.
Aubrey: You're like, “Whoa, okay, so, you lose one pound instead of half a pound at a year? Great.”
Michael: Yeah. You're twice as likely to win the lottery when you buy two lottery tickets. That doesn’t mean [laughs] you can win the lottery.
Aubrey: Ah.
Michael: The actual attendance rates at the school that didn't get deworming pills, the attendance rates are 75%, at the schools that did get deworming, it's 82%.
Aubrey: Oh, God.
Michael: Is that nothing? No, it's a 7% bump in attendance rates, but also, a 7% increase in attendance, much less we must scale this to 140 million children immediately. Another thing that again is in the paper that was published in 2004 is that the deworming pills did not have any effect on cognitive ability or school performance.
Aubrey: So, nothing.
Michael: Nothing. Exam scores, they did some IQ testy pseudoscience bullshit on the kids, nothing. So, it only affected attendance.
Aubrey: I feel I'm resorting to fucking Lamaze breathing to get through this shit.
Michael: [laughs]
Aubrey: I feel I'm this close to deal with my fucking rage about this shit.
Michael: Well, the whole narrative of this intervention doesn't really make sense. The theory of change that deworming is supposed to do is, you give kids the deworming pills, their health improves, their school attendance improves, their test scores improve, and then their life gets better. That's like the chain of events, right?
Aubrey: Yeah.
Michael: But what we have now is, we have, you give the kids deworming pills, it doesn't seem to affect their health, it does affect their school attendance, but it doesn't affect their school performance.
Aubrey: Yeah.
Michael: It just a mess.
Aubrey: I don't know, man, this feels an indicator of the depths of two major appetites. One is, how fucking hungry we are at all times for a silver bullet thing and how much more fun that gets to people and exciting when it's an unexpected thing like, “Oh, if you get this one pill, you get better at school. Imagine.”
Michael: Totally.
Aubrey: And appetite number two that is shaping this is the unending appetite of white people to be like, “We found the thing, we solved it. You're welcome. You couldn't do it yourself. We did it for you.”
Michael: Exactly.
Aubrey: Not a bad impulse to want to help. A terrible fucking impulse to want to save people from themselves.
Michael: Yeah. I actually got radicalized on this. I think a decade ago now, where I was writing an article about how big ideas were ruining international development, I called up one of the charities that was doing one of these projects in India. They were expanding the mass deworming programs to India. One of them was Kafkaesque interviews I've ever had, ask them like, “Okay, how's the project going so far?” They're like, it's great. We've distributed 10 million pills or whatever it was.” I'm like, “Okay, great. What are the results on school attendance?” They were like, “Oh, we're not measuring that.”
Aubrey: What?
Michael: All because it's proven. It's proven that it affects school attendance. We don't really have to measure that anymore. I was like, “Well, it's proven in 75 schools in Kenya 10 years ago, you're doing this in India, which is a completely different educational context of schooling and the government's role in school, everything else.” I'm like, “So, you're just not testing the effects anymore” and they're like, “But we are testing the effects.” I was like, “Well, then, what are the effects?” They're like, “Well, we've distributed 10 million pills.”
Aubrey: Okay, okay. You and I get into this shit often.
Michael: [laughs]
Aubrey: That's a fucking output. It's not a fucking outcome.
Michael: Exactly.
Aubrey: That's step one to do what you set out to do, it's truly fucking wild. How often you and I get into data on this show and the results of a given study, or a clinical trial, or whatever are just, “Here's what we did.”
Michael: We did that.
Aubrey: And then that’s celebrated like something that has an impact on people's health out in the world.
Michael: Aubrey, save your rage.
Aubrey: No, I'm getting too angry too soon, aren’t I?
Michael: This is a three-part debunking and that was only part one. [laughs]
Aubrey: What? That is frankly, nothing short of a personal attack.
Michael: At the same time that this reanalysis of the 2004 paper is going on, I think this is a coincidence. There's also a meta-analysis of every previous study on deworming that's ever been done.
Aubrey: Oh.
Michael: One thing that drives me absolutely batshit about this story and about the way that it's presented in this effective altruism book is, everyone pretends like no one thought of fucking deworming before. There have been deworming projects since the 1970s.
Aubrey: Yeah.
Michael: In 2015, there's a Cochrane review of every previous decent study on this. It covers 45 studies going back to the late 1970s. It includes 20 randomized control trials in Africa. The review says, “There is quite substantial evidence that deworming programs do not show benefit in terms of average nutritional status, hemoglobin, cognition, school performance, or mortality.” Nothing.
Aubrey: That's the good stuff right there.
Michael: The world's largest randomized controlled trial or at least, according to the study itself was performed in India in 2013 on deworming, it finds no effect. There's a huge randomized control trial in China, which also does not find anything. There have been 10 years of various deworming projects in Myanmar and the rates of worm infection are now higher than they were 10 years ago.
Aubrey: What?
Michael: I don't think that has anything to do with the studies itself. I think, again, I think there's something else going on there.
Aubrey: Sure.
Michael: What drives me nuts about this is like, “Why would we still be talking about this 2004 study in Kenya when there's 50 other fucking tests of this by now?”
Aubrey: Yeah.
Michael: People have tested this and it has been found wanting.
Aubrey: Time and time and time and time, again.
Michael: It doesn't have any of the effects that people said that it had.
Aubrey: And that's on any measure, right? We're no longer in school attendance world.
Michael: They mentioned school attendance. This is actually one of the main things of the worm wars, the big methodological debate that comes out is, because Michael Kremer and others say that like, “You can't test whether it affects school attendance and a meta-analysis, because there's not enough studies that do school attendance. It's only based on this one.” They all like they throw accusations back and forth, ba, ba, ba. But there are other studies that look at school attendance and it doesn't seem to have any impact on. And also, even if it did, it's like, why are we talking about school attendance when it doesn't affect school performance and it doesn't affect health?
Aubrey: Right. Then what?
Michael: Then what?
Aubrey: You got them in the room, then what?
Michael: And also, there's also some interesting second order effects that oftentimes, when school attendance rates go up in the developing world, the teachers are then overcrowded and lots of the kids end up doing worse on tests, because they're getting less attention from their teachers. Not to say that school attendance is bad or whatever, but anytime you're doing these development projects, there's always these weird second order effects. And so, there are actually places where even if it did increase school attendance, if it increases it significantly and they're not able to hire teachers, it might not actually have the long-term effects that you're going for.
Aubrey: Yeah.
Michael: Every study on this seems to find that it doesn't really do what they set out to do. A lot of these studies, you can tell they don't want to admit that it didn't have any effect. You and me are trained to say this. Now, you read abstract and they'll be like, “We distributed 50 million pills,” and then like, “We surveyed a bunch of kids and the kids had heard of the program.”
Aubrey: Yeah.
Michael: And you're like, “Wait, [chuckles] that's not like--" No one would look at a deworming program be like, “Have the kids heard of this or not?” You're clearly picking something you can publish.
Aubrey: We know that calorie labeling works, because some people saw a billboard about it one time.
Michael: Exactly. I was sub tweeting this one when we were recording that episode.
Aubrey: [laughs]
Michael: I was sub tweeting the deworming people.
Aubrey: I have to say, I really appreciate how often in the researching of an episode, you or I end up cryptically tweeting about something, so that we don't spoil it only for the other person.
Michael: The thing is, I was going to block you on Twitter, because I was about to tweet something, salty about deworming.
Aubrey: [laughs] Were you fully going to block me?
Michael: I had a whole plan. I was going to be like, “Aubrey, I just want you to know, I blocked you on Twitter, it's not you. I tweeted about deworming.” But then I was like, “I don't want her to know.” So, then I walked away from my computer, ultimately.
Aubrey: Here's the hilarious thing. As you know, I have been very off Twitter for the last, I don't know, four months or so. The thing that is a hilarious to me is, I think the rest of the internet would be more likely to figure that out than I would.
Michael: Yeah, exactly. I know. And then, “Did you know, Mike blocked you on Twitter?”
Aubrey: Yeah, totally. [laughs]
Michael: This is like, Mike blocked Aubrey on Twitter.
Aubrey: Trouble in paradise.
Michael: Okay, final stage of the debunking.
Aubrey: Oh, okay.
Michael: Okay. We've talked about the 2004 paper problems, we've talked about the field of deworming programs in general, meta-analyses problem. We're now talking about the problems of upscaling.
Aubrey: Okay.
Michael: Because this is another thing that as a former development guy, I know that what happens in 75 schools in Kenya, it doesn't happen the same way once you start rolling this out on tens of millions of kids in entire countries.
Aubrey: Sure.
Michael: This is something that happens in 2008 in a region of Tanzania called Morogoro. Tanzania adopted a national deworming strategy and was rolling this out across the country. And so, the idea was that they were going to give kids a bunch of deworming pills and they had a town hall meeting a couple days before this happened to tell the parents about it. But according to the parents later, they didn't really get the message that this was happening or they just had other things happening that day like agricultural duties things at night that they had to do. Very few parents ended up attending this one town hall meeting being like, “We're going to give your kids deworming treatments.”
According to the parents who actually went, the information that they gave the parents there was not even very good, because they didn't really explain to the parents that they were doing prophylactic deworming. Deworming pills are pretty widely available in Sub Saharan Africa. When you get symptoms of a worm infection, you go to the pharmacy and if you can afford it, you buy the pills, and you take the pills, and the worms are gone. These are known drug treatments. But the idea of, we're going to give every kid deworming pills just in case, that hadn't really been done before. It's a little bit like some teacher comeing to you and saying like, “Well, your kid takes Tylenol for headaches, right? Well, we're just going to give every kid Tylenol at 3 pm every day.” You'd be like, “What?” It just doesn't really fit with your understanding of how that particular medication works.
Aubrey: Yeah.
Michael: After this town hall meeting that very few people attend and it doesn't give very useful information anyway, they start doing this deworming with the kids. But another change that they make is that the deworming pills that people in Tanzania are familiar with, I think it's called albendazole. It's a little tiny pill and apparently, it tastes sweet. The kids are used to taking it-- You know, it's difficult to get kids to take pills in the first place?
Aubrey: Sure, absolutely.
Michael: Because it's small, because it's easy to swallow, the kids are used to it. So, it's not that big of a deal to get kids to take this. But there's different species of intestinal worms. They're also having these kids take this other pill that is big and bitter and they have to test the kid’s height before they give it to them to get the dose. And so, some of the kids are taking three of this, some of the kids are taking two of this, it is just big and really unpleasant to swallow.
The other thing, I cannot fucking believe this, but the second pill that they're giving people, you should take it on a full stomach, otherwise, it starts to give you these side effects. But we are in a remote, very poor region of Africa. Most of these kids are only eating one meal a day. So, it’s like they're not taking it on a full stomach. Whether you told the parents that or not, they're not taking it on a full stomach.
Aubrey: What are the side effects?
Michael: It's like stomachache, nausea. It's nothing serious, but the kids start coming home from school and they say, “Oh, they gave us all these pills today. They gave us this normal pill, but they gave us this new pill and they measured our height” and Tanzanians are used to their weight being used to determine doses of various anti-malarials and stuff.
Aubrey: Sure.
Michael: “They are measuring our height instead of our weight, they give us this new pill instead of this old pill and also, I have a stomachache, and maybe I've been vomiting, maybe I have a headache, I have these weird symptoms.”
Aubrey: Yeah.
Michael: The parents obviously like, “What the fuck?” A lot of them have not heard of this at all.
Aubrey: Yeah, totally.
Michael: And another thing that happens is, fairly shortly thereafter, a little girl dies. She dies of malaria, she dies in the hospital, she was not given the pills, but it starts to look to people like these two things are connected. All of a sudden, these kids are taking these weird pills in school, they're getting weird symptoms, and then this girl dies. And so, people on the local talk radio stations start talking about three or four kids have died because of the pills.
Aubrey: Yeah.
Michael: And so, what starts happening is the parents formed these kinds of mobs and start showing up at the schools. This is an excerpt from a PhD dissertation that's written about this. It says, “In some schools the situation turned violent as parents and guardians argued with teachers who had been administering drugs. In one primary school and gym we know, a classroom wall was pulled down, school property was destroyed, the National Flag removed, rocks were thrown at the teachers, and one was severely beaten.”
Aubrey: Holy shit.
Michael: “The list of injuries suffered by teachers included a broken arm, a broken finger, and a serious head injury. In one school, a health worker had the drugs stuffed down her throat causing her to choke and vomit.”
Aubrey: Holy uell.
Michael: I know. The riot police were mobilized and dispersed angry crowds at a number of schools using tear gas.
Aubrey: Jesus.
Michael: The Ministry of Health and Social Welfare officially suspended the program in Morogoro and throughout mainland Tanzania, the same day. I want to be very clear that there's a lot of really fucking ugly tropes about savage Africans and Africans blunt. There's nothing unique to African parents here. We are talking about a program that is giving a medical treatment, a novel medical treatment to children without the knowledge or consent of their parents. People anywhere, [chuckles] I think very rightfully freaked out about this.
Aubrey: Right. And also, white people in the US are losing their shit about their kids have to talk about race, sometimes.
Michael: Exactly.
Aubrey: It is, to me, a deeply understandable reaction that like, you find out from your kid that they've been given a new pill at school, you don't know what that was, you don't feel like you were notified about it, and now, your kid is getting sick as a result of that pill, right?
Michael: Yeah.
Aubrey: I don't know, man. This is every episode of Law & Order: SVU. “If anybody ever tried that with my kid, they've ever held a--.”
Michael: Yeah.
Aubrey: You know what I mean, like, all of that shit. This is a way that we talk about parenting and it's a way that a lot of parents feel which is like, “Do not fuck with my kid.”
Michael: As soon as we start talking about whether a development intervention works or it doesn't work, we're already in the wrong place. Something can work and not be rolled out well. It can work in one place and not work in another place based on really, really obvious social and political conditions.
Aubrey: Yeah.
Michael: One of the things that happens after this in Tanzania in 2008 is the ruling party starts blaming the opposition party and says, “Oh, they're the ones that started the rumors,” and then they use it as an excuse for a crackdown.
Aubrey: Yeah.
Michael: You might want public health not to take place in a political environment, but it does. it just does.
Aubrey: That's right.
Michael: This is obviously unextreme case. This is by far the worst case of deworming being rolled out too fast that I was able to find in the world. But also, these kinds of problems happen pretty frequently with deworming programs, because they're being rolled out so quickly. One of the things that they don't really mention in the follow up literature to this 2004 paper in Kenya is it, in a lot of schools, the teachers aren't distributing the pills. You can do these mass deworming programs, they’re like, “Oh, 80 million kids or whatever.” You give a thousand pills to the teachers and they just don't give them to the kids, because the teachers don't know why the fuck they're doing this. They're not getting paid extra to do it, they're not given the context for it. Then you can go back to your donors and be like, “We distributed a thousand pills in this rural village.” But it's like, “Did they get to the kids?” There're various studies that measure the uptake and some of them, it's like 20%.
Aubrey: Jesus.
Michael: This drives me nuts. Even in the original 2004 study in Kenya, again, in the paper itself, the first thing, they gave kids a couple rounds of pills, it was every six months they gave them deworming. The first time they gave the kids deworming, 70% of the kids, they were eligible, took it. By the fourth time, they're giving kids the pills, only 44% of the kids are taking them. All of the stuff about costs, the pills are only 15 cents. If you hear different numbers. I've heard two cents, I've heard 45 cents. The original 2004 paper says, it costs about a buck 50 per student.
Once you add up that we had to give it to them every six months and then all this distributional stuff, the low cost of this intervention is based almost entirely on pharmaceutical companies giving it away for free. A lot of volunteer labor to roll this out, teachers doing this with no sensitization, all of like the low-cost justification for these programs depends on the fast, cheap, universal rollout that caused the problems in Tanzania. As soon as you start slowing down to actually see if this is the right thing to do, the cost savings often disappear.
Aubrey: Well, and also, this feels very life hack adjacent, right?
Michael: Yes.
Aubrey: It feels doing that, but with entrenched public policy and public health issues.
Michael: Right.
Aubrey: If someone came along and said that shit about like Brexit-
Michael: [laughs]
Aubrey: -they're like, “Oh, we got this one cool thing you do” and then Brexit, “We just done did it. It's just all fixed.”
Michael: Right.
Aubrey: People would rightly be like, “That doesn't seem right.”
Michael: Okay, so, we've done the bunking, and we've done the debunking, and now we have to do a little bit of rebunking.
Aubrey: [laughs] What's getting rebunked?
Michael: Okay. That Guardian article that you read that was like, “Deworming has been completely debunked and it's gone.” It is famous in the worm wars fears for being a little overstated. Journalists only know how to tell two stories about international development. It's either, this is going to solve everything or this is a debunked fucking lie. We went through this with microcredit. Remember, it was like cover of Newsweek, “Microcredit is going to solve poverty.” Then they expanded it again, too quickly, they expanded it to areas where it didn't work as well, and then was like, “Microcredit is a fucking scam.”
Aubrey: Yeah.
Michael: It’s like, ”Neither one of those are true.
Aubrey: Neither one of those are true and also you only get those two stories if you keep buying into the first one that's like, “This is going to fix everything.”
Michael: Exactly. And also, my takeaway from this original 2004 paper that has been dissected so much is it like, after all the statistical reanalysis, the schools that got the deworming treatments have a roughly 6% higher attendance rate than the schools that didn't get deworming. To me, that is a promising result. It's not bullshit and it's not going to save the world. It's promising.
Aubrey: It feels like a grounded result, too.
Michael: Right.
Aubrey: If you think about your life and things that have made differences in your life, my guess is that it's a constellation of a lot of 6% things.
Michael: Exactly. It's the study to me that it's like, “Okay, it seemed to “work in 75 schools.” Let's try it in another 75 schools.”
Aubrey: Yeah.
Michael: It seemed to be fine in this region of Kenya, let's try it next door in Uganda.
Aubrey: Yeah.
Michael: It never should have been expanded to 140 million kids. The thing that I wish that I could impress upon people in development aid is that literally nothing is going to be so good and so proven that you can go from 75 schools to fucking 3 million schools on another continent.
Aubrey: Yeah.
Michael: You're never going to find it.
Aubrey: I'll say in public policy advocacy world, which is the world that I come from, there is such a deep scarcity mindset that you start from a place of constraints rather than starting from a place of like, “Okay, if we really wanted to get to the bottom of school attendance, what would that take?” Instead, we start from a place of like, “We found this one thing and that does it. We're just going to get a shit ton of it out there at the end and just hope for the best” that we get to the passage or non-passage of a policy and we think that's the end of the story. And actually, that's exactly when the rubber hits the road.
Michael: Right.
Aubrey: You have to pay just as much attention to implementation that the least exciting sounding thing.
Michael: I know.
Aubrey: But that's where the promise of the policy comes to life. In this one, it feels folks just completely skipped over program evaluation and assessment stuff.
Michael: Right. Well, there are places where this works. There's a big study in Uganda that finds that it does increase kids’ heights and weights, and it does increase their test scores. Although, it doesn't affect their school attendance, which I thought was really funny. So, it's like the mirror image of the Kenya study, but it's okay, so, it works on these metrics in this place. By far, the most interesting study I read on this was, there's a meta-analysis of 153 deworming projects in Kenya. The whole paper is about like, “Okay, what makes this work?” Because every development intervention works in some places and not others, and it's not always obvious where it's going to work. Like you said, implementation, this is the hard part.
What they find in Kenya is that the specifics of the implementation really matter. Basically, every deworming project that only gives kids one dose of deworming pills doesn't work, because the deworming pills, they kill all the worms that are inside of you just like napalms your entire digestive system, but it doesn't prevent you from getting reinfected. If you only go there, and give kids one round of pills, and you come back in six months, they'll have worms again. It really only works in areas where there's really high infection levels because again if you have a light or moderate worm infection, it sounds really gross to think about, but it often doesn't cause any symptoms. The worm just lives in your body for a year, and then it dies, and you poop it out, and that's it.
Unless you're dealing with an area where there's large numbers of heavy infections, in the original study, it was a 22% heavy infection rate. That's exactly the place where you should be doing deworming. That's really debilitating for kids. But if it's in a place where 10% of people have worm infections only, maybe 1% of people have heavy worm infections, you don't have to give pills to every single kid for that. It's like giving every single kid a Tylenol for one kid having a headache.
Aubrey: Yeah.
Michael: Another thing they find in Kenya is that it works much better when it's coupled with other interventions. When you're giving kids these deworming pills, even if they do have heavy infections, the worms are gone, but then they need extra food to kind of get catchup growth, because oftentimes, they've had stunted growth and stunted weight. They need to eat more for a couple months after that to catch up to where they should be. But if these are really poor kids, and they're in a really poor area, and they don't have access to food--
Aubrey: Right. Good luck.
Michael: Yeah, it's like there's no point in deworming them because whatever benefits they would get from deworming, they want to get because there's not enough food around for them to eat. You also have to do these nutritional interventions and in general deworming works a lot better when it's coupled with this much more holistic program about sanitation, getting people running water. Also, some education programs are part of this. I'm skeptical of those in general, but apparently, they work here in some circumstances. Deworming as part of a larger program of like, “We're going to work on improved sanitation in this area,” then like, “Yeah, they're really an essential component of that because you need to get the fucking worms out of the kids.”
Aubrey: Yeah.
Michael: But then you're back to an expensive project and you're back to a project where you have to actually know the local area, and you have to talk to people beforehand, and you have to build relationships, and donors don't like that. The TED Talk industrial complex Davos, they don't like that, either.
Aubrey: I think part of the reason that that stuff feels so frustrating to folks is that it feels so slow compared to, “I got a million pills out.”
Michael: Right.
Aubrey: What I would argue is that building something slowly and right, and making sure that it works, actually, probably takes less time than going on five of these wild goose chase scenarios-
Michael: Jesus Christ, yeah.
Aubrey: -where everybody thinks we found the silver bullet for whatever. It's the public policy in international aid equivalent of a get rich quick scheme. Do you know what I mean?
Michael: Oh, it is. It literally is for the developing world like, get rich, quick.
Aubrey: Yes.
Michael: The most chilling thing to me about this is the way that this generation of development aid has replicated exactly the mistakes of the previous generation.
Aubrey: Boo. Say more about that.
Michael: When Michael Kremer got into development during this statistical Dark Ages, everything was gut-based, vibe-based. And they've now created a situation where they're basically saying like, “We're just going to ignore all of the data.” The World Health Organization is still doing math deworming, the World Bank still stands behind this, the Copenhagen Consensus, the GiveWell charity, which is a group that I really respect has written these weird defenses of deworming where they're like, “No, no, no, ignore all the meta-analyses. What we're really interested in is the 10-year follow-up study. We know that deworming works because of the 10-year follow-up study,” but now, there's no plausible path to them having better lives. It's completely deranged to say that a program of deworming in which up to half the kids didn't even participate, didn't even get the deworming pills, and had no effect on health, no effect on test scores, and an extremely modest effect on attendance would be affecting people's earnings 10 years later. I'm sorry.
Aubrey: Yeah.
Michael: It isn't plausible to me and also it's very weird for this entire field of development. The whole thing was like the random easter's, that's what they call this generation of mostly economists who came into development aid and were like, “Let's do this in a more rigorous way” that now, these same random easter's, these rigorous people are now telling me to ignore meta-analyses on the basis of a single study. That is so chilling to me. It's like, “What was the point of all this?”
Aubrey: Yeah, imagine being like, “Definitely don't pay attention to a fucking Cochrane Review.”
Michael: Right. Okay. This is what Michael Kremer during the whole warm war, everybody fighting on the internet period in 2015, this isn't a response to the response to the response to the response. This whole thing is fucking Tumblr comments in 2014.
Aubrey: [laughs]
Michael: He's talking about defending deworming. He says, “The estimated value of benefits exceeds the cost by more than a hundredfold. While the results of any one study should not be taken in isolation and effects may differ across environments, even if policymakers believe there's a one in one hundred chance of experiencing effects of this magnitude, the expected benefits of deworming would exceed the costs.”
Aubrey: They're playing our song, Mike.
Michael: He's basically saying like, “Even if this doesn't work, look how cheap it is. Look how cost effective this is.”
Aubrey: This is like the asbestos argument.
Michael: GiveWell says same thing. GiveWell says, “Because mass deworming is so cheap, there's a good case for donating to support deworming, even when in substantial doubt about the evidence.”
Aubrey: This is very similar to some of the responses to the calorie labeling stuff.
Michael: Oh, yeah?
Aubrey: Yeah, absolutely. The people are like, “Well, it doesn't work for everybody, but look how cheap and effective.”
Michael: Right.
Aubrey: And you're like, “Effective measured by what?”
Michael: Right. Effective, how?
Aubrey: We're doing something, we're not doing nothing, but also, the something that you're doing has the effect of nothing and also, plus maybe negatives like, “What?”
Michael: I forgot to mention, Michael Kremer won the Nobel Prize in 2019.
Aubrey: Come on, Michael.
Michael: It's not just for this. It's for other work and it's for this whole kind of the systematization of statistics within international development, and he shared it with some other economists and like, “I actually think that using more randomized controlled trials in development is a good thing. I think statistical rigor is a good thing.” But also, that has come with drawbacks, and side effects, and a lot of weird and bad, coldest acts. I also think just defend the original 2004 paper. There's not going to be a perfect methodology ever.
Aubrey: Sure.
Michael: You can find things to quibble with in anything in a meta-analysis, in any individual study and ultimately, every single thing that we do in public health is going to be a judgment call. In principle, that's not necessarily a bad thing. I think the idea of a judgment free science is always going to be a myth. You're always going to have to have values.
Aubrey: Yeah, and if you don't get explicit about articulating and committing to your own set of values, then values will show up in your work-
Michael: Yeah, exactly. Yeah, that's a good way to put it.
Aubrey: -without you're intending them to, right?
Michael: Right.
Aubrey: If we're thinking about, for example, pretty much all of the weight loss research that we have out in the world starts from a place of assuming that fat people's experiences are immaterial.
Michael: Right.
Aubrey: If I asked some of those researchers like, ”Hey, do you think that the life experiences of people who are trying the things that you're testing out matter?” I imagine quite a few of them would be like, “Well, yes.”
Michael: Yeah.
Aubrey: But they haven't designed a thing that reflects that. That's part of what we're getting into here is, this is research that was not designed in a way that reflected an understanding that there was knowledge on the ground that they need to know about. There are a bunch of things that again, if you don't articulate those values, and then take the next step of going, and here's how we're going to operationalize them, other values will speak on your behalf.
Michael: I think the parallel to fat people is really good because it's driven by the same thing of like, “This is for their own good and I'm helping.”
Aubrey: It's driven by paternalism.
Michael: Yeah. I think people give themselves permission to not think things through.
Aubrey: To not think things through and to not ask for feedback. That's the toxic good guy narrative that were [crosstalk] so much in international development, and in public health, and in lots and lots and lots of places.
Michael: So, that was deworming.
Aubrey: [laughs] Is that where we're going to end it? Just me being like, “Everything's fucked, and we're all done, and we all lost it,” and you're like, “Oh, there we go.”
Michael: Okay, so, this is a digression. You posted on the Maintenance Phase Instagram the other day. It was like a meme about how every millennial podcast is just one person talking excitedly and the other person going, “That's fucked up,” every five minutes.
[laughter]
Aubrey: I was like, “Wow, they've really got our number.”
Michael: I think we both understand the action of owned. Fair enough.
Aubrey: When you're right, you're right.
Michael: I was trying to structure something around like, “How can I get Aubrey to say like, “That's fucked up.”’
Aubrey: [laughs] It's a millennial podcast, that’s fucked up.
Michael: We did it. We got that.
[laughter]
[Transcript provided by SpeechDocs Podcast Transcription]