Community IT Innovators Nonprofit Technology Topics

AI Bias and Data Equity with Heather Krause

• Community IT Innovators • Season 7 • Episode 58

Share episode

Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.

0:00 | 28:15

Carolyn Woodard explores why statistical analysis and evaluations are never neutral with Heather Krause, mathematical statistician and founder of We All Count.

Heather built her career doing statistical consulting, evaluation, and causal analysis in the global south, work that eventually convinced her that the widespread belief in value neutral quantitative analysis is a myth. In this conversation, she and Carolyn dig into why every statistical task, even something as simple as calculating an average, is really a series of choices, and why those choices are never neutral.

Using a deceptively simple classroom size example, Heather shows how the same data can produce two different, equally correct answers depending on whose experience you are measuring for, and why that matters far beyond the classroom. She and Carolyn also talk about a dairy cooperative project in Bangladesh, the discomfort people feel when the myth of neutrality gets challenged, and why transparency, not dumbed down science, is the real fix. 

The conversation wraps with a candid look at how AI fits into all of this: it is not value neutral either, and understanding its hidden choices may be one of the most useful data literacy skills nonprofits can build right now.

Heather and Carolyn discuss:

  • Why statistics and the application of statistics are not the same thing, and how every calculation, like choosing an average's denominator, embeds a value judgment and takes a perspective.
  • A classroom size example showing how measuring from a teacher's perspective versus a student's perspective produces two different, defensible answers, and why the choice matters.
  • A dairy cooperative project in Bangladesh that illustrates how whose worldview gets built into a statistical model shapes what the evidence ends up being used to say.
  • Why transparency about the assumptions behind a data project, explained in plain language, builds more trustworthy evidence than technical jargon or hedged science.
  • Why AI models are not value neutral, how they can hide the choices they make, and how nonprofits might use AI to build more transparent, equitable data practices.

Resources Mentioned:

_______________________________
Start a conversation :)

Thanks for listening. 


Carolyn Woodard

And the average thing, I told my son that, and he was like, You're right.

Heather Krause

It's either a heart emoji or an axe emoji every time.

Carolyn Woodard

Welcome everyone to the community IT Innovators Technology Topics podcast. I'm Carolyn Woodard, your host, and today I'm really excited to be speaking with a new guest on the podcast, Heather Krause, who is the founder of We All Count. So, Heather, would you like to introduce yourself?

Heather Krause

Yes. Thank you so much, Carolyn, for inviting me to join the podcast. I am a big fan, and it was very uh humbling to be asked to have a conversation with you. Um,

Heather Krause

My name is Heather, and I'm a mathematical statistician who started, I started my career as a single mom of three little kids. And so uh I basically uh from the jump built a statistical consulting company because I couldn't afford child care. And so uh and

Heather Krause

I did mostly work as a statistical consultant doing evaluations and impact analysis and causal analysis in uh countries uh in the global south. So my very first job was from in Bangladesh, and uh I was able to bring the kids along and um have an adventurous life. And uh we went from there, and

Heather Krause

I think it was because I was a statistician, but I wasn't working in the same cities or cultures that most of my statistical training came from, nor the same cities and cultures where most of the funding and methodological ideas were coming from that I had an unusual uh an unusual point of view, I guess. Uh, and it was a point of view that led me to become upset by uh what I felt like was people imposing uh values and power dynamics and points of view onto communities in the name of uh statistical objectivity.

Heather Krause

And and I can understand that that was happening with people who wanted to promote the status quo, shall we say? Uh, but I was working for companies and with people who really were had a lot of rhetoric around um wanting to create empowering, you know, they were doing statistics because they wanted to improve the situations or um bring down the structures of marginalization and oppression and poverty. And so that's what the rhetoric was.

Heather Krause

But because I was a statistician, kind of straddling those two roles, I could see that the numbers were not actually, or not the numbers, but the way the data was being used was not aligned with that rhetoric. And it took me a while, a decade at least, to realize that this wasn't actually most of the time because people were bad or stupid or lazy, which is what I thought at first. Um, but

Heather Krause

It turns out that there was just this big gaping hole in our in our education, in our practice, and in our thoughtfulness, uh, that centered around this myth that quantitative analysis is value neutral. Uh, and that if it's a number and it's done with like this magical methodological recipe, then there's nothing we can do but shrug our shoulders and say the numbers don't lie. And um,

Heather Krause

And so that kind of rage uh inspired the rest of my work. And that's how I got here today. I I like to say that uh we all count as a rage-founded organization, but we're now a joy-driven organization.

Carolyn Woodard

I love that. I love that way of putting it. So you thank you so much for coming to here today and talking with me more about I I saw you give this presentation, and I have to laugh because I've been a language person my whole life, and I'm not a math person. So I don't know how I ended up in your session, but it it just blew my mind basically, because I had never really encountered this way of looking at math. Um, and

Carolyn Woodard

You said, you know, in that session that math seems neutral, and like you just said, it seems like it's valued neutral, but it and a lot of people talk about it like it's neutral. The numbers are just the numbers, the numbers don't lie. Um, but it is never ever neutral. So could you just walk ... I know it was an hour-long presentation and we only have half an hour or so, but could you walk us through that presentation, a couple of like maybe the examples that you gave for to help both the math people and the language people out there who may be listening to kind of understand this concept of math not being neutral?

Heather Krause

Yeah, yeah. Thank you so much for coming to that session. That was such a fun conference. I really enjoyed it. And um,

Heather Krause

I just want to clarify one thing that to a certain extent, I I think that math probably is neutral. Math and statistics and the application of statistics are different things.

Heather Krause

Like uh two plus two is four, and that is going to be true um, you know, from any social location and in any part of the world. But the same is not true when we get into um statistical applications.

Heather Krause

So, for example, um, let's let's stay with something as simple as two plus two, which is like, let's pretend that we want to uh understand the average classroom size in a school. And uh that's very simple. And uh that kind of math we do all the time. And let's say that there's three classrooms in that school, and that one classroom has three students in it, one classroom has six students in it, and one classroom has nine students in it. And um, it is very easy to take the average of those three classrooms and say that, well, it's very easy. The um the average classroom size in the school is six.

Heather Krause

And that is correct, that is not a a bad or a wrong answer, but that is not a value neutral answer, that is not the answer that is going to be true, uh, whatever your social location is in this scenario, in this real world. That is true. The average classroom size is six. Uh, if you are assuming that we want to know what it feels like to be a teacher in this school, and

Heather Krause

If you want to know what it feels like to be a student in this school, you have to do the math uh using the denominator of students rather than the denominator of teachers. And I'm not going to walk through the math here because this is an audible podcast, um, but I am happy to give you a link to a YouTube video that does that. Uh, and if you take the average classroom size in this school from the lived experience of students, the answer is seven. And um, those numbers are very close together, but only because we are using a really small school.

Heather Krause

And so the the when we have a task like calculate an average, that task in math and in statistics is comprised of a bunch of choices. And those choices are unavoidable. You cannot take an average without selecting a denominator, and um, that choice is inherently value laden. Uh,

Heather Krause

When you when you get that number, the average classroom size, is it representing what it feels like to be a teacher or what it feels like to be a student? And um, this really matters from a mathematical point of view because it's gonna change the number. And this really matters from a human point of view because only one of those groups of people is gonna have their lived experience kind of elevated as the evidence.

Heather Krause

And sometimes people, like you said, uh see this example and think, oh my gosh, that's so exciting! And and let's talk about this more. And some people um want to chase me down the road in anger and fury because destabilizing this myth of quantitative uh value neutrality kind of crumbles the foundations of their world in a way that they're not comfortable. So, so yeah, uh, but it is true, it's very demonstrably so.

Heather Krause

And this is not saying that science is a hoax um or that, you know, everybody should dumb down their research to make it, you know, palatable and uh politically correct.

Heather Krause

What it actually is saying is the opposite. Uh, we need to toughen up our science, not dumb down our science. We need to adhere more closely to best practices and pay more attention because this stuff is not value neutral and it's really easy to harm communities that you're trying to help with strategic deployment, with developing um new policies and programs, with medicine, uh, with you know, you name it, big tech, of course.

Heather Krause

So uh that bottom line is what I mean when I say um the good news and the bad news is that um quantitative evidence is not value neutral.

Carolyn Woodard

Yeah, yeah. I it's so interesting to me that people really do um get angry.

Heather Krause

Oh, very angry.

Carolyn Woodard

And I imagine people who work in like measurement and evaluation.

Heather Krause

Oh, yes, yeah. All the time. Uh people who work in every sector. There's yeah, but that doesn't bother me because I really do understand it because I went through the very similar education process. I myself definitely got into the profession of mathematical statistics because I wanted to reduce uncertainty. I wanted to get some real answers to what the heck was going on. And uh and

Heather Krause

That's not actually the role of statistics, is not to get rid of uncertainty. It's to help us describe uncertainty, help us to understand where the uncertainty is coming from and which direction it's pushing. Uh, and it took me about 20 years of being a statistician to come to even start to come to peace with that. So I I am not I'm not afraid or worried when people are angry.

Carolyn Woodard

I feel like this does come up in evaluation, that there is an awareness that when you make choices in the evaluation, that you get different outcomes. And you had a great example of, I think it was the women um who were doing the milking and the the what you were trying to measure versus whether that was good or bad for the people who were the milkers.

Heather Krause

Yeah, yeah. Great uh example, uh, which was in Bangladesh, one of the first uh projects that I worked on that helped me start to wrap my head around this. And I believe this project still goes uh actively underway because it turned out to be so successful.

Heather Krause

But essentially, this long story short, this was a project designed to improve the lives of women that live in rural Bangladesh that own dairy cattle. And um part of the pilot, part of the learning program uh was teaching the women uh who owned cows lots of new ways to take care of cows and new ways to measure the uh the quality of the milk and all very, very cool stuff.

Heather Krause

And essentially, I mean again, this may be hard to talk about on an audible podcast, but essentially what really mattered when we were building the causal statistical models or doing the impact analysis was paying attention to whose concept of the way the world works was getting embedded into those mathematical models because those mathematical models uh have to be built and they have to be built by humans. And they

Heather Krause

There's no there's no mathematical model that can tell you this is the way the world works.

Heather Krause

What a model can tell you is, if this is the way that the world works, then your outcome is this, or then your effect size is this, or if your assumptions about the way the world works are correct, this policy is having such and such an effect. No mathematical model in the world can say this is how the world works.

Heather Krause

And that also makes people very uncomfortable and very upset, and it makes other people very excited and overjoyed that we we have an opportunity, that means to sort of, and that's what we did with this um dairy project, check if it works from several different points of view, from several different value systems. And that's true if um we're trying to understand uh whether this program helps women get more milk from their cows, and that's also true if the women are replaced by AI robots, and we want to decide is this program getting more milk from the cows and the AI robots?

Heather Krause

Like it doesn't have to be, it's not about women that live in Bangladesh, it's about any mathematical model and what we assume to be the way the world works and uh the directionality of success. And that's what I try and emphasize.

Heather Krause

Like when I was first starting out with We All Count, uh, I was very, very angry and um would mostly spend my time trying to prove how what people were doing was wrong or was like harming the people that they were claiming to help, uh, and that you know, they had all these choices and they weren't paying attention to the choices.

Heather Krause

And then I realized that that was a an unpleasant way to be in the world for starters, and it also was totally ineffective. Uh, you know, people

Heather Krause

Most of the people that I work with all actually want to make meaning with and for the people that they claim to make meaning with and for. There was just this giant gap in in the way that a lot of data practitioners, myself very much included, uh, were taught how to work through a data project. And um so yeah, it's a lot,

Heather Krause

It's a lot more joyful now. And I try and emphasize that, you know, when people say, Oh my gosh, I mean, I have to start making all of these choices. I say, no, actually, you're already making all of those choices, right? If you're taking the average classroom size, you're already making a choice, you're already choosing a denominator. You don't have to start making more choices, but you do have to start paying attention to your choices, and that's actually an opportunity to do the best uh data project you possibly can, the most useful data project.

Heather Krause

I mean, how many data practitioners get to the end and just watch people not use their answers or not really implement their evidence? And part of that is because it's not usually actually useful. And this paying attention choices will get you more useful, more meaningful, more rigorous evidence. So uh I try to emphasize the upside of it.

Carolyn Woodard

I think, I mean, um, I believe that um I think there's also situations where people don't trust the evidence. I'm gonna put that in air quotes, trust the evidence, because if what your statistical model comes out with, you know, saying is the average class size doesn't relate to their experience or their perspective, then it seems invalid, even though you know you have the math and you did the math, but it does, it's not reflecting their perspective.

Carolyn Woodard

So can you talk a little bit about, you know, going forward? Um, you talked a bit about how you can take different perspectives and compare them so you can run the make different assumptions when you're running your um your math through what you're trying to find out. Um can you talk a little bit about so if we're if we're if you're listening to this and you're kind of stuck in the well, I thought there was just one way to do this. Um, can you give us advice?

Heather Krause

Yes, I can try. Um and it's a I'm so happy, Carolyn, that you're asking me this question because this is another place where people get mad, but from the other, a different perspective, which is even if what you're saying is true, uh you are now feeding people like you know, climate deniers everything that they would ever want to hear, right? Like you were saying that science isn't trustworthy. How you like, stop. And I and I I

Heather Krause

I thought about that for a long, long time. I take that very, very seriously. I don't, I don't just glibly carry on. Um, and I spend a long time, you know, talking to people about the impact on their lives. And and I I think that we are in a very tricky period of time right now where absolutely evidence is being used um nefariously, evidence is being used uh politically, evidence is being manipulated. Um, and so uh I think that not always trusting evidence, regardless of your country or your social location or your politics, um is a pretty good idea at the beginning. Um I

Heather Krause

I spent a long time working with journalists. I really, really love the profession of journalism. And there was this big data journalism wave about a decade ago, and I was a part of that. And um one of the things that I used to try so hard to get the journalists to do was to make the steps that they were taking to make their calculations or get their estimates or build their maps really transparent in a non-technical way. I don't mean transparent in like an academic paper way where we're listing the confidence intervals and we're listing, you know, um Bernoulli adjustments and things like that, but I mean in a non-technical way, like a way that you could explain to your five-year-old why you're making her a bowl of spaghetti. Um,

Heather Krause

Because once you can do that, you really do understand your evidence. And again, the well-resourced, very intelligent, well-meaning journalists that I was working with were terrified to do that because they said if we do that, people will not trust the science. And they were not wrong, but at the same time, not being transparent is not the solution. Um, and so again,

Heather Krause

I'm not a policymaker, I am not an ethicist, I am not necessarily going to be the person that has like the perfect plan to get us out of the conundrum, like the to get us off the double or triple or quadruple edge sword that we are on right now around evidence. But I do know for sure that transparency has to be part of it.

Heather Krause

And that is what we are promoting. We are not saying um to be equitable, you have to use the denominator of teachers, or to be equitable, you have to use the denominator of students. What we're saying is to do good science, you have to tell us which one you use so we can decide whether that answers our question or not, or whether the evidence that you're generating is trustworthy and aligned with the work that we're trying to do. And that is a tough, tough sled right now in a lot of areas.

Carolyn Woodard

You mentioned AI previously, just uh the AI. Robots that are going to milk the cows. And I just, you know, I don't want to end on a downer, but I was very struck in your presentation that you talked about this tendency that we have, especially when it's something that, you know, we're not an expert in, to accept something that's presented in a very confident way, maybe not without all of the assumptions spelled out in a way that we can understand, but it's just presented as like, well, here is the thing.

Carolyn Woodard

And it struck me in that presentation and just in life that that is something that AI does a lot, right? It's that's one of the things that's well known for. It can have a wrong answer, but it'll tell you very confidently.

Heather Krause

Correct.

Carolyn Woodard

So can you, I guess, um, leave us with your thoughts on um the ways that we, you know, like fact-check the AI and the way we should think about those confident answers in the way that we think about science when science is very, very confident about something.

Heather Krause

Yeah, uh, I think it's a great question. And I don't think it's a downer at all. Um

Heather Krause

I think that we are still in the very, you know, very, very, very early stages of AI. And um, I think that we as a society could decide to make the best of it and and really, you know, transform some some things into wonderful situations, or we could decide to make the worst of it and uh really suffer the consequences. Um,

Heather Krause

And again, uh I am not an ethicist or or uh an AI uh expert, but I do know that um at least right now, uh, when you're working on a data project, AI is not doing anything different than you or I or our data teams are doing, right? You ask AI, you give AI the school and say, here's these three classrooms, what's the average classroom size in this school? And the AI is gonna very confidently tell you whatever answer uh it's gonna tell you.

Heather Krause

Most AIs that we have tested are gonna take the uh teacher's position and tell you that the average is six. Uh and it is not gonna be transparent about the choices uh that it made.

Heather Krause

And so part of the good news, if we do decide uh to find a way to use AI that is not gonna destroy the earth or or our our human selves, um would be to use AI to make the choices more transparent. Uh, you know,

Heather Krause

Lots of things about AI do free up uh time. And one of the things we can do with that time is produce more rigorous, more accurate, more equitable, more transparent science. And um, I you know, I hear every single day like I would love to take the time to really consider each of these steps in my data project and document them, but you know, my boss would never give me all this time. And I'm like, well, if you're gonna use AI, you're gonna have more time.

Heather Krause

And you don't, you do not want um AI to be making those choices for you. Because if there's one thing we know for sure, for sure, uh AI is not value neutral.

Heather Krause

No AI model in the world is value neutral, or it's going to be in the future. Um, predictive algorithms cannot be value neutral. It's a mathematical impossibility. Doesn't mean we shouldn't use them, um, but it does mean we shouldn't think that they're value neutral. Uh, and so

Heather Krause

I think that it's possible that AI could be really good news for data equity in that it would help us um think uh in a in a series of structured choices about each task that we're doing uh and make sure that we are making those choices in alignment with the communities, the values, and the purpose of our projects. So I think that that question will is actually ending us on a hopeful note.

Carolyn Woodard

Me too. Like I I see so many opportunities with with AI, and I think like everything, um, especially as you get to know more its limitations and what it's good at, then you can take advantage of that to use it to do the things that it's really good at, like finding patterns and and um, you know, maybe taking some of the busy work off of your plate so you have more time to design better, better, you know, data structures or what have you. Um so yeah, I think that that is that is a positive note. So thank you.

Heather Krause

Yeah, of course.

Carolyn Woodard

Well, thank you, Heather, so much for your time. And I am gonna share in the show notes, uh, you said you would share a um a link uh to the math problems, and then of course we'll share your your company, weallcount.com. You can uh find out more information about Heather and and what uh her company does around this data collection.

Heather Krause

Thank you for having me.