The Answer Was Never Us — with Leilani Battle

Leilani Battle studies how software shapes what we see and believe. The University of Washington professor and co-director of the UW Interactive Data Lab talks with Michael, Hadley, and Wes about an experiment that manipulated people using nothing but loading speed, and why AI models don't seem to recommend charts the way the community that studies charts actually does. Other highlights: rationality's blind spots and a thorough disc golf origin story.

What's inside:

• Loading spinners that quietly change what people find
• AI models that don't recommend charts like humans do
• The blurry line between databases and human factors
• A behavior-change experiment for building fairer models
• Rationality's role in producing unethical outcomes
• Disc golf origin story
• A vegan pancake recipe

Transcript

0:02

Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field. In this episode, we talk with Leilani Battle, who likes things she can repeat, study, and refine. Whether that means finding her disc golf skills, tweaking a recipe, or deriving the perfect vegan buttermilk sourdough pancakes. This same experimental instinct runs through a research at the University of Washington. Leilani built her career straddling two very different research cultures, the computation behind databases and the human centered experimental world of visualization, an experience she describes as essentially getting a PhD in two disciplines. We talk about the magic that happens when those fields meet, how what loads first in a visualization can shape what people find in their data, and how our work now applies that same technical and human perspective to responsible data science. So I'm so excited for you to listen to Leilani. Leilani, welcome to The Test Set. So glad to have you. So Leilani Battle is the Robert E. Denning Endowed Associate Professor of Computer Science at University of Washington and co director of the UW Interactive Data Lab and also a member of the database group. And you do a lot of really interesting work, it seems like on the computational aspects of visualization, but also the cognitive human factor side of it. Why would we say speed up visualization and in and in what situations for people? And also more recently, it seems working a lot on responsible data science as well. So thanks thanks so much for coming on. Thank you for the invitation. So often I feel like I'm like, at the very end, we get to people's hobbies. But I I hope it's okay that we flip it around because I think you have so many incredible, interesting hobbies. I I thought maybe we could go hobby forward, which was, first and foremost, I I do wanna know you mentioned you do a lot of pickling and sourdough, and I think it's incredibly important we get to that. But also, you said your most surprising hobby is probably disc golf. How did you could you just unpack for us your history with disc golf? Because I know you're in the Pacific Northwest, so I don't know if that's related. Or how did you get into disc golf? It's a good question. So long story short, my husband got me into disc golf, and he's been playing for a very long time, since high school, actually. But I didn't really have any interest in disc golf whatsoever until the pandemic when we were all stuck inside. We couldn't necessarily go and do our more social hobbies, But I still wanted to kinda go out, hang out with my husband, just do something outside of my house at the time. And he had been suggesting disc golf to me for years. I think I tried it once in high school or college or something and was like, nope. No thanks. And then he managed to get me to do it again during COVID, because he's like, we could go outside and be in nature and throw Frisbees. And I was like, okay, cool. I'll do it again. And then that time, something stuck, and I really fell in love with it. And I just kept doing it. And something I think that helped at the time, so I was an assistant professor at the University of Maryland College Park, and we had bought a house that was like a fifteen, twenty minute walk from a very beginner friendly disc golf course. So I developed this habit of every weekend going with my husband to this course and playing disc golf. And I just kept doing it, and then when we moved back to Washington, Washington has so many beautiful disc disc golf courses, most of them free to play, actually, too. And there's a so I I never checked out the disc golf scene, like the community in Maryland when I was there, but I started checking it out here because I wanted to be able to play with more women. Most of the people I'd see on the course are men. And I love playing with my husband, but I just wanted to kinda get into it more. And then I started playing tournaments in our local area, and that's really where I kinda shifted and really got obsessed with the sport. Wait. I love that. I didn't realize you're you're a competitor. You're entering the tournaments, disc golf tournaments. Amateur amateur division term or I could go on forever about disc golf. So there are mixed divisions, meaning you could be, like, any gender and play in those divisions, then there are some gender gender and age protected divisions. So there are female specific amateur divisions and professional divisions, and then age protected divisions that start about every decade and eventually go about every five years as you get older. Wes, have you disc golfed at all? I played a little bit of ultimate Frisbee in in high school, but it's been it's been a long time. I'd have to I'd have to really think hard, like, what's the last time I I threw a Frisbee? It's been it's been a while, but, I mean, I think, like, it it it's interesting, like, all the the changes to our hobbies and our personalities that that were caused by caused by COVID. Like, I I think I understand, like, the people that got into pickling and sourdough. And I didn't do sourdough, but I did pickling because all of a sudden I wasn't traveling like I I used to before COVID and I was like, am I I'm stuck at home, like, what can I do that I didn't used to be able to do? Like, oh, sit and watch a thing, you know, do science like inside my my kitchen. So but, you know, now I enjoy you know, I I haven't pickled late lately, but it it is it is, you know, now, like, a new hobby that I indulge in from time to time. And the the pickling I thought this is interesting in a profile that was written on you. They talk a lot about a bit about you doing some life logging. I feel like you mentioned, like, cooking in batches and enjoying sort of doing, like, big pickling batches. I'm I'm curious if you could say a bit more about some of your yeah. Why you pickle and what you like about it. I thought that was an interesting note, some of the batch batch cooking. Yeah. Are these things connected? So what I'll say is probably the most satisfying feeling for me when it comes to cooking and pick, like, any kind of food preparation is when I'm able to come up with recipe or a combination that to me feels optimal. Like, it's really easy to make, it tastes great, and it's repeatable. Maybe it's a scientist to me. I love it when it's repeatable. If it's not repeatable, to me it's a terrible thing to cook because then it implies you can never do it again that well. So I think to me these kinds of hobbies where even disc golf kind of falls into this, where I can just kind of do it over and over and over and hone the recipe, hone the skills. I I know I really like how it feels. And pickling and cooking, even meal prep where I cook in bulk, like, they're the kinds of things where I can do them over and over and over and get better and better and better over time. And also where documenting and just recording my experiences or experiments, if you will, pays dividends. It sounds like you really like you're solving the reproducibility food recipe crisis, but you like the fluency. Like, by yeah. By repeating an action, you can kinda, like, see yourself getting more fluent at it from sort of Yeah. I can experiment with different techniques. I also like to do our call making base recipes where you can take kind of the core of that recipe and then modify it in different ways to get a a different, like, a fresh experience. Because you can imagine if you ate the same food over and over, you might get bored. So having these kind of paste recipes where you can mix things up, it's still repeatable, but gives you variety. Like, I love things like that. Yeah. Yeah. It's it's sweet of you to think about the human factors too that you're like, if I eat the same thing, I'll be bored with it. That's just practical. I like to save money too. I can't can't risk throwing all my food away. I I really love the the Ratio cookbook, I think by Michael Ruhlman, which is like, okay, actually across all of these different types of things, like mostly baked goods, there's just like the same underlying ratio, which I think is really cool. Like, I loved, like, I loved learning, like, a pound cake. It's called a pound cake because it's like a pound of butter, a pound of sugar, a pound of flour, a pound of eggs. And that, like, one one one one ratio is like the basic ratio between basically every type of cake or if there's kind of a like, I think that's just so, like, empowering and, like, cool to cool to know. That's exactly the kind of thing I love. An example of recipe that I worked on for a really long time and really enjoy doing is, like, my buttermilk pancake recipe. So I started making buttermilk pancakes in graduate school, and I got really good at this. But then I became a vegan, which kind of destroyed my buttermilk pancake recipe, at least the old version, and I had to rebuild it. But because I had all those experiences and what makes a good pancake, I figured out what makes a great vegan pancake as well. Do do you kid, do you think we can get that recipe to share with our listeners, Leilani? Like I don't know if it's secret. It's more like I'm probably shy about sharing a recipe because I'm like, oh, what if other people don't like it? Or I don't know. So like a side of my life I'm not used to sharing widely, but I am happy to write up the vegan buttermilk pancake recipe. Yeah. Or just or at the very least Hadley gets it and then it just shares the secret with you. Yeah. I'll make it and then I'll I'll rate it. Yeah. I should also mention it's a vegan buttermilk sourdough pancake recipe, so maybe not all your listeners can make it anyway. I I feel I I just feel like there was there'll be like, there's enough of our listeners who will be able to make it. Who are vegans who have sourdough, I just feel like this this they're probably overrepresented. Yeah. If you have a sourdougher, it's actually pretty easy. So happy to share. I do feel like a lot of this does hit on what seems so interesting about your work is a lot of that balance of, like, there's a lot of computation in it, but also experimentation. Like, how do we for example, in your studies of visualization, like, well, how can we technically speed up visualization? But also, like, in what situations would a person benefit from this? Both, like, the computational problem of how, but also, like, how do we stay tapped into the sort of human centered part? Like, what part of a person's task does this actually improve? Yeah. I'm so curious to get into it. I wonder almost if just leading up to it, you could tell us a bit of how you got sort of to where you are studying these things. Tell us a little bit about your journey to where you are today. I'll try to maybe sprint through kinda my path, career path. Initially, when I was in high school, I was really into drawing and art, and I loved playing video games. So I had assumed that because I was passionate about those things, like drawing and video games or art and video games, that I would end up there somewhere. That being said, I think the influences of interacting with that kind of technology, you know, like game consoles, my family always had a computer at home growing up, things like that. I think I was more open to kind of being in tech in general. And thankfully, I realized right before applying to college, like, universities that, you know, there's more to life than being a game developer or an artist or designer for games. And I, like, based on my dad's interests and experiences around electrical engineering, I figured, oh, that seems like a viable route if my dad was interested in it. And this is kind of the first of some coincidences, I guess, but when I applied to the University of Washington, because I was for me, Washington is home. I grew up in the state of Washington primarily, and the University of Washington would be considered kind of a home university for me. I knew they had a strong engineering program, including electrical engineering and computer science and engineering. I was directly admitted to the computer science and engineering program at UW Because originally I wasn't even going to do computer science. I was gonna do electrical engineering. And with that direct admission, I was like, oh, well this is a pretty competitive program. It's hard to get into. I can give it a try. And if I hate it, I could always do something else. And so that's kind of where I ended up in computer science specifically. So not every part of my life was, you know, carefully engineered to end up exactly where I am. There have been some kind of happy coincidences here and there. I think that is one of them. So I started going through the program, started taking the intro computing courses, ended up loving them. Did not see that happening, but I thought they were incredibly fun. You know, kudos to, back then, the instructors teaching the intros courses. Because I'm at UW now as faculty member, so kinda seeing the other side of it, it's been really fun to see everything that goes into designing a really good CS course, especially at the introductory level and how that had such a huge impact on my life. So I I was enjoying it. My early years as an undergraduate student, I actually struggled to get good, like, competitive internship offers. And my first summer after my freshman year, I didn't have any opportunities to do a tech internship. So I I decided to apply to research internship programs, and I did an on campus research internship program that summer. And that was my first introduction to computer science research. It was actually in professor Rajesh Rao's lab here in the Allen School. And at the time, his he does a lot more, like, kinda neuroscience and engineering related things now. Back then, his lab was a bit more of like a traditional robotics lab. And I I've mentioned this to him too. I'm really grateful for that experience, but I knew I was not a roboticist. So after that, my next summer research experience was actually in databases, in the database group here with Magda Balazinska, who is our school's director. So she's she's been in my life and in my career for a long time, and I really look up to her. But kind of seeing the database group, how it operates, the kinds of research projects they were working on, I started to develop an appreciation for databases. And the work that I contributed to in particular was a little bit more oriented in the human computer interaction direction. So thinking about how we can make people's lives easier when they're trying to use database systems. And I think that kinda set me up for graduate school. So as I was finishing my bachelor's degree, I was looking at what opportunities I had. By then, I was able to get, you know, more competitive internship opportunities and and experience a little bit of what industry is like. And I just felt like I didn't have enough time to explore what I wanted to learn in academia before leaving and joining the workforce. I felt like I got a decent sense of what to expect in industry. I did not get a decent sense of what, you know, what potential I had in academia. So I wanted to keep going. And so Were you pretty set on joining academia by academia by that point? Or I was somewhat conflicted because part of it was my parents, not so I'm the first person in my whole family to get a doctoral degree, and including my extended family. Oh, that's incredible. Yeah. So for them, this is really a foreign concept, and it's even when I had made the decision, it took some convincing of my parents that I wasn't throwing my life away by not automatically taking six figures from Microsoft or Google or whoever, Amazon. And they were really worried, like, we gonna have to pay for this? We already paid to help you go to college. Like, are we gonna have to pay again? Like, just a lot of kind of educating my whole family on what it means to go to grad school in computer science and what the benefits are and what the value could be long term, stuff like that. But, yeah, it's a little conflicted because it's a hard decision to make for a lot of people. It doesn't make sense to go to academia. It doesn't make sense to get a PhD. You you can do awesome in industry without it. So, you know, it's a it's a tough decision to make. But I I landed on the side of I wanna do this. I wanna try. I wanna see if possibilities are out there. The the database group was incredibly helpful and supportive in in helping me apply to graduate programs, and I ended up getting into MIT, which to me was really exciting. That also changed my parents' minds because they've heard of MIT. Yeah. That's incredible. Yeah. They're like, oh, MIT. Well, I guess this grad school thing would be okay. They're like, we could we're we're into it. We're willing to, like, see where this goes. Yeah. Exactly. It was at that point where they were like, oh, okay. Some name brand school, I guess I get it now. I don't no shade to my parents. I love them. It's just they're kind of funny because they don't really they're not academics, so the way they see my job in academia is very different from the way I see my job in academia. But, so then I went to grad school. I started working with Mike Stonebraker. He was my PhD advisor. I was in the MIT database group. Thankfully, MIT also, at the time, and still does, it's even stronger now, but had really great HCI researchers as well. And in fact, it was Rob Miller who doesn't do any viz work, who connected me with another professor at Tufts University, Remco Chang, who is a visualization researcher. And it was with Remco and Mike that I started charting out what I actually wanted to do for my dissertation and learning what it takes to be a researcher, not only in databases but also in visualization. I did get a very small taste in conversations that people in the database group were having at the time at UW around kind of more HCI and visualization oriented work because there were starting to be some interesting intersections. Like, I think at the time, people were talking about, like, Many Eyes, which was, like, a pretty influential platform around back then when I was an undergrad and graduating and moving to MIT. But I didn't actually do any visualization research as an undergrad, and it was really in grad school when I started getting exposure to that and and learning about it. It was a little tough to do it in a database group. I loved being in the database group and I feel like it was a strength as far as my technical ability and my strengths as a researcher. I I feel like I really needed that. But it's hard to say publish papers at visualization conferences when you're used to reading and learning about database papers in conferences. So I really had to find my way. Like, the data the database research community is, like, not by and large, like, not that interested in kinda human factors. Right? It's more about making stuff fast and efficient and correct, like, not, like, easy to use for humans. Is that is that right? Or is it, like was that kind of, like, the bit of database is like a fringe community, or is that kind of fairly central? I love that question actually because it there's some nuance to it. I would say a lot of people in the database community had thought that way when I started my PhD. And I think today, there's much more of an appreciation for user centered database work, And I see more of it getting published at SIGMOD now than before. But on the whole, I would say database researchers do not focus on it. I think they can if you spend time talking to them about it, they can understand the value, but they they just don't prioritize it. And, of course, there's people there's gonna be people out there who are like, oh, this is a waste of time, or this is not my job. I don't see why I should focus on this. And there'll be other people, know, like my PhD advisor who really see the value and opportunity there and don't see it as a sacrifice as far intellectually speaking for database research. I mean, I think what's been really interesting in the last, you know, in the last twenty years has been, like, the that all the work that's happened at the inter intersection of of database systems and data visualization and human computer interaction. And and I think, like, the HCI community has embraced work happening in databases and data processing because without, like, that fast slicing and dicing, like, the cross filtering, you know, being able to, like I think when you see a really, like, a really compelling dashboard with, like, the instantaneous interactions, low you know, having low latency interactivity, like, being able to, like, drag a slider and see, like, sort of start to experience the data through interacting with an interactive data visualization and then seeing, like, oh, the thing that made this possible was, you know, DuckDB, or the thing that made this possible was, like, this analytic database. And so, like, without the database technology, you wouldn't be able to create, like, these compelling, you know, data experiences. And so I feel like there's this virtuous cycle between, like, between the worlds. And so I'm you know, as somebody who's, like, deep inside that world, I I mean, it's just it must be incredibly, you know, incredibly interesting to be able to, like, have influence, like, on both sides of, like, you know, sort of more of the HCI data visualization research as well as, like, the computational side of, like, how do like, what can the computations make possible that wouldn't be possible without it? Absolutely. I think this is one of the most fun parts of my job because I love straddling that boundary. It was really difficult in the beginning just kinda finding my way because you have to I'm I'm fond of saying I basically had to get a PhD in two disciplines to be able to talk to them, but now that I'm there, I really love talking about those research problems and understanding the strengths and weaknesses of both communities, because when you put them together they make magic happen. And one classic example that I love to point to that is kind of very easy the very easy win for that intersection is, you know, Tableau and originally the Stanford project Polaris. That was an example of how putting those two things together can make magic happen in a way that no one community could have possibly done that on their own. With Tableau, just to kind of unpack it a a bit, are you saying so for context, for people to see maybe, like, Tableau is a, in a way, a BI tool where a person can use, like, a graphical interface to click through their data and, like, chart it or present it as a dashboard. What what what are you saying Tableau brought specifically kinda to that to the world? Okay. Cool. So what I love about this because the Tableau paper I think the original Tableau paper came out in, like, two two thousand two. And at the time, it was this product this research prototype called Polaris. And it was kinda funny too when you read the paper because the paper is when you read the paper, the introduction is basically making the case for, hey, viz community, this is why we should care about what databases do and how we can map our visualization primitives onto database primitives to automatically generate SQL queries to process our data. That's basically what that paper was bringing to the viz community. It was saying, a, we should care about this, and b, our theory and database theory could actually be integrated to produce this really nice automated stack going top to bottom and bottom to top. And the grammar that kind of tied it all together is this this grammar called VizQL. Basically, this mapping across the stacks. Because you can think of it as a stack. Right? You have your data processing and storage layers, which database management community typically owns. And then you have the visualization and interaction and is there interface design layers which the visualization community typically owns. And that paper was saying, why bisect the problem in that way when you can actually do this more integrated approach? And nowadays people would look at that, and I even had PhD students after reading the paper read that and be like how is this groundbreaking? Everybody just does this. And I'm thinking that's the point. That's the point. That paper was basically saying everybody should be doing this. And in twenty twenty six, it's very easy to say, oh, but everyone's doing this, whatever. So just the idea that those communities should be working together and integrating what they're doing on a foundational level, I think was a tectonic shift, intellectually speaking, for the viz community and the database community. And I think that's also why Tableau as a company had this, like, wild commercial success soon after that because those communities just weren't connecting in a way that that company was able to kinda put them together at the time. So to be fair, there were definitely companies before and after who've done great things that are kind of us in a similar spirit. Right? I didn't realize though there was a research paper associated with Tableau and a lot of it. That's pretty crazy to hear. I I do find it so so fascinating, like, how we kind of build these, like, boundaries around, like, this is the area in which I work, or this is the area in which we work. And then it becomes almost, like, heretical to kinda, like, step outside of it or combine it with something else. Like, it's very difficult to like, you yeah. Each community builds up its own kind of standards, its own, like, vocabulary, and that's how these you you generate these, like, shibboleths. Like, this is how I advertise myself to other people so they know I'm a serious database researcher or a serious viz researcher. And then when someone comes out from the outside and they don't know how to, like, signal all of these, like, oh, I'm a serious whatever. And just so can be, like, so hard to, like yeah. Like, that I guess I never really understood why, like, interdisciplinary or cross disciplinary work was so hard, but it's not really about the work. It's about the people and the way they define themselves and, like, their communities. Yeah. Absolutely. Yeah. I think so. And lots of times, a community defines itself by the problems it chooses to focus on, that community chooses to focus on. And that work, and then also just my, I like to think the work that I've focused on in my career is trying to break down some of those walls to reclaim some of what we lose by those, you know, those boundaries that we those artificial boundaries that we put up to kind of just define and protect our communities in some ways. Do you feel like today you mostly are the are the conferences you go to today mostly split still, like like, bucketed in one of the two areas, or are there many sort of, like, cross disciplinary conferences for sort of the HCI, viz, and computation stuff? That's a really great question. So as far as conferences I publish in, I will publish wherever I believe the work is a good fit. That could be an HCI conference. That could be a visualization conference. That could be an AI ethics conference. That could be a, you know, per more programming languages oriented conference or a database conference. I think to me, the most important thing is that community cares about the work that we did. So this would be different if I was working really hard to get tenure because I would want one community to know me well, so I'd probably want my home base to be in one or maybe two communities so that when, you know, letters go out recommending my tenure case, they're saying, I know Leilani. She does great work in x. If you're too diffuse in your portfolio, then people can't write those kinds of letters because they'll be like, who's that? Leilani, who's that? Or I don't really know her work. So there's that danger, but if you can avoid that danger, I think it's better to focus more on where the work will be best received and have the most impact rather than worry about, you know, oh, I published all my papers at this. That being said, practically speaking, I can't jet set around the world all year. I have to be practical about what conferences I actually go to. I try to make sure, when possible, try to make sure I go to at least one, like, viz or HCI conference and one database conference a year. The past couple years have been weird, and I've been prioritizing sending my students to conferences instead. But, yeah, typically when I attend a conference, I'll try to go to one of each. And and are you sort of conscious of, like, code switching when you're writing, like, a database paper versus, like, a HCI paper? Yes. I am deliberate, and I do code switch between communities because the things that they talk about, the terminology they use, the things they emphasize are different. That being said, with AI, things are blending a little more. We're sort of ending up in similar places more so than I've seen in the past because we have kind of similar goals we wanna achieve. So I I think things are blending more now, but I do have to code switch when I talk to different communities. And and when you say AI because of AI, you mean, like, the problems it faces. Not that people are using AI to write all their papers now, and hence, all of the papers seem the same regardless of discipline. Yeah. I mean more that, say you want to use AI to help people achieve data science tasks, like complete data science tasks, analyze their data, things like that. Database researchers looking at that problem and visualization, and HCI researchers looking at their problem will end up in similar spaces because there's things like how exactly do we prompt models or design infrastructure around the models to facilitate this. When a model manipulates the data, how do we communicate that to users in a way that they can feel comfortable that they understand what's going on and they can trust it? And then also, if we want to empower users specifically to analyze the data, how can we help them do that in tandem with the model or maybe co plan and coordinate with the model such that, you know, they achieve the goal that they wanna achieve in with hopefully, you know, minimal effort? So with those things in mind, right, database people are starting to have to worry about interfaces a bit more and transparency in the behavior of the system because users are a little less trusting of black boxes and wanna know what's going on. And also, when a mistake happens, it's not super obvious why the mistake happens, so users are gonna need more transparency as well on, oh, why did this AI powered or LLM powered operation do this thing? I didn't understand. And then also on the flip side with these people. Right? Being able to provide enough context to the model can very easily and quickly become a data management problem as well. So there it's interesting how those two sides are blending with this shared interest of maximum utilization of LLMs. I mean, thing I'm curious about is, I mean, is there is there a sense kind of in your in your community that your research community that that, you know, your field is going to experience, like, the same type of you know, or different you know, certain type of disruption compared with more general computer science or or software engineering. Like, I think, you know, software engineering is clearly, like, in a state of upheaval and and disruption. But I imagine, like, in in data visualization and interactive data that that there is disruption, but perhaps we don't know, like, what things are going to look like on the other side. I know, at least in the business world, there's a lot of discussion about, like, headless BI and some and, you know, semantic layers becoming more important, like trying to give agents, like, a simpler language to talk about, you know, relationships and data models and describing data visualizations and and and things like that. And like, I feel like databases have become even more, you know, even more important because they can't be they can't be vibe coded quite so easily. Whereas I I feel like we're gonna see a lot more like just in time, you know, dashboards and like just in time data interfaces where you aren't necessarily looking at, you know, something that was developed over a long period of time, but maybe something that was stood up on top of, you know, stood up on top of DuckDB or on top of one of the interactive, you know, data visualization frameworks, you know, and and like a as like a, you know, throwaway prototype. And, you know, in the past, you you can't you couldn't imagine, you know, building some interactive data application only to throw it away and then rebuild it, you know, the next day. But now that's something that, you know, you could build these, like, just in time interactive data interfaces. And so I feel like that is going to lead to, like, new interesting avenues of research, like how to incorporate, like, how to incorporate LLMs into the, like, you know, essentially into the process. So Absolutely. I think we're already experiencing a massive shift in the way people do data science work with the introduction of large language models. One easy example I can point to is in chart generation and visualization recommendation. A lot of a lot of my prior work has been in visualization recommendation and automated methods to support that, And now we always have to answer this question of, oh, well, how would LLM solve this problem? Or how would someone use an LLM to, you know, generate a chart or generate code to generate charts or even generate dashboards? So when we try to introduce new ideas in that space, we always have to be thinking about it. And also, practically speaking, a lot of people I know, researchers, folks in industry, they'll just very quickly go to a model and ask it to make charts for them. So just on a practical level, people are using LLMs in a way where they would traditionally use maybe viz tools and stuff like that. And also even tools like data science tools themselves are doing it as well. Right? Just automatically using providing AI recommendations. So I think huge, huge change. I can spotlight one ongoing project in our lab around this because we are curious, you know, what the shift looks like. So Will Wong, a PhD student in our lab who's co advised by myself and Jeff Heer, He's investigating this problem by interviewing statistical and and data analysis consultants at universities. Because traditionally, pre LLM days, if you're a scientist and you weren't totally sure how to, you know, analyze and statistically test your data that you collected for an experiment or something, your university, there's a very good chance, like, you could talk to the statisticians there, or there's even a consulting office you could go to, and then a consultant could work with you to figure out what questions you're trying to answer from your data, whether you collected the right data, what statistical models and tests make the most sense, how to process your data properly for it and so on. But you can imagine with LLMs, like, statistical consulting must be seeing a humongous tectonic shift, like, experiencing a huge shift in their, like, client base and how they are utilized, how consultants are utilized. So we are wrapping up an interview study with consultants to see how, like, their perception of language models and how it's affected their jobs. And it's been it's been really interesting. Yeah. It's so interesting to hear that, yeah, that that avenue opening of, like, you're mentioning, like, just being able to ask an LLM to generate a visualization. And in the context of your really thoughtful, careful work on, like, visualization recommendation, still thinking critically about what's the quality of that recommendation and, like, who's asking for it. And it sounds like you're also saying, like, what happens to the people who were who they were going to before for that advice, all those kinds of elements. So from different angles, we've been tackling this problem. So, Will, he the PhD student I mentioned, he also had a a previous paper on evaluating the kinds of recommendations at the time that models were giving when you asked them to generate charts or to complete a partial visualization design. So if you had a partial design, you wanted the model to finish it, what would it recommend? Or if you wanted the model to just recommend charts, you know, what what do you get from it? At the time so to be fair, this is a paper from twenty twenty four. It's twenty twenty six now. Models move fast, update quickly. At the time, his work found that what the models were recommending was mostly fine, but there were some cases where they would recommend weird stuff. Like, would recommend visual encodings that didn't really make sense for the context. And I think kind of the big takeaway for me was that sometimes the models were recommending designs that didn't seem to align with the data that we have in our community. So as visualization researchers, as scientists, we've collected data. We've done tons of experiments seeing what kinds of design decisions make sense, which ones don't make sense. And it just didn't seem like models like, what they were learning on was not that. It didn't seem like it was that. At least not most of the time. And for proprietary models, who knows what exactly they're learning their viz recommendations from? But it seemed like the answer was not us. Sometimes, anyway. Do you think this is changing how the viz community, like, types of tools it makes? Like, my recollection, which is now very out of date, is that, you know, people would tend to produce kind of like, you know, polished UI, point and click kind of interfaces. But obviously, like, an LLM is gonna have a hard time driving that. Do you so do you think this is kinda pushing the viz community to produce tools that are like plain text or JSON or code based that then the LLM can generate the visual specification? It's a good question. So kind of before, during, and now, like this before and during the AI wave, I think we were already in some ways transitioning away from these kind of standalone viz tools, at least in the specific areas I work in. We're sort of transitioning away from these standalone viz tools and more towards, like, modules and things that can integrate into stuff like computational notebooks. And I I personally feel like that's the future, having a standalone. Unless you're providing an environment that other tools would integrate into, I I don't think it's really to your advantage to have a standalone tool that you create as a viz researcher because then you have to argue to everybody why they should leave whatever environment they already use and use your tool instead and then export whatever your tool makes back into their usual workflow. So, yeah, that I mean, that's fascinating. Because, I mean, I I haven't and not that I was ever particularly active in the viz community, but when I was active, that was, like, fifteen years ago. And that was the thing that struck me. Like, why would someone use your cool new visualization when they've gotta do it in a completely different computational environment? Like and so so it just felt like the viz community didn't actually care about people using their visualizations, which is, you know, like, fine. Not all research has to be, like, end user focused, but it did feel, like, weird to me. So that's that's neat that people are more focused and, like, let's integrate this into where people are already working. You still see a lot of stand alone tools just practically speaking because it's easy for a grad student to make a research prototype that way than to build a, like, Jupyter plugin or something. That being said, with large language models, Jupyter plugins are way easier to make now than these to So I think there's less of an excuse now and that integration, because I don't know. I think there is more of a question around computational notebooks years ago, like, earlier in my career. Nowadays, it's like a no brainer computational notebook environment. Like, why wouldn't we use it? So I think the viz community has been forced to adopt it more. Yeah. It's funny because, yeah, because computational notebooks are, like, such a like, in some ways, and, like, I can you know, that's such a terrible environment. But, like, they're they're clearly, like, the least worst environment. Like, everyone knows them. They're just this great, like, you know, ground floor that everyone can rest on. But I'm sure like, I I can just imagine, like, the viz community looking at them and being like this. These are awful. Like, there's so many problems with them that Kinda like MapReduce in the database community. You can't beat them, join them, or come up with a better version. That reminds me of this advice I heard years ago. Like, how like, if you wanna if you wanna pick some, like, machine learning algorithm to use in your actual applied work, like, how do you how do you do what how do you figure out which algorithm should should use? And the advice was, like, go read, like, three of the top recent papers, and then look at what they benchmark their method against, and then use that. Because, like, that's the gold standard. It's so true, actually. That's a smart way of picking a good commercial tool to use. I do I think it'd be nice too to talk about the responsible data science stuff, which I know is a lot more of your recent work. But I figured maybe before we get to that, I'm curious if you have, like, a favorite viz study you've done. Like, do you have a study of yours that's, like, a personal favorite that you've done on viz? I would say there have been, like, takeaways from different studies that I've done that I've really enjoyed. I don't know if I would say, oh, that whole everything about that study, I just love it about everything else and all the rest. You have some favorite takeaways, though? Yeah. I have some favorite takeaways. So one funny takeaway from a study we did in twenty twenty two, I believe. That was the twenty twenty two CHI paper. So this work was in collaboration with Michael Correll. He at Tableau at the time, but now he I believe he's at Northeastern. But the it was around visualization recommendation, and our our idea was to see, okay, how would people like automated recommendations versus, like, human curated recommendations? And what arguments are people making in terms of why they like one versus the other? So what we did was we picked up some picked up some commercial tools like Tableau, Power BI, and such, had them generate recommendations for visualizations of public health data. And this data was specifically a subset of data from NHANES. So the it's basically a national health survey. I'm not gonna make myself regurgitate that acronym right now off the top of my head, but it's basically a national health survey data from the from the, I think from the CDC or or the National Institutes of Health, one or the other. Maybe the National Institutes of Health. But we had the self survey data. We showed it to students, professionals, and researchers in public health. And we showed them recommendations from, you know, automated tools and also from just people crafting recommendations. And the professional like, not the professional. The the participants of our study generally tended to prefer the human created recommendations because they felt that the human created recommendations told more of a story. There's more of a story behind why that recommendation was generated. And at the time, the automated tools, they just they generated pretty overly simplistic recommendations. You could tell they were just putting variables together and then generating charts. And so that, like, is a a fun, like, favorite takeaway of mine. Of course, this is, like, pre LLM, so maybe the automated tools would do better now. So we should redo that experiment. Yeah. It's interesting to hear that it's people they prefer, yeah, recommendations from people. And it sounds like maybe was that surprising to you? What I would say was surprising wasn't even necessarily at the running the study stage, but at the experiment design stage when we were looking at the recommendations. Because we could also see it before we ran the study. We're like, how are these tools generating these recommendations? This seems really generic. Why would anyone want this? Maybe we should've given that feedback to the tool creators, but yeah. That to me was always surprising. But maybe it's because I'm a I'm a researcher who focuses on visualization recommendations, so I could see it when I was trying to use these tools. Oh, you know, these recommendations are not like, I don't understand who they're helping. It doesn't seem that they're helping anybody. It sounds like the thing that really struck you was the as you were running it, you were really struck by the kinda difference between the automated and the humans. Yeah. I was just struck by, like, how some overly simplistic the automated recommendation methods were at the time. Like, I guess they wanted to from a commercial standpoint, I can understand wanting to generate recommendations that will be, you know, consistent. They'll always work. They aren't going to make any customers unhappy, that kind of thing. So I can understand strategically why you would do that to but as a researcher who, you know, likes to do push the boundaries of what computationally we can do, I was thinking there's no way that someone would pick these recommendations over the human curated ones. And I I hope it was not a self fulfilling prophecy. I like to think I designed good experiments, but we we pretty much saw that in the study results. Yeah. It is interesting to hear. It's like you kinda see it coming, but you have to run run it through. Yeah. Yeah. Basically. And then one more study I'll highlight that was really fun, actually, was from the tail end of my dissertation work. Basically, you wanted to see, okay. Well, with recommendations, there could be this issue of performance where, let's say, you're analyzing a large dataset. What if the thing people look at is the thing you can recommend the fastest? Basically, the thing they see first. How could that potentially introduce manipulation of people? Like, what what conclusions they might draw or or what findings they might find in their data? So what we wanted to do is design an abstract experiment to try to test this. Could you, in a sense, manipulate what people found in a visualization purely through latency? Like, how quickly or how slowly people might see the data or different parts of the data. So I designed this, like, really abstracted version of this experiment. I created this image collage and asked people to explore the collage of images, and then they were looking for certain a certain target image. Actually, put two targets, like two copies of that target in the collage, but the goal of the task is to only find one. And you can imagine kinda like Google Maps, hey, you sometimes say you're on your phone and your data speed isn't that great and the map takes a while to load, maybe loads in kinda patchy. Imagine something kinda like that in this experiment, but it's an image collage, so some images appear faster, slower than others. So I designed the latency for the images such that it would look kinda random, but in fact, it was definitely not random. I was manipulating how quickly certain images would appear, and I was trying to make it such that I only use latency to try to incentivize people to find one target over another in the collage. And the the experiments showed that if the latency was pretty low overall, like relatively low, it didn't affect people as much. And then but gradually, as you ratcheted up the latency, like, you made it the tile, the images load in slow enough, you could definitely manipulate people to find a certain target in the collage. So it was just showing how latency matters, how you design a system matters, but also how you recommend things matters. If you do it purely based on performance and don't pay attention to what, you know, what you're prioritizing Like, if you're if you have options to show a or b and you always make a show up faster, depending on how much faster a shows up, you might be incentivizing the user to just always look at a and never look at b. That seems like super relevant in the era of LLMs again where, like, latency is becoming so important again. Absolutely. So that to me was like a fun experiment that I enjoyed. I really enjoyed the design of the experiment. The takeaway was pretty straightforward in my opinion. In my opinion. Like, I guess one thing that was surprising about it was people kind of thought of latency as a sort of binary thing, like, oh, there's either too much or too little, but that experiment was showing. It's actually kind of a gradual effect, And the more latency you have, the worse the effect becomes for people that get become kinda biased. And the, I guess, the latency between two things being relative as well, I guess, where two things are competing? Yeah. Like, in an image collage, there are all these different images you could be looking at, but your eye and your mind is sort of drawn to the stuff that shows up first. And if it shows up way earlier than everything else, you're just gonna kinda predicate your exploration on that. I feel like it's so helpful to hear the, like, surprising pieces of people's research, like, stood out and kind of whether it was the I think it's interesting to think about too, like, does the thing that could stand out to someone or that you might like might be the designing of it or the, like, what you saw early on and and the takeaways. So, yeah, I really appreciate you sharing. I know I know we're getting long, but I definitely wanna get to the responsible data science because I know it's what you're working on now. Could you could you tell us a little bit about that work and how you got into it? Yeah. So responsible data science is this giant umbrella. Lots of people work in this space. At a high level, respond the the whole point of responsible data science is the idea that we want the outcomes of data science work to be responsible. We want that people are not discriminated against. We want that people are not harmed by the outputs of, say, models, large language models and otherwise. We want people to be treated fairly. Okay. So everybody who wants that and is working towards that, they fall under this responsible data science umbrella. What I think is really cool about the work that I'm doing, and this is in collaboration with a few professors who I wanna shout out. So I wanna shout out Emily Wall, kind of my longest standing collaborator on this work, professor Amy Zhang, who Amy and I co advised an awesome PhD student working in this space, Teanna Barrett, and then also professor Yuichi Shoda. So Yuichi is a faculty member in psychology at UW. And without Yuichi, we wouldn't be I wouldn't be where I am now with these projects, so I think his perspective has been really helpful too. But the thing that always struck me about responsible data science work, where I think it's changing a bit more now, but years ago when Emily and I started talking about this problem, what struck us about it and and what struck me about it especially is how it was always sort of treated as an algorithmic problem, a model training problem, or a data cleaning problem. The problem was always someone or something else. The problem was never us. And I was just thinking, wait a minute. So who does data science work? Is the data science worker the model? Is the data science worker the data? Or is it me and you? So if we're going to, you know, try to make data science work more responsible, should we be emphasizing the artifacts? Or should we emphasizing the people doing the work? So what came out of that is this idea of behavior change interventions for responsible data science. So the idea that we should not just focus on whether or not a data set is responsible, a model's responsible. We should also be focusing on, you know, are the practices of the data scientists responsible, and how does that contribute to ethical or unethical outcomes in data science? So that's the perspective that we brought to the community. Because initially, people weren't really thinking about the problem in this way. They weren't talking about it in this way. People were actually kinda skeptical too about it. It took us a bit to actually publish our very first paper in that space, where it was kind of a mixture of a literature review and vision paper describing what behavior change would mean in the context of responsible data science and what kinds of tools in psychology and in computer science that we could use to actually achieve that goal of trying to help people shift their behaviors. And then since then, we've had a couple a few more papers on that line in that line of work where the most recent one is actually a paper at CHI, so top HCI conference, where we actually evaluated two existing behavior change intervention strategies in the context of a controlled data modeling task. So we asked people to train a machine learning model with some provided data that we gave them during the experiment, And then we they they experienced I think they experienced two different intervention strategies. So one was we primed them. So we had them read a little prompt that hopefully motivated them to be more responsible in their actions and training the model. And then a separate intervention where we actually just loaded, like, a a modular plug in into the notebook they were using to train the model. And they could use that tool, that specific tool, to check the fairness of their model and improve it. Yeah. Yeah. So sorry. I have it here. I was trying to grab the but this is evaluating behavior change interventions for responsible data science. Is that Yes. Yeah. Yeah. Evaluating behavior change interventions for responsible data science. Yeah. So a cool outcome of that work was that priming people to think responsibly did shift their behaviors. They they did have an increase in the range of responsible activities they did during the session, But the fairness of their models did not actually get better from a significance perspective. We did not notice a significant improvement in the fairness of the models. So just motivating people to do better, basically, just telling them to do better and them feeling like they wanna do better doesn't actually lead to anything any better. So also, some things change, but not necessarily a measurably better way. So giving them tools to actually show them, like, this is what better looks like in terms of outcomes and pairing that with the motivation, that actually works. I think that behavior change framing is, like, so interesting and, like, powerful. Like, it's not something I don't know. Like, I think in, you know, how a computer science and statistics, it's so easy to get locked in on, like, the the algorithms and the mathematics and forget, like, oh, actually, it's humans doing all this, and we have these levers to pull on how the humans use our tools, and that's just as valuable. Powerful. Yeah. But not more. Yeah. Yeah. What what was the tool called? Is it Aequitas? Oh, Aequitas. Aequitas. Yeah. Okay. Yeah. What does that look like? It runs inside the notebook. Is that It does run inside the notebook. It shows you some bias metrics, and you can pick specific subgroups in your dataset that you wanna compare against. For example, a dataset that we asked participants to train models with was sort of a classic, like, loan application dataset. So a you could a demographic group you could compare would be like men versus women and their loan applications. So you could see if your model would be, say, biased against women and be more likely to reject their loan applications. So it's like versus the prime, which is more of a nudge. It gives them some, like, tools to kinda implement some, like, checks for them. So the the prime is really just a prompt that encourages you to think, like you could think about saying, hey, with the loan the credit loan dataset, it's it's possible that women are more likely discriminated against than men and more likely to have their loans declined. So you could say something like that and like, oh, please keep this in mind when you're training your models. And as I'm understanding, it's like people will the prime actually does nudge people in that they kind of they've both, like, self report. They're like, oh, I did follow these kinds of, like, principles more. Yeah. People would say, oh, yeah. I was definitely more motivated, like, being reminded of that or, oh, as a woman, I can resonate that resonates with me, and I would not want people to be discriminated against in that way. It definitely motivated people, and we did notice a behavioral shift. But I think the important thing is just inducing a change in behavior does not guarantee the outcomes change. You have to induce the behavior in a deliberate direction. Yeah. Gotcha. Like, I guess, like, provide the skill or capability to kinda actually produce a fair model. Yeah. Giving them the the capabilities to measure the fairness of their model and then also to to fix a model when it's it's not at the level that they want or if they want it to be more fair. Yeah. Gotcha. Yeah. It's super interesting. And as I understand it too, the like, the evaluation of like, there's two parts as I understand it. Like, the the responsible data science, you have the framework for it. But this paper is really interesting that it's also attempting to evaluate it and provide, like, a behavioral experiment. And I think I recall, I think the paper says it's, like, one of the first controlled experiments in kinda, like, evaluating yeah. I'd be curious to hear a little bit more about that. Yeah. So behavior change behavior change for responsible data science is just, in my opinion, understudied. There should be way more people than the me and the faculty I mentioned studying this problem. And there are some people studying this problem, so I'll acknowledge this is not an empty bucket, but there should be way more people studying it. And I think part of the challenge in studying this problem is that we are not necessarily well equipped with the tools that we need to tackle these kinds of behavioral problems, but there are other communities that have studied it for a long time, like psychology. Behavior change is a really important aspect of psychology research, And so what we are trying to introduce through these papers are strategies and tools that we could adopt where we maybe we could even collaborate with researchers in psychology, like Yuichi's collaboration with us is an example of that, where we don't have to come up with all the answers as computer scientists. And in fact, we shouldn't because a lot of us are not behavioral scientists, a lot of us are not psycho psychology researchers. We should be collaborating with other people, and we should be learning from them to develop solutions. So all of this work is really trying to build infrastructure intellectually and, you know, mechanically speaking, like, was designing experiments to measure success to see, you know, what could behavior change interventions look like in this context if we were to build more of them. And, you know, what are the right things to do, and what are the kind of the pitfalls we should avoid in those designs? I saw a lot of interesting measures like cognitive load and things like that. So it's cool to see kind of the full array of, like, instrumentation, you know, in this experiment. Yeah. I'm glad you mentioned cognitive load too because an important thing to acknowledge in that work is that we're we're asking people to literally do more work than they would normally do when you're asking them to do data science work responsibly, which also means introducing a cognitive burden on top of what they already have to deal with as a user of data science tools. And the the load measures from that study show that, reflect that. It's not shocking. Right? It makes sense that if you're asking people to not just worry about model performance, but also whether it's discriminating against people, that's more work that they had to do. It was cool that those measures in the paper show that. Yeah. It's a really cool approach. And then I know the other one I saw that you put out recently was wait. What's this?

The African Data Ethics: 1:05:01

A Discursive Framework for Black Decolonial AI. Yeah. I'm I'm really curious to hear more about that paper as well. Yeah. So the I I definitely wanna acknowledge the intellectual leader of that work was Teanna Barrett. So she's a PhD student here at UW, co advised by me and Amy Zhang. That was kinda her her first major project as a PhD student here at UW, and I felt like she did a fantastic job with it. And So what's interesting to me about that work is this idea that, so, know, in responsible data science, we care a lot about what it means to do something right and what it means to do something wrong, ethically speaking. And a lot of how various communities try to write that down are through frameworks, ethics frameworks. What we noticed is that in these ethics frameworks, you have them for a lot of different communities, but it seemed like African countries were being left out of that conversation of what, like, what is right and what is wrong in the context of interacting with, engaging with African peoples. And it seemed like we could develop a framework to reflect the values and the ethics of different African countries to fill that gap. So that was the idea behind that work. And Teanna worked really hard to take into account African like, sub Saharan African philosophy, ethics work, kind of thought leaders in terms of how AI should be incorporated into their cultures, things like that, and kind of putting it all together to review. Okay. If we were to create one framework to exemplify these principles that should be upheld, what would that look like? And what gaps are we filling compared to other ethics frameworks? Like, which of these principles might be covered in current frameworks and which are not? And I would say anyone who's really excited about it, please reach out to Teanna. She loves talking about her work and and that paper, And a lot of people have have been really excited about it, so I I just encourage folks to reach out. Do you know what y'all might do next as next steps building on the paper? Yeah. That's a great question. So one thing that really bothers me about ethics frameworks is that they're so hard to use in a practical sense. I could could go out, read a bunch of books, learn all there is to know about these frameworks, but then how do I actually apply it to my data science work? And then also a a more recent project that Teanna has been working on is this idea of, well, how exactly do data scientists operationalize their ethics and their values in their own work? And the answer is it's really dependent on the environment that they're in and the opportunities that they're afforded to express those values through their work. And some some people haven't thought as deeply about it as others, but even for people who think deeply about it, depending on where you work, you might need, like, outside projects to be able to do that, to really closely align your values with the practices in data science that you follow. Because your your company might be hostile towards it. They might not care. Like, you better meet your your key performance metrics, and I'm not gonna give you any extra resources for this other stuff, or it might be that resources get cut back at work, so then the time and the energy and resources you had to do some more of the responsible aspects might be reduced. So it's very situational, and there are a lot of factors involved. So what we're interested in is, well, for the question of if it's hard to figure out what your ethics are, what values you wanna uphold through your data science work, could frameworks help you and could we make frameworks easier to use in that direction? Another question is if there are these barriers, these environmental barriers to helping you, you know, practice data science more responsibly in the way that you want, From a technology perspective, what can we and what can we not do to support that? You're saying outside projects. They might need outside projects because they just don't have the resources. Yeah. Or the bandwidth or the company is just not interested or doesn't want them to do that work within the context of the company. So then maybe they may have a passion project outside where they're sort of I don't wanna say atoning, but, like, allowing themselves to to focus on where their heart is. Yeah. It's interesting to think about to the in terms of responsibility that I guess if you're a data scientist within a company, the question of, like, what is my responsibility? And if I don't have resources in the company to kinda, like, meet that responsibility, Should I maybe do do I need to put in work to kinda, like, build the skills to to behave responsibly? Or or even just how hard should I fight to try to make these resources available so that people can perform their work responsibly in the way that aligns with my values? Because the organization has values and then the individual has values, and there's this question of how they intersect. Yeah. That's a great point. Have you run into much discussion around or thought about responsible data science? Yeah. I I think we we think a lot about like, don't like, we don't use, like, responsible, but we think about a lot of these similar things. And particularly, like, you know, for for me, I think a lot about the, like, the pit of success and, like, how do you kinda guide people into doing, like, the right thing or the bit like, making the right thing or the better thing, like, easier than the hard thing easier than the the bad thing or the the worst thing. And I think it's fascinating if, like, we now have a lot more levers to pull in terms of, like, prompting LLMs and giving them tools and having, you know, different agents review your work afterwards with different with different lenses. So I I think it's like an it's a exciting time to be thinking about how can we make these frameworks more actionable for for, you know, real data scientists who, of course, wanna do their best but have so many other pressures on their time. Yeah. Absolutely. I was gonna say one direction I'm really interested in moving in the future is this idea of, yeah, it might not necessarily be right versus wrong in this kind of more nuanced space. Something I think we're running into and that we see is that inaction might be the kind of the biggest problem, inaction along different dimensions. Because it's pretty clear, right, that if data science data scientists and engineers and so on do what they believe is rational or what they believe is, like, reasonable and are not actively doing actions where they are like, you know, I should be doing additional things to make this as responsible as possible versus, oh, I don't think I'm doing harm, so I think I'm fine, we still end up in all sorts of bad situations. So it's not that people aren't rational or people aren't reasonable. It's that the rational and reasonable like, rational behavior is not the in the current context, the definition of rationality might not be ethical. So if you define rational as responsible, you end up in a different frame, which might lead a totally separate line of decision making that we come very different from what we do right now. So something I'm Emily Wall and I are interested in following is this idea of rationality in data science and where it might get us into trouble and how other frames, like other definitions of rationality, might help us kind of break out of that cycle. Wanting to make money is rational, for example, but can lead us all sorts of directions. Sticky situations? Yeah. And it's not necessarily bad. Right? But sometimes the outcomes are bad. So the question is, like, how does that frame that definition of rational get us in trouble? Do you have advice for people maybe starting out in data science today or recommendations for them? You mean for, like, responsible data science? Yeah. For responsible data science. What I would what I would say is you don't have to, like, read a bajillion books on it or anything. Maybe pick one book, like Data Feminism, and check it out. And you don't have to necessarily commit yourself to doing everything in that book, but just kinda I think it's useful to know what's out there, and then you yourself can decide how you wanna incorporate it. And then I would say for each tool or feature that you're gonna use to automate your work, see if there is another tool or feature that you can use to evaluate it from a responsible perspective. Like, you're gonna train models, see if there's already tools out there that could help you check how fair they are. If you're data cleaning and you're gonna do some automation, see if there's anything that can help check for, like, bias. It seems like such a helpful rule of thumb. Like, for each thing you use, find a thing to evaluate it as, like, such a practical piece of advice to kinda, like, try to counterbalance. Like, an an easy thing to kinda, like, nudge a person, I feel like that that strikes me as really helpful for being able to kinda like put this into operation. Yeah. If I was starting on the very beginning, that's what I would do. I would think about, you know, almost like unit testing. If I were to do the equivalent of that for any tool I wanna add to my toolbox, you know, what checks and balances can I just automatically put in place as I add to the toolbox? Hadley and Wes, any other questions? Just this I just I mean, I just really appreciate your, like, thinking about, you know, responsible data science, how do we kinda, like, steer people towards it? You know, not about being perfect, but how do we help everyone just get a little bit better? I think that's such a the the framing around behavior interventions to just feel really like it. Absolutely. Agreed. If you're fans of, like, The Good Place, I think of it kinda like that. The Good Place? Yeah. There are no perfect choices in life. You can only, you know, do better. And that's just such, yeah, such an antidote antidote to, like, paralysis. Like, you know, you can't you were never gonna be perfect. You're never gonna do the the best thing you possibly could, but you can do a little bit better than what you're doing right now, and everyone can do that. Exactly. It's so it's relative. I I think of responsibility in relative terms. There aren't good people and bad people. Well, Leilani, thank you so much for coming on. Sorry I've been so mean, the thing is I think you have so so many interesting studies that span, like, computation, cognition, and ethics. Like, I I think that I got too excited in the sauce, but, honestly, it's been so nice having you on and hearing from your work at the intersection of so many interesting areas and really focusing on or being sure to bring in also the human component into so much of it and and to think about the ethical side of it has been so inspiring to hear about. So, yeah, really appreciate you coming on The Test Set, and thanks so much for talking. Yeah. Thank you for having me. It's been a blast. I've loved talking about all the topics we covered today, especially responsible data science. I actually don't get that many opportunities to talk about it. It's not that popular of a topic in industry right now. What's super exciting, I think, if people it's cool that people both have the framework and can read some of the experiments you've done, you know, evaluating it too. Yeah. Awesome. Yeah. Thanks so much. Yeah. Thanks for being here. The Test Set is a production of Posit PBC, an open source and enterprise tooling data science software company. This episode was produced in collaboration with creative studio, Adjy. For more episodes, visit TheTestSet.co or find us on your favorite podcast platform.