Dr. Bryan Wilder - ML/Optimization: Decision Making in Social Settings

Bryan Wilder is an Assistant Professor in the Machine Learning Department at CMU. He received a B.S. in computer science at University of Central Florida, and then started a PhD in computer science at the University of Southern California with advisor Milind Tambe, and then they transferred over to Harvard together. His research focuses on AI for equitable data-driven decision making in high-stakes social settings, and integrating methods from machine learning, optimization, and social networks. He has won loads of awards including a Schmidt AI2050 Early Career Fellowship and Siebel Scholar award.

We discuss his project on HIV-prevention, some work on better integrating ML predictions with optimization models that have some uncertainty, and a brief but nice beginner's lesson in robust optimization.

Check it out!

Transcript

0:00

Hi there, and thanks for tuning in to optimizing you the podcast where we talk about all things optimization. Interested in studying optimization, interested in applications of optimization. Interested in hearing from current PhD students, faculty, and practitioners about what they're up to with optimization. Then listen on my friend. Hi there. Welcome back to optimizing you the podcast where we talk about all things optimization. Today, our guest is Brian wilder and assistant professor in the Machine Learning Department here at Carnegie Mellon University. He received his Bachelor's of Science in Computer Science at the University of Central Florida, and then started his PhD in Computer Science at the University of Southern California with advisor might say it wrong, Milind Tampa Bay. And then they transferred over to Harvard together. His research focuses on AI for equitable data-driven decision-making in high-stakes social settings and integrating methods for machine learning, optimization and social networks. He has won a lot of awards, including recently a Schmidt AI 2050 early career fellowship and Siebel Scholar Award. So I'm very excited to talk to Brian. Thanks for coming on. How are you doing? Good. Thanks for having me. So we normally do this podcast and kind of two sections. The first section we talked a lot about your career. So like how, how your PhD went and why you wanted to pursue this position you're currently in. How you're liking it and it can be as honest as you want about the head. And then the second half is your research. So what kind of work are you doing? Which lines of research are you most excited about? And why is the research you're doing important? Does that sound good? Yeah, let's go for it. Alright. Let's just start with with stuff about you then. Can you give yourself a bit more of an introduction? Sure. Yeah, So my research focuses, Like you mentioned, that the intersection of machine learning and optimization really focused on how we improve decision-making under uncertainty and these socially consequential settings. In particular, on the application side, I do a lot of work into veins that the touch public health or medicine. And so I tried to work really closely with researchers and practitioners in those domains to figure out where we can maybe use computational methods to improve the quality of decision-making. And so then that leads to an array of problems on the methods side, where I think about things like what properties should machine-learning models have if they're gonna be used in downstream decision-making in various ways, e.g. if they're trying to feed into resource allocation that's modeled via optimization, how we construct systems that are going to have the kind of for Boston equitable performance that we want when they're going to be used out there in the real-world, in these, in these kind of important settings. Awesome. And how did you get started in studying and being interested in these kind of things? Yeah. I mean, I was always interested in the intersection with kind of computation and broadly the social world, right? So things like, you know, like social networks and population level interactions. And how we use kind of computation as a lens to sort of study those systems. But then as I was starting my PhD there is this kind of really interesting opportunity in the lab that I was working in to not just sort of study social systems, but to figure out how to change them for the better right? To take on some of the thorny social issues that we see all around us. And think where we as computer scientists or operations researchers, right? Where we have the opportunity to use those, those kinds of toolboxes, right? To maybe, to maybe change things for the better. And so I think this area is really fascinating and I was drawn into it because it combines both, I think a lot of the scientifically interesting questions that are at the interface between algorithms and the kind of messy real-world environments that deal with sort of the end of the interaction with lots of different agents. And all of their incentive is, and that gives rise to and data and all of that. Alongside with them, the additional challenges that when you think about this specifically from the lens of decision-making, right? And figuring out what should a policymaker or what should a social worker designing an intervention? What should those people be able to do with maybe the data that they're starting to have access to. I got drawn into that area as. You know, as a graduate student. And then over the course of the next several years, I had a really cool opportunity to work closely with some colleagues in social work on a problem related to HIV prevention. So this was just like a really cool way of bridging those two very different worlds, right? You got to learn a lot about the social sciences, about policy, about what public health actually it looks like kind of more on the ground, spending time at community organizations and that kind of thing. That really convinced me that there was a lot to do in this area, right? That there was room for people who are willing to spend, spend the time to, to build up the right kind of expertise, right? Because it takes a lot of time to learn about both sides of that world and off to function effectively. But then there's maybe something unique that can be done there. Interesting. Did you go and work? You actually worked with these groups during or after your PhD while you were a professor or separately? So I started this line of work when I was a PhD student. And I would collaborate closely with social workers and the projects that we're working on. Also to involve services offered to youth experiencing homelessness in los Angeles. Which services are offered at drop-in centers that that will offer a range of services. And so over the course of my PhD, I would spend time at those centers helping with bits of data collection or as we were further along in the project, helping pilot test approaches that we were thinking about. And just spending a few weeks in that environment really gives you a better appreciation of what the operational realities are like. What if you're going to claim to improve some sort of intervention with learning here with AI. What, that, what your solution is actually going to have to do to make a difference on the ground, what constraints this is going to have to operate under. And so that's something that I got a taste of as a graduate student in that fairly early on and that continued throughout my PhD. And now as I'm starting to set up my own lab and start start at faculty job this year. It's something that I'm very much trying to build into, into my own group right unit, bringing in kind of interdisciplinary collaborators and public health and medicine and working closely with them to define the problems that we're going after. Absolutely. What was the problem with HIV prevention? So is it like it was an inequitable in some way and you needed to, I guess, like we work in decision-making, then there's usually some sort of limited resources and maybe some inequity. Can you talk about the problem and what kind of solution or changes you made through your research. Definitely. So this particular project dealt with preventative interventions related to HIV spread. So HIV is a huge public health challenge, particularly among populations like youth experiencing homelessness. And so one kind of avenue to try to tackle that problem is to disseminate preventative information, right? Like what you'd need to do to not get HIV and raise awareness about the problem. And so the social workers that we were collaborating with had developed later intervention, right? Where you recruit a small number of youth from the community and you train them to communicate with their friends about these issues, right? And sort of be the advocates for, for, for preventative behaviors in that population. And there's a lot of reasons why this is kind of the model. They might be attractive. Both the message is going to be more effective coming from within the community as opposed to sort of outside fingers just telling you to do it. And also that there are really resource constraints, right? You can't deliver these interventions have actively to all of the youth in the population. But you might be able to train a limited number of people to be peer leaders that are going to be the ambassadors who then go out and help disseminate the message much more widely. And so that's where you get into resource constraints, right? You have to figure out, give it this population who are at the small number of people that you should recruit as your peer leaders to have the biggest overall impact. And so that you have an objective function that's about the total number of people who are reached with information as a result of the intervention. That's fantastic and very cool work. Did Did it work, I guess, I guess what came with the project? Yeah, so we spent a while developing algorithmic approaches for this problem. A lot of the key challenges basically you are that you just don't have access to any of the kind of data that you would have it in other settings. So there's plenty of work kind of an optimization on social networks solving these kinds of problems. But by enlarge it, it kind of assumes that someone's going to give you access to the structure of exactly who's connected to who in the population. And a model basically that describes accurately how information will propagate over that network. When we tried to go out into public health settings, really done with that information is available. And so we spent a while developing algorithms that do sort of a combination of strategically deciding how to gather information in the first place. How to sample people to, in a, in a very parsimonious way to get at that social structure. And then solving robust optimization problems to account for the remaining uncertainty. And so we went through this kind of loop of realizing that maybe there were some of these issues developing algorithmic solutions, pilot testing. We would go with our social work colleagues out to one of these drop-in centers and see what would happen if we tried to roll it at intervention using one of these approaches, we hit some issue that would send us back to the drawing board. And then eventually that process converged, right? And we got something that we felt was really ready to be scaled up and test it out in a larger setting. And so then we ultimately ran a field trial that enrolled about 700 youth over the course of a couple of years. And that allowed us to rigorously evaluate the algorithms that we developed versus the status quo methods for, for intervention. And that allowed us to demonstrate that there was a significant change in behavior as a result of adopting the intervention that was algorithmically assisted. That's amazing, Very cool that you've got to implement your work and test how the algorithms performed for such a good, something for social good. That's very nice. What I guess for maybe people who, who aren't very knowledgeable in this area. What are the algorithms look like? How did you how did you create them? Yeah. What what kind of tools and fancy maths did you use? I think there's a couple of broad areas that we need to run. So one is this problem of, well, you want to maximize the dissemination of information on a network, right? This, you're gonna think if the network is a graph of some sort, but you don't know what that network structure looks like. I would know it just tells you up front exactly who's connected to who. Then we, we approached the problem in two stages. So first we thought about how do you gather information about network structure in the first place, and that happens via surveying. You just go to someone and you ask them who their contacts are. And so that led to a problem that involves a lot of questions about algorithm design networks. A lot of random graph theory where we tried to model that process, right? If you wanna be able to select a optimal set of influences on a network using a very limited number of queries into the structure of the network, right? So a query is where you go to a single node and reveal their set of edges. Then under what conditions can you do that effectively? When can you guarantee that you find a nearly optimal set of influencers using a number of queries that's asymptotically it much smaller than the total number of nodes in the network. There. We basically tried to describe properties of graphs that would enable that. So we went after in particular community structure where graphs are they, real networks tend to have this structure where there is individual subgroups that are much more densely connected and then sparser edges across different communities. And so we analyzed a particular kind of random graph model called the stochastic block model that formalizes that structure and showed that on graphs drawn from that model, we can construct efficient algorithms that basically simulate a series of random walks on the network, right? So you go to someone chosen at random, you ask them who they're connected to, then you follow one of those edges at random, right? Ask one of their neighbors who they're connected to and so on. And showed that by carefully balancing the length of time that you run any one of those random walks for, which gives you information in detail about a single community with any kind of diverse samples. Where you go to lots of different communities by choosing new random starting points. That, that could give you enough information to reconstruct at approximately optimal set of influential nodes. So that's kind of how we dealt with the sampling step to the process. And then we also used some kind of some type, some methods kinda like more of that kind of classic called robust optimization literature to deal with uncertainty about the model of information dynamics. And so we analyzed some models and robust optimization and distribution like robust optimization, where there's these parameters describing information diffusion that are unknown to you. Very cool. With that robust optimization over the parameters that are unknown to you. Is that optimizing the worst-case or the average case? I forget. Yeah, something like that. For most optimization case, it's the worst-case, which of course, you know, sets up a dilemma that you need to define your uncertainties that's in a way that don't lead to solutions that are too conservative, maybe, right? If the worst-case, there's just very, very pessimistic and you can't do anything that you don't get useful answers. But, but yeah, if you, if you can define a uninformative uncertainties out, then you go after the worst-case so that you get performance that's kind of guaranteed simultaneously across all the different scenarios. Very cool, and it's interesting. So there's problem. You worked on it for like HIV prevention. Finding these people to query to get the right information for your network and then hopefully get up, get the word out as much as possible. Is this. You like a more general problem. What came to my mind was like influencers and marketing. You want to get the right influencers to maybe get your product idea to the most people as possible on Instagram or something. Or they like a lot of other problems that have the same structure. Yeah, it's definitely an interesting question. I think to some extent, the specific challenges that we tried to take on were definitely inspired by kind of public health, community health settings. Where you have these kind of bottlenecks around information access. Populations that probably don't use digital social networks to a large extent are where those networks are not likely to be reflective of the real underlying connections in the population. So I think that methods are most applicable to domains that have something got that structure. It's somewhat less likely that you see e.g. marketers going out in a similar kind of model as opposed to being on social media or whatever. But nevertheless, I mean, I think that there probably are like other analogous policy settings, right? Where you think about diffusion over real world, real networks that are difficult to measure in some way. Yeah, so we see examples of this, e.g. in the development economics literature, where people think about diffusion of technology or financial institutions or other things on networks. And I think, I definitely think that there could be analogous problems in that setting. Very cool. Are you still working on this problem or has your research kind of shifted to other things? These days, my research has shifted a little bit. So it'd be, I was still having a little bit of work on social networks over the years. But these days, my main focuses on a different set of projects that are more about trying to be able to both do kind of rigorous public health surveillance about health and equity is right to be able to track disparities in outcomes across different populations from the real-world observational data that we might have access to. That ideally on top of that, being able to build more effective policies for resource allocation in medical settings. Cool. Could you touch on one of those and maybe break down like how you became interested in it and what you're doing with one of those projects? Yeah, definitely. So one example of a problem that we're working on now is how do you do principled inference about health inequities, ideally in real-time in a fairly detailed way. So how do we get information about which populations have worse outcomes or worse burden of disease in different settings. So e.g. with, with COVID, right? We might want to know things about which populations are particularly likely to experience severe disease and what are the mechanisms that drive that, right? Yeah, how much detail can we give about exactly that, that breakdown of disease burden? And this is something that's relatively difficult to get at because you have a dilemma, right? In the sorts of data that you have available to you in this kind of public health research, you either try to go for very high-quality experimental, experimentally collected data sets where you exercise a lot of rigorous researcher and collecting potentially even not all sources of data to study a particular question. And maybe you got an answer that's trustworthy, but it takes a very long time to do that process. Or you have access to datasets that come in real time because they're just kind of things that are observationally available in the health system, right? Like insurance claims or electronic medical records or whatever else. They could potentially give you answers a lot faster. Which could be really nice if you're in something like epidemic and you want answers immediately. But when you're in this world of observational data, suddenly everything gets really messy. You'll have a lot of challenges like selection bias and confounding. That mean that inferences that you draw sort of naively on that data is not likely to be particularly trustworthy, right? And then if you train machine-learning models on top of that data, then you can certainly get into all sorts of additional downstream issues when that data is effectively not representative of the population that you really think is out there. We're focusing on basically trying to synthesize together those two pieces of information, right? Where if you have access to some kind of experimentally collected data that's probably trustworthy, but it's likely to be a lot less rich. You might just get estimates of a couple aggregate summary statistics or something like that in a rigorous way. And then on the other side you have access to these very rich kind of tropes of observational data that might measure a lot of different factors about a lot of different patients. Then how can you effectively de-bias the estimates that you got observationally or how can you place kind of principle of bounds on the amount of error that you might be making in your inferences on the observational data, right? So. And the hope then is that if we can synthesize together all these different sources of evidence that we'll be able to provide much more principal sources of real-time information that would be able to help drive downstream decision-making. Cool. So you have two different sources of information and you're trying to use both of them in the smartest way possible. Can you give me an example of what the two different sources look like? Yeah, sure. So one example of this with this kind of specific example Studying health disparities in COVID is that the, the real-world data, things like insurance claims, medical records, right? This is just as patients are treated, as their carers recorded. A particular set of health systems or a particular set up insurance companies or whatever else, right? Then you get access to that information. But that's going to inevitably leave out a lot because many people, especially if they have relatively mild cases, they don't see the Dr. the first place. Even maybe or maybe for some populations, they're just less likely to show up in your dataset for other reasons. You probably have real kind of selection bias issues. And that makes it difficult to study things like differences in say, hospitalization rates or mortality rates across populations because there's this kind of missing denominator that's probably missing in different ways across different subgroups of the population. Then the other source of information that you have is things like Sarah surveys that the CDC conducted that gives you about his closest, we're gonna get to a ground truth estimate of the total number of people that were infected with COVID. Because they go out and they actually tests large numbers of blood samples taken from, from blood donors or from routine and kind of commercial lab testing. And you serological tests to determine whether the person who contributed that sample was exposed to COVID. And so that gives you a source of information that we think is pretty reliable. But then the flip side is that it doesn't measure a lot of heterogeneity, right? It gives you maybe an estimate of the average number of people infected within different race or ethnicity groups, or maybe different geographic areas. But it's not going to measure the rich clinical history or sociodemographic characteristics that might be available in other sources like EMR or claims data. And so you need to figure out how to piece together those two sources of information that if the inferences that you're going to make on the really rich datasets have to line up at least on the axes that you're able to measure, right, with the experimentally collected data. Interesting, So would it be right to say maybe like you're trying to help the populations that have less information about them by using the more rich information that you have just in general. Yeah, that's definitely, I think 11 consequence of this, right? That you get more, ideally more accurate inferences for populations where that kind of selection bias is more severe rate that, or maybe like last of all represented in kind of traditional medical datasets. Yeah, that sounds great because I can totally see certain populations might not want to go to the Dr. as much or they just might not have as much information available about them. So you have to figure out some way to use the little information that you have in the smartest way possible? Yeah. Yeah, definitely. Okay. So you have a lot of awesome applied work and you use a lot of smart maths to do it. I was reading one paper of yours, recently called melding the decisions pipeline, decision focused learning for combinatorial optimization. And I enjoy combinatorial optimization. The program I'm in is called algorithms combinatorics and optimization. So a couple of those words there. And I'm curious if kind of at a high level you could break down some of the math you're trying to change or develop between the machine learning world and the optimization world. So this is an interface that I find really fascinating. So this kind of line of work was motivated by this recurring archetype that we see with problems that combined prediction and optimization. Where you have optimization problems that model downstream decision-making under, under constraints. But there is perimeters and those optimization problems that are unknown, right? There's coefficients in the objective function that represents how much people each need a different resource or what consumer demands will be in the future or whatever else that are not knowable to you apriori. And so instead, you train a machine learning model, right? You take all the information you have, you feed it into the model and the model predicts that set of coefficients. And so there's definitely a plugin approach here that is maybe the most standard thing you could do, where you just take the ML models predictions, you plug them into the solver and you get a recommended decision. But the question is whether we can do better if we know that the model, that the machine-learning model is gonna be used downstream to inform the way that you solve this particular optimization problem or this particular class of optimization problems, then maybe it should be able to be smarter at informing that particular task. Because all machine learning models make mistakes, there's inevitably errors, especially when you have limits on data that impose term limits on the complexity of the model that you can learn. Errors are inevitable. And so you have a loss function. Tells you how to trade off different kinds of errors. And standard loss functions tell you that all errors count the same, basically, right? In some sense, like a, like a mean squared error, e.g. being accurate about one coefficient and the vector function doesn't really matter much more than being accurate about another or anything else. But really different kinds of errors might have different consequences for downstream decision-making, right? Some errors. There might be some parameters where even a small error changes what decision you will make it. So it's really important to get it right. And others where more error is kind of more acceptable somehow. And that's kind of an intuition. But more generally, you can imagine a lot of different ways that the differences in predictions could have differential impacts on what happens downstream and optimization. And so the goal for this line of work is to bring those worlds closer together to be able to somehow incorporate knowledge about the optimization problem into the way that you're training the machine learning model. So that it learns to focus specifically on producing good downstream decisions induced via optimization. That sounds absolutely brilliant. When I was reading it, I was like This is such a great idea. Why haven't people been doing this all along? Has this been going on for awhile or what's this area looked like? Yes. I think this is an area that's really awesome Over the last call it five years, something like that. Where I'm I mean, I'm certainly not the only person working in this area. There's a really cool community of people and operations research and in machine learning who have become interested in these questions. And I see this as kind of like a sort of natural new focus for the community. Because if you think about the broader context for optimization under uncertainty, right? You have formalizations like stochastic optimization, robust optimization, or maybe more recently distributional a robust optimization. And now, as we see, machine-learning starting to become an increasingly widespread part of that toolkit, right then it makes sense to ask specifically, how do you design systems that incorporate that element of prediction with optimization? Absolutely. Do you think this line of research is the one that is going to keep you, keep you going. Is this kind of like the, the the type of work that you think has been winning you lots of awards. Or is it maybe another research area here? Did we hit the right one? So it's definitely an area that I'm, that I'm still fascinated by. Both because I think, you know, there's, there's tons of applications for these, these kinds of frameworks, right? It's a really recurring challenge. And also because the math is really cool under the hydride, making this sort of, you know, especially when you think about discrete optimization, right? Making that sort of discrete world talk effectively to the continuous world that machine learning models and how bad I definitely don't think that we really are fully understand how to do that yet. So it's definitely an area that I'm excited to continue working in. Nice. Yeah. We were a bit separate in my department sometimes in terms of the discrete optimizers and they continuous optimizers. But when I do see that overlap, it can definitely be quite useful. So on a more human level, I guess, aside from like you're doing all these amazing things. Like what what, what's your goal with all of this? Like what do you hope to accomplish? Yeah, So really, my hope is to advance the science of machine learning for social impact, right? So what that means is that I think that there really is a tremendous opportunity for these kinds of tools in machine learning and optimization to improve the way that we make decisions under uncertainty and the demands that I care about, right? Things like public health and medicine. But there's a lot of scientific progress needed to get to the point where we would really feel comfortable about doing that, right, to having computational systems really have input on important decisions. And so that's the gap that I want to close, right? To build into the methods that are robust enough, that are equitable enough that our sound enough in a variety of ways and high performance. High-performing enough in a variety of ways that they can actually be relied on by practitioners. And so my goal is to build those methods that then people outside of my own community, right, outside of machine learning and optimization, we will actually find it useful, will actually adopt and we'll actually see impact in their own domains. That sounds great. Yeah, Because I have, I feel like there's, there's pushback sometimes on, we can't just let this AI thing do whatever it wants because it might not be equitable or are we don't understand how it works, like in that interpretable AI space. Is there a certain type of practitioner or medical example that you think? Using these techniques will be very helpful in, like in creating cancer detection? Methods are something that was one of our previous guests, Andrew least doing something with that, but like, is there a specific application you have in mind? Yeah. So definitely. I mean, there's I think no no shortage of good places to work in health care. I mean, definitely some things that are I'm I'm excited about these days. You know, touch a variety of places where I think there's real elements of constrained decision-making and health. So that includes public health responses to emerging epidemics, right? Where you have to think about careful use of resources like, like tests or therapeutics or, or, or whatever else. It includes community health settings, both in the US and, and kind of global health contexts, right? Where you have community health workers who are really the backbone of reaching out to patients and trying to improve their care about of course, have limited time and resources and figuring out how those processes can more effectively reach the patients that really need help. Improving maternal and child care, right? Where there's again, a variety of concerns both in the US and around the world and how you more effectively figure out which, which, which mothers aren't likely to experience different kinds of adverse events and how you get them preemptively, the intervention that might help avoid that. All of these are, I think, really pressing problems where there is the real, the real opportunity, maybe for machine-learning to be part of the picture, somehow. Awesome. You're such a good person. So sometimes, I guess, like people want to do things that are very challenging in math and computer science. And people might also want to do things that are very good and helpful for the community. How do you think you were able to land yourself in a role that, that does both of these things while keeping a great job. And you're kind of like doing things that have good social impact and doing things that I really cool and exciting mathematically. Like, how do you make this happen? Do you have advice for people who would who would want to follow a similar track? Yeah, sorry. I think that this is an area where the world needs more people who both have deep technical expertise. And also if really want to sort of realize that concrete impacts did their work right? So I think this is a great area for people to think about getting into. I do think that there's a kind of ongoing tension which maybe your question points to right, between these different objectives, right? Between doing sort of like deeply technical work that advances our scientific understanding of machine learning or whatever other field alongside the kind of the practical impact that we want that work to have on the ground. And I I don't think that there's any clean way of resolving that immediately. There are. It requires really being willing to invest time in both areas and kind of acknowledging that you're not going to be able to spend as much time in either of them, right, as someone who is entirely committed to them. But Still then trying to steer your way towards problems where it kinda being someone who understands both sides of that divide allows you to make a unique contribution, right? And that's not going to be every problem. But I think that there are a lot of problems out there where that's necessary and that's the place where you can make a sort of unique contribution. Got it. And I guess also why did you choose a machine-learning department instead of like, there's always computer science or ECE or like operations research. We can have you over and Tepper. I'm like what what put you in that specific area? Yeah. I mean, I definitely think that I could probably be happy doing the kind of work I want to do in a in a wide variety of departments, right? Amino my work definitely get a span. There's plenty of these these kinds of different areas. So for me it was more just about the kind of like the culture of the individual department that I really like being a part of MIT it Sam, You know, I think the department itself is wonderful. The culture is really great. There was also, I think, a kind of like support for the sort of work that I wanted to do. And it provides a really great platform to do that. Nice. And this is your first year as a professor here. How, how's the transition band? Um, how have you been liking it? Has there been anything unexpected and a positive or negative way that you've been feeling? So far. I like the tropical hot. So I mean, I think it's true that it can be overwhelming at times during the first year that you're you're a sudden late navigating so many different kinds of things you just never thought about as a PhD student. And there's not really an instruction manual to do that. There's no question that there's times where things can be really busy or, or, or overwhelming. But I do think it gets better over time, right? As you sort of learn the ropes. And then it's genuinely just amazing, right? To have a job where you can really carve out the research that you want to focus on to be able to work with the Metro students. Really amazing and rewarding that I think makes it definitely worth it. Nice. One thing that I've been wanting to do on this podcast is have really maybe complicated but useful topics, like the stuff you work on with, with any sort of optimization or machine-learning or robust optimization described in a very simple way. So when someone listens to the podcast, there, wow, I learned something today. Is there a topic that you think you can give a nice summary about for people to understand like robust optimization or some machine learning problem that will teach them about the area. And then like maybe inspire them to read about it further. Maybe even look into it as a research topic. Sure, Sure. Well, I guess you know, you, you mentioned robust optimization. So we can start there. I think robust optimization is really about making good decisions under uncertainty in a particular way. And in particular, you're trying to find a decision that you're going to be at least pretty happy with no matter what happens. Maybe one way to think about this is deciding whether to take an umbrella with you when you, when you leave the house, right? So this is a question that in Pittsburgh we often have to think about because the weather is wildly erratic inheritance at the time today, that would be useful, right? And you probably don't know exactly what's going to happen because weather forecasts and Pittsburgh are not particularly good. So there's always some chance that things go wrong. And so you ask yourself across different scenarios that you think are plausible. Maybe you, you look at the forecast but then you put some error bar, some error bars on the forecast, right? You think maybe there's, I don't trust this exactly. There's probably some wiggle room. You look out at the set of scenarios that you think are possible. And you say like basically in the worst-case what I want to take an umbrella. And that has some costs. You have to carry this thing around. So maybe if, you know, it's pretty reliably sunny, then the other worst-case is that you just pay this cost for no benefit and you don't take it. But if you think there's a big enough chance of rain than you think, you know, rather than trying to calculate the exact probability that there's going to be rain today and optimize my expected utility. Instead, I just sort of plan for the worst-case and then I take the umbrella with me. That's such a nice example. I've, I've, I haven't heard that one for robust optimization, but I like it. It was definitely on my mind today. Okay. And then what does that look like when you have a bunch of you have your error bar of what the possibility of it raining might look like. Maybe this can be extended to something like beyond this application. And you're trying to optimize for the worst-case scenario. So what does it look like when the problem gets a bit more complicated? Yeah, So the, I think the reason that this is a really simple example is that you have one parameter that you're uncertain about, which is, What's the problem? Is it going to range here? What's the probability it's going to rain? And the worst-case is always, it rains or earrings with maximum probability. So planning for the worst-case really means planning for rain in that example. But that structure breaks down when you go to more complicated settings, right? Where you have larger number of parameters that you're reasoning over that might have an uncertainty. Set this to find in a more complicated way that you're optimizing over a higher-dimensional decision, then suddenly you don't, you don't have this sort of just like Clinton closed forum where you can very easily identified that the worst-case is always the same thing and you just play it for that one scenario. You know, suddenly the worst-case maybe changes depending on what you do. Depending on what decision you commit, two different scenarios would be the worst-case for you. Yeah. And then so in those cases, when it gets a little more complicated or there's those dependencies. How are you doing the maybe the search over the possible decisions that you can make to find the optimal value, is it? Yeah. How do you do that? So there's a wide range of techniques that are out there in the literature. So it kind of depends on exactly the sort of like technical properties that your objective function has. A set of scenarios effectively has. The nicest world is where you're solving convex optimization problems. And then if you look at it from thinking about it from the viewpoint of an adversary that's choosing the worst-case scenario to make your life as difficult as possible, right? So if you think about the adversaries optimization problem, that they are optimization problem is then concave, right? Then you're in this world where you can sort of your interests like this family of convex optimization problems. And you can use methods from that literature. And there's nice tricks that you can do with linear programs where you can fold everything into a single linear program using duality and things like that. When we're. Outside that setting, right? And nonconvex settings or settings with its discrete variables or whatever else, then life becomes much more difficult and you sort of, you have, maybe you like more specific families have methods to solve particular classes of herbicides, mixed integer programs, or robust submodular or problems or whatever else. And you kind of got more specialized techniques. That's pretty cool. As someone who works with linear and integer programs. A lot that I liked the way that you describe that. So in terms of one thing that you just said was you could fold together like a linear program and its dual in some special way. Can you touch on that a little bit more? So roughly the idea here is that if you have a sort of a min-max problem where everything is linear in a way that's meant was I can't hear you. I don't have a sheet of paper. I can't write this down. But roughly, if the inner and the outer program problems are both linear programs, then if you take the dual of the inner problem, right? The, the adversaries maximization problem. Then it becomes a minimization problem that has strong duality, the same objective value. And so then you went from a min max to a min, min. And so it collapses to just a single level of optimization. That's really cool. I hadn't seen that before, but that makes a lot of sense. In this min-max setting is this, is this related to when people talk about two-player games? Yeah, it's exactly, I mean, because the sort of duality and stuff, It's very closely related to two-player zero-sum games. So it's very similar concepts just expressed in different language. Very cool. Is there anything else you think some old would want to know about robust optimization? I think that's at least a starting point. It's a really fascinating area. Honestly. There's tons of stuff that I don't think we no at all how to handle yet. Cool. And 11 other tangential question I wanted to ask you is about chat GPT, which you could probably talk about for a long time. But because it's so new and I haven't asked anyone about it yet. How are you feeling about it? And do you think it's very useful and that it's very powerful and we can make one that's even better than it currently is. How do you feel about it? Yeah. So it's an interesting question. I mean, I don't think anyone should have super high competence answers to questions like this right now. But with that preview, so my kind of mental model right now is that the success of Chat GPT in domains where it's successful is showing us that actually just fitting the distribution of texts that's out there is more useful than you thought it was. Basically that if, maybe if you have something that's just absorbed kinda of a bunch of all the different textbooks that are out there that being able to summarize that information and regurgitate it back to you is in a way that's responsive to the specific question that you asked. That actually gets you surprisingly far in a lot of domains. So there's this, maybe the skepticism is that you can point to examples where this fails, right? That it's just kind of having really internalized the training distribution and being able to pair it back at texts from it goes wrong in a variety of circumstances. But at the same time when the training distribution is all texts that's ever been produced. In some sense, then that distribution is pretty wide, right? And there's probably a lot of useful things that you can do with that operate in distribution in some sense. Very cool. I like that. I like the way you say it. Can learn from so much text and spit out the best thing possible. And the Mac could be helpful in a lot of settings, just using all the information available to you. So it's been it's been fantastic talking to you. We hadn't met before today, which is different than in my previous interviewees. So I hope our relationship continues. I'm curious to see what other research you end up doing. Do you have any other other words for the podcast or anything else you want to talk about? Any, any, anything special that's coming up that people should should look out for in terms of your research? Anything? I don't think anything. But yeah, no. I mean, thank you very much for having me on a lot of fun. Absolutely. Thank you, Brian. Have a fantastic evening and weekend. You too.