Exploring AI Matters
Our mission is to help the policy community understand the breadth and richness of AI and the potential for such technologies, wisely applied, to augment all sorts of human endeavors.
Some AI tools are able to assist humans in performing tasks faster, more accurately, or more efficiently. Some, however, are inaccurate and unreliable. Who or what we hold accountable for these flaws, and what incentives we do or do not create for their correction will influence AI’s hand in how we work.
In this series we will refine, sharpen, and clarify your understanding of AI.
Exploring AI Matters
Episode 18 - A Visit to Model Land
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
From ancient times what we now call mathematical models have been used to predict the arrival of seasons, comet returns, and eclipses. With sophisticated mathematics, good data gathering, and modern computers we can now predict things like the tides and the trajectories of interplanetary probes with considerable accuracy. We have also made progress in predicting weather.
For some phenomena, for example earthquakes, we are still seeking to develop predictive models.
As we heard in Episode 2 of this podcast, AI systems can be described as making predictions based on models, models that are trained on vast collections of data. These models seem intuitively different from the others. However, some truths hold for both kinds of models, in particular while "all models are wrong, some models are useful," as is explained by our guest in this episode in which we eplore the world of models with Dr Erica Thompson.
Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that. In addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you.
SPEAKER_03Some ancient mathematical models were used to predict the arrival of seasons, lunar phases, and the occurrences of eclipses. With modern computers and sophisticated data gathering, we have learned how to predict things like the changes of tides and the courses of interplanetary probes with considerable accuracy. We have also made progress in predicting weather and climate change. For some phenomena, for example, earthquakes, we are still seeking to develop predictive models. Many AI systems are described as making predictions based on models, models that are trained on vast collections of data. These models seem intuitively different from the others. However, some truths hold for both kinds of models. In particular, while all models are wrong, some models are useful as will be illuminated by our guest today. In this episode of Mind the Gap, we will be exploring the world of models with Dr. Erica Thompson, a renowned expert. Welcome to Mind the Gap Dialogues on Artificial Intelligence. I am Roland Trophy, a national security lawyer.
SPEAKER_04And I am Charles Palmer, a computer scientist. We are your hosts for this episode of Mind the Gap Dialogues on Artificial Intelligence. In addition, we have two more hosts.
SPEAKER_01Hello, I'm Anna Adams, a national security lawyer.
SPEAKER_05And I'm Mark Donner, a computer scientist.
SPEAKER_03Each episode will be led by two of us, with the others adding impromptu questions and comments as the spirit moves them. We are delighted today to be talking with Dr. Erica Thompson, an expert on decision support models, and author of 2022's highly acclaimed Escape from Model Land: How Mathematical Models Can Lead Us Astray and What We Can Do About It. Her book brings context and perspective to the broad public confronted with predictions made by complex models. Dr. Thompson is an associate professor of modeling for decision making at University College of London, Department of Science, Technology, Engineering, and Public Policy, where she works on a program of research. She is also a fellow of the London Mathematical Laboratory, where she leads the research program on inference from models, and a visiting senior fellow at the London School of Economics Data Science Institute.
SPEAKER_04Thank you for agreeing to join us today, Erica. To get us started, in your book, Model Land is the place where all models are true, as compared to the real world, where all models are wrong, but some are useful. Why should we plan escapes from Model Land?
SPEAKER_00Hi, well, thank you so much for having me on today. It's great to see you all. Well, yes, so why should we plan escapes from Model Land? Essentially because nobody lives in Model Land. You know, we can go away and we can write down our set of equations, we can construct a big fancy computer model, we can do whatever we like, and it could be a really good model. But nobody has any reason to take any action in the real world based on that model until you say that the real world has something to do with the model, you know, that the model is in some way representing the real world. And so what I mean by escaping from model land is making that leap, is saying, okay, I've done this thing on a computer or on the back of an envelope, and now I need to actually give somebody some reason to believe that this is useful, that it has some relationship with the real world. And I ideally I want to be able to quantify that relationship. I want to be able to say how good the model is, when it is good, when it is likely to be useful, and perhaps also when it's likely to be misleading.
SPEAKER_04Well, okay, but if you just made the model, why are why would you say all models are wrong?
SPEAKER_00Well, all models are wrong because they are not the real world. You know, we we make models in order to simplify, in order to gain some tractability on a question, because we can't necessarily experiment with the real world. The future hasn't happened yet, so we can't measure it. And perhaps if we're interested in something that's far away or very small or very large, we can't necessarily measure those. And so we make models in order to simplify, in order to gain access to things that we wouldn't otherwise have access to, like the future. But in doing so, we create simplifications and we make assumptions and we make we make choices, we choose to leave things out because we think they're not important. Hopefully they are not important, and hopefully the model is good enough, but it will never be the real world. You know, the map is not the territory, as the saying goes. The uh the model is not the same as the real world, and that is its power as well as its uh limitation.
SPEAKER_04But okay, if if models are wrong, then how can we say they're still useful?
SPEAKER_00Well, I think this is just a really difficult question, right? I mean, it it's uh in a sense it's trivial because we look back at the history of uh science and the history of technology, and we see models being used everywhere and how you know incredibly good they are. You know, models of physics, of if I throw a basketball or hit a golf ball or something, you know, we can make a really, really incredibly accurate model of that. If I think about the weather and the weather forecast, you know, that's a model, it's predicting the future. And yet, if you get your phone out of your pocket and look at the weather forecast, you are pretty confident that that weather forecast is reasonably good. All right. It's not perfect, it's not going to be perfect. We know that it's not 100% accurate, but it's certainly good enough to inform decision making about what you might decide to do tomorrow. And so that's what I mean by models. You know, they can be wrong, but they can still be useful. We don't need them to be perfect. We don't need to get perfect information about the future, but we can we can make use of uncertain information and we can make use of information that is marginal. And sometimes models are much, much better than that. So, you know, sometimes they really give us extremely good predictions like forecasts of, I don't know, eclipses for the next thousand years.
SPEAKER_03I know we didn't discuss this in our earlier conversations, but there's a couple of quick follow-up questions I'd like to ask based on your answer just now, which I found fascinating. You referred to, if we look back at the history of science and mathematics, is there some period at which humans started or certain civilizations started to use models? Do we have a sense of when that started and how they used them?
SPEAKER_00Well, I guess it depends what you think of as models. And I have quite a broad conception of models essentially as metaphors, that models are standing in for something else. We construct a model as a metaphor for something else, and we hope to extend that metaphor further and further and further, and at some point we find that it breaks down. But that use of metaphor certainly is extremely, you know, it happened a very long time ago in human history. And you can, you know, maybe from the first beginnings of language, I would imagine. I mean, I am certainly not a student of any of these kinds of things, and I don't know, I don't know much about early humans, but I can imagine that the first uses of language began to be metaphorical quite early on, and that that for me, that is a use of a model. If we're thinking more about mathematical models, you know, we could think back to the ancient Greeks, the ancient Egyptians, who were using models to do quite sophisticated astronomy, mathematics, engineering, even. And so those kind of models, it might have been that in order to construct a large building, one might make a scale model of it with smaller stones or out of wood first. That would be a model, standing in for the real thing, allowing us to kind of get an idea of what it looks like and how it might behave and how whether the structure is stable, all of those sorts of things. And again, you would then transfer those judgments about the small world to the real world, the large thing that you're hoping to construct next. So I think models go a really, really long way back. I mean, in a sense, being able to talk to another person is you have to have a theory of mind. You have to have a model of how the other person thinks and how they will behave when you say something to them. You know, if you if you are a, I would I imagine that many of the higher animals have kind of mental models of how other people might behave or how other members of their species might behave. So I think it's really incredibly fundamental.
SPEAKER_03I'm very convinced by what you've just said, especially given the amount of time I've spent studying metaphor. But there's a very interesting distinction, I think, between that and what I find in your book. I don't think metaphors have become more accurate over time. But it seems to me from things I read in your book that models have become more accurate, at least for their limited purposes. Can you explain how that's come about? And that then will lead us to start asking you questions about whether AI is in fact improving our use of models.
SPEAKER_00Yeah, so I mean, certainly quantitative models have benefited from improvements in understanding of mathematics, mathematical techniques, solvers, numerical solvers, and then of course the advent of computers and the ability to put things into computers and use that to predict forward and to do immense calculations that we just wouldn't be able to by hand, or it would be, it would take a prohibitively long time. And so, so, for example, we might think of the weather forecast, which uh, you know, a hundred years ago was being done by people like Lewis Fry Richardson, who did a whole numerical weather forecast by hand on paper, an incredible feat of mathematics. And now, of course, we put it into a computer and it can do it in minutes, something that he took literally years to do on paper for one uh forecast for one day. And so the tools have obviously changed, and perhaps the access to these tools has improved. We have got more ability to make use of these methods, and I suppose it's also people recognizing the value and putting more time and effort into the models. Again, the weather forecast is a good example that uh it it initially came out of military uses, that the the military would be interested in forecasting the weather in order to plan movements. And so the development happened there, and then obviously there are huge civilian uses as well, and so those would have developed on the back of that.
SPEAKER_05The question about early models sort of sparked a uh reflection on my part, which is one of the earliest mathematical models is a calendar. Because it it basically depends on an observation, which is that the the cycle of the moon is 29.53059 days long. And that's pretty constant. And then the modeling thing is saying it's going to keep going that way, it's not going to change to 13 days someday in the future, because you know.
SPEAKER_00Yeah, exactly. And models are essentially inductive in that sense that you you spot a pattern in the past and you either fit that pattern with some statistical model, which might be just saying, well, there's 28 days in between each full moon, or it might be a more dynamical model which says I understand the way in which the earth goes round the sun and the moon goes round the earth, and therefore I can kind of use that to understand what the period ought to be. And in either case, yes, you're projecting forward and you're saying, well, hopefully the future will be like the past, or I've made enough observations in the past that I think I'm safe in making the assumption that the sun will rise tomorrow and the moon will carry on in the same pattern.
SPEAKER_03Erica, could you tell us, given the breadth of your interest, what got you started in your investigations of models and modeling?
SPEAKER_00Yeah, sure. So, I mean, my background is maths and physics, and then I started a PhD on uh North Atlantic storms and what would happen to those in a changing climate. And so the first thing I did as part of my studies was to do a literature review and to look at what everybody else had found, what would happen to North Atlantic storms given climate change in the future. And so I looked at a lot of different models and published results. And what I realized as I kind of tabulated these was that they didn't agree at all, that there were there were models saying that the storm tracks would move north, that they'd move south, that they would get stronger, that they would get weaker, that there would be more frequent storms or less frequent storms. And, you know, not just that there was this spread, but also that they didn't really agree kind of within their own explicit or implicit error bars. And so they were directly conflicting with each other. And so I realized that that had not told me very much at all about North Atlantic storms, and it hadn't told me very much at all about climate change, but maybe it was telling me a lot about the way that we make and use models and the way that perhaps we are overconfident in the outputs of our models. And I just started thinking, well, what does it mean? What you know, what does the output of a model actually mean? What is it? And if I have 20 different models and they say 20 different things, what should I conclude from that? Or maybe if I had 20 different models and they said the same thing, what should I conclude from that? And so really that's how I got started. And since then I've I've looked at models in in a series of different application areas, kind of mostly centering around weather and climate, because that's my background, but in other areas as well. And I think these questions are just fundamental. You know, what are we, what are we actually doing when we construct a model? And therefore, how confident should we be in the output that we get from these models? And obviously, we're making really high impact decisions on the basis of models a lot of the time. There's obviously the weather forecast is extremely consequential, but also you might think of climate change forecasting and the decisions that are being made on the back of that. You might think about pandemic forecasting and the immense decisions that were made given pandemic forecasts over the last few years. And so, you know, these are really important questions, as well as being scientifically and philosophically really interesting, I think.
SPEAKER_03Before we cross the bridge into a discussion of artificial intelligence, I noticed Arma had a question.
SPEAKER_01Uh, just going back to some of the points that you were raising on sort of the improving accuracy of model, I'd love your perspective on whether there's a counterpoint to that in that people are getting better at drawing inferences from models. So the combination of models getting better and our ability to understand what the models and the data are showing us and sort of how that plays into the usefulness of models.
SPEAKER_00Yeah, I think there's certainly a lot of that. I think there's obviously much more powerful and complex and complicated models are available now, and we have better statistical techniques for comparing models with data. So that's one of my interests is how do we, how do we take the model and evaluate it for its for its performance? How do we say whether it's any good or not? And so one way is by looking, looking back at the past and saying, could it have predicted, if it didn't know that data, could it have predicted the things that we've already seen? And then the other way is by thinking about the the quality and thinking about the assumptions. So do we think that the assumptions that we've made are likely to continue to hold? And that's a lot more difficult. You know, that's that's a that's a pretty hard question to answer a lot of the time, because it can be inductive rather than deductive. We are not, we're not necessarily able to just sort of logically derive from the past what will happen in the future. We have to make quite strong assumptions. But yeah, so our models getting better. I mean, certainly many of the quantitative models are getting better, but I think that there is a limit to that. And perhaps that's what you're getting at with the question that how far can we expect these models to continue to improve? And there are some fundamental, I think, mathematical limits in terms of, for example, complexity and chaos, which limit the predictability of many systems, including the weather system. And then there are also questions. So another of my research dreams is about the way that values and politics and social values are become embedded into models. And so you might choose to make different assumptions based on your attitude to risk, for example. What is the risk of getting the answer wrong? You might choose to be very conservative with your assumptions if you are highly risk-averse, or you might choose to be, you know, a little bit more loose with your assumptions if you if you are more risk tolerant. And so you can make perfectly reasonable and justifiable different decisions, which will end up with a different model, a different output, and different levels of uncertainty based on social values such as risk tolerance, which are not, you know, not fully scientific in the sense that we couldn't just write down a priori what the right answer is for how risk tolerant you personally ought to be. And so there's a there's an irreducible uncertainty in that sense as well, because we can't necessarily agree on what the right inputs to these models ought to be.
SPEAKER_03You also seem to be suggesting that our ability to generate higher quality data has changed, and also our ability to recognize when we've got or the criteria for distinguishing high quality from low quality data is improving. Am I right about that?
SPEAKER_00Yeah, I think generally we do have that. I mean, obviously it's quite context-dependent. Certainly, again, if we go back to the weather forecast and think about that, then there are satellites orbiting that were not there 30, 40 years ago. And so we have more data, we have better methods for incorporating that data, and we are able to, for example, look at where the greatest additional value would come from taking more observations. So you could say, you could do an analysis and say, well, where is it that we can improve the forecast the most by investing in, say, six more weather stations? What would give us the best value for money? And you can do that. And so I think that the these kind of techniques are really powerful in helping us to understand where to focus model improvement as well, and also knowing where to stop. You know, if you say, actually, we've got 3,000 weather stations and an extra six isn't going to make any difference, then maybe you put your money into improving the communication of weather warnings to vulnerable groups instead, rather than putting up another six weather stations.
SPEAKER_03Shift our discussion. Artificial intelligence researchers and practitioners and people who are considering using it, talk about training a model. How should we think about AI models in comparison with the models you talk about in Escape from Model Air?
SPEAKER_00Yeah, so I suppose the distinction that I would make here, uh, firstly, is that there is not much of a distinction and that AI models really are one of the set of models that I'm talking about. AI, I suppose, by definition, tends to be data-driven rather than physics-driven or kind of process-driven. So we would expect that AI models are kind of fully statistical in that sense, that they are coming out of the data with less in the way of assumptions being made about structure of the underlying generating function. And so that doesn't make them different. It puts them sort of at one end of a spectrum between what I would call dynamical or process-based models like the weather forecast, versus fully statistical models where we might just say, well, I've got a stream of data coming in and I don't really care what kind of data it is or what the units are. I just want to predict the next value in that series. And so AI is more at this end. But of course, you can you can answer the same questions. You can set AI on the weather forecast in exactly the same way by saying, well, here's just a set of values, here's a field of precipitation or temperature, let's just predict the next value. And it shouldn't be impossible in principle for it basically to do the same thing and for it to learn the physics in an a priori manner by by just sort of predicting the next the next value in the series. And indeed, we do see that AI weather models are being developed. And they are not quite there yet, but they're starting to be on a par with the dynamical models that we already have.
SPEAKER_03You draw a distinction in your book between extrapolatory and interpolatory models. Can you explain for our listeners the importance of that distinction, which I found fascinating when reading?
SPEAKER_00Yeah, so interpolation is when we are, we we have say we've trained a model on a set of data and we are trying to find what the value is basically in the set of the data that we already have. So maybe we know what the answer was on Monday, Tuesday, Thursday, and Friday, and we don't want to know what the answer would have been on Wednesday. And pretty much you just draw a line in between, and you take, you know, you use some method to draw that line in between, whatever it is, might be very complicated, and you work out what Wednesday would have been. And that's interpolatory in the sense that you're kind of in between the stuff that you've already done, the the data that you already observed and the things that you already know. And so you have reasonably, generally reasonably high confidence in that. And it probably also doesn't make a huge amount of difference exactly what functional form you choose to fit in order to sort of join the dots. Between Monday and Tuesday and Thursday and Friday, you'll still get roughly the same answer for Wednesday. But supposing we know what Monday, Tuesday, Wednesday are, and we want to predict Sunday, then we are extrapolating because we can join the dots between the ones we've got, which are all together, but we want to go outside that range in order to get to all the way to Sunday. And so then it makes a very great difference what functional form we chose to fit. And so the key difference between the interpolatory and the extrapolatory models, as far as I'm concerned, is that the interpolatory models, we kind of understand the level of confidence that we ought to have. And it is primarily a function of the data rather than of the model and our assumptions and choices about the model. Whereas the inter the extrapolatory models, sorry, we are making big assumptions and they have a big impact. You know, you can imagine that if you had a slight difference for Monday, Tuesday, Wednesday, and you extended that line, you'd get a big difference on Sunday than if maybe you had a slightly lower value on Wednesday. So you need to be really careful then with extrapolatory models about the functional form that you're fitting, about the type of model and the type of assumptions that you're making, where it doesn't necessarily matter so much for the interpolatory models.
SPEAKER_03Are AI or generative AI models more of one kind or the other, or a blend of the two?
SPEAKER_00So it's not really a function of the model, whether it's extrapolatory or interpolatory. It's a function of the question that you're asking of the model. So supposing I can fit Monday, Tuesday, Thursday, Friday with one model, okay, then the interpolatory question is what was the number on Wednesday? And the extrapolatory question is what will it be next Monday? Yeah. So we've got the same model, but we can ask an extrapolatory question or an interpolatory question. And so we can expect that the model will be good for interpolatory tasks, and we have to be really careful for extrapolatory tasks. So for generative AI models, we can we can apply the same the same kind of distinction and say, what am I trying to ask about from this model? What am I expecting to get? So supposing I take a large language model and I ask it a question which is where we basically know what the answer is. You know, the answer is out there on the internet and it's in the training data, and we expect to be able to get that answer back. So you might say, what is the tallest mountain on planet Earth? And it will tell you the tallest mountain on planet Earth is Mount Everest. Okay, so and we are confident in that because we expect that there was lots of good quality training data, the answer was there, and it's just spitting it out. And then we, you know, we could ask all sorts of other questions. Of course, that's a very simple one. But if we wanted to ask it something extrapolatory, you know, let's ask the large language model who's going to win the next US election. Well, that is certainly extrapolatory. That is something that will require thought. It will be able to give you an answer. And I mean, if if I tried to answer it, I would also be extrapolating. I have no secret knowledge of what the future holds. And so, you know, I should, when I'm asking it that kind of question, I need to be more careful. So it's not, it's not necessarily a property of the model, is what I'm saying. It's a property of the question and the model together and the set of training data that you've used to calibrate that model.
SPEAKER_04Aaron Powell The current hot topic in AI, of course, is the generative AI model, which you seem to be getting really close to. But I I'm not sure, are they strictly extrapolatory or are they both extrapolatory and interpolatory? And what does that tell us about their nature and their usefulness?
SPEAKER_00Aaron Powell I mean, I think they do both. And it depends what we're trying to ask it to do. So, I mean, what it really tells us is that we need to be careful about the kinds of questions that we're asking. So if we have a large language model and we ask it a simple question where we expect that the answer is contained within the training data, then we should have reasonably high confidence that it will give us the right answer. And if we're asking it something for which the answer is not contained within the training data, then maybe we need to examine what are the assumptions that have gone into this, what is it that our model is actually trying to do? You know, what objective function is it minimizing when it is constructing the next, the next word and the next sentence in the response? And so that might that might give us some insight into what kinds of answers it's likely to give and what kinds of errors it might be likely to make. And it would hopefully give us some understanding of when we should have confidence in the output and when we shouldn't, and what we could use the output for and what we couldn't use the output for. So if you ask it who's going to win the next election, you know, that would be interesting information, right? It probably contains useful information there and it would probably be able to tell you, you know, narratively about why it came to that decision and what kinds of information it has available to it and how, you know, how other people think and have thought about that topic. But you certainly wouldn't use it to place a bet, you know, to bet all your money on that being the outcome, because you know that it's not going to be reliable in that sense.
SPEAKER_04So we we we spent a lot of time worrying about the data. In fact, I I you read all the time that we're running out of good data. So instead of pursuing the data, would a deeper understanding of the mathematical models improve the capabilities and limitations or identify the limitations of these AI models?
SPEAKER_00Yeah, I think so. I mean, certainly the limitations. And I think if we can define the limitations a bit more clearly, then we can understand more about the capabilities. Because, you know, what once once we can kind of draw that boundary of where we expect them to be good and where we expect them not to be so good, then we can not only choose not to step beyond that limit, we can also make the most of the information that we do have on this side of the limit. And so I think that's really important for all models, not just AI models, is to understand where that boundary is. How far do we expect to be able to predict before we know that the uncertainty is going to dominate? So, I mean, again, to go back to the weather forecast, you get your weather your phone out of your pocket and you look at the weather forecast, and you actually have a really good intuitive understanding without having to go and read any papers or or look into the technical details of the model. You have a really good intuitive understanding of the decay of information in that model. You know that tomorrow's weather forecast is pretty good, and three days' time is okay, and a week's time is indicative, and three weeks' time, you don't even bother looking at it because it only has marginal information. So maybe if you were trading natural gas futures, you would look at it, but if you're deciding whether to have a garden party, you don't bother looking at it because it doesn't, it's not going to be necessarily helpful to you. And so I think I think that's what's missing in many other spheres of the use of models is that intuitive understanding of what you can and can't use it for. How do you decide whether it is going to be decision-relevant for the specific question that you want to ask? And I think understanding those limitations, as I said, helps us to make use of the capabilities where we have them.
SPEAKER_03You've hit a frustration point of mine because what you just said has different applications for the developers of the models, especially the commercially available generative AI models like ChatGPT and what the public knows about them. Occasionally, a company like OpenAI has issued a system card, but the others haven't done so, and OpenAI has not issued a series of those. So they let us have one glimpse in at the limitations, but otherwise we have to play with them, the tool. And sometimes people cut short that play and start using it for serious work when they shouldn't. And that's when you find out about the capabilities and limitations. Or you read the terms of service, and you can see every time they shift a risk to the user that they're identifying a risk implicitly that the user, unless they are one of those rare human beings who reads terms of use, all the other ones know about it. So do you advocate that the developers should say more publicly about their capabilities and limitations of their tools? And can they do so without damaging the commercial viability of those products?
SPEAKER_00So I mean, this is what I mean by escaping from model land, because if you if you just say, here is the model go, you know, then you're not providing any information about the relation that that model has with the real world. You're not providing any information about the capabilities and the limitations. And so that's sort of fine as far as it goes, but you've got to remember that you're just dealing with some abstract mathematical object that somebody has programmed into a computer. What does it actually mean? Well, the only way to know whether it has any relevance to the real world is to evaluate. You know, we we must do evaluations. And I certainly think that the most qualified people to do the evaluations are the modelers because they probably already know where the weaknesses are and they can they can drill down into those and understand them better. But ultimately, the you know, the bottom line is does it reflect the real world in this in the for wherever we're trying to actually predict the real world? And maybe we're not, you know, if you're interested in using Chat GPT to help you make up stories, it doesn't matter whether it's related to the real world. It's actually not important. And so that is a use case where the evaluation isn't necessarily relevant to you and isn't necessarily useful to you. So having choosing to use or to take that use case is relatively risk-free. Now it's not okay, it's not completely risk-free. I can imagine ways of it going disastrously wrong, and you wouldn't want to just release stories that it generates into a society without looking at them first. But but it doesn't actually matter. You know, there there isn't really an evaluation that you can do relative to is it getting, quotes, the right answer because there isn't a right answer. If you want to get the right answer, then you have to do some sort of quantitative evaluation. So that that ought to be the the onus should be on the modelers to do that quantitative evaluation and to say that their model has some relevance to the real world before it's released into the real world.
SPEAKER_04Okay, you said something there that that kind of made my ears jump up. When I'm a software guy, and when we build software, the good developers will test it. And they'll test it really hard. But we don't usually want the final testing to be done by the authors of the program because they like it's it's like having an engineer write your program. They're gonna test the math stuff, they're not gonna test the UI or anything else. But you suggested that the modelers themselves explore the limits and extend the model. What about the the non-modelers? What don't they have a role to play?
SPEAKER_00Yeah, I mean they they do certainly have a role to play. And when when a model is released in whatever way, then certainly it will them the users are likely to discover additional limitations that the modelers were not aware of and push it in different directions. So I mean, so yes, I certainly agree. But uh but to release a model without having done any evaluation would be, I think, slightly ridiculous. And and yet the users should be informed users. They should be able to say, well, this is no use without an evaluation. So if you want to integrate it into business critical decision systems, then you should jolly well do your own evaluation as well. Whatever they've got, whatever OpenAI provide for you, you're gonna need to do your own evaluation too. But if you just want to use it to generate stories, then maybe it really doesn't matter.
SPEAKER_05I find the um the conception of extrapolatory and interpolatory very powerful and useful in thinking about these things. I learned it from your book, and it causes me to reflect back on some of our previous conversations with uh people in the AI space. So, for example, in episode four, we talked with Robert Erdmann, who's the chief scientist of the Rights Museum in Amsterdam, and they and he did some remarkable work to sort of reconstruct the missing pieces of the night watch. And that's I think of that as a an interpolatory exercise. Whereas in our episode eight, we interviewed the head of uh the stroke unit at Mount Sinai uh hospital in New York, and they're using vision analysis systems to look at CAC scans to predict, make make predictions to the on-call neurologist as to whether or not and where they might look to see evidence of a stroke. So this is a powerful tool for thinking about and reasoning about these AI models. So thank you.
SPEAKER_00And I think that example also highlights the the sort of risk tolerance question as well, because if you if you're interpolating part of a painting, obviously you would love to get the right answer. But if you don't, it's really not the end of the world. But if you're predicting stroke susceptibility and you're expecting to make decisions about medical treatment or allocation of resources based on that, then there's a a much, you know, it's much more critical to get the right answer. And so again, the evaluation question comes back that I think in one situation you would be more keen to evaluate and to check your data and to check its working and to ensure that you are making sensible decisions and cross-check that against whatever systems you had in place beforehand before you actually start doing it, you know, operationally. Whereas with the the painting, you can be much more, you know, you can be looser about what it is that you're expecting to get.
SPEAKER_04So it sounds like you're saying we certainly need better data, but we need to better understand how these things are working. The mathematical models that perhaps were even just creating with the data, that how that those models might eliminate the capabilities and limitations and how much risk we can assign to it with these non-models.
SPEAKER_00And I think also that we have to understand the values that are going into this as well. Because, again, I mean, to just to sort of come back to that previous question, that the the the value judgments involved in interpolating the the rest of a picture, it doesn't really matter. You know, nobody has got really strong political opinions, or probably very few people have got very strong opinions on what the right answer is. But for the medical, in the medical case, obviously there's a lot riding on it, people's lives and decisions and you know the resource allocation within a hospital. And so it makes a great deal of difference what we choose to do. And so I think then then we need to say, well, what is it that the AI is aiming to achieve here? Is the is the the objective function that we're seeking to minimize the thing that we are trying to get the AI to do, is it to maximise diagnoses, is it to minimize lives lost, is it to minimize cost to the medical system? There are many possible objectives that it could have. And, you know, we don't know a priori which one is actually happening. If we've just taken some sort of statistical model off the shelf and said, well, we're going to minimize the, you know, the sum of squared differences between this and that, what values are implied by that? What risk tolerance is implied by that? I think these are really interesting questions, especially as AI and these data-driven models start to be used operationally for decisions such as those, which have got obviously a lot of different opinions and values and interests are represented there. And we need to think about how those different opinions and values and interests are represented or perhaps are not represented within the models that we're using.
SPEAKER_03I want to continue on on the subject of the user's perspective. Both scientifically grounded causal models and AI statistical models present conceptual challenges to non-experts attempting to understand or accept their predictions or to evaluate them. What advice can you give to those of us attempting to grasp the predictions made by statistical AI models?
SPEAKER_00I suppose the first is to understand that it is simply curve fitting. And so it may be, it may be done in a very complex way with lots of different parameters and lots of data, huge amounts of data. But basically, we're just fitting a curve to pass data. And so that is extremely powerful. It can be extremely powerful, it can give us some incredible tools for making good decisions, but it also has risks. So I suppose that the advice that I would give, you know, if you're working specifically with a particular statistical model would be to, would be to ask the modeler whether you're whether you're dealing with an interpolatory or extrapolatory decision question and how good they expect the model to be, especially if you're thinking of an extrapolatory use case, and to try to understand how we get out of model land. You know, what is it that gives us confidence that this model is going to be any good? And so for the scientifically grounded models, causal models as you described them, the thing that gives us confidence that they're going to be good in future, firstly, is that they were good in the past, but secondly, is that we expect the laws of physics to continue. You know, we I expect a model of a basketball's flight to be as good tomorrow as it was yesterday, because I expect the laws of physics to be true tomorrow as they were yesterday. And I have really high confidence in that. But ultimately, I have no sort of reason to believe it, other than that I really have strong belief in the laws of physics. But we can we can all agree on that. Whereas when we come to the statistical models, I think it's much harder to communicate why it is that we expect tomorrow's statistical fit to be the same as yesterday's statistical fit. It's not obvious that that should be the case necessarily. And so the longer the set of training data that we have, and if it fit well for a long time, you know, that gives us some reason to believe that it might continue to be true. But obviously it depends what it is. You know, if you look at the weight of a turkey and you and you plot the weight of the turkey as it grows and grows and grows and grows and grows, and then would you project that forever, or are you going to stop at Thanksgiving? So you do often need to have some dynamical, some causal understanding of what it is that's happening in order to understand these kinds of limitations.
SPEAKER_03That was a wonderful answer. Improving a statistical model usually involves training with more quality data. How does one improve a causal model? And when do you know that the model is starting to drift, degrade, or possibly collapse?
SPEAKER_00Well, you improve a causal model in exactly the same way by taking more data. And typically you would then use that data rather than just to fit the curves more precisely, you would use that additional data to try to understand more about the processes that are happening. So you might say, I'm interested in, I don't know, crowd movement, and you'd think about, well, what are the processes that are actually happening? I've got a crowd of people who are trying to get from A to B, I've got some that are going from A to C, I've got some that are going from C to B, and I've got some that are just wandering about in the middle. So maybe I might try to fit all of those separately, individually, and use my additional data to fit a load of subprocesses and then put those all back into my big model. Whereas if we were taking a totally statistical approach, maybe we would either just fit everything in one go, or we'd hope that some of these subprocesses would kind of emerge in some way statistically out of the data, we'd do some kind of clustering and we'd spot those. I mean, eventually, of course, they they kind of coincide, or they ought to. We would hope philosophically that if we had sufficiently large data sets, that the causal models and the statistical models would become the same models because they would both learn the laws of physics and they would both learn the generating functions and the generating processes that are going on underneath the hood, you know, under the behind the scenes of whatever it is that we're looking at. And so, you know, I don't really see statistical and causal models as being all that different. You know, they're they're very different to begin with, but the more and more we do with them, the more similar they look, and the more we expect to get, hopefully, the same answer, you know, if there is a right answer. And I think it's still slightly an open question for me as to whether in many of these cases there is an answer at the back of the textbook that we can expect to converge on. So that comes to your your second question there about you know, when when do we expect the the progress to perhaps bottom out, to kind of flatten and reach reach some kind of limit of predictability?
SPEAKER_03Or decay.
SPEAKER_00Or decay, yeah. I mean, decay even is is an Interesting one. So for physical models, I suppose the general hope is that as we add more processes and as we add more data, we will put in more and more complexity into a model and that it will get better and better and that it will kind of get monotonically better until we approach the correct answer. Actually, this is something that I've written about in the past, about the perhaps the naivety of that assumption. And it doesn't seem obvious to me that that will always happen, especially for complex systems like the Earth system or like any system involving humans who are incredibly complex and will always throw a spanner in the works. We have difficulties, for example, with models of systems like the global economy. So if you take, if you look at central banks, the Bank of England, for example, projects uh like GDP and inflation every every quarter. But of course, they're they are endogenous to the system because people in the economy look at the Bank of England's forecasts and they will then use those forecasts to help them make decisions. So, so for example, if the Bank of England were to forecast that there will be a recession next quarter, you can be guaranteed that it would happen because people would look at that and go, oh no, what am I going to do? And so people, it it's it's part of the system in itself. And I think this is the case for many of these complex models, although perhaps on longer timescales, we can think of the climate system when people look at a climate model and the forecasts of what will happen to the climate in the future, that changes their behavior and that changes the global emissions trajectory. For example, we look at the Paris Agreement from 2015, which was based on the forecasts and the outcomes of global climate models. People looked at that and they said, oh dear, we don't want that to happen. We're going to have to keep global mean temperature change below two degrees. And the result was the Paris Agreement. And hopefully, fingers crossed, that will result in global mean temperature being kept below two degrees. And so we sort of forecast a counterfactual, we forecast what would have happened, and then we take action to make it not happen. And so this kind of comes on to another key theme of my book, which is the sort of participation of models in creating reality. I think that the way that we construct models, from a scientific perspective, you sort of hope to be a completely independent observer outside of the system. You just say, how do I make the best possible model and get the right answer? But ultimately, in most cases, we are trying to make a model because we actually want to make a decision. We want to influence the system, we want to create an outcome which is different from what would have just happened if we left the system to evolve by itself, whether that's, you know, reducing climate change or I don't know, getting improving energy policy or making better business decisions or deciding how to use artificial intelligence. You know, we we we use these tools in order to change what the real world is going to be like. And so I just think there's loads of interesting questions there about what that means if we are constructing models which are part of a system, then then the goal is not necessarily prediction, is it? So how would you say whether a model is successful or not? Not because, not necessarily that it gets the right answer. You know, you look at the forecasts of COVID from March 2020 and the sort of worst-case scenarios, the curves going up, hundreds of thousands of deaths, etc. You know, if those worst-case scenarios had been followed, that would have been catastrophic. Fortunately, that didn't happen. But the fact that the model made those predictions is part of why it didn't happen.
SPEAKER_05It was advocacy. It's it's fascinating. In a previous project, I worked on very long timescale predictions of various inflation rates. One of the things that I concluded after doing a lot of thinking and analysis and modeling is, for example, that some things seem always to have inflation higher than sort of mainline. So healthcare education costs go up always higher than sort of mainline inflation. And that that's impossible in the long term because at some point, you know, healthcare will become 100% of the of the GDP and and no one will tolerate that. So at that point you realize that what you're modeling is not real, and that inflation models that that predict very long-term stuff are nonsense. So that's sort of in sort of implied by what you just said. Anyway, thank you.
SPEAKER_00Yeah, and just understanding those timescales, you know, saying I understand that the weather forecast is good for a few days, or I understand that a COVID forecast is good for maybe a few weeks, or I understand that an inflation forecast is good for however long, maybe a few months. Um, and understanding where to stop and not not taking it out for a hundred years because you just know that it's going to be rubbish. You know, that that's where you have to apply your expert judgment and say, well, how how long do I expect that it could do this for? How how good how good is it remotely reasonable to think that this model could be and not get taken in by the hype?
SPEAKER_03Well, sometimes that's Mark's example earlier of Dr. Terram um using AI for to decide whether to inject a particular drug to limit the damage caused by a stroke, you're going to get that feedback immediately, and you are trying to affect the outcome in the real world.
SPEAKER_04Well, we're talking about the users now, and that's that's a very dear topic to me because I'm also an instructor at Dartmouth College. And one of the concerns that everyone in academia seems to have is is everybody going to depend on or students going to depend on AI too much? The very first day of class, I had just told my students how to look up something using online resources. And I look around and sure enough, immediately one student had already asked Chat GPT to look up something. So what do you think about the the danger of becoming over reliant on AI, especially in industry and companies where they might be using these tools to build systems that, well, who knows what it was trained on?
SPEAKER_00Yeah, I mean it's a difficult one, isn't it? Because uh sort of as we have tools, it's inevitable that they will affect us when when when you give a child a calculator and let them type in the numbers and get the answer, you want them to be able to do the calculation themselves first in order to understand whether you've typed it in right. You know, you if you if you want to know what 24 times 36 is, and you type it into your calculator and you get an odd number, you know, you want to be able to say, that's wrong. That's obvious, I don't know what the right answer is, but that is obviously wrong. And I think that's the kind of sort of level of user sophistication that we need with these tools as well. We need to have people who aren't just resorting to the AI or the model to give them an answer without an understanding of whether or not that looks plausible and with some idea of how to check it, you know, how to how to go back and look at this and and sanity check it and have have an idea of whether they should or shouldn't be confident in that. So I, you know, I'm fairly optimistic about the capabilities of generative AI. I think that it's it's amazing. I think it has the potential to be extremely useful. Once we understand what these limitations are, then I think we can we can make the most of those capabilities. But I certainly think that there's a danger in developing a generation, a cohort of students who default to that and who don't have the the contextual understanding of both what it's doing and of maybe the question that you're asking to be able to spot the the egregious errors and to work around the mistakes. Yeah, I mean, I don't know what you what you think as an instructor. When you when you see a student doing that, what's your response?
SPEAKER_04Well, I I really don't mind them using it as a reference guide.
SPEAKER_00Yeah.
SPEAKER_04Other than the situations where you might not have access, you might not be online. What disturbs me is when they submit a programming assignment and they can't explain it.
SPEAKER_00Yeah.
SPEAKER_04That's that's the over reliance. Yeah, I mean that's exactly it.
SPEAKER_00You know, why would why would you bother getting a qualification at all if it's just if it's the generative AI getting the qualification and not you?
SPEAKER_01There's nothing wrong with Erica. You yeah, that the conversation you were just having was very interesting and very intriguing. And I'm wondering from both of your perspectives, you know, we're talking about understanding the models, its limitations, how to draw inferences, but thinking of this next generation and their use of generative AI and these new tools is part of what's missing in the conversation, how to talk to people about what these tools are and what they can and cannot do. Right. I think to say to a student, you cannot use generated AI maybe too stark. I don't know. I'm not a professor, right? That's just the reality. They're probably going to use it, right? Even if you have them sign an honor code or you turn, you log on to your computer at work and you say you won't use generated AI, all of these things, right? But it is part of the education here and moving this forward, enhancing the conversation about how to use these models, how to use generative AI, how to use them effectively, and then understanding what overreliance can create or what manipulation of data can lead to.
SPEAKER_00Yeah, definitely, 100%. I mean, and I think, you know, when you think about students, of course, they're in the main, they're paying to be there and they're paying quite a lot of money to be there. So I think the question to pose to the student who wants to make use of AI is, you know, how is this benefiting you? How how what does the student expect to get out of turning in an assignment which has been completely constructed by a generative AI model? So there's nothing wrong with using it. I think it's totally acceptable to turn in an assignment which is generated by AI, but you should expect to be marked down. You know, you should expect that if you've turned in exactly the same essay as 39 other people, that it has no originality, it has nothing of interest. You're saying, I so I like to see generative AI as kind of the baseline of a zero-skill model which tells you what anyone else on the internet could do. You know, anybody with access to this could generate that outcome, that output that you've just handed in. So, how does this give you any useful skill or ability to get a job in the future, for example? What you've got to be able to show is how you can either use the tools better or add value in some way. You know, if you if you have a set of six really interesting insights and you get ChatGPT to write an essay around it for you, fair enough as far as I'm concerned. I think that's okay. I'm interested in the quality of the insights and the original stuff that you can put into that essay and that output. And if we think about, you know, if if you're consulting for an organization and they're interested in some kind of output, they don't want to see what ChatGPT can give because they've already done that. You know, what they're going to pay you for is the added value of additional insights, something that nobody else could have done. So what I think I think the key in terms of the pedagogy and the teaching is to help students understand that the way that they can stand out, the only way that they can stand out is by doing something more than what ChatGPT can do as a baseline. And I think that's really challenging. You know, I think I think that's it's a really interesting new challenge for education and for students, you know, not to be too worried about spelling and grammar and all the rest of it, because there are tools that can do that for you now. And perhaps it's also, you know, articulation and fluency, these are things that ChatGPT can now do for you as well. But if you've got the extra ideas as well, then you plus Chat GPT is going to be amazing. But if you're just at the level of putting the question into Chat GPT, then you should get zero.
SPEAKER_03I want to share with you also the some guidance that was issued by the judiciaries of England and Wales back in December of 2023. And one of the points they made, they said, as with any other information available on the internet in general, AI tools may be useful to find material you would recognize as correct, but have not got to hand or ready at hand, but are a poor way of conducting research to find new information you cannot verify. They may be best seen as a way of obtaining non-definitive confirmation of something rather than providing immediately correct facts. And I think that's a helpful caution. And some of the AI developers put in their terms of service, you need to verify every answer that the product generates. But because people don't read those terms of service, they're not on guard for doing that. Whereas to give you a very complex comparison on submarines in certain navies, when the computer is generating a fire solution, the executive officer is conducting that exercise, the captain is doing it in his or her head as a check, because if the generation of the firing solution and the computer using AI is drifting away from what the captain believes to be the accurate solution, the captain can intervene because you do not want to fire a torpedo using the wrong solution, whether because you'll miss the target or because you'll hit a friendly.
SPEAKER_00Yeah, absolutely. And this comes back to the interpolation versus extrapolation as well, doesn't it? That if you if you sort of know what the answer ought to be, then you're doing interpolation. And if you don't know what the answer ought to be, then you're doing extrapolation.
SPEAKER_05I find your your comments on use of generative AI in education fascinatingly optimistic when you think about it. That's a wonderful. It was a very it's a there's a perspective that came out of your answer that I find very encouraging and kind of positive in a way that I had not expected.
SPEAKER_00Well, I think there's no way to get around them. I mean, you you can't put the genie back in the box, can you? There's uh there's we're gonna have to work out a way to live with this. It is going to be attractive for students to use, it's gonna be attractive for the business world to use, and it is an incredible tool. So we've got to work out what we can use it for, what we can't use it for, and help to educate the next generation to understand where those limits are.
SPEAKER_04Yes, uh I I plan on using your uh analogy there as well. What I've been telling my students is sure, you can use it even to explain difficult parts of code, but just keep in mind it's it's very treat it like a fellow student. You wouldn't ask your fellow student to write your code for you, and plus uh you probably are aware that your fellow student might be stoned sometimes and give you some interesting answers. So, but this it's not just about students. Um the public in general, policymakers, oh my gosh, that's a big problem. They have a hard time grasping and learning when and how to trust these predictions and how to choose the model, how to decide who to listen to. Do you have advice on how to improve decision making based on these models or whether it's improving the models or improving the interpretation?
SPEAKER_00Well, I think we need to do both. I mean, and ultimately that's why I wrote the book, really, and that's why my book is a is a sort of popular science book rather than an academic textbook, because I felt like this is a discussion that needs to happen at that more societal level. You know, we we need to have more understanding of how these, you know, incredibly useful and yet also incredibly complex and obscure and sometimes very much black box in terms of both the data and the model processes and the social and political values that are implied by those models and processes. You know, we we just don't necessarily understand them and we need to understand them. So I think what we need is more model literacy, and perhaps that comes down to partly to mathematics education, but also, you know, more generally, it's civics education, right? Is understanding how some of these things work. Not necessarily the details of every single model, because of course you need technical domain expertise to understand them, but to be able to ask relevant model literate questions. Why is this the case? Why, you know, who made this model? How did they make this model? What are the assumptions? How far can I trust it? What evidence do you have that it's reliable? And how do we escape from model land? You know, how do I know that this isn't just a bunch of equations on a computer? How do I, why should I have reason to believe that it has relevance for my decision in the real world? And so I think it it's a difficult one. We do need more education, and this is something that I've been thinking about myself is what what kinds of training perhaps do we need for decision makers, whether they're in industry or policy making. I mean, I'd be interested to hear your thoughts on what sorts of training would be valuable. And again, for me, it comes down to understanding the kinds of models, you know, when models are likely to be good, when they're likely to be less good, how far we can trust them, and what kind of values are embedded within them. But uh, like I say, I've written a whole book about it. So it's hard to give a short answer on that.
SPEAKER_04There's the whole issue of bias. I mean, that's that's I think this is where that first really kind of appeared, which identified flaws in the selection of training data.
SPEAKER_00Yeah, but it's all the way through. I mean, it's the data, it's the model, it's the people that made the model, it's the social context around the model, it's the interpretation of the model, it's the operationalization of the model into decision-making systems, it's the consequences of that. There's, you know, there's there's a lot there. But but we can kind of discuss those in a in a context-agnostic fashion. We can we can talk about how models are supposed to work, how they do work, what are some common traps and pitfalls, what are some difficulties, and what are the kind of questions you need to ask in order to consider yourself well informed enough to understand whether or not to use these models. So I think I think that is an achievable aim, if not if perhaps an ambitious one.
SPEAKER_03I want to emphasize something because part of our audience is lawyers. And when I talk to lawyers, law firms, or clients about generative AI products, the risks, the capabilities, the limitations, what I keep running into is that they by and large have a pretty good grasp of data quality. They have a pretty good grasp of what they'd like the model to do, but what they don't have a grasp of is models or model literacy. That's what led me to pick up your book after I read a review of it. And I'd like to share with our audience just one basic response, which is your book is amazing. It it was a page turner on a subject that's could have been, you know, very difficult and challenging because I'm not a model builder. This was not something I had a background in. I think Charles had the same reaction when he read it. And it's we would encourage our listeners to get your book if they found this at all interesting. And I thought this has been a fabulously interesting discussion, and really want to thank you for being so clear, articulate, and responsive to our questions.
SPEAKER_00Well, thank you so much for having me.
SPEAKER_05One of the things that I find it the really comes out clearly in this conversation, plus some of the others, is that there's an emerging theme of how to educate people. You know, in the bad old days, policy people were educated in rhetoric and you know, logic and a few other things. Increasingly, I think we need to begin to articulate curricula at the undergraduate level, maybe even in the high school level, that focus on numeracy, I think is the word that people use to deal with sort of reasoning about models. But all you know, people have also been talked about digital literacy. I I think the the young people coming up will have a level of digital literacy that we don't need to worry about, uh, though it would help if they understood what was going on inside of the box. So there's two boxes to make sure people understand. One is what's going on inside of computer broadly, and the other is what's going inside going on inside of models, because those are those are two things that are going to be really important. And the arrival of things like ChatGPT will mean that, oh, if you want a fluent exposition, you don't need to waste many years of your life, okay, that I've wasted and that all all five of us have wasted on becoming fluent in our language. This reminds me of a paper I read in the 80s, which studied Japanese academics, scientists, and showed that their success rate was their their career success rate was much more correlated with their mastery of English than necessarily with the quality of their research. And I could envision things like Chat GPT leveling the playing field for people who are not as fluent in you know in the dominant language, English, as uh as other people might be. I mean, that'll make life harder for us, but I think it'll be better for the world at large. Anyway, sorry, didn't mean to rant.
SPEAKER_03Which isn't to change the fact that mastering the language is a lifelong apprenticeship, and getting better and better at using metaphors and models is also a lifelong apprenticeship.
SPEAKER_04Well, again, Erica, thank you so much for joining us today. This every time I have thought we have one of these, I think, gee, this is the most fascinating one yet, and you continue that stream. Just all kinds of ideas to take out and think about. I'm sure our audience will understand the way you explained it better than if I had explained it. So again, thank you for your time and thank you for your book.
SPEAKER_00Fantastic. Well, thank you so much for the invitation and for a great conversation.
SPEAKER_03We thank the business law section of the American Bar Association for their generous sponsorship of the production of this podcast. Send email to comments at mindthegapdialogues.com. We read all comments and questions and will try to respond in the letters section of a future episode. If you are writing about a particular episode, please do mention a specific episode number. Please also do include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap Dialogues on the AMR.
SPEAKER_02Thank you for listening to the Aviate Business Law Sections Podcast Series to the extent that the section offers a robust collection of content. To explore more about this topic, or to learn about joining the section, visit ambar.org slash bizlaw. That's B I Z L A W.