Exploring AI Matters
Our mission is to help the policy community understand the breadth and richness of AI and the potential for such technologies, wisely applied, to augment all sorts of human endeavors.
Some AI tools are able to assist humans in performing tasks faster, more accurately, or more efficiently. Some, however, are inaccurate and unreliable. Who or what we hold accountable for these flaws, and what incentives we do or do not create for their correction will influence AI’s hand in how we work.
In this series we will refine, sharpen, and clarify your understanding of AI.
Exploring AI Matters
Episode 3 - The Future of AI in Lending Decisions
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Send us comments and questions.
Good credit impacts whether you can rent an apartment, take out a mortgage or car loan, or in some instances, receive a job offer. Financial institutions are increasingly utilizing AI to analyze non-traditional data sources, such as standardized test scores, to assess a borrower’s risk profile. In this episode of Exploring AI Matters Dr. Talia Gillis, Professor of Law at Columbia University, will discuss the opportunities and challenges of AI-based lending. [2022-07-12]
Welcome to Exploring AI Matters. This podcast series, previously known as Mind the Gap, Dialogues on Artificial Intelligence, will continue to appear in the ABA series to the extent that. In addition, all of the episodes, old and new, will now appear under our new podcast name, Exploring AI Matters. Thank you.
SPEAKER_02Our guest today will be Columbia Law Professor Talia Gillis, whose article, The Input Fallacy, published in the Minnesota Law Review in February of 2022, explores the benefits and drawbacks of using AI to predict creditworthiness and determine loan pricing. The growing use of AI in credit decisions expands what is considered creditworthiness data and how it gets used. Credit criteria, like FICO scores, traditionally used only loan payments to large and established financial institutions to determine creditworthiness. But as Professor Gillis' article points out, lenders now use AI increasingly to assess non-traditional data inputs not generally available in credit files, including information on the applicant's education and digital footprint data, such as the device and operating system a consumer uses when visiting an online purchasing site. While AI-generated alternative credit scores have proved in many instances to achieve improved accuracy in predicting credit worthiness, AI does not uniformly generate improved predictions. It is also not clear whether AI can reduce the occurrences of discrimination in lending practices. To help us break down the challenges in AI's use of high-dimensionality data in the consumer loan context, we are fortunate to have as our guest a scholar qualified as a computer scientist, a lawyer, and an economist. Professor Gillis earned a law degree from Hebrew University, clerked on the Israeli Supreme Court, earned a graduate degree from Oxford University in law, and is now currently teaching at Columbia Law School and earning her doctorate in economics at Harvard University. Welcome, Professor Gillis. Thank you for making time to join us in this conversation.
SPEAKER_01Thank you, and thank you so much for having me.
SPEAKER_04Hello, I'm Mark Donner, a computer scientist.
SPEAKER_02And I'm Roland Trope, a national security lawyer. We are your hosts for this episode of Mind the Gap Dialogues on Artificial Intelligence.
SPEAKER_04In addition, we have two more hosts.
SPEAKER_00Hello, I'm Mama Adams, a national security lawyer.
SPEAKER_05And I'm Charles Palmer, a computer scientist.
SPEAKER_04Each episode will be led by two of us, with the others adding impromptu questions and comments as the spirit moves them. So back when I first applied for a loan, back in the days of the dinosaurs, I filled out a two-page paper application. Subsequently, I met with a loan officer who talked to me while looking at the two pieces of paper I had filled out. How has institutional lending evolved since then?
SPEAKER_01So I'd say that the institutional lending has changed in many ways. So if we think of kind of um, we often talk about algorithmic lending, um, but that kind of refers in many ways to three distinct but very much related trends that we're seeing going on today in the world of consumer lending. So the first kind of an important trend is towards automation. So while traditionally a lot of lending decisions involves kind of human discretion and human involvement in the process. So perhaps you'd meet with that loan officer, they would engage in kind of extracting some of that information. Sometimes they they maybe um would kind of learn something about your credit worthiness or something that might be relevant to the decision whether to provide a loan or not. Um, we see a trend towards all that process being automated, the decision making of whether to give a loan and under which terms happening um automatically rather than the human uh discretion. I think the second trend is relevant to kind of the information that's used to reach these lending decisions. So we can think of kind of traditional lending information, such as credit score information, income information. And to some extent, this is related to the first trend of automation, because traditionally I think we loan offices were thought of uh as kind of an art in a way of also inferring information about credit worthiness. But today we're seeing um kind of much more use of non-traditional data. Now, non-traditional data could be um information about people that that kind of maybe intuitively seems more relevant to kind of ability to pay and credit worthiness. So that might be information like rental, timely rental payments that today isn't considered by kind of mainstream credit scores. It could be information on cash flow, but also kind of a lot of information that we really would think is kind of non-traditional, perhaps hardest to see its relationship to credit worthiness, whether it's kind of consumer behavior or the type of kind of metadata that you talked about to do with um uh what operating system you use, how you use the mouse, a mouse and information like that. And then I think the third trend that's very relevant is towards kind of the use of advanced prediction technologies. So even credit scores, you've some kind of statistical technology to kind of take information and transform it into some score or some prediction of credit worthiness. But what we see today is kind of use of kind of machine learning and much more advanced prediction technologies, which again is very much related to this idea of kind of the move to non-traditional data, high-dimensional data that needs a kind of more sophisticated statistical technique to take that information and to create predictions of credit worthiness that would be relevant to consumer loans.
SPEAKER_05Why has it changed?
SPEAKER_01Yeah, that's that's an excellent question. I think there's two reasons why we see so many changes taking place in the world of consumer credit. The first is that um consumer credit or the prediction of credit worthiness, unlike a lot of domains in which we hear the use of AI, is a domain in which it's actually quite natural to reduce the question to one of a prediction problem, right? It's not in many domains, for example, the use of AI in employment decisions, um, the use of AI in judicial decisions, really, you're using a prediction problem within kind of a broader set of some decision-making context, whereas very often in the context of consumer credit and the prediction of credit worthiness, it's quite easy to define it as a prediction problem. And so it's a domain in which kind of these new technologies kind of to some extent um kind of shine the brightest and perhaps relate to kind of the fact that creditworthiness in the past as well has related to a prediction problem that uses data. I think the the other reason why we're seeing uh this recent explosion in consumer credit is that consumer credit is a domain uh which traditionally has been very problematic for consumers, particularly in the US. So in the US, um where we rely very heavily on credit scores to access mainstream um lending institutions, it essentially has meant that consumers that don't have uh a credit score, consumers who have very thin credit files, have been excluded from mainstream lending. Um, but we now know that many uh borrowers are very credit worthy even if um they don't have the kind of traditional files. So, for example, the um the case of consumers that pay timely rental payments, they may be very credit worthy even if they don't have kind of a mortgage and they weren't paying, uh repaying a mortgage on a on a timely schedule. And so when those kind of behaviors that are shown to predict creditworthiness are kind of excluded from mainstream credit scores, it does create um kind of this um kind of block to access to credit to um large um segments of the population, particularly kind of more vulnerable or minority, racial minority populations. And so I think that part of the push towards these kind of alternative ways to assess creditworthiness is very motivated by this idea that credit, consumer credit markets currently are slightly broken in the US.
SPEAKER_04So given that AI accuracy and credit assessment is imperfect, how do lenders know when it is and is not working?
SPEAKER_01Yeah, so I think that's that's actually very tricky to know whether a prediction of credit worthiness is actually producing the results that we want it to produce. So for some borrowers, you know, the borrowers or the applicants that receive a loan, one of the ways we know whether we've done a better job at predicting credit worthiness is to see whether, in fact, um people who received a loan defaulted or not, right? If the prediction is ultimately with someone makes a timely payment or defaults in a loan, then potentially we have that information, we learn of the the two, the true outcome over time, and we're able to say how good that prediction was. But of course, the difficulty is if we're trying to predict someone will default on a loan, we only learn about that outcome, whether someone defaulted or not, if in fact they receive a loan. If an applicant doesn't receive a loan, then we don't know whether the prediction was right or wrong because we simply don't know the outcome. And that's a big challenge in all domains in which we have prediction problems, in which we don't observe outcomes, but it's particularly problematic with consumer lending. So if there's, for example, a certain population that are more likely to be denied credit systematically, we don't have the ability to learn about um their creditworthiness over time. And potentially we create the situation in which kind of the predictions themselves are biased by this fact that we're not learning anything about those populations. So it's actually pretty tricky this question of how we actually learn whether our algorithm is doing a good job or not.
SPEAKER_04Very interesting point. Um, so there was a recent incident in the press in which each member of a married couple applied for a line of credit from the same institution. The male received a much higher credit line than the female, despite the fact that her income and assets far exceeded his. What do you think this teaches us?
SPEAKER_01Yeah, so so this came up um in the context of um of AppleCard that's issued um by Goldman Sachs. And um Goldman Sachs' initial response to this accusation that a wife and a husband receiving different terms is that it couldn't possibly be um discrimination or driven by gender because they don't know the gender of their applicants. Um and so I think the first thing to kind of note about this incident is in a way that was a very unconvincing response. And I think Goldman Sachs experienced that given the backlash that they um received after they kind of announced that it couldn't possibly be discrimination because they don't ask applicants about their gender. I think that um it was pretty clear to everyone that that in of itself isn't an indication, isn't necessarily an indication that gender is not influencing the decision. I think the other thing um to note about that incident is that it didn't just end with um the accusation um on um on Twitter. There actually was an investigation by the New York State Um Department of Financial Services that looked into it and concluded that there wasn't discrimination. However, that report itself was very opaque. Um it's very hard to figure out exactly what the reasons were behind its conclusion. And I think it's very important going forward that regulators themselves are also transparent about the types of analyses they're using to establish um fair lending.
SPEAKER_05Charles. Oh, I I gotta jump in here because this is this is really scary stuff. But you know, is the US alone here? Is this a US problem or is this happening everywhere?
SPEAKER_01So it's definitely not a US problem. So um countries all over the world are grappling with this issue of how do we kind of capture the benefits of AI while making sure that they um don't discriminate and kind of disproportionately affect populations that are already vulnerable. Um, and so we've seen a lot of initiatives of um of private companies, we've seen initiatives um of non-governmental bodies. Um, perhaps one of the kind of um most important developments that we've seen over the past few months is um the European Union um releasing its proposed rule for the regulation of AI. So it's a proposed rule, it hasn't been adopted yet, but it's um but it's really kind of the most ambitious and broad attempt to regulate AI. And it's very important for our purposes of consumer lending because um what the proposal does is it distinguishes between different types of um use cases and kind of adapts the level of scrutiny or the level of regulation depending on kind of how high risk that use is. And it actually designates credit worthiness assessments as high risk. So it's definitely one of those domains that we could probably kind of expect that in Europe at least will receive quite a bit of attention.
SPEAKER_02And so you try to model your behavior in a way to improve your credit worthiness present and future. What worries me from what you've been describing is that if FinTech companies start to use things like SAT scores, which when you're taking the SAT, you're certainly not saying to yourself, this will improve my credit worthiness. You're letting AI draw on data that people can't change their behavior to modify and improve. And it creates a different kind of adverse unintended consequence. So that leads me to ask um fintech companies seem to have access to a tremendous amount of this kind of information about us that we didn't realize that they had. Uh, when Mark would meet with his loan officer, he didn't come in with his SAT scores or his IQ tests scores from grammar school. Um, how are the fintech companies getting these data? Do they purchase it? Do they license it? Do they develop it themselves? What's the process? And is the process one that's uniform or is it itself changing rapidly?
SPEAKER_01Yeah, so I think that that very much varies by the loan type and even kind of the specific firm providing the loan. So there's a lot of variation in the types of practices or the types of information that's being collected under kind of this heading of algorithmic, um, algorithmic lending. So sometimes information is provided directly by consumers to lenders. And I think a lot of the examples in which we're seeing the use of SAT scores are related to contexts in which we're dealing with populations that may not have a credit score, typically because they're quite young, they haven't had time to build up their credit file, but they nonetheless might be credit worthy. And the use of GPAs or SATs might be a way to kind of signal or indicate credit worthiness when other information isn't available. Um it may be that kind of over a person's kind of decades later than when they go to college, an SAT and a GPA may not be the best indicator of their credit worthiness if kind of over time they've engaged in these other kinds of uh either lending practices or um or um uh payment practices. But very often that information on an SAT or GPA will be provided at kind of an at an early stage. And very often that will kind of be directly provided by the borrower who's interested in the information. Of course, there are many contexts in which it's not kind of directly asked of the consumer, but rather either automatically collected. So, for example, when we talk about use of cash flow or um information about the bank account, that might be information the bank has collected independently of the credit decision, um, but then kind of uses for its credit decision. And finally, I think another example that kind of sits in between these two extremes of kind of upfront asking for certain information versus kind of it being collected automatically and perhaps unknown to the consumer, a situation in which the consumer will actually allow a lender access to their bank account or to uh an IRS form and gather information in that way. And so then there's some kind of automatic process that um gets that information, but it's known to the consumer because they kind of provide the permission for that information to be collected.
SPEAKER_00So as I hear you guys kind of going back and forth on this use of non-traditional data, you know, one point, Talia, that you were raising that struck with me was you know, while there can be some perhaps negative inferences as we talk about sort of SAT and GPAs and what that means, what I hear you saying, and please correct me if I'm wrong, is non-traditional data inputs can be helpful in this process, but it really depends on sort of the quality and quantity of the non-traditional data inputs that we're using to assess these credit rules and creditworthiness. For example, you gave the example of if someone makes rent payments consistently over a period of time, that can be a very helpful non-traditional data input. Um, whereas when we talk about GPAs and SATs, it doesn't seem so natural to us, but that's just one of many, perhaps, um, that are being used by fintech companies.
SPEAKER_01Yes, I think that's that's exactly right. Um, so much of this depends on the quality of the data. Um, and um that's something we we sometimes overlook. We just assume that if you're using this information, then the good quality information is out there. But it's actually very complicated. And even in the easier example that I provided of um this idea of using timely rental payments, it's really not that straightforward. And so um FAMI may recently announced that it would use rental payments in their own models of making determinations whether they um will kind of would guarantee or purchase a loan from mortgage originators. And they are essentially now grappling with this issue of how exactly are we able to verify these rental payments and make sure that we have that high quality information. So, yes, a lot depends on having um high quality um data. And it gets particularly complicated when um the lack of quality isn't spread equally among all groups. So if quality was poor across all groups, then lenders would probably say, well, this information isn't very useful. Where it gets tricky is when there's some populations that perhaps kind of um the way they pay rent is much more straightforward through third companies, easily verifiable, versus kind of other groups that perhaps don't have it so easily verifiable. And then we have a policy that disproportionately affects um different groups.
SPEAKER_02But there's a question we're sort of jumping over. We're saying that the fintech companies get access to all this data. They're using AI to connect dots that we wouldn't have connected otherwise. Um, but there seems to be a question of do they determine whether that data is in fact giving them greater accuracy? And second, is it ethically sound for them to be using that data even if it is accurate? If the data is, let's say, SAT scores again, and there are certain populations that just don't get a leg up early on doing well on the SAT scores, that may not predict, in fact, their success at college, but it may be damaging to their credit worthiness. Can you address both of those questions for me?
SPEAKER_01Yeah, so at a certain level, um there's um we can think of uh lenders themselves having a business interest in the information being good information that's predictive of credit worthiness. So, kind of the first level to recognize is that um in in some domains, the the interests of consumers and lenders might be somewhat aligned in this respect, um, that we want information that's able to predict creditworthiness. I think there are two important caveats to that. The first is that going back to the point before about the fact that it could be that the quality could vary for different groups. It could be that for lenders, it's good enough when they're able to capture enough of the borrowers and say something accurate about enough of the borrowers. And be less concerned about the fact that they have inaccurate data about some other groups and perhaps lose out on the profitability of those loans, but perhaps it's not worth the extra effort of making sure that that data is accurate. And in fact, that kind of people who study the history of credit scoring in the US, that I think that story very much resonates with how kind of credit scores were developed over time. The concern very much with credit scores would be that even if kind of for the majority or the mainstream people with mainstream financial lives, the credit score did accurately capture their information, the concern were always people at the tails. So I think that's the first caveat. The second caveat is that lenders might have an interest in pricing or taking into account factors that kind of from a normative perspective, we don't think that that's okay. So Roland, you express that sentiment with respect to SATs, but of course, the mainstream examples of factors that borrowers that sorry that lenders might want to take into account, but nonetheless are problematic from a normative perspective, are all those protected characteristics that exist in our fair lending legislation, like race, gender, age, marital status. And in those cases, we can imagine that a purely rational borrower or lender might want to take those things into account and yet they're limited in their ability to do it. So kind of business interests alone, right? The business interest of kind of making a prediction that's accurate, because if a borrower defaults, um, that is a loss for a lender. Business interests alone, I think, don't capture all the things that we would care about from a normative perspective when it comes to this algorithmic lending.
SPEAKER_02This sounds very worrisome to me because I think of AI metaphorically as a kind of black box. And now the lending practices seem like a black box surrounding that black box. So to ask about that second kind, do fintech or financial companies disclose how they verify the accuracy or completeness of the information they're obtaining and using with their AI systems.
SPEAKER_01So this again varies by kind of loan, uh loan type and loan institutions. Um, sometimes with larger loans like mortgages, the verification of consumer information is a very, very important factor. So I could report my credit score, I can report my assets and my income, but then that would have to kind of be very much verified by the mortgage originators. And there are a number of technologies that are now being used in that verification um uh process. And that will kind of extend to some of the information that's obtained, kind of the non-traditional information with algorithmic lending. Of course, with traditional credit reports, uh, you as a consumer can view your report and you can correct your mistake. Um, we still don't have that equivalent mechanism, perhaps, in the world of algorithmic lending. But one thing to keep in mind is that there is a requirement in all types of consumer lendings to provide consumers with what's known as an adverse action modus, which is essentially a document that provides some explanation of why a credit decision was taken, such as a denial of credit or kind of a change in terms. And that may provide us in the future with some information that will make us understand more about how these decisions are made by algorithm lenders.
SPEAKER_04Mark? So is there a risk that this type of information might serve as a proxy for protected characteristics like gender or race?
SPEAKER_01Yes, I think there is a real risk that information um will be used as a proxy for protected characteristics and essentially kind of undermine the fact that we're trying to avoid the differential uh pricing or granting of loans based on protected characteristics. Now, this is something that has always troubled fair lending. So um so um a typical example is um the use of zip codes in credit pricing that has always been suspicious because zip codes are often thought of as a proxy for race. And so this idea that you have to think about how other information uh becomes a proxy for those protected characteristics has always been a concern. Of course, it's much more complicated when we talk of kind of big data and the use of machine learning. And that's because it's not really about these um individual variables that we're able to isolate and recognize them as being proxies, but rather um complex interaction between variables that are often substitutable that then form um a protected characteristic. So I think this is something that we should be very much concerned about and thinking of how to how to mitigate this concern.
SPEAKER_02Could you give an example would you just describe, just because I'm not sure what those variables might be?
SPEAKER_01That's exactly the point. But it is unclear what those variables would be to some extent. So something I've done in my work is I've said I've taken variables um from the Hamda data set. So um mortgage originators in the US are uh required to report information about uh mortgage applicants. Um, and so I have this data set um from the Boston Fed um the Boston Fed Hamda dataset that essentially has all this information about uh borrowers and the loaners that they apply to. And then I ask, well, how accurately can I predict the race of an applicant from all this other information other than race, but pretty fairly standard variables like um past credit history, um income information, the loan information. So I try to predict uh race from that pretty standard data set, and I compare that to the prediction of race from zip codes. Again, keeping in mind that zip codes has always been the kind of the archetype for a proxy for race. And I actually find that I can predict race more accurately from those traditional very loan variables and borrowed variables than from zip codes. And I think that that should kind of um that should worry us that um in the interaction between these variables, we're able to kind of reproduce these protected variables in a way that's even more accurate than what we've always kind of traditionally thought of as classic proxies.
SPEAKER_04So in 2002, Andrew Pol, a statistician working for the retailer Target, developed a remarkable technique for predicting when a household included a pregnant woman purely by scoring a variety of purchases. While we know that the predictions were disturbingly accurate, we don't know what percentage of the pregnant women in the target audience it actually found. Can we appropriately shield protected information in the output of processes like this? How might we achieve that?
SPEAKER_01Yes, so the example of whether a woman's pregnant or not is um is a really fantastic example of what I was suggesting before, that where things could could kind of go wrong is when we don't actually observe the outcome and we have kind of we don't have that feedback loop or that new training data that we can use to then form kind of better predictions. And so if Target doesn't know ultimately about whether they actually were able to target pregnant women or not, um, then that will you know put into question of kind of how accurate this algorithm was and whether they can train if they wanted to train, or if we wanted them to train a more accurate algorithm, whether they'd be able to do that um in the future, because they essentially, in many cases, won't observe um the outcome. But I think also one of the things that uh this example of um target clearly um shows is that it probably wasn't a particular product or kind of one individual feature that was um kind of indicating to Target this is a pregnant woman, but really kind of the interaction of products that perhaps many, each individual product is is bought by many people um who are not pregnant women. Um and so I think that that idea kind of that those were that was early days of kind of thinking about that, but we that would very much play into kind of um uh in algorithmic lending um nowadays. I think that the question of how we might um shield ourselves against that situation, assuming that we we don't want Target to learn or act um or treat people differently based on whether they've they've managed to predict that someone's pregnant or not, is a quite a complicated question. But I think the key takeaway, in a way, from from um the Target example is that telling Target not to use any individual per purchase or individual behavior, probably in of itself in isolation, would not greatly impact that prediction that they were making about kind of the status of the women.
SPEAKER_04So so this sort of a sort of a key question. You titled your article, The Input Fallacy. Can you explain the title and give us the gist of the argument?
SPEAKER_01Yeah, so the article um focuses on the fact that um many of the kind of policy proposals um that we see, both by um researchers but also um by governmental organizations, have kind of focused on what I call input-based approaches to regulation of AI. Um, and that essentially means policing the inputs that go into an algorithm as a way to guarantee that outcomes are fair or non-discriminatory. So that policing of inputs may be through saying, let's um make sure you don't include a protected characteristic as an input. It might be through saying, let's make sure you don't use a proxy as an input or just limiting the types of information that can go into an algorithm. And so the focus of the paper is essentially saying why this kind of input-focused approach is misguided, um, both because it's likely to be ineffective for some of the reasons that we've discussed, because um, when a protected characteristic can be inferred from other variables or um and and where proxies are not easily identifiable and we can't isolate them, then thinking that we are able to kind of guarantee non-discriminatory outcomes through this process just simply um isn't true. Um, but also it's potentially harmful. And it's harmful for a few reasons. It's harmful, first of all, because sometimes we actually want to be race aware when it comes to fair outcomes. Now, this is quite radical, perhaps, within our current thinking of um of discrimination, which obviously sometimes is characterized um characterized by what I'd call kind of a myth of um a myth of algorithmic colour blindness, this idea that when we exclude these protective characteristics, we've kind of guaranteed non-discriminatory outcomes. But then very um very often there's types of information that should be interpreted differently for different groups, and perhaps being kind of more uh race-aware or gender aware when we develop these algorithms might lead to kind of more equitable outcomes. Um, the second reason that it's potentially harmful is that we have a lot to gain through the use of alternative data. Um, as I mentioned before, consumer lending really doesn't have a very good history in terms of excluding certain populations from mainstream credit markets. And so actually, the potential for using alternative data to expand credit is a really important benefit of AI and machine learning. And so we really don't want to kind of let go of that benefit, um, given that the status quo is not one that we should be proud of or hold on to.
SPEAKER_02It sounds like you're also advocating for avoiding premature regulation of an industry that's still trying to figure out what it can do best. Charles?
SPEAKER_05Yeah, I agree with that. So one of the you you raised a very good point is that, and we're we find this in other areas of uh machine learning as well. You just whatever data you've got, you pour it in and see what comes out. So the good news is you may get really good results. The bad news might be as far as the explainability. When you come back and say, all right, that's really cool. Uh what did you use? Um have you had questions and situations where the people ask, how did you do that? and there was really no way to know.
SPEAKER_01Yes, I think the explainability is a key concern. And um it's both a concern for consumers who want to understand um credit outcomes. And as I mentioned, there's um regulation that requires that lenders provide um consumers, um, that requires that lenders provide information to consumers. Um, but there's also the explainability to the regulator, right? If the regulator is monitor, monitoring and overseeing fair lending, um then they also need to understand kind of what's got going on and whether outcomes are okay. And so I think explainability is a real challenge. One thing, though, I think that's important to remember is that there are a lot of questions that we care about that aren't necessarily about opening the black box of AI, but really the broader decision-making context and the actual outcomes of decisions. So we may learn a lot about kind of the outcomes of the use, for example, of cash flow data just by kind of observing the the outcomes of an algorithm that kind of prices based on cash flow um data. Um, and so if you're a regulator, maybe that's what you truly care about and not necessarily exactly the type of model that the algorithm used to produce that outcome. The second thing I'll mention is that there are a lot of other questions that I think we really would care about that are about how we set up the algorithm and not necessarily what goes on kind of inside the algorithm or the black box. So just to give you one example, um, we've focused so far in our conversation on the use of AI to predict someone's credit worthiness, whether they'll default or not. But very often in credit markets, one of the things that lenders want to know is how much, what's the maximum amount they can charge you for a loan, and you'll still accept the offer. That may be to do with kind of the competition you're facing, how financially literate you are, if you're the kind of person who will shop around for a loan. Now, suppose that instead of having a loan officer that I come to and I say, hey, I've got all these alternative offers and they adjust my price based on the fact that they've just learned I have an alternative office and I've shopped around. Let's assume an algorithm can do it. They can figure out based on my shopping history whether I'm someone who will just accept the first offer I get or will shop around. If I'm regulating fair lending, I might care, I might think differently of a pricing algorithm that tries to infer that kind of how much a person is willing to pay for a loan and whether that creates, let's say, racial disparities versus if what they were doing was just predicting credit worthiness. And so as a regulator, I would want to know what was your target variable, what were you trying to predict? And that doesn't necessarily have to do with the black box, it doesn't necessarily have to do with what's going on in the algorithm, but it's how the algorithm is set up. So I think we shouldn't overlook the many important questions that we can learn that don't directly go into this um issue of kind of the black box and explainability that the regulator nonetheless would very much care about.
SPEAKER_00So, yeah, based on my shopping history, I am one that shops around and that would be very clearly figured out and defined. But uh one thing I was picking up, and I'd be interested in your thoughts on this, is you as you were kind of talking about the inputs, the decision making, and the outcome, right? And maybe in some instances there'd been more of a hyper focus kind of on the inputs, but perhaps depending on the situation, the scenarios where you're using these types of algorithms, the balance between those three could be different, right? Um depending on what information or what predictions you're trying to make, the types of data that's available to you, sort of that correlation between input decision making and the outcome analysis could be different. And that's okay.
SPEAKER_01I think that's right. I think that um, you know, although I kind of um um kind of warn against um this overfocus on the inputs, I think it would be kind of naive to overlook the fact that we care very much about um what information is um is used to make different determinations. And even if there was um some relationship between a certain input and creditworthiness, it might not mean that it should be used. That could be true for privacy reasons. Um perhaps we're concerned about kind of information from other domains kind of then being used in new domains. Um, you know, that might be a concern if um we're perpetuating some um kind of pre-existing disadvantage in one domain in a new domain. Um, it could also be relevant just for the trust in in lending institutions. So I think there are reasons to think about those inputs. I guess my bigger concern would be to kind of too heavily rely on that input scrutiny to get us the kind of results that we want to, um, but also kind of this idea that there is some real and concrete gain to be made from expanding what kind of inputs we use for these decisions.
SPEAKER_04So I I'm intrigued by by something that that your entire thesis suggests to me, which is um one of the challenges that you highlighted earlier on is that we don't know what the outcome would be if we gave a different answer. And as a result, we can't really improve things over time. I could imagine that in addition to regulatory sophistication suggested by some of the things you said, one could construct what I'll call an insurance program which works basically like this. It's funded by Fannie Mae, Ginny May, something like that. Um that basically says, look, I'm going to I'm going to ask for a dial that lets me, the government, increase your risk tolerance. In exchange for that, I will buy some of your failures. I will make you whole for some percentage of your failures in exchange for being able to see as a regulator the result. So that will then let us as an industry, not me as a particular lender, will let us as an industry learn how to improve the outputs. Okay. Um rather than basically say, well, all I can work from is the inputs. Okay. Is that is that something that that one could put together? Does it make sense?
SPEAKER_01I love that idea of creating some mechanism to learn about outcomes. Um, and I would add to that idea that um to some extent we should incentivize through this program random lending that isn't informed by um by credit worthiness predictions at all. It's completely random because through that randomness, you know, we typically think of noise as a bad thing, but through that randomness, we might potentially learn a lot about these populations that in just using the algorithm again and again and again, that's providing the loans the same types of population again and again, we're just simply not learning. And so I do think that that is um a form of that idea, I think has to be part of how we think about this type of algorithmic lending going forward.
SPEAKER_02Talia, um, we haven't asked you what we sometimes ask guests, which is what is it that that brought you to be interested in AI? And yet you've written this article, and I congratulate you getting it published in Minnesota Law Review. That's really exciting. What are you trying to do, what audience are you hoping to reach, whether it's policymakers or others, with the article? And it would be helpful to know, um, as both a lawyer and an economist, what drew you to be interested in and to learn about AI?
SPEAKER_01Yeah, so I um, as you kind of mentioned in the beginning, my my training is um in economics, um, which we don't traditionally think of as a discipline that uses um kind of machine learning, which is very much a prediction tool, um, as opposed to a lot of the techniques we use in economics, which uh are about causal inference. But something that um was taking place during my years as um a doctoral student in the economics department, was a lot of um of inquiry and thinking about how we use um these advanced statistical techniques in some of the empirical research that we do as an economist, as economists. And key to that were was understanding in what ways machine learning is different than, let's say, a linear regression. Um, that That's used by economists. And that kind of, I'm going to call it soul searching because it often did feel like soul searching. That soul searching occurred to me at some point, that's very relevant to policy and how we regulate algorithms, understanding how this technology is different than things than what we've been kind of familiar with over the years. So if traditionally kind of credit scoring used more standard kind of linear regressions, what does it mean that all of a sudden credit worthiness is assessed using machine learning and big data? And that I think very much kind of motivated me to explore these ideas in my legal work about the regulation of AI. I think that I would say the primary two audiences for my paper, both kind of policymakers and regulators and other legal scholars. So kind of to go back to something that you mentioned before, Roland, about the risks of premature regulation, I think we're facing another risk. And that risk is that there's so much uncertainty from alternative lenders with respect to kind of what fair lending means and how one would kind of comply with fair lending, that I think to some extent it's preventing a lot of beneficial innovation in the space. And so while we we need to keep in mind that kind of um uh premature or problematic regulation isn't going to help things, we also, when there's a lack of guidance and some kind of key questions, it also kind of we get an industry that's that's a little defensive. And that happens with financial institutions, right? Financial institutions are heavily regulated. Um, sometimes they tend to be more kind of risk-averse. Um, and so um, we're currently in the state where there isn't kind of much guidance coming out, um, both in the US but also in Europe on issues related to kind of um algorithms of fairness and discrimination. Um, and so um I think my audience, um, you know, when I try to speak to regulators and and policymakers by saying, look, we need to have some kind of understanding of how this would be analyzed, but inputs is probably not the way, and we need to kind of think of alternative ways. But I'm also talking to legal scholars, and I think kind of our responsibility of legal scholars as kind of we think through these challenges, um, is really not necessarily to go with our first instinct of what kind of perhaps feels the cleanest or or kind of the most um related to kind of traditional analysis of fair lending, which I think is input analysis. I think um it would be pretty fair to say that um traditional fair lending analysis was very much kind of focused on the inputs into the decision. And so I think as legal scholars, we need to be getting a little more creative in how we think about um these kind of legal issues that arise in this context.
SPEAKER_02Well, I think when you talk about, you know, what difference does AI make, that's going to be something of interest to all of our audience. And I, in reading your article, was very impressed by how sophisticated and nuanced your approach was. You didn't say the immediately obvious is the end of the answer. You came back and said, well, that may seem to be good, but now look at it from another angle. So let me do that. Let me kind of lift up a rock and see what's underneath it. And it it it may make the financial institutions a little defensive to be asked this question. But from your perspective, social media companies say they do not collect protected characteristics. But recent revelations in the press suggest that they can target protected groups with shocking precision and accuracy. Is this the same issue, or is this the same issue that you identify in your article, or do you come at it a different way?
SPEAKER_01Yes, I definitely think that um this focus on do you collect do you know or not know a protected characteristic really um should not be the focus of any analysis of um considering whether protected characteristics are actually used in decision making. Um in fact, I also think that there are many contexts in which it's beneficial that um uh protected characteristics be collected for the purposes of an analyzing disparate impact. So if disparate impact, um, which is a doctrine, uh legal discrimination doctrine that deals with uh policies that on their face look neutral but may have a discriminatory effect, it's very hard to analyze unless you have information about people's um protected characteristics. So I think to some extent, not only is it insufficient to say we don't know protected characteristics and therefore our pricing, our decisions are not problematic, but in fact, in many ways, um we should be requiring the collection of protected characteristics to characteristics to put in the correct kind of safeguards and mechanisms that we need to examine things like fair lending. Now, it's in the US, mortgage lenders are required to collect information about race, because as I mentioned, that's something that's reported applicant information, including race. Um, but that's really a kind of an outlier in consumer lending. It's just kind of mortgage lending.
SPEAKER_02Well, you know, you remind me in what you've just said that discrimination is obviously not usually a good thing. It's usually not. And but bias is a term that we we need to keep neutral because in the AI context, uh, it may be important to have intended bias that has a positive outcome, to get certain imagery to focus on malignant tumors and not other kinds of growths. That's not a bad bias. And I think we it's important to keep bias as a neutral term when we're in the AI context. Mark, you look like you had a question.
SPEAKER_04So, so move on to the sort of last uh sort of wrapping up process. You know, when you think about if you can't manipulate or control the inputs, then the only other place is the algorithms and the outputs. So I'm that made me think about the output question. So let me let me take as an analogy the postal service. The postal service is not physically secure, it doesn't really try to be physically secure. Nonetheless, everyone trusts it with highly sensitive information all the time. Why do they do that? Because the penalties for violation are severe, they're known to be severe, and they're vigorously enforced. Is a model like this applicable to consumer lending?
SPEAKER_01I think what's key in the example that you give of the um the postal service is there's essentially a negative behavior that we can easily define that we want to deter. Um and um so the use of kind of high high sanctions is to kind of to determine to deter that um the definable um and potentially verifiable um um behavior. I think what's tricky when we talk about kind of fair lending is I think in most cases, the concern is not intentional discrimination in the sense of animus towards a certain group. I think in many ways, kind of what maybe traditionally was a concern, in which kind of you have a loan officer that would feel some animus towards a certain group and kind of price differently based on that. I think that to a large extent the use of kind of automated algorithms um may alleviate some of our concerns in those domains. So I think that um when we talk about what concerns us with fair lending, it's not to the same extent that kind of intentional kind of animosity towards a group. Um I think what's tricky, therefore, is kind of to define what exactly and under what circumstances is a pricing algorithm leading to an outcome that we don't want. And um we're still missing a lot of that definitional piece and a lot of that the guidelines with respect to how do we how do we define that problematic behavior before we even kind of impose very strong sanctions, we need some sense of of how we define that problematic behavior. And I would say that's kind of, I think, a lot of the ambiguity um that we we live in at the moment, um, that we we don't have those clear definitions of what we're after. Now, once we do, maybe the right next step is kind of severe sanctions for violators. But I think that we're stuck to some extent on that first step.
SPEAKER_02Can I ask also when when we were making determinations about intended or desirable outcomes, is AI at the stage yet where we can design it not to generate unintended or undesired outcomes? Because that problem has always been attached to designs of software. You can design software to generate desired outcomes, but it's often difficult to design it so it doesn't generate undesired or unintended outcomes, won't that same problem reoccur in AI? And don't we have to worry about that once we define the intended outcome?
SPEAKER_01So I think part of what's tricky is that what is desirable or not kind of um can vary on the situation and can be somewhat contradictory. So just to give you an example, um, if um let's say a 25-year-old man and woman were to kind of uh go to a mortgage originator, apply for the same loan, let's assume they went to the same college and they studied the same thing and they have the same job and they have the same income. So everything at age 25 looks the same other than their gender. And let's say we were setting up an algorithm to predict their income in 20, 30 years. Um, so mortgages, you know, they they have a long horizon. And so a big part of what you want to know is kind of what will this the income of this person be in kind of 20, 30 years. So let's say we set up an algorithm to predict um their future income. Now, and I and I apologize for the personification of an algorithm, but an algorithm might really want to know their gender, right? If we have labor market discrimination, if women are less likely to be promoted, if women are more likely to kind of um uh take a step back in their career after having kids, then actually a person's gender has a direct relationship to that prediction of their future income. So we say, hey, algorithm, predict future income. Now, from the perspective of the algorithm, a desirable outcome or desirable prediction uses gender in reaching an accurate prediction. But from a normative perspective and from a legal perspective, we can't price differently for a man and for a woman. And so this issue of how do we deal with these two opposing goals in which there's something that's empirically relevant to assessment of creditworthiness and a future income, but we don't want pricing to differ along that dimension. Um, it's a very complicated question of kind of how we set up an algorithm with these two potentially contradictory um goals. Um, and so one approach, and I think um uh one that makes things a lot more transparent is saying let's predict creditworthiness and let's impose these constraints if we see these differences between how women and men are treated by this pricing algorithm.
SPEAKER_04So to wrap up, AI is a rapidly evolving field. All of us, including both technologists and policy people, are working hard to keep up with the changes. What most concerns you about the future of AI and what most excites you about its future?
SPEAKER_01So I'd say my largest concern, and this is, I think, a concern I share with people who work in other domains, um, kind of outside consumer lending, is um that this kind of ability for um for better prediction or more accurate prediction um could cause serious, um, raise serious distributional concerns. The the moment we can better distinguish between different people, you know, that those people that benefited from the lumping together with other people, the less granular, granular predictions. And so we have, of course, the kind of classic discrimination concerns that we've discussed, but they're distributional concerns that just have to be um have to do with the fact that we may kind of increase gaps between those people at the top who are going to get very beneficial terms, gonna get loans, and those that now, because um, you know, maybe we learn more about them, but they kind of end up in the situation in which kind of they no longer have access to credit or kind of have access to worse credit. And it's very important to keep that in mind when we when we think about um kind of access to credit, because um credit terms or getting a loan is not only reflective of your current assets or your current wealth, but to some extent it's a producer of future wealth. And so we want to be very sensitive to the situation in which kind of we're potentially perpetuating a situation of disadvantage that pre-exists, even when it's just along um social economic lines and not kind of uh protected characteristics. So that's I guess what I what I'd be somewhat concerned about. But there's also a lot that I'm optimistic about the move to algorithmic pricing. Um, some of it has to do with what we've discussed, this idea that potentially um through um considering a broader range of characteristics, a broader range of kind of aspects of people's financial lives, uh, we're potentially allowing or including people um within um certain markets that they've traditionally be excluded. But I'm also um I think there's reasons to kind of be optimistic uh for other reasons as well. I think that um a lot of traditional lending has happened in a very opaque way. So we've talked about an algorithm being a black box, but um, one can't imagine a bigger black box than the human brain. And in a world in which kind of loan officers were making decisions about um credit and decisions about loans, we we really didn't know much at all. We perhaps they provide an explanation, but you'd have to even trust the explanation they provide. And so I think that there are real gains in transparency through a world that operates with automation, in which we kind of can learn a lot more about what's going into these decisions and how these decisions are made. I also think that um related to this transparency, there's a lot of potential for prospective testing of different um algorithms or um the use of different inputs. So, in a traditional kind of lending situation, we might not have a sense of whether kind of a practice was leading to some kind of um unfair outcome or discriminatory outcome until a decision-making policy was actually implemented and people actually received loans or were denied loans. In an algorithmic setting, we can perhaps learn a lot about the effects, uh, the distributional discriminatory effects ahead of time through prospective lending. And so there's a lot of opportunities to learn more at an earlier stage. And I think for regulators to um to know and learn and regulate in a much more kind of effective and informed way. And so I think that's a real promise.
SPEAKER_02Well, let me say I seldom hear somebody who has thought as long, hard, and clearly as you obviously have done, given the answers to the questions, only some of which you saw in advance, and many of which we sprang on you. Um, it also makes me wish that you were teaching at Columbia online, even though it's in the same city as I am, because I would like to sit in. And I think I would find that really worthwhile. You've been very generous with your time. Uh, I still don't know how you do all that you're doing, both the doctorate teaching and raising, you know, with your husband, three children, two of which are twins. Um, to me, it it's a balancing act that I I find really admirable admirable. But I would also just like to thank you for taking the time and being willing to accept an invitation to talk to all four of us. Because I, for one, and I think I'm not I'm not going to put words in anybody else's mouth. I thoroughly enjoyed this conversation.
SPEAKER_01Thank you so much. Um, I really appreciate uh these questions and comments, and it's given me a lot to think about. Thank you.
SPEAKER_02The views expressed in these podcasts are solely those of the speakers and not of their employers or organizations. We thank the American Bar Association for their generous sponsorship and support of the production of this podcast. Our theme music was composed and performed by the very talented Ben Rosenblum. We welcome questions and comments from listeners. Send email to comments at MindtheGap Dialogues.com. We read all comments and questions and will try to respond in the letters section of a future episode. If you are writing about a particular episode, please do mention a specific episode number. Please also do include pronunciation tips to help us properly say your name when we reply in a subsequent episode. See you next time on Mind the Gap Dialogues on AI.
SPEAKER_03Thank you for listening to the AVA Business Law Section Podcast Series to the extent that the section offers a robust collection of content. To explore more about this topic or to learn about joining the section, visit ambar.org slash bizlaw. That's B I Z L A W.