Himan Abdollahpouri is currently a postdoc at Northwestern University. Most recently, he received his PhD in the area of machine learning and recommender systems from the University of Colorado, Boulder. His particular area of focus is the multiple stakeholder recommendation paradigms.
Himan Abdollahpouri is currently a postdoc at Northwestern University. Most recently, he received his PhD in the area of machine learning and recommender systems from the University of Colorado, Boulder. His particular area of focus is the multiple stakeholder recommendation paradigms.
For us. I know you're a huge sin following where you watch a lot of movies as do I, I'm curious to know what your favorite streaming services, the pirate Bay. That was that what's that it's this great site where you can get all the content ever made for free TV movies, sports. The only drawback is that it's a little illegal. Cause, you know, I've been thinking about that. Is that illegal anymore? I mean, I feel like there's a hundred of these sides. What's the likelihood you'd go to jail. Yeah, I haven't heard. I remember when I was growing up, I used to hear these new stories or is it like this 14 year old kid who was sentenced to like$500 million? And the FBI was like kinda threatened to put him in, but I think that was just propaganda created by them. The recording industry. I doubt that actually happened. And I haven't heard anything like that in the last like five, 10 years. So I think they lost the war against the pirates. The boogeyman stories your parents told you we were growing was in relation to the FTC and like fair use of platforms. No, I th I think, I think you make a great point. I mean, why pay for Netflix, HBO and support all these artists and support Hollywood and the creatives out there when we can get it for free? I mean, you it's, it's, it's a foul, it's a valid point. Exactly. No, seriously though. What's your favorite streaming service? I don't really have a strong feeling one way or another. I usually chase the content. So if I hear that there's some interesting show being produced, then I'll watch it. Like I heard about one division and then I checked out the free episode they had on YouTube and I thought, wow, this is very different. So then I watched that. I don't have any sort of. Ties to any particular streaming service. I don't think that any of them are that much better than the other. So you don't really care about discovery on a particular platform. You do the discovery on your own and then you come to the platform second. Okay. Well, I have done some discovery on Netflix, so they do recommend a lot of anime, some of which I already watched. But then I watched it like 10 years ago and I watched it again and I'm like, wow, actually this is really solid. And sometimes it's completely new stuff. So yeah, I would say that almost entirely. The discovery that I do is anime on Netflix. I mean, I value the recommendation systems on Netflix a lot. I think I've found some interesting things, especially foreign movies and TV shows. But that being said, I think this is a great segue into introducing our guests today. Humanae Abdulla Puri is currently. Post-doc at Northwestern university. He recently received his PhD in the area of machine learning and recommender systems from the university of Colorado, Boulder. He focuses on this area called the multiple stakeholder recommend patient systems or paradigm, which I think is super interesting. And we'll definitely get into that during the episode. We'll get his thoughts on Netflix as well, I guess. Let's get into it. Humanae, Thanks for joining us. Thanks for having me great place to start. This would be a, sort of a quick story. How I came across you a few weeks ago, I was on clubhouse. The title of the room was pretty unique. It said, can AI help us find love? And at the point of the conversation, when I joined, they were talking about dating apps and I think that's when you also started talking, it was really, those are just a really interesting take. If you could. Give us your take right now on what you think about dating apps and sort of the algorithms that run them. And sort of what you think about that business. Yeah. So I think if I remember correctly, the whole like discussion that I made was like usually in a Western culture, they kind of criticize some Eastern values of finding love and they say, you know, some sort of arranged marriage. I mean, we, like, we might have some kind of weird arranged marriage, but there are good ones as well that just like maybe two parents they find love for there. Sons and daughters and they meet, and then they might like each other and they get married. So I was really kind of interested why people like, in like modern countries, they think it's okay for an algorithm to find love for them, but it's not okay for two human beings to find love for them. So like the whole discussion started from there, which to me, honestly, It kind of sounds weird that we trust an algorithm that, I mean, I've, I've built algorithms myself and they are nowhere perfect to like how a human can understand my, you know, my interests and what I really like about finding a love partner. But we think it's okay that we trust algorithm, we install dating apps and we use them to find love. And I mean, quite frankly, it never, I mean, it, it kind of rarely works. I mean, but yeah, so that was more like political discussion, not really about AI, more about the cultural political things, but it was related to AI as well. Sure. But you believe that the underlying sort of algorithms that operate behind Bumble and Tinder, they're just not up to par in comparison to human discretion. Absolutely not. Yeah. I mean, because like humans, we are much more complex than just like some likes and dislikes that we might express on a dating app or swipe left swipe, right. Or something like that. So like someone that knows us personally, let's say. A close friend or parents, you know, they have definitely much more idea of what we like and what kind of things we enjoy in a partner. So I think the success rate, if like, if let's say a person finds someone for us, it's much higher than if an algorithm finds for us. I mean, that's my personal opinion and it's based on how I understand algorithms work. And in general, how applications work. And this is also well, the other question here is when your friends are trying to set you up with someone, their goal is for both of you to have a good match, both of you to be happy, but is that even the goal for a lot of these systems, you know, or are they looking at something else? Like engage? That's a good question. So I mean, if we. Think about it. I mean, these are businesses, right? So they are like what they say, billion dollar businesses overall across the world. So that's, I mean, they definitely care about money for sure. And the, I mean the whole thing that are they okay. If you find someone and just leave the platform. Or are they more happy if they keep you around? So that's a very important question to kind of think about, I mean, because like if they really want me to find someone that they would be happy if I just find someone to leave the platform or they might be more happy if I don't and I just keep. Installing do not install and then install again and then on install and then just be on these apps for years and years. What is the recommender system and is that what powers dating apps? Yeah. So if I want to explain like in a simple way, so a recommender system basically is a system that tries to find you interesting and relevant. Things for you that it might be difficult for you to find it on your own. For any reason, either there are many possibilities that you can not really explore all of them and find which one is right for you, or you just don't have enough knowledge about what's available. So basically they just try to kind of diminish the work that you want to explore things, and they just help you to find things that you might be interested. So rather than you, we're looking at lots of potential options, they make use of the big data of what previous users have selected to just say, Hey, these are the two or three or a few things that you think that we think you might. So basically it's not kind of only based on what previous people have used. I mean, they need your. Historical information as well. I mean, if you start, if you just start the app, yes. Basically you might like the app might ask you for like certain information. Like for example, you know, I'm interested in this age range and these like, kind of, you know, so that will be the start, but then the more you use these applications, I mean, let's say. On dating app, they use this swipe left, right. Kind of information. They, they, they learn, okay, this person is interested in this kind of thing or on music, you know, like scape, thumbs down, thumbs up. So like over time, the system like tries to learn your preferences by how you are interacting with these recommendations. Well, yeah, it sounds like you're a bit skeptical of how effective dating apps recommendation systems are. Why is that? Well I'm skeptical. Not because the algorithms are not really strong enough to find you someone. I, it's also kind of related to the psychology of human being that usually when we think. Almost everybody is a potential option. That's actually not good way of thinking about how you find law, right? So historically your, you find someone to date or to marry because you were in a approximate interaction with them either they were your neighbors or they were living in your workplace, but, but these dating apps. Makes you think there is always a better person out there and you have better chance of finding a better person. Right? So that kind of a circle kind of a cycle is going on because you, you, you feel never happy with the person that you are kind of. Kind of like the app thinks you're connected to that person. Right. So that kind of illusion that you think you are such a special person and there is someone better for you, you know, that's actually, to me, it's very dangerous. And if you look at the statistics in like, I've seen some statistics online. I'm not sure how reliable they are, but generally based on my observations of like people around me, my friends and everyone, usually they are not really successful. But generally speaking, I mean, I think the real. Traditional way of finding people was much better. I mean, I mean, it's just, it's a weird thing that we want to use algorithm for everything. I mean, to me, well, let's say Spotify or Pandora or Netflix, these are perfect examples of how algorithms is necessary. Right, because they are really helping you to find things that you would enjoy and you just, you know, it's, it's just a nice experience, you know? Yeah. And we'll, we'll definitely get back to Spotify and Netflix. But in terms of multiple stakeholder recommender systems, which is what you did your dissertation on, how does that apply to dating apps? So there's multiple parties involved and there everybody's getting recommended something. So if you could speak to that a little bit. Yeah. So the multi-stakeholder approach or concept in recommender systems kind of we care about different stakeholders when we have recommendations. So let's say historically the, the most important factor for evaluating whether our recommendation is successful. Was okay. Is that good for the user or it's not good for the user? So the user probably was the main stakeholder here. But I'm, and by the user, I mean, the person who receives the recommendation. Right. But there are many examples that that's not that simplistic way is not really the case. I need some more complex. So there are other stakeholders that they benefit from either being recommended or getting the recommendations. So get back to the dating apps. So let's say we have a two-sided. Kind of a market for like, for simplicity, let's say we have only women and men. So on one side, men are looking for women and on the other side, women are the same looking for men. So the algorithm did recommend me, someone that I might like. Right. That's not the only factor. Okay. I might like supermodels from Victoria secret, for example, but are they also interested in me? You know, so like the whole goal is to find nine, six S full matches that will end up in possibly a date, or even more than like more than decent, just like becoming a serious relationship. So that's what they call reciprocal recommendation. Right. So that's recommendation should be accepted from the person who is receiving it. And also from the person who is being recommended. Right. So it's like a two-sided match. And actually, it sounds like the more recommender system that we see in dating apps, we should stay away from more manual the process is the better. Yeah. I mean, because, you know, I mean, Sometimes we say, okay, more options is good, but it's a very well known concept that the politics of choice. Right. So, I mean, it, it could apply to other domains as well, but for when we are talking about humans, so like, I mean, dating apps is about human you're finding humans, right? So a human is much more interesting and complex than a profile. And that's what actually, even in, in West, like in America and other Western countries now like there are businesses that actually there is someone it's like an agency and they will, they will meet you. They will meet you the other person and they, they try to connect you. So that's probably much more, it's going to be more successful. You know, it's very interesting that as somebody who actually understands the ones and zeros behind how these algorithms work, you are skeptical of using them for this field. And I think we've found that the more people know about technology, maybe the more skeptical they become. For example, there's a lot of early employees and VPs at Facebook who do not let their kids use Facebook, or even let them use social media. I think that's very interesting. Yeah, exactly. I mean like Facebook, I think I remember like the first year that I was using it, I mean, you know, like usually these apps, they start really good. So they start by connecting people, like basically friends connecting to each other. No ads, no news pages, no busy-ness pages, basically real people connecting to real people. That was fine, honestly, in the beginning, but then they never stopped them. They are so greedy. They just add other things in. They just like add, you know, like for example, news websites now, now did not like news agencies. Now they want to join Facebook and have a page. Right? So like the whole thing become political and then they become like so confusing with people that they just wanted to connect with their friends as. Facebook claims we are connecting people around the world, which basically has worked in the opposite. They have disconnected people around the world. Honestly. Can you expand on that? Yeah, so, I mean, I don't know how old you are, but when I was a kid I'm like 35 years old now, you know, people didn't, they were much more connected before all these technologies, like, or these social media, Instagram, Facebook, we used to like, hang out with my cousins, you know, with my family all the time I do on weekends or like, you know, at night and play games and without even getting distracted. For, for our phones and cell phones and smartphones, you know, but these days we think we are connected. We think we are connecting to all the, all around the world, but honestly, are we really connected or are we really feeling happy that we feel connected to everyone? Or are we really feeling, feeling isolated? And we think we are not really connected to anyone. I personally feel that it's the opposite. I mean, we've, I personally feel much disconnected to the people now. Then before that there was no social media, right? I would definitely say as well that I feel worse about my life on, in periods where I'm using my phone heavily. And when I'm constantly on social media than when I'm not. And I feel like that's not just, I feel like that there's a causal correlation there. Yeah, let me maybe present a devil's advocate sort of position for Facebook, which is Facebook has become sort of necessary for day-to-day life. For many people. This is where they get their news. This is where news beyond just a sort of like local and what's happening around them with their friends, but at a very like international level, people are maybe more educated because Facebook exists and and I think that's partially due to their recommender systems and the way their business operates. I mean, that's a good point, but we can also have, I have a current, like kind of a, kind of a counter argument for that. And actually last year at the, we have a very famous conference on recommender systems, ACM conference and recommender system or rec seas. And there was a keynote keynote speaker. I think he was from university of Indiana and he's really. He has done a lot of research on misinformation and everything. So I remember I asked a question on, so we use this app because it was online, the, the conference. So I asked the question, how do you, like, what do you recommend for people that use social media? From kind of protecting themselves from misinformation. And his answer was really interesting. He said, well, I recommend not to use social media for your information, use it for what it was designed to be, which was posting the paint, the picture of your pet or you, or anything it's not designed for. I mean, it wasn't really the social media wasn't initially designed for information, right. It was for like connecting people, which as I said, It didn't really work in that way either. So if you really want to be informed, read news, read like newspapers. And I personally recommend reading local news actually, because they are much more, they have less incentive to give you misinformation, then large kind of corporations. So to follow up on that a little bit is Facebook powered by recommender systems? Absolutely. I mean like Facebook has recommender systems in many different components of Facebook. So, you know, like Facebook is just, it's not only, let's say one. As I, as I could say, wall that you see on your, you know, it's like a wall that you see, so it has different components. And in many of those components, recommender system exists. So in the world definitely exists. And actually people have complained that they are not seeing posts from their friends. But they are seeing some promoted or some news posts. Right? So that's because of the algorithm. So the algorithm things, it's, those posts have more priority for you. Two for you or more priority for Facebook, maybe because, and they put them on your wall versus they are ignoring the organic posts that your friends have posted on, on their timeline. And they might be actually really looking for some interactions from their friends. And me maybe, maybe what, maybe your friend is depressed and they just posted something and they really need to get some interaction. Maybe some likes some, maybe some comments. But they are being ignored, other posts that probably might help Facebook make more money or like some promoted content, they will show up on your time. So that's algorithm that decides that. So in your professional opinion, how do you think Facebook's algorithm is performing and not just in terms of the user impact or rather in terms of whatever they're optimizing for? Yeah. Swell. I mean, it's been four years that I'm not using Facebook. I do use Instagram, which I might delete that as well, because the reason that I used Instagram and I liked it more than Facebook was. The wall of Instagram like that kind of the home page or that wall that you basically see your friends pictures that used to be only your friend's pictures. So, and if you wanted more, you, you, you used to go to the explore tab and you see so many other things. But so I personally, I was using the. My personal wall, because it was mainly the people that I followed. And I mean, not mainly actually only the people that I followed and I liked it because, you know, I follow people because I enjoy to see their posts. But like recently they incorporated promoted posts and in many other things in that tab as well. Right. So can they ruin the whole thing for some users including me? So, yeah, I mean, I personally think I dunno. It's, it's, it's interesting. I mean, they have no reason to be so greedy. They are all of the, one of the, the richest companies in the world. Why they don't just stay at least where some, many people, they were okay with it and they could still make money, but I don't know why they still want to, you know, it's just weird to me. I just wanted to add some context. I think you're right. I think Facebook is greedy. I mean, as are a lot of the companies in the United States I think they're really motivated by or influenced by what's happening in China. And we're typically like five or 10 years behind the curve in terms of social, in comparison to China in China, they have these super apps quote unquote, and what these are an example of a super app is a Weebo. And on this one app, you can get food delivery. You can look at social media of your friends, you can order things. Similar to the way you can order things on Amazon prime. You can listen to music. So it's sort of five or six apps that are sort of individually United States all in one. And I think that's where Facebook's headed. Facebook's incorporated shopping. So now you can shop for apparel and all sorts of stuff like that. You have a Tik TOK component where you can just reel through videos and then you can also, you also exactly what you're saying. A lot of like promotional posts for other businesses. I still think we're really early stages. I think that's where Facebook has headed. They greedy is one way of sort of characterizing it, but it's almost like if you don't do this, somebody else is going to do it. Maybe Snapchat does it. And so the competition is always. Sort of at the back. I mean, I think these are a bit different, so you could be ambitious, but not greedy. So the idea of super app, I like this idea, but I don't think, I mean, a super app, you could still have a super app that does so many things, but it doesn't really like like it's not like based on ads or, you know, like so many, many population and not a good user user experience. Right. So you could have an app, as you said, it, music, you could, you could pay, you could buy, you could listen to, you could watch movies and everything, but it's still a great app. And I would be okay to pay for this app. Right. But, but the whole, I mean, the whole problem that I'm talking about is when basically like in a system that you are not paying for it. They want to like, so basically they want your eyes, right? Even outside. It seems everybody is just trying to somehow engage you to have your eyes and look at them. Because, because they want to show you ads, so that, to me, that's the problem that honestly, it's kind of, it's kind of really bad. And I think it's in the future, like a busy-ness that would create a great social media, like a useful social media. And people probably would even okay to pay. I mean, people they would pay to, I mean, Spotify, Netflix, people, they pay for this kind of thing, because it's a great service. So do you, think Twitter significantly better than Instagram? I mean, I'm okay with Twitter's recommendation more than Instagram, per se. At some point, I remember, I think I was part of an AB test for Instagram. And like for a couple of days, my recommendation were so great. I think they were testing some other algorithms. And I probably, I was one of those people that they were testing it on me. So like, everything was so clean. I would, I was only seeing that I was interested, not my friends' interests and I wish that okay now, okay. They solve the problem. But then I went back to the, you know, where it was. So how do you know if you test just because the quality suddenly spiked. I mean, it was my assumption because everything changed. So like dramatically, like, so now that I use Instagram, let's say, or like before that, when I was using Instagram, it was so clear, like so many posts that it was like maybe Persians and like things that my friends in Iran they were interacting with, you know, not really the one that I enjoyed, but, but in like maybe a week, I remember last year, a week, All my recommendation were things that I'm really interested now. Like, I mean, it's just like, you know, like nature science, this kind of thing. And I, it was so amazing and I was so happy that okay. They, they, they finally changed it, but now it's back to how it was. So probably they didn't like it. So like a person you could say, Hey, let's keep on getting this type of content, but a computer algorithm, there's not so much, there's not so much clarity. It's not so much transparency. I mean, let's say, let's assume I was part of these AB tests. For example, let's say, let's say I was so they're like the way that they decide whether they should keep an algorithm or they shouldn't, they don't care about me as a person. I mean, on average, how much engagement they got with algorithm a versus how much they got with algorithm B. So let's say for some reason algorithm, Hey, like the one that they had previously still outperformed, let's say so they kept it for example. I mean, I'm not sure exactly if that happened, but this is how they decide. It's not just this person. Oh, did he did an enjoy, let's have algorithm a for this person on or algorithm B for this specific person. So shifting gears a little bit, I watched a documentary, I think a year or two ago called coded bias. So this documentary was about this woman who was a PhD student at MIT at the time. And what she discovered was that a lot of algorithms that we didn't have too much transparency over, but we're controlling lots of people's lives in terms of in terms of how healthcare operates in terms of. Sentencing, the judges used had racial bias and other types of bias baked into them. And so she advocated for this creation of an FDA for algorithms. So how do you feel about that? Having a regulatory body in the U S which studies the algorithms to make sure that they're both fair and that they don't optimize for the wrong things. Yeah, that's a good question. And I personally, yes, I agree that, you know, algorithms, they are biased towards certain things and, but it's actually much more complex than that. So let's, let's just for one example, let's talk about the, the face recognition that may at the airports, or like some other places they might use for having some extra check, you know, some like more security check. So in machine learning, you have. False positive and false negative. So false positive is you, your classifier or your algorithm thinks this is the person of interest, but it's not. And false negative is the algorithm thing. Okay. This person is clean and it's not the person of interest. But it is right. So each of these has some kind of penalty, some one might be more important than the other, like, so let's say a medical one might be more important in an airport. One might be more important. So the idea is it's very difficult to have a zero false positives and zero false negatives. You have always error. In these machine learning algorithms. And a lot of the complaints often come from like one of these. And basically it more comes from the false positive. Like someone is not a person of interest, but the algorithm thinks things it is, but we never see or hear from the false negative that, you know, someone was. A person of interest and the algorithm said that person was not right. So, you know, yeah. I mean, again, it's, it's very complex, but like usually media and like some people that they are not really expert in machine learning, they claim some things that scientifically it's much more complex to explain. Yeah. I've heard some buzzwords around this of like explainability and things like that. Is that just a buzzword or do you, do you buy it that those, that work is being done and it w and it's successful and explaining how these algorithms work? I'm not sure if it's successful because it's very, actually difficult to really explain. Especially some algorithms are much more difficult to explain what happened inside of the algorithm than the others. Let's, let's say for example, decision tree, let's say so when we say decision tree, we mean we make decision as. Let's say, you know how we played the famous 20 question, you know we have, is it 20 or 21 question that we have something in mind and the person asked us, is it a human or it's not a human and you say yes or no. And then he said, okay, is it a famous person or it's not? So that's the idea behind a decision tree that you ask questions each round to come up to the final. Answer. So those are much easier to explain because you know exactly how you made the decisions, right. But let's say deep learning or, you know, so those are basically optimal, like some mathematical optimization and weight adjustment of some like neurons and edges and things like that. And that is that those are much more difficult to explain how. And algorithm thought this image is an image of a dark, or this image is an image of a cat, for example, because the decision process was not as straight forward as a decision tree algorithm, for example. Right? So that's actually a very interesting debate within the machine learning community as well. Like deep learning. They perform really well. For certain applications, let's say for a tumor detection or like anything that are about images or signal processing. And for many of those, you probably don't need explanation as long as you perform really well. I think it's fine that you don't know how you got to the point. So I'm detecting a tumor with 99% accuracy. It doesn't matter how I came up with this decision. I mean, it's, I mean, I just say it's 99% accurate. But if I say I came to the decision that this person should be in jail for 10 years and I'm 99% accurate. I think this is, this is more difficult here. Right? So explainability people have worked on it. Even in recommender system. There is actually a broad research area, explainability to explain to people why these things is recommended to you. So we've spent a lot of time talking about where these decision-making and recommendation algorithms fail. Maybe we could talk about when they're successful. Absolutely. For sure they are successful in many domains. I mean, I personally like for example, music is my favorite. Example, because, you know, we have like maybe hundreds of millions of songs available. Right. And I personally really cannot listen to all of them and find out which one I like, but I use some music apps. Maybe for a week or two. I, you know, I enjoy and I thumbs up thumbs down if they, they, they learn my preferences. And then all of a sudden I get something, one recommendation that I get surprised. This is amazing. I really liked the song, you know, so to me, this is a very safe and interesting way of using recommendations. And honestly, the same thing could happen for. E-commerce right. You know, it's, it's kind of interesting, like, let's say I'm searching for something and let's say I search for Tablecloth. And then I go to the page and then under the page, I see some other interest, just a recommendation that could go well with my purchase. I wouldn't thought I need those, but they might be interesting for me too, actually, by with these tablecloths that I'm buying, right. Sometimes they might be too persuasive that I wouldn't really want to spend that much money, but sometimes I'm fine because they show me things that I liked. And I would buy, so did this, this is one other example of what I like about recommendations. What else I could say? Yeah, we can, we can dive a little deeper into Apple music and Spotify. I know you've worked at Spotify mean for us. We've had a lot of conversations about this. I'm sort of, of the opinion I've used both. With Apple music. They take a very human centric approach to how they recommend. Music and how they put their playlists together. Spotify is entirely different. Their discover weekly is as far as I know, completely out algorithmically generated. I'm a huge fan of Spotify. Do you think in systems where recommender systems are really great in the case of music, like you just described, is it better to have some sort of human input, some component where humans are involved? That's a great question. And the answer I think is yes. And I think as far as I know, even at Spotify, there are some human curated playlists that I mean, I'm not a hundred percent sure, but when I was there, I think that's the case. So I think, you know, like, like music is a very. Interesting domain, because you know, like if you have expertise in the music, let's say maybe like, let's say if you are a musician or you actually know scientifically how in a different type of music and things like that, that's probably really helps to put together some recommendation for people as well. So I think, I think yes, downstairs. Yes. I think I'll go through that. I mean, to some extent they could find you interesting music, but I think the combination would really work well in, in some examples. And as they know that they actually do that. So I think Pandora also does that. They have curated playlist as well. I think. Sure. I think actually the more interesting question here is, is Spotify as algorithm optimized for the musician or for the listener. That's a very good question again, you are asking good questions. So historically I mean, it was for users. Or for the listeners, but Spotify over the recent years has kind of changed its mission to be a, two-sided a truly two-sided market that they really want to help also the artists to kind of be exposed to different audience and make a living, as they say in their mission, make a living off their art. Right. And at the same time people who listen to Spotify or, you know, they also want them to have a good experience and, you know, like listen to the songs that they really enjoy or podcasts that they really enjoy. So they definitely do care about both sides artists and the listeners. Do you know, if musicians are changing, how they record music so that the song itself will align with the recommendation system put out by Spotify. That's also a good question. Again, I'm not sure about that I know that let's say Netflix did something like that. So they. They analyzed so many years of users watching history and they created a TV show that the algorithm there, their analysis realized, okay, this type of content would have a lot of engagement and they created the TV show based on that information. I think house of cards was I think if I'm not wrong. Yeah. That was the first Netflix original that wasn't a comedy special. Yeah. I think, I think you're right. Yeah. Not surprisingly. When I hear that they made it, they made a TV show based off of what the algorithms said. I would expect it to be awful, awful TV. The fact that it was prestige TV, that won awards is very surprising. Yeah. I mean they, I mean, that's also, I mean like many things that I say here, I mean, I've never worked for Netflix, so these are the things that I've read online on some news. So how credible they are, it's how credible those news were. What I read was they, they analyzed, you know, what people have liked before. Like what genre, what kind of thing, what kind of content was the most appealing to users? And they try to. Make a TV show that captures these type of different aspects that would have more engagement. Yeah. Yeah. Who might actually I had a really particular question. This is something I personally care about. On Netflix. I don't know if you remember this, like a few years ago, they used to have reviews for each movie. You could go and see what people were writing. There there'd be like five stars and three stars. This was the case, like a few years back, maybe four or five years ago. And then Netflix got rid of that. Now they have the likelihood indicator. It will say like, Oh, there's a 75% chance. You'll like this. In your opinion, when people see this like, Oh, this has an 80% chance that I'll like this, are they more likely to watch something? Does that even affect the psychology of the user? I honestly, I'm so glad you mentioned that because I can, I even forgot about that. To me, this is, I mean, I mean, I'm pretty sure they have good reasons for it because they are Netflix, but for me, this is so weird. That you give me a probability that I would like how much I would like this. So let's assume you would give me something that I like. I would probably like this for like 40%. They didn't. Why do you even show it to me? If it's 40%, for example, right? Or you could even manipulate these numbers and say, this is something that you would like 95% all the time. And then I would probably click on it. Right. So to me, honestly, to me, the just give me the average rating of what other people have, have, have rated it. It's much more useful to me to decide whether I want to watch this movie, then your on trend non-transparent algorithm that I don't know how you came with this probability number. And tell me how you think I would like this movie. Right? So, yeah, I think I liked the. Rating more that it was, you know, people like on average people like this movie, four out of five star. Okay, good. Then maybe I would also like it four out of five or maybe not. So it's more clear to me than, right. Yeah. Yeah. I think so Amazon prime right now still shows the reviews, but they've hidden it before used to be like one click. You could get to it and see the reviews, but now you have to go to two or three clicks. I've noticed. So I'm curious to see if you know this sort of our earlier stage, a streaming company. If like five years down the line, they come to their same conclusion that Netflix came too. And they remove all the reviews and they implement the likelihood indicator. I think that will be a real sort of a confirmation that this is, this is better for the company. Actually I have colleagues that they work on this and there is actually a strong connection. Like those, like in those, a website that they use reviews. And people read the reviews. There is a more likelihood that th like a purchase could happen or like some kind of other things would happen. So I'm kind of surprised. I mean, maybe they removed it, not for this reason. I mean, you know, it's Netflix, they might, they have not a lot of analysis why this is necessary or why it's not necessary. Maybe it wasn't because maybe it was just to simplify the web page. I dunno. It was maybe just to make things more clean. Yeah. You know I remember reading like 10 years ago that Netflix tried to crowdsource the recommendation system. So they put out a million dollar challenge and they gave away a ton of data. I think this was in like 2011 and they said that anyone who can get above a 10% successful recommendation rating wins a million bucks. Yeah, I mean, that was 2006, which a Netflix prize, it was a famous challenge that actually that kind of challenge initiated so much interest in the recommender system community. And a lot of people, they mentioned that as a good, starting like a milestone for the research that people got attracted to this domain. So DSD, they had this challenge. They said, if anyone can improve. Our recommendation algorithm by 10%, which back then they were using a metric called root mean squared error or RMSE, which basically is if I put it simply, it means if you rate something four out of five, And my algorithm thinks it's three out of five. We have one error, like it's four minus three is one. And then you kind of average all errors for different users. Squared root, you get a number. That's how much error you are making. So they said, if you can reduce this by 10%, we give you$1 million and it took three years for people to finally beat. That algorithm by 10% and like several teams that they couldn't win individually. Over three years, they got, they got together, they built an ensemble or like a combined method and they won$1 million. So that's, that's one of the great moments in recommender systems history. Yes, exactly. Yes. I think, you know, one other sort of business that I wanted to discuss was Uber eats or door dash. I think if you think about really a multi-stakeholder issue Uber eats and DoorDash, they have to service multiple holders in the form of a driver the restaurant tour the person that is buying. The food and then their own business right there. They got to optimize for their own business. What do you think about that? Yeah, that's the perfect example of a multi-stakeholder actually environment. So, you know, like an Uber eats, as you mentioned, I mean, the goal of the app is initially let's say to make good recommendations for people to buy food, you know? So that's definitely like people who use the app to find restaurants or like order food are definitely stakeholders, but. That app wouldn't exist if there was no restaurant. Right? So the restaurant on the other side, they want to make sure that they are being shown to enough customers and they get enough orders. Right. So for them it's very important to be fairly recommended to different people. Right. So even though let's say if the app thinks many users like you like restaurant a and B and C, you know, let's say your data shows that. But it's not a good idea to recommend these three to everybody. Because even if you do that, these three restaurant, they don't have the capacity to handle all these, all these orders. So that's where that also comes into play. So how can you distribute this orders that you show fairly? Different restaurants to people and such that people get what they are interested in and they get good food recommendation and restaurant also are being exposed to different people. And it doesn't end there. As you said, the people who take the foods from the restaurants to the people, there are also stakeholders. So each person. Can only handle a certain amount of delivery, right? So either in some locations we have, so let's say 10 delivery people in, in one location, we might have 50 or in one location we might have not like even one delivery. So the bit that algorithm should really take into account all this constraint that okay. We should make sure that different delivery partners. We give them a fair load of work because you know, they are also making a living of their job. Right. So you can't overload someone, even though they cannot handle. All of these work and not giving enough work to another person that they are not doing anything and they want some order to kind of, you know, make some money. So that's a perfect example, you know, I mean, your business cannot sustain without making sure that you are taking into account the preferences and the needs of all these different three stakeholders. Does the quality of the recommendation go down? The more stakeholders you have, because I'd imagine if you're trying to make all four groups happy versus me just going on Yelp and figuring out what restaurant I want to go to. The Yelp recommendation would probably be better for just me. Right? I mean, again, it's, it's, it's more complex than that. So as I said, let's imagine your, your main interest is rest is a restaurant, a and B or C. And let's assume these are the best restaurant in the world. They are just top-notch. Everybody likes them. Right. So everybody orders and they cannot handle all the, all these orders and you don't get your order. So are you happy now that you didn't get your order? Fair point? Yeah. It's more than just the quality of the food. There are other factors as well. It's a good point. Yeah. Yeah. Yeah. And I think this is a great use case where you have multiple competitors and the differentiating factor between them is their recommender system or a huge differentiating factor. I'm sure there's a lot of differentiating factors, but the value add is really developed on the recommender system platform. One other thing that I wanted to ask you is you are doing a postdoc in this area. What is some of the cutting edge research that's being done on recommender systems? There are so many different research areas in recommender systems and people, they work on the algorithmic part. The, as I said, the explanation, the user interface, design the fairness part, different biases. So it's actually gaining a lot of attention more and more because. We are seeing recommendation systems in many different domains and many different applications. So, yeah, but if I say, if I want to say I personally, what I work on, so I work on the kind of the use of recommendation systems and personalization in news. And how can we make sure that. News is actually informative and we are recommending things to people that fulfill their needs for information that they should know, or they want to know kind of giving them a package, a nice well to get put together package of information that are useful for someone who read the news. Yeah, Humanae will. Thank you so much for joining us. We like to wrap up our episodes by asking the guest one last question, which is what's the best piece of software created either in recent memory or all time in your opinion I personally like Spotify a lot, so. I mean, I could think more and pick another one, but I like Spotify. I like Uber as well. Honestly, you know, Uber, I think it's a perfect app that is solving people's problem. It's not creating problems for people and try to solve it. You know, it's, it's really solving people's problems. Like, you know, I remember back in the time he used to stand on the street in the rain, raise your hand for a cab, you know, now it's solved. I think it's a great example of a great technology that it's solving people's people's lives and yeah. Well that, yeah. That's I think that's very true. But again, thank you for joining us. Yeah. Very interesting episode. Thank you very much. I had enjoyed our conversation a lot. That's our episode for this week. Thank you so much for listening. Make sure to subscribe to us and rate us on Apple podcasts. We would really appreciate the support. You can also follow me on Twitter at F Z from Cupertino and Busan. The ad next facade. See you guys next week.