Pragmatic Data Scientists
Make Data Useful
Pragmatic Data Scientists
Extreme distribution and black swan events
Hello everyone. Hello. Hi this is Sid. Welcome to our channel, Sid.
Sid:Hey YZ it's great to be here.
YZ:Yeah. Sid is recently got his M B A and from U S C and he noticed a lot of mistakes or misconceptions in data-driven decisions.
Sid:Yeah, especially in finance, I noticed that a lot of our material focused on using historic information to make predictions about the future. Mm-hmm. And time and time again that actually ends up costing us. Mm-hmm. And so I thought, you know, we should chat a bit about that in today's video. Wow.
YZ:So was raw is getting wisdom from the
Sid:history. I came across this interesting concept in the Black Swan by Nasim Talib called the Turkey Thanksgiving
YZ:Paradox. Oh, so. For audience who haven't heard about, about the story, can you give us a, like, a very brief introduction of what it is
Sid:for sure. So from the perspective of a Turkey that's being raised for Thanksgiving it thinks throughout its life that humans are great. Life is great, humans are here to take care of me. They look out for my best interests and with every passing day mm-hmm. It's conviction. About this worldview only strengthens. Mm-hmm. Like each time its owner feeds it more food and takes care of it, it thinks that, you know, humans are indeed looking out for the greater good of, Turkey kind. Yeah. Until one day just out of the blue, it gets caught by surprise and it is slaughtered just before Thanksgiving. Yeah. And that was completely unexpected. And it just didn't have the full picture.
YZ:Nice. So what is a real life example of such a
Sid:Yeah, so these, these are black swan events and I think, obviously recently a lot of us heard this word during the pandemic, but it could be anything from, you know, a terrorist attack like nine 11 or you know, the financial crises of 2007, or even just the advent of. A new innovative technology. Like when the internet came up in the late nineties, it just disrupted companies like Blockbuster. Mm-hmm. Who got caught off guard because they just didn't know, didn't see this coming based on historic data.
YZ:I see. So like the small probability events, everything looks good until that happens. So that is the Black Swan event. But yeah, like I'm thinking if I'm a Turkey, how can I learn, you know, my destiny because like, The, the, because we can only observe the history, right? Yeah. And the, this small event may be just a waiting us in the future. Yeah. But there is, it's very difficult to predict.
Sid:Yeah. I mean, I think it's, hard to be at the Turkey in that situation, but I think we can learn some important lessons that we can apply which is that a lot of the time we. Make decisions based on concepts like the mean or, or standard deviation. And, and that's a specific realm of problems, which is different from problems that have more fat tail events. Yeah. So In the, in the book The Black Swan the author talks about two categories of problems. One is called the Medioc Christian World, which is basically normal distributions where the extreme outliers don't affect the mean that much. Yeah. So this could be. Heights. So
YZ:fu
Sid:Yeah. Heights, weights, you have a thousand random samples and then you add a very heavy person. Mm-hmm. Or a very tall person, it doesn't really affect the picture. Yep. And you can use concepts like standard deviation and and mean to do some modeling. Forecasting predictions. Yeah. But then there are a whole other set of problems called extreme problems. Mm-hmm. Wherein. The outlier and you can think of something like wealth, for example. Mm-hmm. If you add Jeff Bezos or Elon Musk to mm-hmm. A random distribution mm-hmm. It skews the results so much that you shouldn't be using concepts like mean and standard deviation Yeah. To, to make predictions. Yep. And in fact, I think in the world of finance, people have not realized that there are a lot of fat tail events. Mm-hmm. And they still use the old modeling based on historic information. Yeah. Like
YZ:e even if I'm not a finance person, I think related to our daily life is we make investment decisions. Yeah. A lot of time according to me. Yeah. And also we make income assumptions. Like for example, I plan my life and a lot of people including myself, Would tend to plan our life according to a linear growth of our income. But yeah, extremely events happen that can make us extremely poor. Yeah. Or extremely rich compared to our mean, and then we should actually plan
Sid:accordingly. Yeah. And in fact, the keys that you actually can't time some of these things because No, kind of, at least there, that doesn't exist any data science technique that you can use to predict and time the stock market, for example. Yep. So if you. Look at this study by that Bank of America published in 2020 where they looked at the returns of the s and p 500 from 1930 all the way to 2020. They said that, okay, if you just held the index for those nine decades, your return would be something like 17000%. Mm-hmm. Okay. That's a lot. Mm-hmm. But guess what? If you missed the 10 days of the biggest gains, every decade, your return would be just 28%. Right. Instead of 17,000. Okay. So what that tells us is that just 1% of the days in the stock market actually are significant in terms of what they do to your portfolio, and you can just do anything else on the remaining days and it wouldn't matter. Yeah. In fact, If you were smart enough to predict and avoid the 10 worst days in that period from 1930 to 2020, your return would've been close to 4 million percentage. Mm-hmm. So just think about that. You do nothing. You're at 17000% you, you miss the good day. The 10 best days. Each decade, you're only at 28%, and you avoid the 10 worst days each decade. You're at 4000000%. So in fact, in such situations, the best idea was to just do nothing. Mm-hmm. And active fund managers who try to use some kind of predictions to say, oh, I'm gonna like get out of the market during a recession or, or get in during the good times, they actually end up missing those 1% of the days and they perform worse than the person. Who had no financial knowledge and did nothing at all.
YZ:I see. So I guess if I try to apply this to my daily wisdom Yeah. Of how I act what I learn, understanding there is this huge extremity extremist Yeah. Extremist, yeah. Understanding this extremist. Story, event. Yeah. And just have the mental model around the impact of the extreme events. Yeah. Can actually change, should change my behavior. And two, understand is hard to predict. Yeah. And so maybe either why I gave up or maybe in certain events, Yeah, I, I'm like, I'm, I'm not able to predict the stock market. Yeah. That's too complex. But maybe my daily work, yeah. Maybe 1% of my work will yield most of the return. Then maybe I should pay more attention to funding that 1% of work. Yeah. And just do that. You said?
Sid:Yeah. And I think having I think in the concept, like you, you've done a lot of product work. I've done a lot of marketing in my career. In fact, it's just like one or two campaigns that go viral that does all the gains. You've probably seen it in social media and in in YouTube as well, that it's, it's not that every single. Yeah. Thing that you do
YZ:use yourself. The one, the one, one or two video or my top line, one or two video. Probably give me half of my subscribers. Yeah. And I have 200
Sid:videos. Yeah. And I, and I think it, one of the lessons that I learned was to have options. So in life, in, in your personal life, in professional life, if you always have multiple options that you can exercise and even in finance, like if you always have the option to, at a given point, get in or out of the stock market, that allows you to then capitalize on that tail event. Yeah. And if, but if you're not so diversified and your entire strategy, Focused on just one specific thing. It doesn't work because, and you might see certain brands, which maybe put all their eggs in one basket and they decided to support a specific celebrity. Mm-hmm. And then that celebrity now gets like canceled or something. Yeah. And then the entire. Business. Yeah. Could be a shoe brand, for example. And then the entire business is gone. Falls apart. Yeah. Yeah.
YZ:Alright. Okay. So the, I guess the, the knowledge we, we learned there is diversify. Yeah. Make many best, like even small banks. Yeah. Like instead of making like some big be and he hope, hope that we can capture the 1%. Yeah. Maybe make a lot of small bets. Yeah. Because just just admit the fact that we are not good at predicting
Sid:that 1% ex Exactly. Yeah. I think if we try to focus on the wrong problem, and a lot of the times in finance people focus on, oh, can I figure out exactly when the recession happens or when the stock market is gonna crash? They're chasing the wrong problem. Yeah. And they actually end up performing worse. Whereas the real problem that you should try to solve is, can I just ride the wave so that I can gain from, you know, those tail events and because only 1% of the days in the stock market actually contribute to my portfolio returns. Yeah.
YZ:Alright. Thank you for the wisdom.
Sid:Thanks wise. Thanks for having me. Yeah.
YZ:Bye.