{"version":"1.0.0","segments":[{"startTime":0.494,"endTime":3.577,"body":"(graphics whooshing)"},{"startTime":5.011,"endTime":7.5,"body":"- So this year I've been"},{"startTime":5.011,"endTime":7.5,"body":"giving a series of lectures"},{"startTime":7.5,"endTime":10.95,"body":"on how technology is affecting"},{"startTime":7.5,"endTime":10.95,"body":"the world of finance."},{"startTime":10.95,"endTime":12.81,"body":"This is the fourth in that series,"},{"startTime":12.81,"endTime":16.14,"body":"and it's about how big data"},{"startTime":12.81,"endTime":16.14,"body":"is being used in business"},{"startTime":16.14,"endTime":18.12,"body":"to make inferences."},{"startTime":18.12,"endTime":22.127,"body":"So let's start with some"},{"startTime":18.12,"endTime":22.127,"body":"popular examples, right?"},{"startTime":22.127,"endTime":24.06,"body":"One of the most famous examples,"},{"startTime":24.06,"endTime":26.76,"body":"and many of you may have"},{"startTime":24.06,"endTime":26.76,"body":"heard of this example before,"},{"startTime":26.76,"endTime":29.55,"body":"is that of the American company, Target."},{"startTime":29.55,"endTime":32.37,"body":"Now, for those of you who"},{"startTime":29.55,"endTime":32.37,"body":"have never shopped at Target,"},{"startTime":32.37,"endTime":34.47,"body":"it's a typical big box retailer,"},{"startTime":34.47,"endTime":36.63,"body":"a little bit more like"},{"startTime":34.47,"endTime":36.63,"body":"Waitrose than Tesco,"},{"startTime":36.63,"endTime":37.563,"body":"a little upscale,"},{"startTime":38.49,"endTime":42.39,"body":"but they had a big problem in 2012."},{"startTime":42.39,"endTime":47.39,"body":"And the problem was that"},{"startTime":42.39,"endTime":47.39,"body":"people shopping is \"sticky.\""},{"startTime":47.49,"endTime":48.36,"body":"That means, in other words,"},{"startTime":48.36,"endTime":51.36,"body":"when you go to a supermarket to buy stuff,"},{"startTime":51.36,"endTime":53.61,"body":"you walk in almost an autopilot."},{"startTime":53.61,"endTime":55.44,"body":"You walk in, you look for the milk,"},{"startTime":55.44,"endTime":56.94,"body":"you look for the groceries,"},{"startTime":56.94,"endTime":59.28,"body":"you just know where these things are."},{"startTime":59.28,"endTime":61.86,"body":"And so you don't actually, you know,"},{"startTime":61.86,"endTime":64.89,"body":"go from one supermarket to another."},{"startTime":64.89,"endTime":69.09,"body":"You know where things are"},{"startTime":64.89,"endTime":69.09,"body":"available at your own supermarket."},{"startTime":69.09,"endTime":71.97,"body":"So Target, in trying"},{"startTime":69.09,"endTime":71.97,"body":"to persuade you to move"},{"startTime":71.97,"endTime":75.12,"body":"to their supermarket, had"},{"startTime":71.97,"endTime":75.12,"body":"a great deal of difficulty."},{"startTime":75.12,"endTime":78.36,"body":"So they spent an enormous"},{"startTime":75.12,"endTime":78.36,"body":"amount of time and money"},{"startTime":78.36,"endTime":82.65,"body":"collecting details using"},{"startTime":78.36,"endTime":82.65,"body":"Target loyalty cards,"},{"startTime":82.65,"endTime":85.53,"body":"not on phones, but using your loyalty card"},{"startTime":85.53,"endTime":88.2,"body":"to try to figure out"},{"startTime":85.53,"endTime":88.2,"body":"exactly what you were doing"},{"speaker":"in order to predict one thing","startTime":91.02,"endTime":94.83,"body":"That is, were you, as a woman,"},{"speaker":"in order to predict one thing","startTime":94.83,"endTime":97.2,"body":"were you likely to be pregnant?"},{"speaker":"in order to predict one thing","startTime":97.2,"endTime":99.57,"body":"Now, why is that so interesting?"},{"speaker":"in order to predict one thing","startTime":99.57,"endTime":102.12,"body":"Because if you're pregnant"},{"speaker":"in order to predict one thing","startTime":99.57,"endTime":102.12,"body":"for the first time,"},{"speaker":"in order to predict one thing","startTime":102.12,"endTime":103.2,"body":"you're doing something brand new."},{"speaker":"in order to predict one thing","startTime":103.2,"endTime":104.28,"body":"You've never done this before."},{"speaker":"in order to predict one thing","startTime":104.28,"endTime":106.77,"body":"You're buying stuff you've"},{"speaker":"in order to predict one thing","startTime":104.28,"endTime":106.77,"body":"never bought before."},{"speaker":"in order to predict one thing","startTime":106.77,"endTime":108.57,"body":"So in other words,"},{"speaker":"in order to predict one thing","startTime":108.57,"endTime":111.99,"body":"if Target can predict"},{"speaker":"in order to predict one thing","startTime":108.57,"endTime":111.99,"body":"when you are pregnant,"},{"speaker":"in order to predict one thing","startTime":111.99,"endTime":115.86,"body":"they can hit you with, you"},{"speaker":"in order to predict one thing","startTime":111.99,"endTime":115.86,"body":"know, baby-related material."},{"speaker":"in order to predict one thing","startTime":115.86,"endTime":119.1,"body":"If you start shopping at Target then,"},{"speaker":"in order to predict one thing","startTime":119.1,"endTime":120.87,"body":"well, stickiness again,"},{"speaker":"in order to predict one thing","startTime":120.87,"endTime":123.56,"body":"you never leave Target again, right?"},{"speaker":"in order to predict one thing","startTime":123.56,"endTime":125.46,"body":"So that was the idea."},{"speaker":"in order to predict one thing","startTime":125.46,"endTime":128.34,"body":"And so they collected a lot"},{"speaker":"in order to predict one thing","startTime":125.46,"endTime":128.34,"body":"of information and, you know,"},{"speaker":"in order to predict one thing","startTime":128.34,"endTime":131.46,"body":"apparently the story is"},{"speaker":"in order to predict one thing","startTime":128.34,"endTime":131.46,"body":"they had came up with data,"},{"speaker":"in order to predict one thing","startTime":131.46,"endTime":133.53,"body":"for example, it turns out"},{"speaker":"in order to predict one thing","startTime":133.53,"endTime":136.65,"body":"that if you suddenly start"},{"speaker":"in order to predict one thing","startTime":133.53,"endTime":136.65,"body":"buying unscented hand lotion,"},{"speaker":"in order to predict one thing","startTime":136.65,"endTime":139.5,"body":"where previously you were"},{"speaker":"in order to predict one thing","startTime":136.65,"endTime":139.5,"body":"using scented hand lotion,"},{"speaker":"in order to predict one thing","startTime":139.5,"endTime":142.95,"body":"that's an indicator that your"},{"speaker":"in order to predict one thing","startTime":139.5,"endTime":142.95,"body":"sense of smell has changed."},{"speaker":"in order to predict one thing","startTime":142.95,"endTime":145.62,"body":"That's an indicator for"},{"speaker":"in order to predict one thing","startTime":142.95,"endTime":145.62,"body":"a possible pregnancy."},{"speaker":"in order to predict one thing","startTime":145.62,"endTime":148.59,"body":"Things like, are you using"},{"speaker":"in order to predict one thing","startTime":145.62,"endTime":148.59,"body":"more moisturizer than before?"},{"speaker":"in order to predict one thing","startTime":148.59,"endTime":150.0,"body":"Things like that."},{"speaker":"in order to predict one thing","startTime":150.0,"endTime":152.07,"body":"In fact, apparently they were so accurate,"},{"speaker":"in order to predict one thing","startTime":152.07,"endTime":154.41,"body":"and this is a book by a"},{"speaker":"in order to predict one thing","startTime":152.07,"endTime":154.41,"body":"guy called Charles Duhigg,"},{"speaker":"in order to predict one thing","startTime":154.41,"endTime":156.81,"body":"who wrote something called"},{"speaker":"in order to predict one thing","startTime":154.41,"endTime":156.81,"body":"\"The Power of Habit,\""},{"speaker":"in order to predict one thing","startTime":157.68,"endTime":159.54,"body":"apparently they were so successful"},{"speaker":"in order to predict one thing","startTime":159.54,"endTime":163.35,"body":"that this man apparently"},{"speaker":"in order to predict one thing","startTime":159.54,"endTime":163.35,"body":"stormed into a Target one day,"},{"speaker":"in order to predict one thing","startTime":163.35,"endTime":166.11,"body":"and he said, \"What are you guys doing?\""},{"speaker":"in order to predict one thing","startTime":166.11,"endTime":167.467,"body":"And they said,"},{"speaker":"in order to predict one thing","startTime":167.467,"endTime":169.56,"body":"\"I'm sorry, I don't know"},{"speaker":"in order to predict one thing","startTime":167.467,"endTime":169.56,"body":"what you're talking about.\""},{"speaker":"in order to predict one thing","startTime":169.56,"endTime":171.127,"body":"And so he goes to the manager and he says,"},{"speaker":"in order to predict one thing","startTime":171.127,"endTime":173.25,"body":"\"Look, my daughter is 16 years old,"},{"speaker":"in order to predict one thing","startTime":173.25,"endTime":175.77,"body":"and you're giving her all"},{"speaker":"in order to predict one thing","startTime":173.25,"endTime":175.77,"body":"this baby-related stuff."},{"speaker":"in order to predict one thing","startTime":175.77,"endTime":178.17,"body":"What the hell do you think you're doing?\""},{"speaker":"in order to predict one thing","startTime":178.17,"endTime":179.797,"body":"And the manager apologizes, he says,"},{"speaker":"in order to predict one thing","startTime":179.797,"endTime":181.92,"body":"\"I'm so sorry, I don't"},{"speaker":"in order to predict one thing","startTime":179.797,"endTime":181.92,"body":"know how this happened.\""},{"speaker":"in order to predict one thing","startTime":181.92,"endTime":185.46,"body":"And, you know, gave the guy"},{"speaker":"in order to predict one thing","startTime":181.92,"endTime":185.46,"body":"a $100 coupon or whatever."},{"speaker":"in order to predict one thing","startTime":185.46,"endTime":187.473,"body":"And the guy goes away, you know?"},{"speaker":"in order to predict one thing","startTime":189.077,"endTime":190.77,"body":"A week later he comes back"},{"speaker":"in order to predict one thing","startTime":189.077,"endTime":190.77,"body":"and he's shopping again."},{"speaker":"in order to predict one thing","startTime":190.77,"endTime":192.753,"body":"The manager notices him and says,"},{"speaker":"in order to predict one thing","startTime":192.753,"endTime":196.41,"body":"\"Oh, I'm so sorry about last"},{"speaker":"in order to predict one thing","startTime":192.753,"endTime":196.41,"body":"week, I really apologize.\""},{"speaker":"in order to predict one thing","startTime":196.41,"endTime":197.243,"body":"And the guy says,"},{"speaker":"in order to predict one thing","startTime":197.243,"endTime":198.93,"body":"\"Well, there were some"},{"speaker":"in order to predict one thing","startTime":197.243,"endTime":198.93,"body":"things happening in my home"},{"speaker":"in order to predict one thing","startTime":198.93,"endTime":200.67,"body":"that I really didn't know about."},{"speaker":"in order to predict one thing","startTime":200.67,"endTime":202.74,"body":"I'm afraid my daughter"},{"speaker":"in order to predict one thing","startTime":200.67,"endTime":202.74,"body":"is indeed pregnant.\""},{"speaker":"in order to predict one thing","startTime":202.74,"endTime":207.093,"body":"So, you know, Target knew about"},{"speaker":"in order to predict one thing","startTime":202.74,"endTime":207.093,"body":"this before the family did."},{"speaker":"in order to predict one thing","startTime":207.96,"endTime":209.79,"body":"Now, of course, now Target tries to be"},{"speaker":"in order to predict one thing","startTime":209.79,"endTime":211.56,"body":"much less creepy about it, right?"},{"speaker":"in order to predict one thing","startTime":211.56,"endTime":212.91,"body":"I mean, they don't want"},{"speaker":"in order to predict one thing","startTime":211.56,"endTime":212.91,"body":"to hit you with this."},{"speaker":"in order to predict one thing","startTime":212.91,"endTime":214.95,"body":"So what they do now is basically"},{"speaker":"in order to predict one thing","startTime":214.95,"endTime":217.29,"body":"when they send you their ad circulars,"},{"speaker":"in order to predict one thing","startTime":217.29,"endTime":219.03,"body":"they have baby-related stuff,"},{"speaker":"in order to predict one thing","startTime":219.03,"endTime":222.12,"body":"but they also put in things"},{"speaker":"in order to predict one thing","startTime":219.03,"endTime":222.12,"body":"that no mother is going to buy,"},{"speaker":"in order to predict one thing","startTime":222.12,"endTime":225.72,"body":"like a chainsaw or snow"},{"speaker":"in order to predict one thing","startTime":222.12,"endTime":225.72,"body":"tires or something like that."},{"speaker":"in order to predict one thing","startTime":225.72,"endTime":228.93,"body":"But just the proportion"},{"speaker":"in order to predict one thing","startTime":225.72,"endTime":228.93,"body":"of baby-related stuff"},{"speaker":"in order to predict one thing","startTime":228.93,"endTime":231.87,"body":"is higher in your circular"},{"speaker":"in order to predict one thing","startTime":228.93,"endTime":231.87,"body":"than in other people."},{"speaker":"in order to predict one thing","startTime":231.87,"endTime":234.06,"body":"I mean, you know, nobody compares"},{"speaker":"in order to predict one thing","startTime":234.06,"endTime":236.22,"body":"your ad circulars with"},{"speaker":"in order to predict one thing","startTime":234.06,"endTime":236.22,"body":"your neighbors, right?"},{"speaker":"in order to predict one thing","startTime":236.22,"endTime":238.8,"body":"So that's the idea here."},{"speaker":"in order to predict one thing","startTime":238.8,"endTime":240.497,"body":"So that was one example."},{"speaker":"in order to predict one thing","startTime":240.497,"endTime":245.46,"body":"It's a very famous example of"},{"speaker":"in order to predict one thing","startTime":240.497,"endTime":245.46,"body":"companies able to use big data"},{"speaker":"in order to predict one thing","startTime":245.46,"endTime":248.613,"body":"to predict things that we"},{"speaker":"in order to predict one thing","startTime":245.46,"endTime":248.613,"body":"don't know about ourselves."},{"speaker":"in order to predict one thing","startTime":249.54,"endTime":251.52,"body":"Another example is Facebook."},{"speaker":"in order to predict one thing","startTime":251.52,"endTime":254.37,"body":"Facebook earns billions"},{"speaker":"in order to predict one thing","startTime":251.52,"endTime":254.37,"body":"of dollars every year"},{"speaker":"in order to predict one thing","startTime":254.37,"endTime":258.87,"body":"by categorizing you as a"},{"speaker":"in order to predict one thing","startTime":254.37,"endTime":258.87,"body":"particular type of consumer:"},{"speaker":"in order to predict one thing","startTime":258.87,"endTime":261.6,"body":"somebody who drinks beer,"},{"speaker":"in order to predict one thing","startTime":258.87,"endTime":261.6,"body":"somebody who's a Republican,"},{"speaker":"in order to predict one thing","startTime":261.6,"endTime":263.19,"body":"somebody who's something else."},{"speaker":"in order to predict one thing","startTime":263.19,"endTime":267.69,"body":"And they charge ad companies"},{"speaker":"in order to predict one thing","startTime":263.19,"endTime":267.69,"body":"fortunes to get your ad"},{"speaker":"in order to predict one thing","startTime":267.69,"endTime":270.483,"body":"in front of precisely the"},{"speaker":"in order to predict one thing","startTime":267.69,"endTime":270.483,"body":"right type of customer."},{"speaker":"in order to predict one thing","startTime":271.5,"endTime":273.84,"body":"Another example was this professor"},{"speaker":"in order to predict one thing","startTime":273.84,"endTime":276.513,"body":"who bought tickets to fly on a plane,"},{"speaker":"in order to predict one thing","startTime":277.8,"endTime":280.29,"body":"bought well in advance, because"},{"speaker":"in order to predict one thing","startTime":277.8,"endTime":280.29,"body":"he wanted cheap tickets."},{"speaker":"in order to predict one thing","startTime":280.29,"endTime":281.61,"body":"But when he was actually flying,"},{"speaker":"in order to predict one thing","startTime":281.61,"endTime":283.11,"body":"he checked with his neighbors,"},{"speaker":"in order to predict one thing","startTime":283.11,"endTime":285.03,"body":"both his neighbors"},{"speaker":"in order to predict one thing","startTime":283.11,"endTime":285.03,"body":"apparently had bought tickets"},{"speaker":"in order to predict one thing","startTime":285.03,"endTime":286.38,"body":"just a few weeks before,"},{"speaker":"in order to predict one thing","startTime":286.38,"endTime":288.6,"body":"but they paid much lower prices than him."},{"speaker":"in order to predict one thing","startTime":288.6,"endTime":292.47,"body":"So he got mad, he downloaded"},{"speaker":"in order to predict one thing","startTime":288.6,"endTime":292.47,"body":"all the data from Sabre,"},{"speaker":"in order to predict one thing","startTime":292.47,"endTime":294.96,"body":"which was the airline reservation system,"},{"speaker":"in order to predict one thing","startTime":294.96,"endTime":296.94,"body":"and tried to see what patterns"},{"speaker":"in order to predict one thing","startTime":296.94,"endTime":299.88,"body":"he could find out about airline prices."},{"speaker":"in order to predict one thing","startTime":299.88,"endTime":302.46,"body":"So he came up with a"},{"speaker":"in order to predict one thing","startTime":299.88,"endTime":302.46,"body":"company called Farecast."},{"speaker":"in order to predict one thing","startTime":302.46,"endTime":304.56,"body":"So some of you may have used this."},{"speaker":"in order to predict one thing","startTime":304.56,"endTime":306.99,"body":"And the way it works is"},{"speaker":"in order to predict one thing","startTime":304.56,"endTime":306.99,"body":"you try to buy a ticket"},{"speaker":"in order to predict one thing","startTime":306.99,"endTime":308.82,"body":"and it'll tell you, \"Don't buy yet,"},{"speaker":"in order to predict one thing","startTime":308.82,"endTime":311.34,"body":"prices are likely to drop"},{"speaker":"in order to predict one thing","startTime":308.82,"endTime":311.34,"body":"in the next one week.\""},{"speaker":"in order to predict one thing","startTime":311.34,"endTime":313.95,"body":"Or, \"Buy right now, because"},{"speaker":"in order to predict one thing","startTime":311.34,"endTime":313.95,"body":"prices are likely to go up.\""},{"speaker":"in order to predict one thing","startTime":313.95,"endTime":316.5,"body":"They actually do a good job using big data"},{"speaker":"in order to predict one thing","startTime":316.5,"endTime":318.903,"body":"to predict what the"},{"speaker":"in order to predict one thing","startTime":316.5,"endTime":318.903,"body":"airlines are going to do."},{"speaker":"in order to predict one thing","startTime":320.04,"endTime":325.02,"body":"Google Flu is another example"},{"speaker":"in order to predict one thing","startTime":320.04,"endTime":325.02,"body":"where Google apparently said,"},{"speaker":"in order to predict one thing","startTime":325.02,"endTime":328.447,"body":"okay, if people search on"},{"speaker":"in order to predict one thing","startTime":325.02,"endTime":328.447,"body":"Google for things like,"},{"speaker":"in order to predict one thing","startTime":328.447,"endTime":330.217,"body":"\"What are the symptoms of flu?\""},{"speaker":"in order to predict one thing","startTime":330.217,"endTime":333.06,"body":"\"How do I cure a stuffed nose,"},{"speaker":"in order to predict one thing","startTime":333.06,"endTime":336.72,"body":"or a blocked nose, or a sore throat?\""},{"speaker":"in order to predict one thing","startTime":336.72,"endTime":337.86,"body":"And things like that,"},{"speaker":"in order to predict one thing","startTime":337.86,"endTime":342.01,"body":"that's a leading indicator, a"},{"speaker":"in order to predict one thing","startTime":337.86,"endTime":342.01,"body":"real-time indicator for flu."},{"speaker":"in order to predict one thing","startTime":343.29,"endTime":345.3,"body":"So that is much better than waiting"},{"speaker":"in order to predict one thing","startTime":345.3,"endTime":347.76,"body":"till people actually go to the doctor"},{"speaker":"in order to predict one thing","startTime":347.76,"endTime":350.853,"body":"and find out whether they have flu or not."},{"speaker":"in order to predict one thing","startTime":352.11,"endTime":355.59,"body":"Cambridge Analytica, which"},{"speaker":"in order to predict one thing","startTime":352.11,"endTime":355.59,"body":"is based in Cambridge,"},{"speaker":"in order to predict one thing","startTime":355.59,"endTime":357.48,"body":"took a lot of this data from Facebook"},{"speaker":"in order to predict one thing","startTime":357.48,"endTime":359.28,"body":"and apparently tried to sell it"},{"speaker":"in order to predict one thing","startTime":359.28,"endTime":362.52,"body":"to political parties in"},{"speaker":"in order to predict one thing","startTime":359.28,"endTime":362.52,"body":"America to try to target"},{"speaker":"in order to predict one thing","startTime":362.52,"endTime":365.58,"body":"particular types of"},{"speaker":"in order to predict one thing","startTime":362.52,"endTime":365.58,"body":"people for political ads."},{"speaker":"in order to predict one thing","startTime":365.58,"endTime":367.02,"body":"And we have here lots more examples."},{"speaker":"in order to predict one thing","startTime":367.02,"endTime":367.92,"body":"We have Amazon,"},{"speaker":"in order to predict one thing","startTime":367.92,"endTime":371.37,"body":"which makes recommendations"},{"speaker":"in order to predict one thing","startTime":367.92,"endTime":371.37,"body":"to you about what to buy."},{"speaker":"in order to predict one thing","startTime":371.37,"endTime":374.34,"body":"We have Netflix, which"},{"speaker":"in order to predict one thing","startTime":371.37,"endTime":374.34,"body":"makes also recommendations"},{"speaker":"in order to predict one thing","startTime":374.34,"endTime":376.89,"body":"about what to watch next on Netflix."},{"speaker":"in order to predict one thing","startTime":376.89,"endTime":381.27,"body":"And all these companies are"},{"speaker":"in order to predict one thing","startTime":376.89,"endTime":381.27,"body":"famous examples of big data."},{"speaker":"in order to predict one thing","startTime":381.27,"endTime":384.54,"body":"But there's one thing which"},{"speaker":"in order to predict one thing","startTime":381.27,"endTime":384.54,"body":"we don't know yet, right?"},{"speaker":"in order to predict one thing","startTime":384.54,"endTime":386.97,"body":"For example, what is big data?"},{"speaker":"in order to predict one thing","startTime":386.97,"endTime":389.1,"body":"What is \"big\" about big data?"},{"speaker":"in order to predict one thing","startTime":389.1,"endTime":392.19,"body":"Why do we call it \"big data,\" right?"},{"speaker":"in order to predict one thing","startTime":392.19,"endTime":395.583,"body":"So one way to think"},{"speaker":"in order to predict one thing","startTime":392.19,"endTime":395.583,"body":"about it is very simple."},{"speaker":"in order to predict one thing","startTime":398.859,"endTime":401.34,"body":"When we analyze something,"},{"speaker":"in order to predict one thing","startTime":398.859,"endTime":401.34,"body":"we have a choice."},{"speaker":"in order to predict one thing","startTime":401.34,"endTime":404.55,"body":"We can analyze a sample from a population,"},{"speaker":"in order to predict one thing","startTime":404.55,"endTime":407.61,"body":"or analyze the whole population."},{"speaker":"in order to predict one thing","startTime":407.61,"endTime":410.1,"body":"If you are analyzing the whole population,"},{"speaker":"in order to predict one thing","startTime":410.1,"endTime":411.72,"body":"that's a form of big data."},{"speaker":"in order to predict one thing","startTime":411.72,"endTime":414.51,"body":"If you're analyzing a"},{"speaker":"in order to predict one thing","startTime":411.72,"endTime":414.51,"body":"sample from the population,"},{"speaker":"in order to predict one thing","startTime":414.51,"endTime":416.313,"body":"that's regular data, okay?"},{"speaker":"in order to predict one thing","startTime":417.291,"endTime":421.47,"body":"But all our lives, at"},{"speaker":"in order to predict one thing","startTime":417.291,"endTime":421.47,"body":"least till a few years ago,"},{"speaker":"in order to predict one thing","startTime":421.47,"endTime":422.82,"body":"we were analyzing samples."},{"speaker":"in order to predict one thing","startTime":422.82,"endTime":426.36,"body":"We were not analyzing"},{"speaker":"in order to predict one thing","startTime":422.82,"endTime":426.36,"body":"the complete population."},{"speaker":"in order to predict one thing","startTime":426.36,"endTime":428.13,"body":"So what does that mean?"},{"speaker":"in order to predict one thing","startTime":428.13,"endTime":431.46,"body":"First requirement, if"},{"speaker":"in order to predict one thing","startTime":428.13,"endTime":431.46,"body":"you're analyzing a sample,"},{"speaker":"in order to predict one thing","startTime":431.46,"endTime":435.66,"body":"not a population, is that"},{"speaker":"in order to predict one thing","startTime":431.46,"endTime":435.66,"body":"you have to have clean data."},{"speaker":"in order to predict one thing","startTime":435.66,"endTime":439.44,"body":"I spend years sometimes cleaning databases"},{"speaker":"in order to predict one thing","startTime":439.44,"endTime":441.33,"body":"just to make sure there are no errors"},{"speaker":"in order to predict one thing","startTime":441.33,"endTime":443.19,"body":"when you actually run a regression,"},{"speaker":"in order to predict one thing","startTime":443.19,"endTime":444.75,"body":"or do something like that, right?"},{"speaker":"in order to predict one thing","startTime":444.75,"endTime":449.46,"body":"So cleaning the data is first step."},{"speaker":"in order to predict one thing","startTime":449.46,"endTime":453.06,"body":"But beyond that, the second step is,"},{"speaker":"in order to predict one thing","startTime":453.06,"endTime":455.913,"body":"okay, how do we choose the sample, right?"},{"speaker":"in order to predict one thing","startTime":456.93,"endTime":458.88,"body":"And there are lots of problems here."},{"speaker":"in order to predict one thing","startTime":458.88,"endTime":463.88,"body":"The first problem is how do we"},{"speaker":"in order to predict one thing","startTime":458.88,"endTime":463.88,"body":"get precision in our sample?"},{"speaker":"in order to predict one thing","startTime":464.52,"endTime":467.28,"body":"And precision doesn't come"},{"speaker":"in order to predict one thing","startTime":464.52,"endTime":467.28,"body":"with the size of the sample,"},{"speaker":"in order to predict one thing","startTime":467.28,"endTime":470.61,"body":"it comes with the"},{"speaker":"in order to predict one thing","startTime":467.28,"endTime":470.61,"body":"randomness in the sample."},{"speaker":"in order to predict one thing","startTime":470.61,"endTime":473.22,"body":"Two of the most famous"},{"speaker":"in order to predict one thing","startTime":470.61,"endTime":473.22,"body":"examples, for example,"},{"speaker":"in order to predict one thing","startTime":473.22,"endTime":477.127,"body":"was in 1948 when the"},{"speaker":"in order to predict one thing","startTime":473.22,"endTime":477.127,"body":"New York Post announced"},{"speaker":"in order to predict one thing","startTime":477.127,"endTime":480.69,"body":"\"Dewey defeats Truman,\""},{"speaker":"in order to predict one thing","startTime":477.127,"endTime":480.69,"body":"based on early polls."},{"speaker":"in order to predict one thing","startTime":480.69,"endTime":482.19,"body":"But the sample was wrong,"},{"speaker":"in order to predict one thing","startTime":482.19,"endTime":484.32,"body":"and in fact, Truman had defeated Dewey."},{"speaker":"in order to predict one thing","startTime":484.32,"endTime":487.83,"body":"Or the Trump election in"},{"speaker":"in order to predict one thing","startTime":484.32,"endTime":487.83,"body":"2016, where a lot of pollsters"},{"speaker":"in order to predict one thing","startTime":487.83,"endTime":490.53,"body":"had no idea they were going to win, right?"},{"speaker":"in order to predict one thing","startTime":490.53,"endTime":492.87,"body":"So why do these things happen?"},{"speaker":"in order to predict one thing","startTime":492.87,"endTime":495.51,"body":"Well, let's take an example of a sample."},{"speaker":"in order to predict one thing","startTime":495.51,"endTime":496.86,"body":"Suppose you're a pollster,"},{"speaker":"in order to predict one thing","startTime":496.86,"endTime":499.62,"body":"and you have a list of"},{"speaker":"in order to predict one thing","startTime":496.86,"endTime":499.62,"body":"everybody's landlines,"},{"speaker":"in order to predict one thing","startTime":499.62,"endTime":500.79,"body":"and you call them"},{"speaker":"in order to predict one thing","startTime":500.79,"endTime":505.053,"body":"to find out what their"},{"speaker":"in order to predict one thing","startTime":500.79,"endTime":505.053,"body":"voting intentions are."},{"speaker":"But who are you getting","startTime":506.1,"endTime":509.46,"body":"people with landlines, right?"},{"speaker":"But who are you getting","startTime":509.46,"endTime":511.35,"body":"I mean, and who has landlines?"},{"speaker":"But who are you getting","startTime":511.35,"endTime":514.08,"body":"A particular type of people,"},{"speaker":"But who are you getting","startTime":511.35,"endTime":514.08,"body":"possibly on the older side."},{"speaker":"But who are you getting","startTime":514.08,"endTime":516.18,"body":"I mean, I don't have a landline myself,"},{"speaker":"But who are you getting","startTime":516.18,"endTime":519.27,"body":"but, you know, I use a"},{"speaker":"But who are you getting","startTime":516.18,"endTime":519.27,"body":"mobile for everything."},{"speaker":"But who are you getting","startTime":519.27,"endTime":521.31,"body":"And people who answer the landlines"},{"speaker":"But who are you getting","startTime":521.31,"endTime":523.26,"body":"once they're actually called, right?"},{"speaker":"But who are you getting","startTime":523.26,"endTime":525.81,"body":"So that's a particular group of people."},{"speaker":"But who are you getting","startTime":525.81,"endTime":527.7,"body":"And so they may not be the kind of people"},{"speaker":"But who are you getting","startTime":527.7,"endTime":529.74,"body":"who are actually going to be voting."},{"speaker":"But who are you getting","startTime":529.74,"endTime":534.12,"body":"And it gets worse as you go"},{"speaker":"But who are you getting","startTime":529.74,"endTime":534.12,"body":"into finer and finer samples."},{"speaker":"But who are you getting","startTime":534.12,"endTime":536.55,"body":"For example, if you think about"},{"speaker":"But who are you getting","startTime":536.55,"endTime":538.53,"body":"trying to find out in an area,"},{"speaker":"But who are you getting","startTime":538.53,"endTime":541.41,"body":"are the Republican"},{"speaker":"But who are you getting","startTime":538.53,"endTime":541.41,"body":"women with two children,"},{"speaker":"But who are you getting","startTime":541.41,"endTime":544.32,"body":"are they going to vote, and"},{"speaker":"But who are you getting","startTime":541.41,"endTime":544.32,"body":"who are they going to vote for?"},{"speaker":"But who are you getting","startTime":544.32,"endTime":546.63,"body":"That's a very specific sample."},{"speaker":"But who are you getting","startTime":546.63,"endTime":549.45,"body":"So again, randomness is gone"},{"speaker":"But who are you getting","startTime":549.45,"endTime":551.793,"body":"once you go to more and more precise data."},{"speaker":"But who are you getting","startTime":553.59,"endTime":557.07,"body":"And we have a lot of"},{"speaker":"But who are you getting","startTime":553.59,"endTime":557.07,"body":"ways to handle that data."},{"speaker":"But who are you getting","startTime":557.07,"endTime":559.17,"body":"What's the average customer like?"},{"speaker":"But who are you getting","startTime":559.17,"endTime":560.91,"body":"What's the average voter like?"},{"speaker":"But who are you getting","startTime":560.91,"endTime":562.98,"body":"What's the modal voter like?"},{"speaker":"But who are you getting","startTime":562.98,"endTime":566.01,"body":"So there are ways of"},{"speaker":"But who are you getting","startTime":562.98,"endTime":566.01,"body":"summarizing the distribution."},{"speaker":"But who are you getting","startTime":566.01,"endTime":568.56,"body":"But the key part about all of this is,"},{"speaker":"But who are you getting","startTime":568.56,"endTime":571.74,"body":"we are trying to solve"},{"speaker":"But who are you getting","startTime":568.56,"endTime":571.74,"body":"the problem of causation."},{"speaker":"But who are you getting","startTime":571.74,"endTime":574.26,"body":"Does A cause B, right?"},{"speaker":"But who are you getting","startTime":574.26,"endTime":576.48,"body":"I mean, this is a fundamental"},{"speaker":"But who are you getting","startTime":574.26,"endTime":576.48,"body":"problem in science."},{"speaker":"But who are you getting","startTime":576.48,"endTime":579.21,"body":"We talk about causation all the time."},{"speaker":"But who are you getting","startTime":579.21,"endTime":582.24,"body":"We have a model in our"},{"speaker":"But who are you getting","startTime":579.21,"endTime":582.24,"body":"heads, and we collect data"},{"speaker":"But who are you getting","startTime":582.24,"endTime":586.53,"body":"to prove or disprove a"},{"speaker":"But who are you getting","startTime":582.24,"endTime":586.53,"body":"particular hypothesis, okay?"},{"speaker":"But who are you getting","startTime":586.53,"endTime":589.143,"body":"Now, what about big data?"},{"speaker":"But who are you getting","startTime":589.98,"endTime":592.71,"body":"Well, the first thing is,"},{"speaker":"But who are you getting","startTime":589.98,"endTime":592.71,"body":"what's the big difference"},{"speaker":"But who are you getting","startTime":592.71,"endTime":595.653,"body":"when analyzing a population"},{"speaker":"But who are you getting","startTime":592.71,"endTime":595.653,"body":"and analyzing a sample?"},{"speaker":"But who are you getting","startTime":597.06,"endTime":598.98,"body":"Let's take some examples."},{"speaker":"But who are you getting","startTime":598.98,"endTime":601.23,"body":"One example, which I've"},{"speaker":"But who are you getting","startTime":598.98,"endTime":601.23,"body":"referred to before,"},{"speaker":"But who are you getting","startTime":601.23,"endTime":605.01,"body":"is the Domesday Book here"},{"speaker":"But who are you getting","startTime":601.23,"endTime":605.01,"body":"in England, around 1066,"},{"speaker":"But who are you getting","startTime":605.01,"endTime":607.953,"body":"after William the Conqueror"},{"speaker":"But who are you getting","startTime":605.01,"endTime":607.953,"body":"came in to England."},{"speaker":"But who are you getting","startTime":608.85,"endTime":613.85,"body":"He decided to count the number"},{"speaker":"But who are you getting","startTime":608.85,"endTime":613.85,"body":"of people in his kingdom"},{"speaker":"But who are you getting","startTime":614.22,"endTime":619.22,"body":"and check how much property"},{"speaker":"But who are you getting","startTime":614.22,"endTime":619.22,"body":"they had, how much cattle,"},{"speaker":"But who are you getting","startTime":619.89,"endTime":623.55,"body":"how many, you know, heads of"},{"speaker":"But who are you getting","startTime":619.89,"endTime":623.55,"body":"livestock, things like that."},{"speaker":"But who are you getting","startTime":623.55,"endTime":626.223,"body":"There was an impending war with Denmark."},{"speaker":"But who are you getting","startTime":627.72,"endTime":630.33,"body":"He wanted to make sure"},{"speaker":"But who are you getting","startTime":627.72,"endTime":630.33,"body":"that he had enough people"},{"speaker":"But who are you getting","startTime":630.33,"endTime":631.89,"body":"to actually fight in that war."},{"speaker":"But who are you getting","startTime":631.89,"endTime":636.48,"body":"So they were trying to get the"},{"speaker":"But who are you getting","startTime":631.89,"endTime":636.48,"body":"data on the whole population."},{"speaker":"But who are you getting","startTime":636.48,"endTime":639.84,"body":"Unfortunately, it was an"},{"speaker":"But who are you getting","startTime":636.48,"endTime":639.84,"body":"incredibly labor-intensive job."},{"speaker":"But who are you getting","startTime":639.84,"endTime":642.78,"body":"And when William died,"},{"speaker":"But who are you getting","startTime":639.84,"endTime":642.78,"body":"they stopped doing it."},{"speaker":"But who are you getting","startTime":642.78,"endTime":644.88,"body":"So that was pretty much the last attempt"},{"speaker":"But who are you getting","startTime":644.88,"endTime":647.91,"body":"at trying to get data"},{"speaker":"But who are you getting","startTime":644.88,"endTime":647.91,"body":"on the whole population."},{"speaker":"But who are you getting","startTime":647.91,"endTime":649.38,"body":"But that was revived."},{"speaker":"But who are you getting","startTime":649.38,"endTime":651.42,"body":"So for example, in America,"},{"speaker":"But who are you getting","startTime":651.42,"endTime":654.57,"body":"they decided to have 10-year censuses."},{"speaker":"But who are you getting","startTime":654.57,"endTime":657.24,"body":"The problem is, till about 1880,"},{"speaker":"But who are you getting","startTime":657.24,"endTime":659.37,"body":"the amount of data they were collecting"},{"speaker":"But who are you getting","startTime":659.37,"endTime":662.04,"body":"on people from around America was so much,"},{"speaker":"But who are you getting","startTime":662.04,"endTime":665.07,"body":"it was taking more than 10 years"},{"speaker":"But who are you getting","startTime":665.07,"endTime":666.72,"body":"before the data could be analyzed."},{"speaker":"But who are you getting","startTime":666.72,"endTime":669.36,"body":"By that time, the next"},{"speaker":"But who are you getting","startTime":666.72,"endTime":669.36,"body":"census was already started."},{"speaker":"But who are you getting","startTime":669.36,"endTime":670.71,"body":"I mean, what do you do, right?"},{"speaker":"But who are you getting","startTime":670.71,"endTime":673.53,"body":"I mean, there's no way to get that data"},{"speaker":"But who are you getting","startTime":673.53,"endTime":675.39,"body":"analyzed in real time."},{"speaker":"But who are you getting","startTime":675.39,"endTime":677.58,"body":"So that's when they contacted Hollerith,"},{"speaker":"But who are you getting","startTime":677.58,"endTime":679.2,"body":"and he came up with punch cards"},{"speaker":"But who are you getting","startTime":679.2,"endTime":681.69,"body":"to actually collate the data,"},{"speaker":"But who are you getting","startTime":681.69,"endTime":685.38,"body":"that created in the 1890"},{"speaker":"But who are you getting","startTime":681.69,"endTime":685.38,"body":"census, in about two years,"},{"speaker":"But who are you getting","startTime":685.38,"endTime":688.44,"body":"rather than 13 years, as it was happening."},{"speaker":"But who are you getting","startTime":688.44,"endTime":691.965,"body":"Of course, Hollerith went"},{"speaker":"But who are you getting","startTime":688.44,"endTime":691.965,"body":"on, and then it became IBM."},{"speaker":"But who are you getting","startTime":691.965,"endTime":693.72,"body":"So IBM was one of the first companies"},{"speaker":"But who are you getting","startTime":693.72,"endTime":697.713,"body":"which was actually created to"},{"speaker":"But who are you getting","startTime":693.72,"endTime":697.713,"body":"solve the problem of big data."},{"speaker":"But who are you getting","startTime":699.12,"endTime":700.98,"body":"Cambridge Analytica, I've"},{"speaker":"But who are you getting","startTime":699.12,"endTime":700.98,"body":"already talked about,"},{"speaker":"But who are you getting","startTime":700.98,"endTime":703.17,"body":"I'll spend a little more time later."},{"speaker":"But who are you getting","startTime":703.17,"endTime":705.93,"body":"But one of the things"},{"speaker":"But who are you getting","startTime":703.17,"endTime":705.93,"body":"that we need to emphasize"},{"speaker":"But who are you getting","startTime":705.93,"endTime":707.4,"body":"is it's a population,"},{"speaker":"But who are you getting","startTime":707.4,"endTime":710.4,"body":"but the population doesn't need to be big."},{"speaker":"But who are you getting","startTime":710.4,"endTime":713.04,"body":"There's a book by, you"},{"speaker":"But who are you getting","startTime":710.4,"endTime":713.04,"body":"know, called \"Freakonomics,\""},{"speaker":"But who are you getting","startTime":713.04,"endTime":716.58,"body":"which some of you may have"},{"speaker":"But who are you getting","startTime":713.04,"endTime":716.58,"body":"read, by Levitt and Dubner,"},{"speaker":"But who are you getting","startTime":716.58,"endTime":718.11,"body":"and they talked about the population"},{"speaker":"But who are you getting","startTime":718.11,"endTime":720.66,"body":"of sumo wrestling bouts in Japan."},{"speaker":"But who are you getting","startTime":720.66,"endTime":723.15,"body":"And because they looked"},{"speaker":"But who are you getting","startTime":720.66,"endTime":723.15,"body":"at the whole population,"},{"speaker":"But who are you getting","startTime":723.15,"endTime":726.09,"body":"they were able to detect"},{"speaker":"But who are you getting","startTime":723.15,"endTime":726.09,"body":"instances of cheating"},{"speaker":"But who are you getting","startTime":726.09,"endTime":727.56,"body":"between sumo wrestlers."},{"speaker":"But who are you getting","startTime":727.56,"endTime":730.05,"body":"You can only see that if you"},{"speaker":"But who are you getting","startTime":727.56,"endTime":730.05,"body":"look at the whole population,"},{"speaker":"But who are you getting","startTime":730.05,"endTime":733.65,"body":"not if you look at a sample"},{"speaker":"But who are you getting","startTime":730.05,"endTime":733.65,"body":"from that population."},{"speaker":"But who are you getting","startTime":733.65,"endTime":736.62,"body":"But what's the common theme"},{"speaker":"But who are you getting","startTime":733.65,"endTime":736.62,"body":"about all of this, right?"},{"speaker":"But who are you getting","startTime":736.62,"endTime":741.36,"body":"Just think about these issues,"},{"speaker":"But who are you getting","startTime":736.62,"endTime":741.36,"body":"it's messy as hell, right?"},{"speaker":"But who are you getting","startTime":741.36,"endTime":743.58,"body":"So the data varies in quality."},{"speaker":"But who are you getting","startTime":743.58,"endTime":745.29,"body":"It's collected by different people"},{"speaker":"But who are you getting","startTime":745.29,"endTime":747.06,"body":"at different points in time."},{"speaker":"But who are you getting","startTime":747.06,"endTime":749.22,"body":"It's kept in a wide variety of places."},{"speaker":"But who are you getting","startTime":749.22,"endTime":752.31,"body":"It's incredibly messy,"},{"speaker":"But who are you getting","startTime":749.22,"endTime":752.31,"body":"it's not clean data."},{"speaker":"But who are you getting","startTime":752.31,"endTime":756.36,"body":"And that means you can't"},{"speaker":"But who are you getting","startTime":752.31,"endTime":756.36,"body":"really make precise inferences."},{"speaker":"But who are you getting","startTime":756.36,"endTime":759.18,"body":"You can sort of point"},{"speaker":"But who are you getting","startTime":756.36,"endTime":759.18,"body":"to general directions,"},{"speaker":"But who are you getting","startTime":759.18,"endTime":762.033,"body":"and I'm going to talk a little"},{"speaker":"But who are you getting","startTime":759.18,"endTime":762.033,"body":"bit more about that as well."},{"speaker":"But who are you getting","startTime":763.35,"endTime":765.603,"body":"But where is this big data coming from?"},{"speaker":"But who are you getting","startTime":766.62,"endTime":768.45,"body":"Well, one of the big areas"},{"speaker":"But who are you getting","startTime":768.45,"endTime":771.54,"body":"is we reuse the data from"},{"speaker":"But who are you getting","startTime":768.45,"endTime":771.54,"body":"other sources, right?"},{"speaker":"But who are you getting","startTime":771.54,"endTime":773.76,"body":"I mean, and that's one of"},{"speaker":"But who are you getting","startTime":771.54,"endTime":773.76,"body":"the reasons why it's messy,"},{"speaker":"But who are you getting","startTime":773.76,"endTime":777.24,"body":"because unlike a sample,"},{"speaker":"But who are you getting","startTime":773.76,"endTime":777.24,"body":"where we are collecting data"},{"speaker":"But who are you getting","startTime":777.24,"endTime":779.64,"body":"to test a particular hypothesis,"},{"speaker":"But who are you getting","startTime":779.64,"endTime":781.5,"body":"here, there's already data out there."},{"speaker":"But who are you getting","startTime":781.5,"endTime":784.05,"body":"We repurpose it for a new question"},{"speaker":"But who are you getting","startTime":784.05,"endTime":786.96,"body":"which that data was never meant to answer."},{"speaker":"So let's take an example","startTime":786.96,"endTime":790.683,"body":"machine translation."},{"speaker":"So let's take an example","startTime":791.73,"endTime":796.203,"body":"Machine translation used to"},{"speaker":"So let's take an example","startTime":791.73,"endTime":796.203,"body":"be an amazingly difficult job."},{"speaker":"So let's take an example","startTime":797.04,"endTime":799.08,"body":"So the earliest people who did it"},{"speaker":"So let's take an example","startTime":799.08,"endTime":802.14,"body":"tried to codify the rules of language."},{"speaker":"So let's take an example","startTime":802.14,"endTime":803.047,"body":"What they said was,"},{"speaker":"So let's take an example","startTime":803.047,"endTime":806.1,"body":"\"Here's a noun, here's an"},{"speaker":"So let's take an example","startTime":803.047,"endTime":806.1,"body":"adjective, here's a verb."},{"speaker":"So let's take an example","startTime":806.1,"endTime":808.59,"body":"This needs to be conjugated in this way.\""},{"speaker":"So let's take an example","startTime":808.59,"endTime":810.84,"body":"But language is messy, right?"},{"speaker":"So let's take an example","startTime":810.84,"endTime":813.06,"body":"I mean, there are"},{"speaker":"So let's take an example","startTime":810.84,"endTime":813.06,"body":"exceptions to every rule."},{"speaker":"So let's take an example","startTime":813.06,"endTime":815.19,"body":"There's ways to express yourself"},{"speaker":"So let's take an example","startTime":815.19,"endTime":817.02,"body":"that don't really fit well with the rules."},{"speaker":"So let's take an example","startTime":817.02,"endTime":820.44,"body":"It was really tough to"},{"speaker":"So let's take an example","startTime":817.02,"endTime":820.44,"body":"do machine translation."},{"speaker":"So let's take an example","startTime":820.44,"endTime":822.903,"body":"So even with the biggest dataset they had,"},{"speaker":"So let's take an example","startTime":824.13,"endTime":827.67,"body":"which was in Canada, at"},{"speaker":"So let's take an example","startTime":824.13,"endTime":827.67,"body":"the Canadian parliament."},{"speaker":"So let's take an example","startTime":827.67,"endTime":830.46,"body":"What they had was about 30,000 speeches"},{"speaker":"So let's take an example","startTime":830.46,"endTime":833.13,"body":"of Canadian parliamentarians,"},{"speaker":"So let's take an example","startTime":833.13,"endTime":837.0,"body":"which had been translated in"},{"speaker":"So let's take an example","startTime":833.13,"endTime":837.0,"body":"both French and English, right?"},{"speaker":"So let's take an example","startTime":837.0,"endTime":840.6,"body":"And so, very precise"},{"speaker":"So let's take an example","startTime":837.0,"endTime":840.6,"body":"translations, very well done."},{"speaker":"So let's take an example","startTime":840.6,"endTime":842.857,"body":"So they took the 30,000 pages and said,"},{"speaker":"So let's take an example","startTime":842.857,"endTime":845.1,"body":"\"Let's see if you can find patterns.\""},{"speaker":"So let's take an example","startTime":845.1,"endTime":846.09,"body":"And they couldn't."},{"speaker":"So let's take an example","startTime":846.09,"endTime":848.94,"body":"It was impossible to do."},{"speaker":"So let's take an example","startTime":848.94,"endTime":851.49,"body":"So what did Google do that was different?"},{"speaker":"So let's take an example","startTime":851.49,"endTime":853.44,"body":"Well, they already had a project"},{"speaker":"So let's take an example","startTime":853.44,"endTime":856.95,"body":"called digitizing the world's"},{"speaker":"So let's take an example","startTime":853.44,"endTime":856.95,"body":"books, Google Books, right?"},{"speaker":"So let's take an example","startTime":856.95,"endTime":859.02,"body":"And they took all that data"},{"speaker":"So let's take an example","startTime":859.02,"endTime":861.57,"body":"and they just shoved it into an algorithm,"},{"speaker":"So let's take an example","startTime":861.57,"endTime":863.37,"body":"which we'll talk about in a bit,"},{"speaker":"So let's take an example","startTime":863.37,"endTime":866.28,"body":"and they said, \"Can you find patterns?\""},{"speaker":"So let's take an example","startTime":866.28,"endTime":867.747,"body":"And they were able to find patterns."},{"speaker":"So let's take an example","startTime":867.747,"endTime":869.16,"body":"And the data's messy,"},{"speaker":"So let's take an example","startTime":869.16,"endTime":871.44,"body":"but why were they able to find patterns?"},{"speaker":"So let's take an example","startTime":871.44,"endTime":874.71,"body":"Because they had over 3 billion pages,"},{"speaker":"So let's take an example","startTime":874.71,"endTime":877.95,"body":"not 30,000, 3 billion."},{"speaker":"So let's take an example","startTime":877.95,"endTime":879.42,"body":"That's an order of magnitude,"},{"speaker":"So let's take an example","startTime":879.42,"endTime":881.64,"body":"several orders of magnitude more"},{"speaker":"So let's take an example","startTime":881.64,"endTime":884.82,"body":"than you would expect with"},{"speaker":"So let's take an example","startTime":881.64,"endTime":884.82,"body":"just, you know, 30,000 pages."},{"speaker":"So let's take an example","startTime":884.82,"endTime":886.59,"body":"And now this is pretty accurate."},{"speaker":"So let's take an example","startTime":886.59,"endTime":888.513,"body":"I use Google Translate all the time."},{"speaker":"So let's take an example","startTime":889.68,"endTime":893.13,"body":"So, it's not that the machine"},{"speaker":"So let's take an example","startTime":889.68,"endTime":893.13,"body":"knows what it's translating,"},{"speaker":"So let's take an example","startTime":893.13,"endTime":894.45,"body":"it's just looking for patterns,"},{"speaker":"So let's take an example","startTime":894.45,"endTime":898.83,"body":"the way these things correlate"},{"speaker":"So let's take an example","startTime":894.45,"endTime":898.83,"body":"in different languages."},{"speaker":"So let's take an example","startTime":898.83,"endTime":902.04,"body":"Another example, a couple"},{"speaker":"So let's take an example","startTime":898.83,"endTime":902.04,"body":"of professors at MIT"},{"speaker":"So let's take an example","startTime":902.04,"endTime":904.68,"body":"launched this Billion Prices Project"},{"speaker":"So let's take an example","startTime":904.68,"endTime":907.29,"body":"to solve the problem of inflation."},{"speaker":"So let's take an example","startTime":907.29,"endTime":909.09,"body":"And how do you figure out"},{"speaker":"So let's take an example","startTime":909.09,"endTime":911.913,"body":"whether inflation is taking"},{"speaker":"So let's take an example","startTime":909.09,"endTime":911.913,"body":"off or not within a country?"},{"speaker":"So let's take an example","startTime":912.84,"endTime":916.02,"body":"Usually, the Bank of England or the Fed"},{"speaker":"So let's take an example","startTime":916.02,"endTime":919.327,"body":"looks at a particular"},{"speaker":"So let's take an example","startTime":916.02,"endTime":919.327,"body":"basket of goods and says,"},{"speaker":"So let's take an example","startTime":919.327,"endTime":922.71,"body":"\"What are the change in the"},{"speaker":"So let's take an example","startTime":919.327,"endTime":922.71,"body":"price of that basket of goods?\""},{"speaker":"So let's take an example","startTime":922.71,"endTime":925.17,"body":"So you have to have a"},{"speaker":"So let's take an example","startTime":922.71,"endTime":925.17,"body":"representative sample."},{"speaker":"So let's take an example","startTime":925.17,"endTime":928.98,"body":"You have to monitor those"},{"speaker":"So let's take an example","startTime":925.17,"endTime":928.98,"body":"prices very, very carefully."},{"speaker":"So let's take an example","startTime":928.98,"endTime":931.14,"body":"These guys did something very different."},{"speaker":"So let's take an example","startTime":931.14,"endTime":934.05,"body":"They just took every price they"},{"speaker":"So let's take an example","startTime":931.14,"endTime":934.05,"body":"could find on all websites,"},{"speaker":"So let's take an example","startTime":934.05,"endTime":937.11,"body":"on all retailers they"},{"speaker":"So let's take an example","startTime":934.05,"endTime":937.11,"body":"could find in America."},{"speaker":"So let's take an example","startTime":937.11,"endTime":938.91,"body":"That means over a billion prices."},{"speaker":"So let's take an example","startTime":938.91,"endTime":940.08,"body":"They're not comparable, right?"},{"speaker":"So let's take an example","startTime":940.08,"endTime":943.14,"body":"I mean, what one retailer"},{"speaker":"So let's take an example","startTime":940.08,"endTime":943.14,"body":"might call a suitcase,"},{"speaker":"So let's take an example","startTime":943.14,"endTime":945.0,"body":"somebody else might call a briefcase."},{"speaker":"So let's take an example","startTime":945.0,"endTime":946.83,"body":"They're all over the place."},{"speaker":"So let's take an example","startTime":946.83,"endTime":949.92,"body":"But the point is, this"},{"speaker":"So let's take an example","startTime":946.83,"endTime":949.92,"body":"is realtime big data."},{"speaker":"So let's take an example","startTime":949.92,"endTime":953.04,"body":"And so they were able to make"},{"speaker":"So let's take an example","startTime":949.92,"endTime":953.04,"body":"inferences about inflation,"},{"speaker":"So let's take an example","startTime":953.04,"endTime":955.59,"body":"that it takes a longer, much longer time,"},{"speaker":"So let's take an example","startTime":955.59,"endTime":958.17,"body":"for entities like the Bank of England"},{"speaker":"So let's take an example","startTime":958.17,"endTime":961.023,"body":"or the Federal Reserve"},{"speaker":"So let's take an example","startTime":958.17,"endTime":961.023,"body":"to solve these problems."},{"speaker":"So let's take an example","startTime":961.92,"endTime":964.56,"body":"And of course, there's lots of new data."},{"speaker":"Examples","startTime":965.46,"endTime":968.76,"body":"Satellite photos, this"},{"speaker":"Examples","startTime":965.46,"endTime":968.76,"body":"is a classic example."},{"speaker":"Examples","startTime":968.76,"endTime":973.76,"body":"So if you want to know, for"},{"speaker":"Examples","startTime":968.76,"endTime":973.76,"body":"example, how busy a retailer is,"},{"speaker":"Examples","startTime":974.04,"endTime":976.14,"body":"one possibility is wait"},{"speaker":"Examples","startTime":974.04,"endTime":976.14,"body":"for the annual report."},{"speaker":"Examples","startTime":976.14,"endTime":977.994,"body":"So their quarterly reports, right?"},{"speaker":"Examples","startTime":977.994,"endTime":979.26,"body":"It'll tell you their sales."},{"speaker":"Examples","startTime":979.26,"endTime":982.26,"body":"Another possibility is"},{"speaker":"Examples","startTime":979.26,"endTime":982.26,"body":"take satellite photos"},{"speaker":"Examples","startTime":982.26,"endTime":983.58,"body":"of their parking lots."},{"speaker":"Examples","startTime":983.58,"endTime":987.06,"body":"How many cars are there in their"},{"speaker":"Examples","startTime":983.58,"endTime":987.06,"body":"parking lot at any one day?"},{"speaker":"Examples","startTime":987.06,"endTime":989.4,"body":"And that's available on a daily basis."},{"speaker":"Examples","startTime":989.4,"endTime":992.01,"body":"So you can have a much faster estimate"},{"speaker":"Examples","startTime":992.01,"endTime":994.47,"body":"of just how much traffic"},{"speaker":"Examples","startTime":994.47,"endTime":998.07,"body":"is happening at these big box retailers."},{"speaker":"Examples","startTime":998.07,"endTime":1000.713,"body":"Another example, open source intelligence."},{"speaker":"Examples","startTime":1001.58,"endTime":1004.04,"body":"For example, in the Ukraine war,"},{"speaker":"Examples","startTime":1004.04,"endTime":1007.7,"body":"before the Ukraine war started,"},{"speaker":"Examples","startTime":1004.04,"endTime":1007.7,"body":"you could use Google Maps"},{"speaker":"Examples","startTime":1007.7,"endTime":1011.39,"body":"to figure out where the"},{"speaker":"Examples","startTime":1007.7,"endTime":1011.39,"body":"invasion was happening."},{"speaker":"Examples","startTime":1011.39,"endTime":1012.223,"body":"How?"},{"speaker":"Examples","startTime":1012.223,"endTime":1014.537,"body":"Well, it turns out that"},{"speaker":"Examples","startTime":1012.223,"endTime":1014.537,"body":"a lot of the people"},{"speaker":"Examples","startTime":1014.537,"endTime":1018.26,"body":"who were in those areas"},{"speaker":"Examples","startTime":1014.537,"endTime":1018.26,"body":"were reporting traffic jams,"},{"speaker":"Examples","startTime":1018.26,"endTime":1020.03,"body":"because the tanks, the Russian tanks,"},{"speaker":"Examples","startTime":1020.03,"endTime":1022.88,"body":"were all blocking up the"},{"speaker":"Examples","startTime":1020.03,"endTime":1022.88,"body":"traffic on those days."},{"speaker":"Examples","startTime":1022.88,"endTime":1025.61,"body":"So you could look at Google"},{"speaker":"Examples","startTime":1022.88,"endTime":1025.61,"body":"Maps and get a good sense"},{"speaker":"Examples","startTime":1025.61,"endTime":1029.18,"body":"of where the tanks were massing"},{"speaker":"Examples","startTime":1025.61,"endTime":1029.18,"body":"to come across the border."},{"speaker":"Examples","startTime":1029.18,"endTime":1031.22,"body":"Another example of, you know,"},{"speaker":"Examples","startTime":1031.22,"endTime":1034.46,"body":"data being repurposed for something else."},{"speaker":"Examples","startTime":1034.46,"endTime":1036.2,"body":"And of course, smartphones, right?"},{"speaker":"Examples","startTime":1036.2,"endTime":1037.7,"body":"All of us have smartphones,"},{"speaker":"Examples","startTime":1037.7,"endTime":1041.18,"body":"which is collecting an"},{"speaker":"Examples","startTime":1037.7,"endTime":1041.18,"body":"immense amount of data on us."},{"speaker":"Examples","startTime":1041.18,"endTime":1044.78,"body":"They collect data on what"},{"speaker":"Examples","startTime":1041.18,"endTime":1044.78,"body":"searches we have carried out,"},{"speaker":"Examples","startTime":1044.78,"endTime":1048.08,"body":"what books we have bought,"},{"speaker":"Examples","startTime":1044.78,"endTime":1048.08,"body":"what vacations you shop for."},{"speaker":"Examples","startTime":1048.08,"endTime":1051.653,"body":"Literally everything we"},{"speaker":"Examples","startTime":1048.08,"endTime":1051.653,"body":"do is on our smartphones."},{"speaker":"Examples","startTime":1052.52,"endTime":1055.16,"body":"And of course now, most"},{"speaker":"Examples","startTime":1052.52,"endTime":1055.16,"body":"of us have Oura Rings,"},{"speaker":"Examples","startTime":1055.16,"endTime":1056.24,"body":"we have Apple watches,"},{"speaker":"Examples","startTime":1056.24,"endTime":1059.42,"body":"I have a Google Pixel"},{"speaker":"Examples","startTime":1056.24,"endTime":1059.42,"body":"watch, we have Fitbits."},{"speaker":"Examples","startTime":1059.42,"endTime":1060.98,"body":"They're all adding stuff,"},{"speaker":"Examples","startTime":1060.98,"endTime":1062.57,"body":"which we don't even know ourselves,"},{"speaker":"Examples","startTime":1062.57,"endTime":1065.42,"body":"all being captured by big data, right?"},{"speaker":"Examples","startTime":1065.42,"endTime":1067.91,"body":"So your phone knows, for"},{"speaker":"Examples","startTime":1065.42,"endTime":1067.91,"body":"example, when you're stressed."},{"speaker":"Examples","startTime":1067.91,"endTime":1069.41,"body":"It knows when you're low on sugar."},{"speaker":"Examples","startTime":1069.41,"endTime":1070.61,"body":"It knows when you like a person"},{"speaker":"Examples","startTime":1070.61,"endTime":1072.5,"body":"of the same or the opposite sex."},{"speaker":"Examples","startTime":1072.5,"endTime":1073.973,"body":"It's all there."},{"speaker":"Examples","startTime":1074.9,"endTime":1077.66,"body":"And we willingly carry"},{"speaker":"Examples","startTime":1074.9,"endTime":1077.66,"body":"some of these things"},{"speaker":"Examples","startTime":1077.66,"endTime":1079.37,"body":"with us wherever we go, right?"},{"speaker":"Examples","startTime":1079.37,"endTime":1083.06,"body":"I mean, most of us would"},{"speaker":"Examples","startTime":1079.37,"endTime":1083.06,"body":"feel lost without these,"},{"speaker":"Examples","startTime":1083.06,"endTime":1085.45,"body":"in a way, tracking devices, right?"},{"speaker":"Examples","startTime":1085.45,"endTime":1088.826,"body":"In the old days, the only"},{"speaker":"Examples","startTime":1085.45,"endTime":1088.826,"body":"people who would be allowed to,"},{"speaker":"Examples","startTime":1088.826,"endTime":1090.92,"body":"who would be made to wear"},{"speaker":"Examples","startTime":1088.826,"endTime":1090.92,"body":"these tracking devices"},{"speaker":"Examples","startTime":1090.92,"endTime":1093.32,"body":"were convicted felons on parole."},{"speaker":"Examples","startTime":1093.32,"endTime":1095.24,"body":"But now, you know, we have it,"},{"speaker":"Examples","startTime":1095.24,"endTime":1098.3,"body":"all generate our own data from it."},{"speaker":"Examples","startTime":1098.3,"endTime":1102.71,"body":"So if you classify these"},{"speaker":"Examples","startTime":1098.3,"endTime":1102.71,"body":"different types of data,"},{"speaker":"Examples","startTime":1102.71,"endTime":1103.97,"body":"there's several different types."},{"speaker":"Examples","startTime":1103.97,"endTime":1105.8,"body":"There's geospatial data,"},{"speaker":"Examples","startTime":1105.8,"endTime":1109.52,"body":"which tells you how you are"},{"speaker":"Examples","startTime":1105.8,"endTime":1109.52,"body":"moving from place to place."},{"speaker":"Examples","startTime":1109.52,"endTime":1111.47,"body":"There is sociometric data,"},{"speaker":"Examples","startTime":1111.47,"endTime":1114.77,"body":"which is the study of"},{"speaker":"Examples","startTime":1111.47,"endTime":1114.77,"body":"your social relationships."},{"speaker":"Examples","startTime":1114.77,"endTime":1116.72,"body":"There is the study of psychometric data."},{"speaker":"Examples","startTime":1116.72,"endTime":1118.97,"body":"What's your mental state,"},{"speaker":"Examples","startTime":1116.72,"endTime":1118.97,"body":"what's your personality?"},{"speaker":"Examples","startTime":1118.97,"endTime":1120.95,"body":"And of course, there's biometric data,"},{"speaker":"Examples","startTime":1120.95,"endTime":1124.73,"body":"which looks at your biological"},{"speaker":"Examples","startTime":1120.95,"endTime":1124.73,"body":"characteristics, right?"},{"speaker":"Examples","startTime":1124.73,"endTime":1127.04,"body":"I'll take examples of each of those."},{"speaker":"Examples","startTime":1127.04,"endTime":1130.52,"body":"But before I show you those examples,"},{"speaker":"Examples","startTime":1130.52,"endTime":1133.34,"body":"why are we collecting so much data?"},{"speaker":"Examples","startTime":1133.34,"endTime":1135.05,"body":"I think a popular phrase,"},{"speaker":"Examples","startTime":1135.05,"endTime":1136.96,"body":"which a lot of people"},{"speaker":"Examples","startTime":1135.05,"endTime":1136.96,"body":"have been talking about,"},{"speaker":"Examples","startTime":1136.96,"endTime":1140.242,"body":"is they say, \"Data is the new oil,\" right?"},{"speaker":"Examples","startTime":1140.242,"endTime":1143.72,"body":"It's like if you have data,"},{"speaker":"Examples","startTime":1140.242,"endTime":1143.72,"body":"it's like you are going to be"},{"speaker":"Examples","startTime":1143.72,"endTime":1147.14,"body":"the next powerful"},{"speaker":"Examples","startTime":1143.72,"endTime":1147.14,"body":"company around the world."},{"speaker":"Examples","startTime":1147.14,"endTime":1151.04,"body":"So a lot of companies believe"},{"speaker":"Examples","startTime":1147.14,"endTime":1151.04,"body":"the more data we have,"},{"speaker":"Examples","startTime":1151.04,"endTime":1153.32,"body":"the more the valuable"},{"speaker":"Examples","startTime":1151.04,"endTime":1153.32,"body":"insights we're going to get."},{"speaker":"Examples","startTime":1153.32,"endTime":1155.36,"body":"And of course, regulators also assume"},{"speaker":"Examples","startTime":1155.36,"endTime":1156.92,"body":"that more data is better, right?"},{"speaker":"Examples","startTime":1156.92,"endTime":1158.75,"body":"I mean, people usually assume"},{"speaker":"Examples","startTime":1158.75,"endTime":1160.88,"body":"that more is always better, right?"},{"speaker":"Examples","startTime":1160.88,"endTime":1163.1,"body":"Bigger house, better than a smaller house."},{"speaker":"Examples","startTime":1163.1,"endTime":1164.9,"body":"More money, better than less money."},{"speaker":"Examples","startTime":1164.9,"endTime":1165.86,"body":"More children..."},{"speaker":"Examples","startTime":1165.86,"endTime":1167.3,"body":"No, it doesn't work for children,"},{"speaker":"Examples","startTime":1167.3,"endTime":1168.77,"body":"but it works for everything else."},{"speaker":"Examples","startTime":1168.77,"endTime":1172.04,"body":"Okay, well, so if you have, for example,"},{"speaker":"Examples","startTime":1172.04,"endTime":1175.55,"body":"companies listed on the"},{"speaker":"Examples","startTime":1172.04,"endTime":1175.55,"body":"stock market, right?"},{"speaker":"Examples","startTime":1175.55,"endTime":1177.59,"body":"You have to file quarterly information."},{"speaker":"Examples","startTime":1177.59,"endTime":1179.42,"body":"Banks and investment funds"},{"speaker":"Examples","startTime":1179.42,"endTime":1182.21,"body":"have to have stringent"},{"speaker":"Examples","startTime":1179.42,"endTime":1182.21,"body":"reporting of obligation."},{"speaker":"Examples","startTime":1182.21,"endTime":1184.52,"body":"Certain sectors are"},{"speaker":"Examples","startTime":1182.21,"endTime":1184.52,"body":"additional information."},{"speaker":"Examples","startTime":1184.52,"endTime":1188.45,"body":"So everybody is being given"},{"speaker":"Examples","startTime":1184.52,"endTime":1188.45,"body":"more and more information."},{"speaker":"Examples","startTime":1188.45,"endTime":1191.06,"body":"The problem is we can't"},{"speaker":"Examples","startTime":1188.45,"endTime":1191.06,"body":"interpret this big data"},{"speaker":"Examples","startTime":1191.06,"endTime":1191.96,"body":"just like that."},{"speaker":"Examples","startTime":1191.96,"endTime":1196.22,"body":"It's just generating data"},{"speaker":"Examples","startTime":1191.96,"endTime":1196.22,"body":"at us, what do we do?"},{"speaker":"Examples","startTime":1196.22,"endTime":1200.51,"body":"So what we have to do is we"},{"speaker":"Examples","startTime":1196.22,"endTime":1200.51,"body":"need to process the data."},{"speaker":"Examples","startTime":1200.51,"endTime":1202.07,"body":"We're going to talk about three steps"},{"speaker":"Examples","startTime":1202.07,"endTime":1204.53,"body":"we need to process that data."},{"speaker":"Examples","startTime":1204.53,"endTime":1206.54,"body":"The first step is a language."},{"speaker":"Examples","startTime":1206.54,"endTime":1210.47,"body":"We need to come up with a"},{"speaker":"Examples","startTime":1206.54,"endTime":1210.47,"body":"language to describe that data."},{"speaker":"Examples","startTime":1210.47,"endTime":1214.22,"body":"Without that language,"},{"speaker":"Examples","startTime":1210.47,"endTime":1214.22,"body":"big data is useless, okay?"},{"speaker":"Examples","startTime":1214.22,"endTime":1215.18,"body":"Second thing,"},{"speaker":"Examples","startTime":1215.18,"endTime":1216.77,"body":"we need to look at our preferences"},{"speaker":"Examples","startTime":1216.77,"endTime":1218.36,"body":"along multiple dimensions."},{"speaker":"Examples","startTime":1218.36,"endTime":1220.49,"body":"And I'll talk about that using an example."},{"speaker":"Examples","startTime":1220.49,"endTime":1221.51,"body":"And finally,"},{"speaker":"Examples","startTime":1221.51,"endTime":1224.39,"body":"we need to capture all"},{"speaker":"Examples","startTime":1221.51,"endTime":1224.39,"body":"those preferences to predict"},{"speaker":"Examples","startTime":1224.39,"endTime":1227.273,"body":"what we're going to do at"},{"speaker":"Examples","startTime":1224.39,"endTime":1227.273,"body":"any one point in time, right?"},{"speaker":"Examples","startTime":1228.29,"endTime":1230.15,"body":"So those are our three steps."},{"speaker":"Examples","startTime":1230.15,"endTime":1233.6,"body":"So, why do we need a language?"},{"speaker":"Examples","startTime":1233.6,"endTime":1236.393,"body":"Well, let's just take a cup of tea, right?"},{"speaker":"Examples","startTime":1237.53,"endTime":1241.49,"body":"So what dimensions could we"},{"speaker":"Examples","startTime":1237.53,"endTime":1241.49,"body":"have with this cup of tea?"},{"speaker":"Examples","startTime":1241.49,"endTime":1243.08,"body":"The type of tea, of course,"},{"speaker":"Examples","startTime":1243.08,"endTime":1246.02,"body":"but it could also be even the type of tea,"},{"speaker":"Examples","startTime":1246.02,"endTime":1247.31,"body":"where is the tea coming from?"},{"speaker":"Examples","startTime":1247.31,"endTime":1249.8,"body":"Is it coming from China,"},{"speaker":"Examples","startTime":1247.31,"endTime":1249.8,"body":"is it coming from Assam?"},{"speaker":"Examples","startTime":1249.8,"endTime":1251.06,"body":"What's the carbon footprint?"},{"speaker":"Examples","startTime":1251.06,"endTime":1253.13,"body":"Is it organic, is it something else?"},{"speaker":"Examples","startTime":1253.13,"endTime":1255.53,"body":"There's so many dimensions"},{"speaker":"Examples","startTime":1253.13,"endTime":1255.53,"body":"you can classify"},{"speaker":"Examples","startTime":1255.53,"endTime":1257.51,"body":"just even a cup of tea."},{"speaker":"Examples","startTime":1257.51,"endTime":1262.1,"body":"So that means that you need"},{"speaker":"Examples","startTime":1257.51,"endTime":1262.1,"body":"to quantify those dimensions."},{"speaker":"Examples","startTime":1262.1,"endTime":1266.57,"body":"You need a way to say, convert"},{"speaker":"Examples","startTime":1262.1,"endTime":1266.57,"body":"each dimension into a number,"},{"speaker":"Examples","startTime":1266.57,"endTime":1269.33,"body":"because a computer"},{"speaker":"Examples","startTime":1266.57,"endTime":1269.33,"body":"doesn't understand words,"},{"speaker":"Examples","startTime":1269.33,"endTime":1270.89,"body":"it understands numbers."},{"speaker":"Examples","startTime":1270.89,"endTime":1273.863,"body":"So we need to quantify"},{"speaker":"Examples","startTime":1270.89,"endTime":1273.863,"body":"each of those dimensions."},{"speaker":"Examples","startTime":1274.85,"endTime":1279.8,"body":"So what does that quantification"},{"speaker":"Examples","startTime":1274.85,"endTime":1279.8,"body":"process consist of?"},{"speaker":"Examples","startTime":1279.8,"endTime":1281.24,"body":"Well, first of all,"},{"speaker":"Examples","startTime":1281.24,"endTime":1285.11,"body":"all those different dimensions"},{"speaker":"Examples","startTime":1281.24,"endTime":1285.11,"body":"have to have uniform tags."},{"speaker":"Examples","startTime":1285.11,"endTime":1286.76,"body":"You would have to say, okay,"},{"speaker":"Examples","startTime":1286.76,"endTime":1289.85,"body":"it's a tea bag as opposed to loose tea."},{"speaker":"Examples","startTime":1289.85,"endTime":1293.45,"body":"It's a Chinese tea as opposed"},{"speaker":"Examples","startTime":1289.85,"endTime":1293.45,"body":"to an Assam tea, and so on."},{"speaker":"Examples","startTime":1293.45,"endTime":1296.15,"body":"So there are specific tags you need."},{"speaker":"Examples","startTime":1296.15,"endTime":1299.87,"body":"And those tags can be incredibly complex."},{"speaker":"Examples","startTime":1299.87,"endTime":1303.74,"body":"For example, this is a company"},{"speaker":"Examples","startTime":1299.87,"endTime":1303.74,"body":"called Zappos in America,"},{"speaker":"Examples","startTime":1303.74,"endTime":1305.81,"body":"which just sells shoes, that's it."},{"speaker":"Examples","startTime":1305.81,"endTime":1308.75,"body":"If you type in \"men's"},{"speaker":"Examples","startTime":1305.81,"endTime":1308.75,"body":"sneakers and athletic shoes,\""},{"speaker":"Examples","startTime":1308.75,"endTime":1312.233,"body":"it comes up with 9,260 choices."},{"speaker":"There are subcategories","startTime":1313.88,"endTime":1316.55,"body":"lifestyle sneakers,"},{"speaker":"There are subcategories","startTime":1316.55,"endTime":1319.16,"body":"athletic shoes, sizes, width,"},{"speaker":"There are subcategories","startTime":1319.16,"endTime":1321.56,"body":"brand, prices, colors."},{"speaker":"There are subcategories","startTime":1321.56,"endTime":1323.36,"body":"You name it, there are categories"},{"speaker":"There are subcategories","startTime":1323.36,"endTime":1325.763,"body":"which you can classify this data into."},{"speaker":"There are subcategories","startTime":1326.87,"endTime":1330.89,"body":"Now, that's easy in a"},{"speaker":"There are subcategories","startTime":1326.87,"endTime":1330.89,"body":"specialized marketplace,"},{"speaker":"There are subcategories","startTime":1330.89,"endTime":1335.36,"body":"like a shoe, like a washing"},{"speaker":"There are subcategories","startTime":1330.89,"endTime":1335.36,"body":"machine, like a hard disk,"},{"speaker":"There are subcategories","startTime":1335.36,"endTime":1337.82,"body":"there's a specific number of dimensions."},{"speaker":"There are subcategories","startTime":1337.82,"endTime":1340.73,"body":"But once the data becomes"},{"speaker":"There are subcategories","startTime":1337.82,"endTime":1340.73,"body":"more unstructured,"},{"speaker":"There are subcategories","startTime":1340.73,"endTime":1343.91,"body":"it becomes more and more difficult to do."},{"speaker":"There are subcategories","startTime":1343.91,"endTime":1346.85,"body":"For example, think about YouTube."},{"speaker":"There are subcategories","startTime":1346.85,"endTime":1349.707,"body":"If you're trying to find"},{"speaker":"There are subcategories","startTime":1346.85,"endTime":1349.707,"body":"something on YouTube,"},{"speaker":"There are subcategories","startTime":1349.707,"endTime":1351.863,"body":"\"how to juggle,\" right?"},{"speaker":"There are subcategories","startTime":1353.0,"endTime":1354.86,"body":"How does YouTube know"},{"speaker":"There are subcategories","startTime":1354.86,"endTime":1357.23,"body":"how to get you that kind"},{"speaker":"There are subcategories","startTime":1354.86,"endTime":1357.23,"body":"of information, right?"},{"speaker":"There are subcategories","startTime":1357.23,"endTime":1361.55,"body":"How does it find a video"},{"speaker":"There are subcategories","startTime":1357.23,"endTime":1361.55,"body":"about juggling to you?"},{"speaker":"There are subcategories","startTime":1361.55,"endTime":1363.41,"body":"Well, it looks for multiple things."},{"speaker":"There are subcategories","startTime":1363.41,"endTime":1367.013,"body":"For example, is the word"},{"speaker":"There are subcategories","startTime":1363.41,"endTime":1367.013,"body":"\"juggling,\" is it relevant?"},{"speaker":"There are subcategories","startTime":1368.15,"endTime":1370.16,"body":"Which part of the video is that, right?"},{"speaker":"There are subcategories","startTime":1370.16,"endTime":1373.04,"body":"It looks at how many people"},{"speaker":"There are subcategories","startTime":1370.16,"endTime":1373.04,"body":"have engaged with the video."},{"speaker":"There are subcategories","startTime":1373.04,"endTime":1376.79,"body":"It looks at the quality of that, right?"},{"speaker":"There are subcategories","startTime":1376.79,"endTime":1378.65,"body":"And whether it can be personalized."},{"speaker":"There are subcategories","startTime":1378.65,"endTime":1380.33,"body":"Let me give you an example here."},{"speaker":"There are subcategories","startTime":1380.33,"endTime":1383.72,"body":"So it looks at what videos"},{"speaker":"There are subcategories","startTime":1380.33,"endTime":1383.72,"body":"you watched in the past."},{"speaker":"There are subcategories","startTime":1383.72,"endTime":1386.84,"body":"What are the videos which are"},{"speaker":"There are subcategories","startTime":1383.72,"endTime":1386.84,"body":"typically watched together?"},{"speaker":"There are subcategories","startTime":1386.84,"endTime":1389.15,"body":"So if you have one video on juggling,"},{"speaker":"There are subcategories","startTime":1389.15,"endTime":1390.89,"body":"what is it also watched by?"},{"speaker":"There are subcategories","startTime":1390.89,"endTime":1392.81,"body":"So there's a group up there."},{"speaker":"There are subcategories","startTime":1392.81,"endTime":1394.94,"body":"And topically related videos."},{"speaker":"There are subcategories","startTime":1394.94,"endTime":1398.97,"body":"For example, if you hunt for"},{"speaker":"There are subcategories","startTime":1394.94,"endTime":1398.97,"body":"the word \"cricket,\" right?"},{"speaker":"There are subcategories","startTime":1398.97,"endTime":1401.03,"body":"There are two possibilities, right?"},{"speaker":"There are subcategories","startTime":1401.03,"endTime":1402.89,"body":"You have a cricket chirping at night,"},{"speaker":"There are subcategories","startTime":1402.89,"endTime":1404.84,"body":"or you could be playing cricket."},{"speaker":"There are subcategories","startTime":1404.84,"endTime":1406.43,"body":"Now, which one will it give you?"},{"speaker":"There are subcategories","startTime":1406.43,"endTime":1408.17,"body":"It depends again on things like"},{"speaker":"There are subcategories","startTime":1408.17,"endTime":1409.533,"body":"what have you watched in the past?"},{"speaker":"There are subcategories","startTime":1409.533,"endTime":1411.29,"body":"If you watched a lot of insect videos,"},{"speaker":"There are subcategories","startTime":1411.29,"endTime":1412.34,"body":"you're going to get that."},{"speaker":"There are subcategories","startTime":1412.34,"endTime":1414.92,"body":"If you're watching a lot"},{"speaker":"There are subcategories","startTime":1412.34,"endTime":1414.92,"body":"of, you know, sports videos,"},{"speaker":"There are subcategories","startTime":1414.92,"endTime":1416.24,"body":"you're going to get that, right?"},{"speaker":"There are subcategories","startTime":1416.24,"endTime":1421.07,"body":"So it has to infer from all"},{"speaker":"There are subcategories","startTime":1416.24,"endTime":1421.07,"body":"these bases, what is it?"},{"speaker":"There are subcategories","startTime":1421.07,"endTime":1426.07,"body":"How does it search for a"},{"speaker":"There are subcategories","startTime":1421.07,"endTime":1426.07,"body":"concept in a tag database?"},{"speaker":"There are subcategories","startTime":1426.95,"endTime":1431.06,"body":"Now, in this particular,"},{"speaker":"There are subcategories","startTime":1426.95,"endTime":1431.06,"body":"in YouTube's case,"},{"speaker":"There are subcategories","startTime":1431.06,"endTime":1436.04,"body":"it has millions of possible"},{"speaker":"There are subcategories","startTime":1431.06,"endTime":1436.04,"body":"videos which it can serve you,"},{"speaker":"There are subcategories","startTime":1436.04,"endTime":1439.01,"body":"but it looks at your user"},{"speaker":"There are subcategories","startTime":1436.04,"endTime":1439.01,"body":"history and your context."},{"speaker":"There are subcategories","startTime":1439.01,"endTime":1441.71,"body":"It looks at other candidate"},{"speaker":"There are subcategories","startTime":1439.01,"endTime":1441.71,"body":"sources to winnow that down."},{"speaker":"There are subcategories","startTime":1441.71,"endTime":1444.47,"body":"There's millions of sources into hundreds,"},{"speaker":"There are subcategories","startTime":1444.47,"endTime":1447.17,"body":"eventually ranks them on"},{"speaker":"There are subcategories","startTime":1444.47,"endTime":1447.17,"body":"the basis of video features."},{"speaker":"There are subcategories","startTime":1447.17,"endTime":1450.623,"body":"And what you get is just"},{"speaker":"There are subcategories","startTime":1447.17,"endTime":1450.623,"body":"the tip of the iceberg."},{"speaker":"There are subcategories","startTime":1451.79,"endTime":1453.95,"body":"So here are some ways you can do this."},{"speaker":"There are subcategories","startTime":1453.95,"endTime":1457.46,"body":"For example, you want to"},{"speaker":"There are subcategories","startTime":1453.95,"endTime":1457.46,"body":"be as precise as possible"},{"speaker":"There are subcategories","startTime":1457.46,"endTime":1458.66,"body":"in your name, right?"},{"speaker":"There are subcategories","startTime":1458.66,"endTime":1462.233,"body":"So if you have"},{"speaker":"There are subcategories","startTime":1458.66,"endTime":1462.233,"body":"laparoscopic-appendectomy.mov,"},{"speaker":"There are subcategories","startTime":1463.16,"endTime":1464.36,"body":"it's looking for that name there."},{"speaker":"There are subcategories","startTime":1464.36,"endTime":1467.24,"body":"So if you're looking for"},{"speaker":"There are subcategories","startTime":1464.36,"endTime":1467.24,"body":"a video on appendectomy,"},{"speaker":"There are subcategories","startTime":1467.24,"endTime":1469.4,"body":"that's where it'll show"},{"speaker":"There are subcategories","startTime":1467.24,"endTime":1469.4,"body":"up in the name, right?"},{"speaker":"There are subcategories","startTime":1469.4,"endTime":1471.41,"body":"It'll look for the video title."},{"speaker":"There are subcategories","startTime":1471.41,"endTime":1472.243,"body":"You want to say,"},{"speaker":"There are subcategories","startTime":1472.243,"endTime":1475.07,"body":"\"A real life step by step"},{"speaker":"There are subcategories","startTime":1472.243,"endTime":1475.07,"body":"laparoscopic appendectomy.\""},{"speaker":"There are subcategories","startTime":1475.07,"endTime":1477.59,"body":"Much better than if you"},{"speaker":"There are subcategories","startTime":1475.07,"endTime":1477.59,"body":"have a very vague thing."},{"speaker":"There are subcategories","startTime":1477.59,"endTime":1480.65,"body":"So this is up to you when you're uploading"},{"speaker":"There are subcategories","startTime":1480.65,"endTime":1483.44,"body":"to give it as many cues as possible"},{"speaker":"There are subcategories","startTime":1483.44,"endTime":1488.27,"body":"to let people find that"},{"speaker":"There are subcategories","startTime":1483.44,"endTime":1488.27,"body":"video for you, okay?"},{"speaker":"There are subcategories","startTime":1488.27,"endTime":1493.27,"body":"Now, this is our first step,"},{"speaker":"There are subcategories","startTime":1488.27,"endTime":1493.27,"body":"classifying our data, right?"},{"speaker":"There are subcategories","startTime":1494.12,"endTime":1497.187,"body":"We are developing a data ontology."},{"speaker":"There are subcategories","startTime":1497.187,"endTime":1499.52,"body":"\"Ontology\" just means a language."},{"speaker":"There are subcategories","startTime":1499.52,"endTime":1503.57,"body":"So developing a data"},{"speaker":"There are subcategories","startTime":1499.52,"endTime":1503.57,"body":"language is our first thing."},{"speaker":"There are subcategories","startTime":1503.57,"endTime":1504.83,"body":"And this is really important."},{"speaker":"There are subcategories","startTime":1504.83,"endTime":1507.8,"body":"Like eBay, if you try to"},{"speaker":"There are subcategories","startTime":1504.83,"endTime":1507.8,"body":"find anything on eBay,"},{"speaker":"There are subcategories","startTime":1507.8,"endTime":1509.33,"body":"it's a complete mess, right?"},{"speaker":"There are subcategories","startTime":1509.33,"endTime":1511.49,"body":"Very difficult to find stuff on eBay."},{"speaker":"There are subcategories","startTime":1511.49,"endTime":1514.25,"body":"Much easier to find stuff on Amazon,"},{"speaker":"There are subcategories","startTime":1514.25,"endTime":1518.24,"body":"because Amazon started with"},{"speaker":"There are subcategories","startTime":1514.25,"endTime":1518.24,"body":"a classification system"},{"speaker":"There are subcategories","startTime":1518.24,"endTime":1520.79,"body":"for the first items"},{"speaker":"There are subcategories","startTime":1518.24,"endTime":1520.79,"body":"they sold: books, right?"},{"speaker":"There are subcategories","startTime":1520.79,"endTime":1523.1,"body":"The Dewey Decimal classification system,"},{"speaker":"There are subcategories","startTime":1523.1,"endTime":1526.25,"body":"easily available, easy to classify."},{"speaker":"There are subcategories","startTime":1526.25,"endTime":1528.98,"body":"But this is a really hot"},{"speaker":"There are subcategories","startTime":1526.25,"endTime":1528.98,"body":"field for jobs, by the way."},{"speaker":"There are subcategories","startTime":1528.98,"endTime":1532.01,"body":"So, you know, when you"},{"speaker":"There are subcategories","startTime":1528.98,"endTime":1532.01,"body":"guys were growing up,"},{"speaker":"There are subcategories","startTime":1532.01,"endTime":1533.39,"body":"I mean, how many of you,"},{"speaker":"There are subcategories","startTime":1533.39,"endTime":1536.277,"body":"when your parents or your"},{"speaker":"There are subcategories","startTime":1533.39,"endTime":1536.277,"body":"uncles or whatever asked you,"},{"speaker":"There are subcategories","startTime":1536.277,"endTime":1537.98,"body":"\"What do you want to be when you grow up?\""},{"speaker":"There are subcategories","startTime":1537.98,"endTime":1541.727,"body":"How many of you said, \"I"},{"speaker":"There are subcategories","startTime":1537.98,"endTime":1541.727,"body":"want to be a librarian!\""},{"speaker":"There are subcategories","startTime":1542.66,"endTime":1543.493,"body":"Anybody?"},{"speaker":"There are subcategories","startTime":1544.46,"endTime":1548.36,"body":"Well, it turns out, this is a"},{"speaker":"There are subcategories","startTime":1544.46,"endTime":1548.36,"body":"really hot field these days."},{"speaker":"There are subcategories","startTime":1548.36,"endTime":1551.93,"body":"Being a librarian is"},{"speaker":"There are subcategories","startTime":1548.36,"endTime":1551.93,"body":"a very, very good job,"},{"speaker":"There are subcategories","startTime":1551.93,"endTime":1555.02,"body":"because it's about understanding data,"},{"speaker":"There are subcategories","startTime":1555.02,"endTime":1557.51,"body":"not about just books anymore."},{"speaker":"There are subcategories","startTime":1557.51,"endTime":1559.61,"body":"Okay, so that's the first step, right?"},{"speaker":"There are subcategories","startTime":1559.61,"endTime":1560.84,"body":"So you've got all this data,"},{"speaker":"There are subcategories","startTime":1560.84,"endTime":1563.42,"body":"you've nicely classified this data."},{"speaker":"There are subcategories","startTime":1563.42,"endTime":1566.24,"body":"And the second step is how do you match"},{"speaker":"There are subcategories","startTime":1566.24,"endTime":1568.7,"body":"the different dimensions of"},{"speaker":"There are subcategories","startTime":1566.24,"endTime":1568.7,"body":"the data you've collected,"},{"speaker":"There are subcategories","startTime":1568.7,"endTime":1573.207,"body":"the metadata, into something"},{"speaker":"There are subcategories","startTime":1568.7,"endTime":1573.207,"body":"that is one number saying,"},{"speaker":"There are subcategories","startTime":1573.207,"endTime":1575.567,"body":"\"You will like this, or"},{"speaker":"There are subcategories","startTime":1573.207,"endTime":1575.567,"body":"you won't like this.\""},{"speaker":"There are subcategories","startTime":1576.41,"endTime":1578.39,"body":"So this is an algorithm"},{"speaker":"There are subcategories","startTime":1578.39,"endTime":1581.337,"body":"which looks at your multiple"},{"speaker":"There are subcategories","startTime":1578.39,"endTime":1581.337,"body":"preferences and says,"},{"speaker":"There are subcategories","startTime":1581.337,"endTime":1582.65,"body":"\"This is the highest preference.\""},{"speaker":"There are subcategories","startTime":1582.65,"endTime":1584.33,"body":"How is that done?"},{"speaker":"There are subcategories","startTime":1584.33,"endTime":1586.61,"body":"Well, we already have those algorithms."},{"speaker":"There are subcategories","startTime":1586.61,"endTime":1589.4,"body":"We have the same algorithms to"},{"speaker":"There are subcategories","startTime":1586.61,"endTime":1589.4,"body":"manage your photo collection."},{"speaker":"There are subcategories","startTime":1589.4,"endTime":1592.22,"body":"We have Siri, Alexa,"},{"speaker":"There are subcategories","startTime":1589.4,"endTime":1592.22,"body":"understand a voice command."},{"speaker":"There are subcategories","startTime":1592.22,"endTime":1594.65,"body":"Or our smart watches detect"},{"speaker":"There are subcategories","startTime":1594.65,"endTime":1598.49,"body":"when our heart isn't"},{"speaker":"There are subcategories","startTime":1594.65,"endTime":1598.49,"body":"beating rhythmically, right?"},{"speaker":"There are subcategories","startTime":1598.49,"endTime":1600.53,"body":"So basically it's just a data stream."},{"speaker":"There are subcategories","startTime":1600.53,"endTime":1601.64,"body":"So what you're going to do"},{"speaker":"There are subcategories","startTime":1601.64,"endTime":1604.55,"body":"is to find a pattern-matching algorithm"},{"speaker":"There are subcategories","startTime":1604.55,"endTime":1606.77,"body":"to get that data stream, you know,"},{"speaker":"There are subcategories","startTime":1606.77,"endTime":1611.77,"body":"to whatever it is trying"},{"speaker":"There are subcategories","startTime":1606.77,"endTime":1611.77,"body":"to do to make sure"},{"speaker":"There are subcategories","startTime":1612.11,"endTime":1615.53,"body":"it understands your"},{"speaker":"There are subcategories","startTime":1612.11,"endTime":1615.53,"body":"preferences in one number."},{"speaker":"So let's take an example","startTime":1615.53,"endTime":1619.37,"body":"Netflix."},{"speaker":"So let's take an example","startTime":1619.37,"endTime":1621.02,"body":"And in this example,"},{"speaker":"So let's take an example","startTime":1621.02,"endTime":1625.7,"body":"I'm actually going to go back"},{"speaker":"So let's take an example","startTime":1621.02,"endTime":1625.7,"body":"to Netflix in 2001, right?"},{"speaker":"So let's take an example","startTime":1625.7,"endTime":1627.023,"body":"Right when it started."},{"speaker":"So let's take an example","startTime":1628.31,"endTime":1629.93,"body":"Some of us may remember this,"},{"speaker":"So let's take an example","startTime":1629.93,"endTime":1632.72,"body":"but Netflix was not always"},{"speaker":"So let's take an example","startTime":1629.93,"endTime":1632.72,"body":"a streaming service."},{"speaker":"So let's take an example","startTime":1632.72,"endTime":1635.21,"body":"It started out with DVDs by mail."},{"speaker":"So let's take an example","startTime":1635.21,"endTime":1637.73,"body":"In the UK they called it LoveFilm."},{"speaker":"So let's take an example","startTime":1637.73,"endTime":1640.55,"body":"But basically the idea"},{"speaker":"So let's take an example","startTime":1637.73,"endTime":1640.55,"body":"was, you have a list"},{"speaker":"So let's take an example","startTime":1640.55,"endTime":1643.73,"body":"of all the movies you want"},{"speaker":"So let's take an example","startTime":1640.55,"endTime":1643.73,"body":"to watch on the website,"},{"speaker":"So let's take an example","startTime":1643.73,"endTime":1647.33,"body":"and Netflix will mail"},{"speaker":"So let's take an example","startTime":1643.73,"endTime":1647.33,"body":"you three DVDs, right?"},{"speaker":"So let's take an example","startTime":1647.33,"endTime":1649.13,"body":"And then you watch them at your own pace,"},{"speaker":"So let's take an example","startTime":1649.13,"endTime":1651.59,"body":"and then send them back and, you know,"},{"speaker":"So let's take an example","startTime":1651.59,"endTime":1654.65,"body":"Netflix charges you 10"},{"speaker":"So let's take an example","startTime":1651.59,"endTime":1654.65,"body":"bucks a month, right?"},{"speaker":"So let's take an example","startTime":1654.65,"endTime":1657.83,"body":"So what it was trying at that time"},{"speaker":"So let's take an example","startTime":1657.83,"endTime":1662.51,"body":"was persuading people to keep"},{"speaker":"So let's take an example","startTime":1657.83,"endTime":1662.51,"body":"paying that 10 bucks a month,"},{"speaker":"So let's take an example","startTime":1662.51,"endTime":1665.27,"body":"because it didn't want you to"},{"speaker":"So let's take an example","startTime":1662.51,"endTime":1665.27,"body":"actually borrow any movies,"},{"speaker":"So let's take an example","startTime":1665.27,"endTime":1667.7,"body":"so if you had, you know,"},{"speaker":"So let's take an example","startTime":1665.27,"endTime":1667.7,"body":"if you kept the same movie"},{"speaker":"So let's take an example","startTime":1667.7,"endTime":1671.63,"body":"for one month, two months, no"},{"speaker":"So let's take an example","startTime":1667.7,"endTime":1671.63,"body":"problem, there's no late fees."},{"speaker":"So let's take an example","startTime":1671.63,"endTime":1673.97,"body":"But at the same time,"},{"speaker":"So let's take an example","startTime":1673.97,"endTime":1676.04,"body":"it didn't want you to"},{"speaker":"So let's take an example","startTime":1673.97,"endTime":1676.04,"body":"stop paying the 10 bucks."},{"speaker":"So let's take an example","startTime":1676.04,"endTime":1679.73,"body":"So it wants to make you"},{"speaker":"So let's take an example","startTime":1676.04,"endTime":1679.73,"body":"continue watching the movies."},{"speaker":"So let's take an example","startTime":1679.73,"endTime":1680.563,"body":"Fine."},{"speaker":"So let's take an example","startTime":1682.16,"endTime":1684.32,"body":"How you persuade them to watch movies"},{"speaker":"So let's take an example","startTime":1684.32,"endTime":1686.51,"body":"is you recommend good movies to them."},{"speaker":"So let's take an example","startTime":1686.51,"endTime":1688.7,"body":"But how do you get those recommendations?"},{"speaker":"So let's take an example","startTime":1688.7,"endTime":1693.7,"body":"The first one was recommendations"},{"speaker":"So let's take an example","startTime":1688.7,"endTime":1693.7,"body":"by employees or editors."},{"speaker":"So let's take an example","startTime":1693.71,"endTime":1695.09,"body":"You know how well that works."},{"speaker":"So let's take an example","startTime":1695.09,"endTime":1697.43,"body":"I mean, you know how many"},{"speaker":"So let's take an example","startTime":1695.09,"endTime":1697.43,"body":"times do we read The Guardian"},{"speaker":"So let's take an example","startTime":1697.43,"endTime":1701.87,"body":"or we read, you know, the"},{"speaker":"So let's take an example","startTime":1697.43,"endTime":1701.87,"body":"FT or the New York Times,"},{"speaker":"So let's take an example","startTime":1701.87,"endTime":1703.91,"body":"and we get these movie recommendation,"},{"speaker":"So let's take an example","startTime":1703.91,"endTime":1704.847,"body":"then we watch it and you're like,"},{"speaker":"So let's take an example","startTime":1704.847,"endTime":1708.41,"body":"\"What the...this is awful!\" Right?"},{"speaker":"So let's take an example","startTime":1708.41,"endTime":1709.97,"body":"It happens to me all the time."},{"speaker":"So let's take an example","startTime":1709.97,"endTime":1711.46,"body":"At least, let me put it this way,"},{"speaker":"So let's take an example","startTime":1711.46,"endTime":1713.3,"body":"it happens more with my wife than with me,"},{"speaker":"So let's take an example","startTime":1713.3,"endTime":1714.68,"body":"because I pick the movies"},{"speaker":"So let's take an example","startTime":1714.68,"endTime":1716.547,"body":"and she looks at the movies and she says,"},{"speaker":"So let's take an example","startTime":1716.547,"endTime":1718.88,"body":"\"That was a horrible, terrible movie.\""},{"speaker":"So let's take an example","startTime":1718.88,"endTime":1721.37,"body":"I'm like, \"But the New"},{"speaker":"So let's take an example","startTime":1718.88,"endTime":1721.37,"body":"York Times loved it!\""},{"speaker":"So let's take an example","startTime":1721.37,"endTime":1723.47,"body":"I never want to see a New"},{"speaker":"So let's take an example","startTime":1721.37,"endTime":1723.47,"body":"York Times movie again."},{"speaker":"So let's take an example","startTime":1723.47,"endTime":1726.68,"body":"So the moral of the story is"},{"speaker":"So let's take an example","startTime":1723.47,"endTime":1726.68,"body":"this doesn't work very well."},{"speaker":"So let's take an example","startTime":1726.68,"endTime":1729.47,"body":"And of course people were hacking"},{"speaker":"So let's take an example","startTime":1726.68,"endTime":1729.47,"body":"these things all the time."},{"speaker":"So let's take an example","startTime":1729.47,"endTime":1731.78,"body":"So I'll give you another example."},{"speaker":"So let's take an example","startTime":1731.78,"endTime":1734.09,"body":"If you have a brand new"},{"speaker":"So let's take an example","startTime":1731.78,"endTime":1734.09,"body":"movie, just been released,"},{"speaker":"So let's take an example","startTime":1734.09,"endTime":1736.46,"body":"like, let's say \"Avatar,\" right?"},{"speaker":"So let's take an example","startTime":1736.46,"endTime":1738.83,"body":"Just released a couple of months ago."},{"speaker":"So let's take an example","startTime":1738.83,"endTime":1740.78,"body":"The recent DVD,"},{"speaker":"So let's take an example","startTime":1740.78,"endTime":1743.15,"body":"if two people have it on their list,"},{"speaker":"So let's take an example","startTime":1743.15,"endTime":1745.28,"body":"who gets that movie first?"},{"speaker":"So let's take an example","startTime":1745.28,"endTime":1747.53,"body":"Well, the newer person,"},{"speaker":"So let's take an example","startTime":1747.53,"endTime":1750.86,"body":"because I don't have"},{"speaker":"So let's take an example","startTime":1747.53,"endTime":1750.86,"body":"enough data on him yet."},{"speaker":"So let's take an example","startTime":1750.86,"endTime":1754.16,"body":"The person who's already"},{"speaker":"So let's take an example","startTime":1750.86,"endTime":1754.16,"body":"been in the system for a year"},{"speaker":"So let's take an example","startTime":1754.16,"endTime":1755.75,"body":"does not get a loyalty bonus."},{"speaker":"So let's take an example","startTime":1755.75,"endTime":1757.52,"body":"I discriminate against them"},{"speaker":"So let's take an example","startTime":1757.52,"endTime":1759.77,"body":"by sending them a different"},{"speaker":"So let's take an example","startTime":1757.52,"endTime":1759.77,"body":"movie from their list"},{"speaker":"So let's take an example","startTime":1759.77,"endTime":1761.81,"body":"because I'm trying to"},{"speaker":"So let's take an example","startTime":1759.77,"endTime":1761.81,"body":"build up my information"},{"speaker":"So let's take an example","startTime":1761.81,"endTime":1763.61,"body":"on the new guy first."},{"speaker":"So let's take an example","startTime":1763.61,"endTime":1765.26,"body":"So people knew this, of course."},{"speaker":"So let's take an example","startTime":1765.26,"endTime":1766.88,"body":"So what they would do is, you know,"},{"speaker":"So let's take an example","startTime":1766.88,"endTime":1768.77,"body":"cancel your Netflix subscription"},{"speaker":"So let's take an example","startTime":1768.77,"endTime":1770.42,"body":"and then sign up under a new name."},{"speaker":"So let's take an example","startTime":1770.42,"endTime":1773.24,"body":"So you are always a new subscriber."},{"speaker":"So let's take an example","startTime":1773.24,"endTime":1776.6,"body":"Or you would match together"},{"speaker":"So let's take an example","startTime":1773.24,"endTime":1776.6,"body":"a group of your friends,"},{"speaker":"So let's take an example","startTime":1776.6,"endTime":1778.85,"body":"you order the movie, one of you gets it,"},{"speaker":"So let's take an example","startTime":1778.85,"endTime":1781.4,"body":"you share it first, and"},{"speaker":"So let's take an example","startTime":1778.85,"endTime":1781.4,"body":"then you return it, right?"},{"speaker":"So let's take an example","startTime":1781.4,"endTime":1783.83,"body":"So pretty standard way."},{"speaker":"So let's take an example","startTime":1783.83,"endTime":1786.47,"body":"So they were trying to solve"},{"speaker":"So let's take an example","startTime":1783.83,"endTime":1786.47,"body":"these problems like this."},{"speaker":"So let's take an example","startTime":1786.47,"endTime":1788.99,"body":"Keep people hooked, keep people staying,"},{"speaker":"So let's take an example","startTime":1788.99,"endTime":1792.47,"body":"pay the $10 a month and"},{"speaker":"So let's take an example","startTime":1788.99,"endTime":1792.47,"body":"stay within the system."},{"speaker":"So let's take an example","startTime":1792.47,"endTime":1795.353,"body":"So they came up with a"},{"speaker":"So let's take an example","startTime":1792.47,"endTime":1795.353,"body":"system called Cinematch."},{"speaker":"So let's take an example","startTime":1796.67,"endTime":1798.32,"body":"How did that work?"},{"speaker":"So let's take an example","startTime":1798.32,"endTime":1800.3,"body":"Something called"},{"speaker":"So let's take an example","startTime":1798.32,"endTime":1800.3,"body":"\"collaborative filtering.\""},{"speaker":"So let's take an example","startTime":1800.3,"endTime":1802.97,"body":"And collaborative"},{"speaker":"So let's take an example","startTime":1800.3,"endTime":1802.97,"body":"filtering is a popular way"},{"speaker":"So let's take an example","startTime":1802.97,"endTime":1805.76,"body":"in which you can actually try to find out,"},{"speaker":"So let's take an example","startTime":1805.76,"endTime":1809.6,"body":"you know, how do you"},{"speaker":"So let's take an example","startTime":1805.76,"endTime":1809.6,"body":"recommend things to people?"},{"speaker":"So let's take an example","startTime":1809.6,"endTime":1811.7,"body":"So let's take a simple example."},{"speaker":"So let's take an example","startTime":1811.7,"endTime":1814.34,"body":"You have two people, two movies."},{"speaker":"So let's take an example","startTime":1814.34,"endTime":1816.59,"body":"We have Pauline who has rated"},{"speaker":"So let's take an example","startTime":1816.59,"endTime":1818.63,"body":"both \"Avengers\" and \"Spiderman.\""},{"speaker":"So let's take an example","startTime":1818.63,"endTime":1821.33,"body":"2 for \"Avengers,\" 2.5 to \"Spiderman.\""},{"speaker":"So let's take an example","startTime":1821.33,"endTime":1823.88,"body":"Julien has rated \"Avengers\" as 3."},{"speaker":"So let's take an example","startTime":1823.88,"endTime":1828.47,"body":"The question is, what would"},{"speaker":"So let's take an example","startTime":1823.88,"endTime":1828.47,"body":"Julien rate \"Spiderman\" as?"},{"speaker":"So let's take an example","startTime":1828.47,"endTime":1831.08,"body":"The easiest one, the"},{"speaker":"So let's take an example","startTime":1828.47,"endTime":1831.08,"body":"slope one strategy says,"},{"speaker":"So let's take an example","startTime":1831.08,"endTime":1832.58,"body":"these two people are alike,"},{"speaker":"So let's take an example","startTime":1832.58,"endTime":1834.11,"body":"they will rate it in the same way."},{"speaker":"So let's take an example","startTime":1834.11,"endTime":1838.52,"body":"So you could say 2 to"},{"speaker":"So let's take an example","startTime":1834.11,"endTime":1838.52,"body":"2.5; 3 to 3.5, maybe."},{"speaker":"So let's take an example","startTime":1838.52,"endTime":1843.52,"body":"Or this is a 25% increase,"},{"speaker":"So let's take an example","startTime":1838.52,"endTime":1843.52,"body":"3 to 3.75, right?"},{"speaker":"So let's take an example","startTime":1843.83,"endTime":1846.893,"body":"You have to pick a strategy,"},{"speaker":"So let's take an example","startTime":1843.83,"endTime":1846.893,"body":"but that's the basic story."},{"speaker":"So let's take an example","startTime":1848.12,"endTime":1849.92,"body":"But then it becomes more complicated."},{"speaker":"So let's take an example","startTime":1849.92,"endTime":1851.54,"body":"So for example, now let's suppose"},{"speaker":"So let's take an example","startTime":1851.54,"endTime":1854.483,"body":"we have three movies and three people."},{"speaker":"So let's take an example","startTime":1855.56,"endTime":1857.54,"body":"Cesar has seen all three."},{"speaker":"So let's take an example","startTime":1857.54,"endTime":1858.92,"body":"Blanche has only seen two."},{"speaker":"So let's take an example","startTime":1858.92,"endTime":1860.81,"body":"And Emma has only seen two."},{"speaker":"So let's take an example","startTime":1860.81,"endTime":1863.75,"body":"The question is, how"},{"speaker":"So let's take an example","startTime":1860.81,"endTime":1863.75,"body":"does Emma rate \"Avengers\""},{"speaker":"So let's take an example","startTime":1863.75,"endTime":1867.02,"body":"versus how does Blanche"},{"speaker":"So let's take an example","startTime":1863.75,"endTime":1867.02,"body":"rate \"Wonder Woman?\""},{"speaker":"So let's take an example","startTime":1867.02,"endTime":1868.64,"body":"Okay, so what do we do?"},{"speaker":"So let's take an example","startTime":1868.64,"endTime":1871.7,"body":"One way is, okay, let's"},{"speaker":"So let's take an example","startTime":1868.64,"endTime":1871.7,"body":"take a look at these two."},{"speaker":"So let's take an example","startTime":1871.7,"endTime":1873.89,"body":"Cesar likes \"Avengers\""},{"speaker":"So let's take an example","startTime":1871.7,"endTime":1873.89,"body":"more than \"Spiderman.\""},{"speaker":"So let's take an example","startTime":1873.89,"endTime":1876.56,"body":"He gives it a rating of 2 higher,"},{"speaker":"So let's take an example","startTime":1876.56,"endTime":1879.65,"body":"but Blanche gives it a rating of 1 lower."},{"speaker":"So let's take the average","startTime":1881.15,"endTime":1884.39,"body":"2 higher, 1 lower, divided by two, is 0.5."},{"speaker":"So let's take the average","startTime":1884.39,"endTime":1888.71,"body":"So Emma would be 1 plus 0.5, is 1.5."},{"speaker":"So let's take the average","startTime":1888.71,"endTime":1890.6,"body":"That's one way of doing it."},{"speaker":"So let's take the average","startTime":1890.6,"endTime":1892.58,"body":"But you could also go with Cesar."},{"speaker":"So let's take the average","startTime":1892.58,"endTime":1897.58,"body":"Cesar likes \"Avengers\" 4 over 1."},{"speaker":"So let's take the average","startTime":1897.89,"endTime":1899.3,"body":"So there's a difference of 3."},{"speaker":"So let's take the average","startTime":1899.3,"endTime":1903.74,"body":"So this is 4 plus 3 is"},{"speaker":"So let's take the average","startTime":1899.3,"endTime":1903.74,"body":"7, that's another number."},{"speaker":"So let's take the average","startTime":1903.74,"endTime":1906.98,"body":"So you've got 1.5 under"},{"speaker":"So let's take the average","startTime":1903.74,"endTime":1906.98,"body":"one matching algorithm."},{"speaker":"So let's take the average","startTime":1906.98,"endTime":1910.07,"body":"You've got 7 under a"},{"speaker":"So let's take the average","startTime":1906.98,"endTime":1910.07,"body":"different matching algorithm."},{"speaker":"So let's take the average","startTime":1910.07,"endTime":1911.63,"body":"And then you weight them."},{"speaker":"So let's take the average","startTime":1911.63,"endTime":1913.22,"body":"Two-thirds the weight for the first,"},{"speaker":"So let's take the average","startTime":1913.22,"endTime":1914.9,"body":"one-third the weight for the second,"},{"speaker":"So let's take the average","startTime":1914.9,"endTime":1917.21,"body":"because there are two"},{"speaker":"So let's take the average","startTime":1914.9,"endTime":1917.21,"body":"sources for the first one,"},{"speaker":"So let's take the average","startTime":1917.21,"endTime":1918.44,"body":"one source for the second."},{"speaker":"So let's take the average","startTime":1918.44,"endTime":1921.65,"body":"And, you know, you can get"},{"speaker":"So let's take the average","startTime":1918.44,"endTime":1921.65,"body":"as complicated as you want."},{"speaker":"So let's take the average","startTime":1921.65,"endTime":1922.94,"body":"Then you start saying, okay,"},{"speaker":"So let's take the average","startTime":1922.94,"endTime":1925.37,"body":"which of these people are"},{"speaker":"So let's take the average","startTime":1922.94,"endTime":1925.37,"body":"so close to each other?"},{"speaker":"So let's take the average","startTime":1925.37,"endTime":1927.83,"body":"Whose ratings are really"},{"speaker":"So let's take the average","startTime":1925.37,"endTime":1927.83,"body":"close to each other?"},{"speaker":"So let's take the average","startTime":1927.83,"endTime":1928.97,"body":"Let's take those."},{"speaker":"So let's take the average","startTime":1928.97,"endTime":1933.97,"body":"Or use, you know, customers"},{"speaker":"So let's take the average","startTime":1928.97,"endTime":1933.97,"body":"only in the same cluster"},{"speaker":"So let's take the average","startTime":1934.1,"endTime":1935.3,"body":"to make predictions."},{"speaker":"So let's take the average","startTime":1935.3,"endTime":1940.3,"body":"So the algorithm starts becoming"},{"speaker":"So let's take the average","startTime":1935.3,"endTime":1940.3,"body":"more and more complicated."},{"speaker":"So let's take the average","startTime":1941.75,"endTime":1942.95,"body":"All right."},{"speaker":"So let's take the average","startTime":1942.95,"endTime":1946.46,"body":"They used what they called an"},{"speaker":"So let's take the average","startTime":1942.95,"endTime":1946.46,"body":"ordinal logit model, right?"},{"speaker":"So let's take the average","startTime":1946.46,"endTime":1947.9,"body":"Just a technical term."},{"speaker":"So let's take the average","startTime":1947.9,"endTime":1950.87,"body":"All that means is the dependent variable,"},{"speaker":"So let's take the average","startTime":1950.87,"endTime":1952.04,"body":"which you're trying to predict"},{"speaker":"So let's take the average","startTime":1952.04,"endTime":1954.32,"body":"how much these people like the system,"},{"speaker":"So let's take the average","startTime":1954.32,"endTime":1957.392,"body":"is on a scale of one to five, right?"},{"speaker":"So let's take the average","startTime":1957.392,"endTime":1960.26,"body":"So that all we know is"},{"speaker":"So let's take the average","startTime":1957.392,"endTime":1960.26,"body":"five is greater than four,"},{"speaker":"So let's take the average","startTime":1960.26,"endTime":1962.15,"body":"three is greater than two, and so on."},{"speaker":"So let's take the average","startTime":1962.15,"endTime":1966.83,"body":"But the key difference is, the"},{"speaker":"So let's take the average","startTime":1962.15,"endTime":1966.83,"body":"differences are not the same."},{"speaker":"So let's take the average","startTime":1966.83,"endTime":1970.857,"body":"For example, if you think,"},{"speaker":"So let's take the average","startTime":1966.83,"endTime":1970.857,"body":"some people might say,"},{"speaker":"So let's take the average","startTime":1970.857,"endTime":1974.63,"body":"\"Five is really, really"},{"speaker":"So let's take the average","startTime":1970.857,"endTime":1974.63,"body":"unbelievably good.\""},{"speaker":"So let's take the average","startTime":1974.63,"endTime":1977.54,"body":"Very few movies will get a rating of five."},{"speaker":"So let's take the average","startTime":1977.54,"endTime":1980.81,"body":"So the difference between"},{"speaker":"So let's take the average","startTime":1977.54,"endTime":1980.81,"body":"four and five is really big,"},{"speaker":"So let's take the average","startTime":1980.81,"endTime":1983.48,"body":"the difference between"},{"speaker":"So let's take the average","startTime":1980.81,"endTime":1983.48,"body":"three and four not so big."},{"speaker":"So let's take the average","startTime":1983.48,"endTime":1985.76,"body":"Below three, they're all"},{"speaker":"So let's take the average","startTime":1983.48,"endTime":1985.76,"body":"the same, it doesn't matter."},{"speaker":"So let's take the average","startTime":1985.76,"endTime":1987.32,"body":"So you know, you just..."},{"speaker":"So let's take the average","startTime":1987.32,"endTime":1991.07,"body":"So you have to adjust"},{"speaker":"So let's take the average","startTime":1987.32,"endTime":1991.07,"body":"for all of these things."},{"speaker":"So let's take the average","startTime":1991.07,"endTime":1994.07,"body":"So what are the problems they had?"},{"speaker":"So let's take the average","startTime":1994.07,"endTime":1996.95,"body":"First problem was the"},{"speaker":"So let's take the average","startTime":1994.07,"endTime":1996.95,"body":"cold start problem, right?"},{"speaker":"So let's take the average","startTime":1996.95,"endTime":1999.89,"body":"The cold start problem is,"},{"speaker":"So let's take the average","startTime":1996.95,"endTime":1999.89,"body":"a movie's just been added,"},{"speaker":"So let's take the average","startTime":1999.89,"endTime":2001.18,"body":"how do you rate it?"},{"speaker":"So let's take the average","startTime":2001.18,"endTime":2002.8,"body":"'Cause nobody's watched it yet."},{"speaker":"So let's take the average","startTime":2002.8,"endTime":2006.28,"body":"Or some new user's just"},{"speaker":"So let's take the average","startTime":2002.8,"endTime":2006.28,"body":"arrived, he has no preferences."},{"speaker":"So let's take the average","startTime":2006.28,"endTime":2009.55,"body":"How do we rate him, what"},{"speaker":"So let's take the average","startTime":2006.28,"endTime":2009.55,"body":"do you predict for him?"},{"speaker":"So let's take the average","startTime":2009.55,"endTime":2012.16,"body":"Second problem, popularity bias."},{"speaker":"So let's take the average","startTime":2012.16,"endTime":2015.52,"body":"So the more popular a movie is"},{"speaker":"So let's take the average","startTime":2012.16,"endTime":2015.52,"body":"the more people will rate it."},{"speaker":"So let's take the average","startTime":2015.52,"endTime":2017.2,"body":"So you'll end up"},{"speaker":"So let's take the average","startTime":2017.2,"endTime":2019.81,"body":"just recommending popular"},{"speaker":"So let's take the average","startTime":2017.2,"endTime":2019.81,"body":"movies to everybody."},{"speaker":"So let's take the average","startTime":2019.81,"endTime":2022.303,"body":"The older movies, nobody really gets."},{"speaker":"So let's take the average","startTime":2023.14,"endTime":2025.57,"body":"Third one, sparse data problem."},{"speaker":"So let's take the average","startTime":2025.57,"endTime":2029.89,"body":"It's a huge database, but"},{"speaker":"So let's take the average","startTime":2025.57,"endTime":2029.89,"body":"most people don't rate movies."},{"speaker":"So let's take the average","startTime":2029.89,"endTime":2034.12,"body":"So you have mostly zeros, only"},{"speaker":"So let's take the average","startTime":2029.89,"endTime":2034.12,"body":"a few data points in between."},{"speaker":"So let's take the average","startTime":2034.12,"endTime":2036.58,"body":"And finally, noisy data problem."},{"speaker":"So let's take the average","startTime":2036.58,"endTime":2038.44,"body":"You might love a movie,"},{"speaker":"So let's take the average","startTime":2038.44,"endTime":2039.94,"body":"but you might feel a little embarrassed"},{"speaker":"So let's take the average","startTime":2039.94,"endTime":2042.04,"body":"to tell your friends you"},{"speaker":"So let's take the average","startTime":2039.94,"endTime":2042.04,"body":"really liked that movie."},{"speaker":"So let's take the average","startTime":2042.04,"endTime":2045.85,"body":"So you tell Netflix that"},{"speaker":"So let's take the average","startTime":2042.04,"endTime":2045.85,"body":"you liked it, or you didn't,"},{"speaker":"So let's take the average","startTime":2045.85,"endTime":2049.66,"body":"but you're thinking, that movie, you know,"},{"speaker":"So let's take the average","startTime":2049.66,"endTime":2053.11,"body":"very different from what I"},{"speaker":"So let's take the average","startTime":2049.66,"endTime":2053.11,"body":"really felt about the movie."},{"speaker":"So let's take the average","startTime":2053.11,"endTime":2055.57,"body":"Okay, so how did Cinematch deal with that?"},{"speaker":"So let's take the average","startTime":2055.57,"endTime":2057.73,"body":"They called it the alternative"},{"speaker":"So let's take the average","startTime":2055.57,"endTime":2057.73,"body":"least squares model."},{"speaker":"So let's take the average","startTime":2057.73,"endTime":2059.26,"body":"This is getting more and more complex."},{"speaker":"So let's take the average","startTime":2059.26,"endTime":2061.36,"body":"It's just ways of manipulating data."},{"speaker":"So let's take the average","startTime":2061.36,"endTime":2063.46,"body":"And what you have is"},{"speaker":"So let's take the average","startTime":2061.36,"endTime":2063.46,"body":"a complicated database"},{"speaker":"So let's take the average","startTime":2063.46,"endTime":2064.87,"body":"that looks like this."},{"speaker":"So let's take the average","startTime":2064.87,"endTime":2066.67,"body":"People, movies."},{"speaker":"So let's take the average","startTime":2066.67,"endTime":2069.43,"body":"And you're trying to fill"},{"speaker":"So let's take the average","startTime":2066.67,"endTime":2069.43,"body":"in these missing bits."},{"speaker":"So let's take the average","startTime":2069.43,"endTime":2072.82,"body":"So what they did was"},{"speaker":"So let's take the average","startTime":2069.43,"endTime":2072.82,"body":"separate into two databases."},{"speaker":"So let's take the average","startTime":2072.82,"endTime":2074.68,"body":"One is called a user matrix"},{"speaker":"So let's take the average","startTime":2074.68,"endTime":2076.63,"body":"and the other is called a movie matrix."},{"speaker":"So let's take the average","startTime":2076.63,"endTime":2079.24,"body":"So you fill in average numbers here,"},{"speaker":"So let's take the average","startTime":2079.24,"endTime":2080.77,"body":"random numbers here,"},{"speaker":"So let's take the average","startTime":2080.77,"endTime":2083.11,"body":"and then you multiply these two together"},{"speaker":"So let's take the average","startTime":2083.11,"endTime":2084.79,"body":"to see how close you're getting"},{"speaker":"So let's take the average","startTime":2084.79,"endTime":2087.4,"body":"to the final database, right?"},{"speaker":"So let's take the average","startTime":2087.4,"endTime":2091.09,"body":"That difference is what"},{"speaker":"So let's take the average","startTime":2087.4,"endTime":2091.09,"body":"you're trying to minimize."},{"speaker":"So let's take the average","startTime":2091.09,"endTime":2094.12,"body":"So that's called the root"},{"speaker":"So let's take the average","startTime":2091.09,"endTime":2094.12,"body":"mean square error, right?"},{"speaker":"So let's take the average","startTime":2094.12,"endTime":2095.83,"body":"So that's what we are going to try to do."},{"speaker":"So let's take the average","startTime":2095.83,"endTime":2097.54,"body":"We're going to minimize that."},{"speaker":"So let's take the average","startTime":2097.54,"endTime":2101.08,"body":"All right, so what did Netflix"},{"speaker":"So let's take the average","startTime":2097.54,"endTime":2101.08,"body":"need this big data for?"},{"speaker":"So let's take the average","startTime":2101.08,"endTime":2104.26,"body":"As I said, they wanted to keep"},{"speaker":"So let's take the average","startTime":2101.08,"endTime":2104.26,"body":"you within their ecosystem,"},{"speaker":"So let's take the average","startTime":2104.26,"endTime":2105.34,"body":"but they had other issues."},{"speaker":"So let's take the average","startTime":2105.34,"endTime":2107.507,"body":"They wanted new shows to commission."},{"speaker":"So let's take the average","startTime":2107.507,"endTime":2111.1,"body":"\"House of Cards\" was probably"},{"speaker":"So let's take the average","startTime":2107.507,"endTime":2111.1,"body":"the one which people say,"},{"speaker":"So let's take the average","startTime":2111.1,"endTime":2113.86,"body":"okay, that was the one where"},{"speaker":"So let's take the average","startTime":2111.1,"endTime":2113.86,"body":"Netflix really used big data"},{"speaker":"So let's take the average","startTime":2113.86,"endTime":2118.0,"body":"to predict that, you know,"},{"speaker":"So let's take the average","startTime":2113.86,"endTime":2118.0,"body":"this is a great movie,"},{"speaker":"So let's take the average","startTime":2118.0,"endTime":2120.07,"body":"a great series, which people will watch."},{"speaker":"So let's take the average","startTime":2120.07,"endTime":2123.01,"body":"Every other studio had tuned it down."},{"speaker":"So let's take the average","startTime":2123.01,"endTime":2126.013,"body":"Or, how much to pay for new films, right?"},{"speaker":"So let's take the average","startTime":2126.97,"endTime":2130.72,"body":"Or, how do you get people"},{"speaker":"So let's take the average","startTime":2126.97,"endTime":2130.72,"body":"to actively rate a movie,"},{"speaker":"So let's take the average","startTime":2130.72,"endTime":2134.29,"body":"as opposed to just passively"},{"speaker":"So let's take the average","startTime":2130.72,"endTime":2134.29,"body":"watching the movie, right?"},{"speaker":"So let's take the average","startTime":2134.29,"endTime":2137.38,"body":"Now, of course, it's"},{"speaker":"So let's take the average","startTime":2134.29,"endTime":2137.38,"body":"much more sophisticated."},{"speaker":"So let's take the average","startTime":2137.38,"endTime":2139.693,"body":"Netflix has tagged every movie."},{"speaker":"So let's take the average","startTime":2141.07,"endTime":2143.237,"body":"Somebody has watched it"},{"speaker":"So let's take the average","startTime":2141.07,"endTime":2143.237,"body":"and tagged everything:"},{"speaker":"So let's take the average","startTime":2143.237,"endTime":2147.4,"body":"\"Chinese American blinks eyes"},{"speaker":"So let's take the average","startTime":2143.237,"endTime":2147.4,"body":"while doing this,\" right?"},{"speaker":"So let's take the average","startTime":2147.4,"endTime":2150.19,"body":"And so when you are watching a movie,"},{"speaker":"So let's take the average","startTime":2150.19,"endTime":2151.6,"body":"and you pause or you click,"},{"speaker":"So let's take the average","startTime":2151.6,"endTime":2153.4,"body":"it knows exactly when you're pausing."},{"speaker":"So let's take the average","startTime":2153.4,"endTime":2156.4,"body":"And it's got all the tags"},{"speaker":"So let's take the average","startTime":2153.4,"endTime":2156.4,"body":"on the movies already."},{"speaker":"So let's take the average","startTime":2156.4,"endTime":2159.28,"body":"So it knows everything"},{"speaker":"So let's take the average","startTime":2156.4,"endTime":2159.28,"body":"about who watches a movie,"},{"speaker":"So let's take the average","startTime":2159.28,"endTime":2160.54,"body":"at what point they stop,"},{"speaker":"So let's take the average","startTime":2160.54,"endTime":2163.66,"body":"when they go on to a different"},{"speaker":"So let's take the average","startTime":2160.54,"endTime":2163.66,"body":"movie, everything, okay?"},{"speaker":"So let's take the average","startTime":2163.66,"endTime":2165.55,"body":"So more sophisticated."},{"speaker":"So let's take the average","startTime":2165.55,"endTime":2167.71,"body":"But at that time, they didn't have this."},{"speaker":"So let's take the average","startTime":2167.71,"endTime":2169.66,"body":"This was the DVD days."},{"speaker":"So let's take the average","startTime":2169.66,"endTime":2173.32,"body":"So what they did was"},{"speaker":"So let's take the average","startTime":2169.66,"endTime":2173.32,"body":"announce a Netflix prize."},{"speaker":"So let's take the average","startTime":2173.32,"endTime":2174.73,"body":"It wasn't a new thing, right?"},{"speaker":"So let's take the average","startTime":2174.73,"endTime":2177.88,"body":"Because remember, this has"},{"speaker":"So let's take the average","startTime":2174.73,"endTime":2177.88,"body":"been going on for centuries."},{"speaker":"So let's take the average","startTime":2177.88,"endTime":2180.87,"body":"So in 1418, the Duomo in Firenze..."},{"speaker":"So let's take the average","startTime":2184.33,"endTime":2187.72,"body":"The city announced a"},{"speaker":"So let's take the average","startTime":2184.33,"endTime":2187.72,"body":"competition to build the Duomo,"},{"speaker":"So let's take the average","startTime":2187.72,"endTime":2190.54,"body":"won by Brunelleschi, at that time."},{"speaker":"So let's take the average","startTime":2190.54,"endTime":2194.5,"body":"In 1741, the Longitude Act was"},{"speaker":"So let's take the average","startTime":2190.54,"endTime":2194.5,"body":"to try to find a precise way"},{"speaker":"So let's take the average","startTime":2194.5,"endTime":2196.63,"body":"of getting longitude from"},{"speaker":"So let's take the average","startTime":2194.5,"endTime":2196.63,"body":"anywhere in the world."},{"speaker":"So let's take the average","startTime":2196.63,"endTime":2198.31,"body":"Won by John Harrison."},{"speaker":"So let's take the average","startTime":2198.31,"endTime":2201.79,"body":"The X-Prize in 1995 offered"},{"speaker":"So let's take the average","startTime":2198.31,"endTime":2201.79,"body":"a whole bunch of prizes"},{"speaker":"So let's take the average","startTime":2201.79,"endTime":2203.23,"body":"for different things."},{"speaker":"So let's take the average","startTime":2203.23,"endTime":2205.81,"body":"And you can still see other prizes today."},{"speaker":"So let's take the average","startTime":2205.81,"endTime":2207.76,"body":"Go to InnoCentive.com."},{"speaker":"So let's take the average","startTime":2207.76,"endTime":2209.26,"body":"You can go in there"},{"speaker":"So let's take the average","startTime":2209.26,"endTime":2212.47,"body":"and try to find examples of prizes to win."},{"speaker":"So let's take the average","startTime":2212.47,"endTime":2215.32,"body":"So Netflix had a big problem, right?"},{"speaker":"So let's take the average","startTime":2215.32,"endTime":2216.507,"body":"How do we design this thing?"},{"speaker":"So let's take the average","startTime":2216.507,"endTime":2218.32,"body":"We are trying to collect this data."},{"speaker":"So let's take the average","startTime":2218.32,"endTime":2220.84,"body":"We're trying to solve"},{"speaker":"So let's take the average","startTime":2218.32,"endTime":2220.84,"body":"the big data problem."},{"speaker":"So let's take the average","startTime":2220.84,"endTime":2222.52,"body":"Should I use my own platform?"},{"speaker":"So let's take the average","startTime":2222.52,"endTime":2225.31,"body":"Then people will say, \"They"},{"speaker":"So let's take the average","startTime":2222.52,"endTime":2225.31,"body":"can manipulate the data.\""},{"speaker":"So let's take the average","startTime":2225.31,"endTime":2228.85,"body":"Should I use an outside"},{"speaker":"So let's take the average","startTime":2225.31,"endTime":2228.85,"body":"host to be more neutral?"},{"speaker":"So let's take the average","startTime":2228.85,"endTime":2230.443,"body":"How much data should I release?"},{"speaker":"So let's take the average","startTime":2231.692,"endTime":2233.17,"body":"If I release too much data,"},{"speaker":"So let's take the average","startTime":2233.17,"endTime":2235.97,"body":"it's confidential data, it'll"},{"speaker":"So let's take the average","startTime":2233.17,"endTime":2235.97,"body":"be available to everybody."},{"speaker":"So let's take the average","startTime":2237.25,"endTime":2238.54,"body":"What about customer privacy?"},{"speaker":"So let's take the average","startTime":2238.54,"endTime":2240.82,"body":"They can back-engineer"},{"speaker":"So let's take the average","startTime":2238.54,"endTime":2240.82,"body":"who these people are."},{"speaker":"So let's take the average","startTime":2240.82,"endTime":2242.86,"body":"And that actually did happen to Netflix."},{"speaker":"So let's take the average","startTime":2242.86,"endTime":2243.693,"body":"They got sued."},{"speaker":"So let's take the average","startTime":2244.69,"endTime":2246.43,"body":"Intellectual property, right?"},{"speaker":"So let's take the average","startTime":2246.43,"endTime":2249.913,"body":"Is it an anonymous contest,"},{"speaker":"So let's take the average","startTime":2246.43,"endTime":2249.913,"body":"is it an open contest?"},{"speaker":"So let's take the average","startTime":2251.982,"endTime":2253.427,"body":"What happens if the winner says,"},{"speaker":"So let's take the average","startTime":2253.427,"endTime":2255.49,"body":"\"I own the intellectual property here."},{"speaker":"So let's take the average","startTime":2255.49,"endTime":2258.04,"body":"You want me to give you"},{"speaker":"So let's take the average","startTime":2255.49,"endTime":2258.04,"body":"my intellectual property,"},{"speaker":"So let's take the average","startTime":2258.04,"endTime":2259.48,"body":"give me more money.\""},{"speaker":"So let's take the average","startTime":2259.48,"endTime":2261.64,"body":"Or what about the losing solution?"},{"speaker":"So let's take the average","startTime":2261.64,"endTime":2262.93,"body":"It may be interesting."},{"speaker":"So let's take the average","startTime":2262.93,"endTime":2265.51,"body":"Just not good enough, but"},{"speaker":"So let's take the average","startTime":2262.93,"endTime":2265.51,"body":"it may be interesting."},{"speaker":"So let's take the average","startTime":2265.51,"endTime":2268.0,"body":"So who owns the intellectual property?"},{"speaker":"So let's take the average","startTime":2268.0,"endTime":2270.01,"body":"And what happens if the algorithm"},{"speaker":"So let's take the average","startTime":2270.01,"endTime":2271.96,"body":"is stolen by your competitors, right?"},{"speaker":"So let's take the average","startTime":2271.96,"endTime":2274.753,"body":"So a huge number of problems they had."},{"speaker":"So let's take the average","startTime":2275.68,"endTime":2278.08,"body":"They had worries about"},{"speaker":"So let's take the average","startTime":2275.68,"endTime":2278.08,"body":"designing the problem as well."},{"speaker":"So let's take the average","startTime":2278.08,"endTime":2279.73,"body":"How do you specify the problem?"},{"speaker":"So let's take the average","startTime":2279.73,"endTime":2282.61,"body":"It has to be very clearly defined, right?"},{"speaker":"So let's take the average","startTime":2282.61,"endTime":2284.2,"body":"How big an increase"},{"speaker":"So let's take the average","startTime":2284.2,"endTime":2286.93,"body":"in the root mean square"},{"speaker":"So let's take the average","startTime":2284.2,"endTime":2286.93,"body":"error should they have?"},{"speaker":"So let's take the average","startTime":2286.93,"endTime":2289.0,"body":"How long should the contest be kept open?"},{"speaker":"So let's take the average","startTime":2289.0,"endTime":2291.43,"body":"If you say, \"I want a 10% increase,\""},{"speaker":"So let's take the average","startTime":2291.43,"endTime":2293.65,"body":"but nobody manages to get there,"},{"speaker":"So let's take the average","startTime":2293.65,"endTime":2296.02,"body":"people will lose interest"},{"speaker":"So let's take the average","startTime":2293.65,"endTime":2296.02,"body":"in two or three years."},{"speaker":"So let's take the average","startTime":2296.02,"endTime":2299.14,"body":"So how long do you keep"},{"speaker":"So let's take the average","startTime":2296.02,"endTime":2299.14,"body":"the contest open, right?"},{"speaker":"So let's take the average","startTime":2299.14,"endTime":2300.88,"body":"What happens if there"},{"speaker":"So let's take the average","startTime":2299.14,"endTime":2300.88,"body":"are multiple winners?"},{"speaker":"So let's take the average","startTime":2300.88,"endTime":2302.8,"body":"What should the size of the award be?"},{"speaker":"So let's take the average","startTime":2302.8,"endTime":2304.51,"body":"They had to solve all these problems."},{"speaker":"So let's take the average","startTime":2304.51,"endTime":2307.51,"body":"And eventually, in October, 2006,"},{"speaker":"So let's take the average","startTime":2307.51,"endTime":2310.0,"body":"they announced a Netflix"},{"speaker":"So let's take the average","startTime":2307.51,"endTime":2310.0,"body":"prize, a million dollars,"},{"speaker":"So let's take the average","startTime":2310.0,"endTime":2312.425,"body":"invited the public to devise"},{"speaker":"So let's take the average","startTime":2310.0,"endTime":2312.425,"body":"a recommendation algorithm"},{"speaker":"So let's take the average","startTime":2312.425,"endTime":2316.99,"body":"that could beat their"},{"speaker":"So let's take the average","startTime":2312.425,"endTime":2316.99,"body":"Cinematch program by 10%."},{"speaker":"So let's take the average","startTime":2316.99,"endTime":2319.24,"body":"And the contest would"},{"speaker":"So let's take the average","startTime":2316.99,"endTime":2319.24,"body":"be open for five years."},{"speaker":"So let's take the average","startTime":2320.23,"endTime":2322.12,"body":"That was the thing they put up."},{"speaker":"So let's take the average","startTime":2322.12,"endTime":2323.71,"body":"That was the Netflix prize."},{"speaker":"So let's take the average","startTime":2323.71,"endTime":2325.09,"body":"You could just click there."},{"speaker":"So let's take the average","startTime":2325.09,"endTime":2328.24,"body":"They basically had every"},{"speaker":"So let's take the average","startTime":2325.09,"endTime":2328.24,"body":"piece of data that they had,"},{"speaker":"So let's take the average","startTime":2328.24,"endTime":2330.82,"body":"they gave it to the public, right?"},{"speaker":"So let's take the average","startTime":2330.82,"endTime":2332.44,"body":"And if no one won within a year,"},{"speaker":"So let's take the average","startTime":2332.44,"endTime":2334.9,"body":"they'd give 50,000 to whoever"},{"speaker":"So let's take the average","startTime":2332.44,"endTime":2334.9,"body":"had the highest achievement,"},{"speaker":"So let's take the average","startTime":2334.9,"endTime":2336.37,"body":"the best they win."},{"speaker":"So let's take the average","startTime":2336.37,"endTime":2338.5,"body":"And they keep awarding"},{"speaker":"So let's take the average","startTime":2336.37,"endTime":2338.5,"body":"the same amount every year"},{"speaker":"So let's take the average","startTime":2338.5,"endTime":2341.35,"body":"until someone won the grand prize, right?"},{"speaker":"So let's take the average","startTime":2341.35,"endTime":2342.55,"body":"And it released the database."},{"speaker":"So let's take the average","startTime":2342.55,"endTime":2344.77,"body":"So a hundred million customer ratings"},{"speaker":"So let's take the average","startTime":2344.77,"endTime":2346.72,"body":"for anyone interested"},{"speaker":"So let's take the average","startTime":2344.77,"endTime":2346.72,"body":"in cracking the code."},{"speaker":"So let's take the average","startTime":2346.72,"endTime":2349.193,"body":"You can try this yourself, by the way."},{"speaker":"So let's take the average","startTime":2349.193,"endTime":2350.89,"body":"I'll put the website out,"},{"speaker":"So let's take the average","startTime":2350.89,"endTime":2352.75,"body":"but it's called GroupLens.com."},{"speaker":"So let's take the average","startTime":2352.75,"endTime":2355.48,"body":"You can download that"},{"speaker":"So let's take the average","startTime":2352.75,"endTime":2355.48,"body":"data and try winning."},{"speaker":"So let's take the average","startTime":2355.48,"endTime":2359.59,"body":"You won't win a million because"},{"speaker":"So let's take the average","startTime":2355.48,"endTime":2359.59,"body":"it was already won in 2009."},{"speaker":"So let's take the average","startTime":2359.59,"endTime":2362.29,"body":"So these are the two top scorers."},{"speaker":"So let's take the average","startTime":2362.29,"endTime":2364.24,"body":"They merged with each other,"},{"speaker":"So let's take the average","startTime":2364.24,"endTime":2367.14,"body":"and they came up with"},{"speaker":"So let's take the average","startTime":2364.24,"endTime":2367.14,"body":"an algorithm and won."},{"speaker":"So let's take the average","startTime":2367.14,"endTime":2369.49,"body":"It was called BellKor's Pragmatic Chaos."},{"speaker":"So let's take the average","startTime":2369.49,"endTime":2371.44,"body":"And they won a million bucks."},{"speaker":"So let's take the average","startTime":2371.44,"endTime":2374.98,"body":"Now, they increased the"},{"speaker":"So let's take the average","startTime":2371.44,"endTime":2374.98,"body":"accuracy of the Netflix system"},{"speaker":"So let's take the average","startTime":2374.98,"endTime":2376.51,"body":"by at least 10%."},{"speaker":"So let's take the average","startTime":2376.51,"endTime":2377.74,"body":"Excellent."},{"speaker":"So let's take the average","startTime":2377.74,"endTime":2379.03,"body":"What was the problem?"},{"speaker":"So let's take the average","startTime":2379.03,"endTime":2381.04,"body":"Well, the problem was"},{"speaker":"So let's take the average","startTime":2379.03,"endTime":2381.04,"body":"there were some movies"},{"speaker":"So let's take the average","startTime":2381.04,"endTime":2383.26,"body":"which nobody could classify."},{"speaker":"Example","startTime":2383.26,"endTime":2386.44,"body":"\"Napoleon Dynamite.\""},{"speaker":"Example","startTime":2386.44,"endTime":2388.57,"body":"Has anyone seen this movie?"},{"speaker":"Example","startTime":2388.57,"endTime":2390.07,"body":"Yeah, some people have."},{"speaker":"Example","startTime":2390.07,"endTime":2393.79,"body":"Okay, it turns out this is a"},{"speaker":"Example","startTime":2390.07,"endTime":2393.79,"body":"movie which is unclassifiable."},{"speaker":"Example","startTime":2393.79,"endTime":2396.52,"body":"Nobody knows why people like this movie."},{"speaker":"Example","startTime":2396.52,"endTime":2398.89,"body":"People who like romantic"},{"speaker":"Example","startTime":2396.52,"endTime":2398.89,"body":"comedies like this movie;"},{"speaker":"Example","startTime":2398.89,"endTime":2400.78,"body":"action movies, like this movie."},{"speaker":"Example","startTime":2400.78,"endTime":2402.4,"body":"You can't predict this."},{"speaker":"Another movie","startTime":2402.4,"endTime":2405.01,"body":"\"Miss Congeniality.\""},{"speaker":"Another movie","startTime":2405.01,"endTime":2407.14,"body":"Nobody knows why people like that movie."},{"speaker":"Another movie","startTime":2407.14,"endTime":2409.48,"body":"You take these movies and you"},{"speaker":"Another movie","startTime":2407.14,"endTime":2409.48,"body":"keep them in the database,"},{"speaker":"Another movie","startTime":2409.48,"endTime":2413.5,"body":"it screws up the accuracy of"},{"speaker":"Another movie","startTime":2409.48,"endTime":2413.5,"body":"the entire algorithm by 5%."},{"speaker":"Another movie","startTime":2413.5,"endTime":2416.32,"body":"So the eventual thing"},{"speaker":"Another movie","startTime":2413.5,"endTime":2416.32,"body":"is they had to take out"},{"speaker":"Another movie","startTime":2416.32,"endTime":2418.36,"body":"all these movies which"},{"speaker":"Another movie","startTime":2416.32,"endTime":2418.36,"body":"cannot be classified."},{"speaker":"Another movie","startTime":2418.36,"endTime":2420.58,"body":"Nobody knows, in spite of the big data,"},{"speaker":"Another movie","startTime":2420.58,"endTime":2423.133,"body":"nobody knows why these things work."},{"speaker":"Another movie","startTime":2424.21,"endTime":2426.67,"body":"Okay, final part."},{"speaker":"Another movie","startTime":2426.67,"endTime":2429.37,"body":"We've got the data, we"},{"speaker":"Another movie","startTime":2426.67,"endTime":2429.37,"body":"have classified the data."},{"speaker":"Another movie","startTime":2429.37,"endTime":2431.17,"body":"We have merged all"},{"speaker":"Another movie","startTime":2429.37,"endTime":2431.17,"body":"those preference streams"},{"speaker":"Another movie","startTime":2431.17,"endTime":2432.52,"body":"using an algorithm."},{"speaker":"Another movie","startTime":2432.52,"endTime":2435.19,"body":"Next question is, can we actually use that"},{"speaker":"Another movie","startTime":2435.19,"endTime":2437.623,"body":"to make predictions about our behavior?"},{"speaker":"Another movie","startTime":2438.73,"endTime":2441.19,"body":"So we have the huge volumes of data."},{"speaker":"Another movie","startTime":2441.19,"endTime":2443.293,"body":"We have frequent feedback for the system."},{"speaker":"Another movie","startTime":2444.19,"endTime":2446.83,"body":"And the system can self-adjust."},{"speaker":"Another movie","startTime":2446.83,"endTime":2449.5,"body":"So we don't know why the"},{"speaker":"Another movie","startTime":2446.83,"endTime":2449.5,"body":"system is self-adjusting."},{"speaker":"Another movie","startTime":2449.5,"endTime":2451.12,"body":"All the system is trying to do"},{"speaker":"Another movie","startTime":2451.12,"endTime":2454.42,"body":"is to keep errors within"},{"speaker":"Another movie","startTime":2451.12,"endTime":2454.42,"body":"a certain boundary,"},{"speaker":"Another movie","startTime":2454.42,"endTime":2456.34,"body":"but it changes the weight by itself."},{"speaker":"Another movie","startTime":2456.34,"endTime":2458.38,"body":"You don't know what it's doing."},{"speaker":"Another movie","startTime":2458.38,"endTime":2460.727,"body":"So that means it throws"},{"speaker":"Another movie","startTime":2458.38,"endTime":2460.727,"body":"out some data, says,"},{"speaker":"Another movie","startTime":2460.727,"endTime":2463.6,"body":"\"That data is no longer fresh,"},{"speaker":"Another movie","startTime":2460.727,"endTime":2463.6,"body":"I'm going to throw it out.\""},{"speaker":"Another movie","startTime":2463.6,"endTime":2466.0,"body":"We don't know why it's"},{"speaker":"Another movie","startTime":2463.6,"endTime":2466.0,"body":"doing many of that thing."},{"speaker":"Another movie","startTime":2466.87,"endTime":2469.12,"body":"But let's take some examples."},{"speaker":"Another movie","startTime":2469.12,"endTime":2471.76,"body":"There's a paper on Twitter"},{"speaker":"Another movie","startTime":2469.12,"endTime":2471.76,"body":"and Foursquare data,"},{"speaker":"Another movie","startTime":2471.76,"endTime":2473.71,"body":"a paper on Facebook data,"},{"speaker":"Another movie","startTime":2473.71,"endTime":2475.87,"body":"and a paper on loan application data."},{"speaker":"Another movie","startTime":2475.87,"endTime":2478.27,"body":"Just three examples here."},{"speaker":"Another movie","startTime":2478.27,"endTime":2480.58,"body":"So the first one, they say Hristova"},{"speaker":"Another movie","startTime":2480.58,"endTime":2483.31,"body":"was actually a PhD student at Cambridge"},{"speaker":"Another movie","startTime":2483.31,"endTime":2485.05,"body":"when I came across this paper."},{"speaker":"Another movie","startTime":2485.05,"endTime":2486.28,"body":"So it's an interesting paper"},{"speaker":"Another movie","startTime":2486.28,"endTime":2489.956,"body":"because it looks at data from Twitter,"},{"speaker":"Another movie","startTime":2489.956,"endTime":2493.18,"body":"data from Foursquare, and"},{"speaker":"Another movie","startTime":2489.956,"endTime":2493.18,"body":"data from Flickr, right?"},{"speaker":"Another movie","startTime":2493.18,"endTime":2496.48,"body":"So these were all geotagged"},{"speaker":"Another movie","startTime":2493.18,"endTime":2496.48,"body":"tweets, geotagged photos,"},{"speaker":"Another movie","startTime":2496.48,"endTime":2497.71,"body":"and when you check-in."},{"speaker":"Another movie","startTime":2497.71,"endTime":2500.597,"body":"For example, if you have someone who says,"},{"speaker":"Another movie","startTime":2500.597,"endTime":2503.95,"body":"\"I'm at El Palacio De Hierro"},{"speaker":"Another movie","startTime":2500.597,"endTime":2503.95,"body":"in Mexico City\" on Twitter,"},{"speaker":"Another movie","startTime":2503.95,"endTime":2505.27,"body":"and she checks-in on Foursquare,"},{"speaker":"Another movie","startTime":2505.27,"endTime":2507.52,"body":"you merge the two together."},{"speaker":"Another movie","startTime":2507.52,"endTime":2510.4,"body":"What she ended up with was 38,000 users"},{"speaker":"Another movie","startTime":2510.4,"endTime":2511.237,"body":"of Foursquare and Twitter."},{"speaker":"Another movie","startTime":2511.237,"endTime":2514.69,"body":"433,000 connections on Twitter."},{"speaker":"Another movie","startTime":2514.69,"endTime":2516.31,"body":"550 check-ins."},{"speaker":"Another movie","startTime":2516.31,"endTime":2517.81,"body":"3 million user transactions."},{"speaker":"Another movie","startTime":2517.81,"endTime":2522.81,"body":"She had converted a social"},{"speaker":"Another movie","startTime":2517.81,"endTime":2522.81,"body":"network into a place network."},{"speaker":"Another movie","startTime":2522.94,"endTime":2524.86,"body":"Now why is this useful?"},{"speaker":"Another movie","startTime":2524.86,"endTime":2527.86,"body":"Well, one of the things"},{"speaker":"Another movie","startTime":2524.86,"endTime":2527.86,"body":"it's really useful for"},{"speaker":"Another movie","startTime":2527.86,"endTime":2529.33,"body":"is real estate."},{"speaker":"Another movie","startTime":2529.33,"endTime":2531.19,"body":"Let's take an example."},{"speaker":"Another movie","startTime":2531.19,"endTime":2533.53,"body":"This is London."},{"speaker":"Another movie","startTime":2533.53,"endTime":2535.54,"body":"And so this is the deprivation rank,"},{"speaker":"Another movie","startTime":2535.54,"endTime":2537.61,"body":"how deprived a particular area is."},{"speaker":"Another movie","startTime":2537.61,"endTime":2539.98,"body":"And this is a diversity rank."},{"speaker":"Another movie","startTime":2539.98,"endTime":2543.28,"body":"And the one I want to"},{"speaker":"Another movie","startTime":2539.98,"endTime":2543.28,"body":"point out is Hackney."},{"speaker":"Another movie","startTime":2543.28,"endTime":2546.67,"body":"Hackney is apparently an"},{"speaker":"Another movie","startTime":2543.28,"endTime":2546.67,"body":"incredibly deprived part of London,"},{"speaker":"Another movie","startTime":2546.67,"endTime":2548.47,"body":"second most after Newham."},{"speaker":"Another movie","startTime":2548.47,"endTime":2552.67,"body":"But it's also ended up being"},{"speaker":"Another movie","startTime":2548.47,"endTime":2552.67,"body":"incredibly diverse, right?"},{"speaker":"Another movie","startTime":2552.67,"endTime":2554.47,"body":"So everybody was going to Hackney."},{"speaker":"Another movie","startTime":2554.47,"endTime":2556.87,"body":"Younger people were going,"},{"speaker":"Another movie","startTime":2554.47,"endTime":2556.87,"body":"older people were going,"},{"speaker":"Another movie","startTime":2556.87,"endTime":2559.9,"body":"you know, gay people, straight"},{"speaker":"Another movie","startTime":2556.87,"endTime":2559.9,"body":"people, doesn't matter."},{"speaker":"Another movie","startTime":2559.9,"endTime":2562.27,"body":"Hackney was full of people."},{"speaker":"Another movie","startTime":2562.27,"endTime":2564.13,"body":"And so what these guys predicted"},{"speaker":"Another movie","startTime":2564.13,"endTime":2566.32,"body":"was that house prices in Hackney"},{"speaker":"Another movie","startTime":2566.32,"endTime":2568.3,"body":"would go up over a period of five years,"},{"speaker":"Another movie","startTime":2568.3,"endTime":2569.71,"body":"and it nearly doubled."},{"speaker":"Another movie","startTime":2569.71,"endTime":2573.55,"body":"So it went from 326,000 to 546,000,"},{"speaker":"Another movie","startTime":2573.55,"endTime":2575.713,"body":"just on the basis of big data."},{"speaker":"Another movie","startTime":2576.94,"endTime":2579.253,"body":"Another example, psychometric data."},{"speaker":"Another movie","startTime":2580.09,"endTime":2582.1,"body":"This is a paper by David Stillwell,"},{"speaker":"Another movie","startTime":2582.1,"endTime":2584.56,"body":"who's a colleague of mine at"},{"speaker":"Another movie","startTime":2582.1,"endTime":2584.56,"body":"the Judge Business School."},{"speaker":"Another movie","startTime":2584.56,"endTime":2587.8,"body":"So David, as part of his PhD program,"},{"speaker":"Another movie","startTime":2587.8,"endTime":2591.16,"body":"developed a personality test."},{"speaker":"Another movie","startTime":2591.16,"endTime":2592.3,"body":"So what the idea was,"},{"speaker":"Another movie","startTime":2592.3,"endTime":2593.747,"body":"you look at pictures like this and say,"},{"speaker":"Another movie","startTime":2593.747,"endTime":2595.657,"body":"\"Who do you prefer, A or B?\""},{"speaker":"Another movie","startTime":2596.53,"endTime":2599.77,"body":"Or, \"What word cloud is more you?\""},{"speaker":"Another movie","startTime":2599.77,"endTime":2602.68,"body":"Is that \"weekend, home, happy,\""},{"speaker":"Another movie","startTime":2602.68,"endTime":2606.34,"body":"or is it, \"universe, music, dreams?\""},{"speaker":"Another movie","startTime":2606.34,"endTime":2607.48,"body":"What is more you?"},{"speaker":"Another movie","startTime":2607.48,"endTime":2609.43,"body":"So people will answer those question,"},{"speaker":"Another movie","startTime":2609.43,"endTime":2612.76,"body":"not just one or two, it'll"},{"speaker":"Another movie","startTime":2609.43,"endTime":2612.76,"body":"be a hundred or so questions."},{"speaker":"Another movie","startTime":2612.76,"endTime":2614.77,"body":"And you come up with a scale, right?"},{"speaker":"Another movie","startTime":2614.77,"endTime":2617.68,"body":"Their are five scales, I'll"},{"speaker":"Another movie","startTime":2614.77,"endTime":2617.68,"body":"talk about them in a bit,"},{"speaker":"Another movie","startTime":2617.68,"endTime":2619.18,"body":"but this is the openness scale."},{"speaker":"Another movie","startTime":2619.18,"endTime":2621.85,"body":"So if you pick A, A, A, A all the way,"},{"speaker":"Another movie","startTime":2621.85,"endTime":2623.38,"body":"you're conservative and traditional,"},{"speaker":"Another movie","startTime":2623.38,"endTime":2626.86,"body":"if you pick B, B, B all the way"},{"speaker":"Another movie","startTime":2623.38,"endTime":2626.86,"body":"you're liberal and artistic."},{"speaker":"Another movie","startTime":2626.86,"endTime":2628.84,"body":"That's an example, okay?"},{"speaker":"Another movie","startTime":2628.84,"endTime":2630.91,"body":"Now, what he then did"},{"speaker":"Another movie","startTime":2630.91,"endTime":2633.82,"body":"was people started sending"},{"speaker":"Another movie","startTime":2630.91,"endTime":2633.82,"body":"those quizzes to their friends,"},{"speaker":"Another movie","startTime":2633.82,"endTime":2636.16,"body":"and their friends, and"},{"speaker":"Another movie","startTime":2633.82,"endTime":2636.16,"body":"eventually he came up with an app"},{"speaker":"Another movie","startTime":2636.16,"endTime":2637.93,"body":"called the Mypersonality app,"},{"speaker":"Another movie","startTime":2637.93,"endTime":2640.27,"body":"where it turned out that 6 million people"},{"speaker":"Another movie","startTime":2640.27,"endTime":2643.453,"body":"signed up to take their"},{"speaker":"Another movie","startTime":2640.27,"endTime":2643.453,"body":"personality through the app."},{"speaker":"Another movie","startTime":2644.62,"endTime":2646.247,"body":"And he asked them, \"Okay guys,"},{"speaker":"Another movie","startTime":2646.247,"endTime":2648.28,"body":"you know, you were taking this app,"},{"speaker":"Another movie","startTime":2648.28,"endTime":2650.32,"body":"would you mind giving me your data,"},{"speaker":"Another movie","startTime":2650.32,"endTime":2652.87,"body":"allow me to take your data from Facebook?\""},{"speaker":"Another movie","startTime":2652.87,"endTime":2654.467,"body":"And a significant chunk of them said,"},{"speaker":"Another movie","startTime":2654.467,"endTime":2657.1,"body":"\"Yeah, sure, you can take"},{"speaker":"Another movie","startTime":2654.467,"endTime":2657.1,"body":"my data from Facebook.\""},{"speaker":"Another movie","startTime":2657.1,"endTime":2658.9,"body":"And that's what he did."},{"speaker":"Another movie","startTime":2658.9,"endTime":2662.5,"body":"So what he then did was,"},{"speaker":"Another movie","startTime":2658.9,"endTime":2662.5,"body":"you've taken the test,"},{"speaker":"Another movie","startTime":2662.5,"endTime":2665.35,"body":"so I know your personality, right?"},{"speaker":"Another movie","startTime":2665.35,"endTime":2668.56,"body":"But I also know what stuff you're writing"},{"speaker":"Another movie","startTime":2668.56,"endTime":2670.06,"body":"on your Facebook page."},{"speaker":"Another movie","startTime":2670.06,"endTime":2673.06,"body":"The question is, can I use the"},{"speaker":"Another movie","startTime":2670.06,"endTime":2673.06,"body":"stuff on your Facebook page"},{"speaker":"Another movie","startTime":2673.06,"endTime":2674.83,"body":"to predict your personality?"},{"speaker":"Another movie","startTime":2674.83,"endTime":2676.75,"body":"That's what he wanted to find out."},{"speaker":"Another movie","startTime":2676.75,"endTime":2678.4,"body":"So if you're an extrovert,"},{"speaker":"you would use words like this","startTime":2680.627,"endTime":2684.76,"body":"\"party, great, I'm"},{"speaker":"you would use words like this","startTime":2680.627,"endTime":2684.76,"body":"missing you guys so much,\""},{"speaker":"you would use words like this","startTime":2684.76,"endTime":2686.683,"body":"really elongate the vowels over there."},{"speaker":"you would use words like this","startTime":2687.55,"endTime":2690.01,"body":"If you're an introvert, on the other hand,"},{"speaker":"you would use words like this","startTime":2691.487,"endTime":2694.33,"body":"\"anime, internet,"},{"speaker":"you would use words like this","startTime":2691.487,"endTime":2694.33,"body":"computer, Pokemon, manga.\""},{"speaker":"you would use words like this","startTime":2694.33,"endTime":2696.4,"body":"You would use elongated vowels,"},{"speaker":"you would use words like this","startTime":2696.4,"endTime":2699.317,"body":"but usually it's about"},{"speaker":"you would use words like this","startTime":2696.4,"endTime":2699.317,"body":"negatives, so, you know,"},{"speaker":"you would use words like this","startTime":2699.317,"endTime":2702.763,"body":"\"dammit, noo, ooh,\" whatever, right?"},{"speaker":"you would use words like this","startTime":2703.63,"endTime":2706.51,"body":"So, he went beyond that."},{"speaker":"you would use words like this","startTime":2706.51,"endTime":2708.31,"body":"He said, \"Well, let's"},{"speaker":"you would use words like this","startTime":2706.51,"endTime":2708.31,"body":"see how the predictions"},{"speaker":"you would use words like this","startTime":2708.31,"endTime":2710.02,"body":"compare to human beings.\""},{"speaker":"you would use words like this","startTime":2710.02,"endTime":2712.6,"body":"So what you have over"},{"speaker":"you would use words like this","startTime":2710.02,"endTime":2712.6,"body":"here is the accuracy."},{"speaker":"you would use words like this","startTime":2712.6,"endTime":2715.63,"body":"And what you have over here is"},{"speaker":"you would use words like this","startTime":2712.6,"endTime":2715.63,"body":"the number of Facebook likes."},{"speaker":"you would use words like this","startTime":2715.63,"endTime":2718.54,"body":"Now what you see is your"},{"speaker":"you would use words like this","startTime":2715.63,"endTime":2718.54,"body":"average work colleague"},{"speaker":"you would use words like this","startTime":2718.54,"endTime":2720.94,"body":"is not really very accurate"},{"speaker":"you would use words like this","startTime":2718.54,"endTime":2720.94,"body":"about your personality,"},{"speaker":"you would use words like this","startTime":2720.94,"endTime":2723.37,"body":"because most of us are"},{"speaker":"you would use words like this","startTime":2720.94,"endTime":2723.37,"body":"professional at work, right?"},{"speaker":"you would use words like this","startTime":2723.37,"endTime":2724.57,"body":"We don't scream at our colleagues."},{"speaker":"you would use words like this","startTime":2724.57,"endTime":2726.1,"body":"We are pretty professional."},{"speaker":"you would use words like this","startTime":2726.1,"endTime":2729.97,"body":"So they're about 27% accurate"},{"speaker":"you would use words like this","startTime":2726.1,"endTime":2729.97,"body":"about your true personality."},{"speaker":"you would use words like this","startTime":2729.97,"endTime":2731.2,"body":"Your friends are a little more accurate,"},{"speaker":"you would use words like this","startTime":2731.2,"endTime":2734.26,"body":"but still edging on the"},{"speaker":"you would use words like this","startTime":2731.2,"endTime":2734.26,"body":"positive side because, again,"},{"speaker":"you would use words like this","startTime":2734.26,"endTime":2736.96,"body":"they're our friends because we"},{"speaker":"you would use words like this","startTime":2734.26,"endTime":2736.96,"body":"don't scream at them, right?"},{"speaker":"you would use words like this","startTime":2736.96,"endTime":2739.33,"body":"So that's about 45%."},{"speaker":"you would use words like this","startTime":2739.33,"endTime":2741.64,"body":"Your family's about 50% accurate."},{"speaker":"you would use words like this","startTime":2741.64,"endTime":2744.85,"body":"They can see both the"},{"speaker":"you would use words like this","startTime":2741.64,"endTime":2744.85,"body":"positives and the negatives."},{"speaker":"you would use words like this","startTime":2744.85,"endTime":2749.02,"body":"David says that the computer's"},{"speaker":"you would use words like this","startTime":2744.85,"endTime":2749.02,"body":"average accuracy is 56%."},{"speaker":"To put that in another way","startTime":2750.7,"endTime":2753.05,"body":"Facebook knows you better"},{"speaker":"To put that in another way","startTime":2750.7,"endTime":2753.05,"body":"than your own mother."},{"speaker":"To put that in another way","startTime":2754.87,"endTime":2757.18,"body":"Okay, well, he went further."},{"speaker":"To put that in another way","startTime":2757.18,"endTime":2759.34,"body":"So he had data on their Facebook likes,"},{"speaker":"To put that in another way","startTime":2759.34,"endTime":2761.92,"body":"on art, CNN, BMW, and so on."},{"speaker":"To put that in another way","startTime":2761.92,"endTime":2765.13,"body":"And then he collected a hundred components"},{"speaker":"To put that in another way","startTime":2765.13,"endTime":2767.65,"body":"using a singular value"},{"speaker":"To put that in another way","startTime":2765.13,"endTime":2767.65,"body":"decomposition method."},{"speaker":"To put that in another way","startTime":2767.65,"endTime":2769.96,"body":"And then he would predict,"},{"speaker":"To put that in another way","startTime":2767.65,"endTime":2769.96,"body":"\"What can I predict"},{"speaker":"To put that in another way","startTime":2769.96,"endTime":2773.14,"body":"based on the stuff which"},{"speaker":"To put that in another way","startTime":2769.96,"endTime":2773.14,"body":"you like on Facebook?\""},{"speaker":"To put that in another way","startTime":2773.14,"endTime":2774.37,"body":"So the dependent variable"},{"speaker":"To put that in another way","startTime":2774.37,"endTime":2777.43,"body":"is your age, gender,"},{"speaker":"To put that in another way","startTime":2774.37,"endTime":2777.43,"body":"political view, and so on."},{"speaker":"To put that in another way","startTime":2777.43,"endTime":2779.38,"body":"And this is what he can do."},{"speaker":"To put that in another way","startTime":2779.38,"endTime":2781.9,"body":"Predict whether you're"},{"speaker":"To put that in another way","startTime":2779.38,"endTime":2781.9,"body":"single or in a relationship."},{"speaker":"To put that in another way","startTime":2781.9,"endTime":2784.417,"body":"Do you smoke cigarettes? 73% accurate."},{"speaker":"To put that in another way","startTime":2784.417,"endTime":2788.56,"body":"Are you Caucasian versus"},{"speaker":"To put that in another way","startTime":2784.417,"endTime":2788.56,"body":"African American? 95% accurate."},{"speaker":"To put that in another way","startTime":2788.56,"endTime":2790.813,"body":"Are you gay? 88% accurate."},{"speaker":"To put that in another way","startTime":2791.65,"endTime":2795.52,"body":"Or Christianity versus Islam,"},{"speaker":"To put that in another way","startTime":2791.65,"endTime":2795.52,"body":"82% accurate, and so on."},{"speaker":"To put that in another way","startTime":2795.52,"endTime":2797.89,"body":"So this is all based on the data"},{"speaker":"To put that in another way","startTime":2797.89,"endTime":2799.843,"body":"which you put on your Facebook page."},{"speaker":"To put that in another way","startTime":2800.98,"endTime":2803.68,"body":"Last one, loan application data."},{"speaker":"To put that in another way","startTime":2803.68,"endTime":2807.88,"body":"This is a paper which looked at people"},{"speaker":"To put that in another way","startTime":2807.88,"endTime":2811.81,"body":"who are applying for a loan"},{"speaker":"To put that in another way","startTime":2807.88,"endTime":2811.81,"body":"on a website called Prosper,"},{"speaker":"To put that in another way","startTime":2811.81,"endTime":2813.85,"body":"which is a peer-to-peer lending site."},{"speaker":"To put that in another way","startTime":2813.85,"endTime":2816.2,"body":"People would come in and"},{"speaker":"To put that in another way","startTime":2813.85,"endTime":2816.2,"body":"they would say, \"Okay,"},{"speaker":"To put that in another way","startTime":2817.57,"endTime":2821.74,"body":"let me try to borrow money"},{"speaker":"To put that in another way","startTime":2817.57,"endTime":2821.74,"body":"from a bunch of strangers.\""},{"speaker":"To put that in another way","startTime":2821.74,"endTime":2825.79,"body":"So I write an essay about"},{"speaker":"To put that in another way","startTime":2821.74,"endTime":2825.79,"body":"why I need the money, okay?"},{"speaker":"To put that in another way","startTime":2825.79,"endTime":2828.82,"body":"And so what these guys did was to,"},{"speaker":"To put that in another way","startTime":2828.82,"endTime":2830.59,"body":"we know that 13% of the borrowers"},{"speaker":"To put that in another way","startTime":2830.59,"endTime":2832.48,"body":"eventually defaulted on the loan."},{"speaker":"To put that in another way","startTime":2832.48,"endTime":2835.18,"body":"So they say, \"Can I predict"},{"speaker":"To put that in another way","startTime":2832.48,"endTime":2835.18,"body":"who's going to default"},{"speaker":"To put that in another way","startTime":2835.18,"endTime":2838.78,"body":"based on a description of"},{"speaker":"To put that in another way","startTime":2835.18,"endTime":2838.78,"body":"what they would write?\""},{"speaker":"To put that in another way","startTime":2838.78,"endTime":2840.223,"body":"Let's take two examples."},{"speaker":"Borrower one writes","startTime":2841.3,"endTime":2843.55,"body":"\"I'm"},{"speaker":"Borrower one writes","startTime":2841.3,"endTime":2843.55,"body":"a hardworking person,"},{"speaker":"Borrower one writes","startTime":2843.55,"endTime":2846.61,"body":"married for 25 years,"},{"speaker":"Borrower one writes","startTime":2843.55,"endTime":2846.61,"body":"have two wonderful boys."},{"speaker":"Borrower one writes","startTime":2846.61,"endTime":2848.59,"body":"Please let me explain why I need help."},{"speaker":"Borrower one writes","startTime":2848.59,"endTime":2851.14,"body":"I use a $2,000 loan to fix our roof."},{"speaker":"Borrower one writes","startTime":2851.14,"endTime":2854.26,"body":"Thank you, God bless you, and"},{"speaker":"Borrower one writes","startTime":2851.14,"endTime":2854.26,"body":"I promise to pay you back.\""},{"speaker":"Borrower one writes","startTime":2854.26,"endTime":2855.52,"body":"That's one."},{"speaker":"Borrower two writes","startTime":2856.727,"endTime":2858.28,"body":"\"While the past year in our new place"},{"speaker":"Borrower two writes","startTime":2858.28,"endTime":2860.74,"body":"has been more than great,"},{"speaker":"Borrower two writes","startTime":2858.28,"endTime":2860.74,"body":"the roof is now leaking,"},{"speaker":"Borrower two writes","startTime":2860.74,"endTime":2863.77,"body":"and I need to borrow $2,000 to"},{"speaker":"Borrower two writes","startTime":2860.74,"endTime":2863.77,"body":"cover the cost of the repair."},{"speaker":"Borrower two writes","startTime":2863.77,"endTime":2868.24,"body":"I pay all bills, car loans,"},{"speaker":"Borrower two writes","startTime":2863.77,"endTime":2868.24,"body":"cable, utilities on time.\""},{"speaker":"Borrower two writes","startTime":2868.24,"endTime":2873.07,"body":"How many of you would say"},{"speaker":"Borrower two writes","startTime":2868.24,"endTime":2873.07,"body":"borrower one is a better borrower,"},{"speaker":"Borrower two writes","startTime":2873.07,"endTime":2875.653,"body":"will more likely pay you"},{"speaker":"Borrower two writes","startTime":2873.07,"endTime":2875.653,"body":"back than borrower two?"},{"speaker":"Borrower two writes","startTime":2877.42,"endTime":2879.043,"body":"Okay, some hands up there."},{"speaker":"Borrower two writes","startTime":2880.12,"endTime":2882.04,"body":"Okay, some hands up there."},{"speaker":"Borrower two writes","startTime":2882.04,"endTime":2883.24,"body":"All right, borrower two?"},{"speaker":"Borrower two writes","startTime":2884.59,"endTime":2886.72,"body":"Okay, turns out actually borrower two"},{"speaker":"Borrower two writes","startTime":2886.72,"endTime":2889.06,"body":"is in fact much better than borrower one."},{"speaker":"Borrower two writes","startTime":2889.06,"endTime":2890.5,"body":"And it turns out,"},{"speaker":"Borrower two writes","startTime":2890.5,"endTime":2892.45,"body":"if someone tells you"},{"speaker":"Borrower two writes","startTime":2890.5,"endTime":2892.45,"body":"they will pay you back,"},{"speaker":"Borrower two writes","startTime":2892.45,"endTime":2893.8,"body":"they will not pay you back."},{"speaker":"Borrower two writes","startTime":2894.91,"endTime":2895.897,"body":"The more assertive the promise,"},{"speaker":"Borrower two writes","startTime":2895.897,"endTime":2897.34,"body":"the more likely they're going to break it."},{"speaker":"Borrower two writes","startTime":2897.34,"endTime":2898.173,"body":"If someone says,"},{"speaker":"Borrower two writes","startTime":2898.173,"endTime":2901.18,"body":"\"I promise I will pay you"},{"speaker":"Borrower two writes","startTime":2898.173,"endTime":2901.18,"body":"back, so help me, God!\""},{"speaker":"Borrower two writes","startTime":2901.18,"endTime":2902.38,"body":"Gone, your money's gone."},{"speaker":"Borrower two writes","startTime":2903.82,"endTime":2905.05,"body":"Okay?"},{"speaker":"Borrower two writes","startTime":2905.05,"endTime":2905.95,"body":"Religion doesn't matter."},{"speaker":"Borrower two writes","startTime":2905.95,"endTime":2906.94,"body":"Someone who mentions God"},{"speaker":"Borrower two writes","startTime":2906.94,"endTime":2909.763,"body":"is 2.2 times more likely"},{"speaker":"Borrower two writes","startTime":2906.94,"endTime":2909.763,"body":"to default on their loans."},{"speaker":"Borrower two writes","startTime":2913.12,"endTime":2914.62,"body":"Another example of using, you know,"},{"speaker":"Borrower two writes","startTime":2914.62,"endTime":2917.17,"body":"big data to make"},{"speaker":"Borrower two writes","startTime":2914.62,"endTime":2917.17,"body":"inferences about us, right?"},{"speaker":"Borrower two writes","startTime":2917.17,"endTime":2919.12,"body":"So the ultimate goal of big data"},{"speaker":"Borrower two writes","startTime":2919.12,"endTime":2921.433,"body":"is complete personalization."},{"speaker":"Borrower two writes","startTime":2923.08,"endTime":2924.16,"body":"This is fake."},{"speaker":"Borrower two writes","startTime":2924.16,"endTime":2926.58,"body":"It was tweeted around that time"},{"speaker":"Borrower two writes","startTime":2926.58,"endTime":2928.99,"body":"as an example of personalized news."},{"speaker":"Borrower two writes","startTime":2928.99,"endTime":2931.007,"body":"So you have the Wall"},{"speaker":"Borrower two writes","startTime":2928.99,"endTime":2931.007,"body":"Street Journal saying,"},{"speaker":"Borrower two writes","startTime":2931.007,"endTime":2934.78,"body":"\"Trump softens his tone\""},{"speaker":"Borrower two writes","startTime":2931.007,"endTime":2934.78,"body":"sent to a Democrat area,"},{"speaker":"Borrower two writes","startTime":2934.78,"endTime":2937.81,"body":"and \"Trump talks tough on wall\""},{"speaker":"Borrower two writes","startTime":2934.78,"endTime":2937.81,"body":"sent to a Republican area."},{"speaker":"Borrower two writes","startTime":2937.81,"endTime":2939.07,"body":"This wasn't really true,"},{"speaker":"Borrower two writes","startTime":2939.07,"endTime":2942.58,"body":"because actually it was different"},{"speaker":"Borrower two writes","startTime":2939.07,"endTime":2942.58,"body":"editions on the same day."},{"speaker":"Borrower two writes","startTime":2942.58,"endTime":2944.29,"body":"At the beginning he started talking tough,"},{"speaker":"Borrower two writes","startTime":2944.29,"endTime":2946.36,"body":"later he softened his tone."},{"speaker":"Borrower two writes","startTime":2946.36,"endTime":2948.88,"body":"But they put it together"},{"speaker":"Borrower two writes","startTime":2946.36,"endTime":2948.88,"body":"to make it look like that."},{"speaker":"Borrower two writes","startTime":2948.88,"endTime":2950.2,"body":"But anyway, the point is,"},{"speaker":"Borrower two writes","startTime":2950.2,"endTime":2952.84,"body":"this is the ultimate"},{"speaker":"Borrower two writes","startTime":2950.2,"endTime":2952.84,"body":"goal of big data, right?"},{"speaker":"Borrower two writes","startTime":2952.84,"endTime":2957.133,"body":"Give you exactly what you want, like this."},{"speaker":"Borrower two writes","startTime":2958.6,"endTime":2962.56,"body":"But this means that"},{"speaker":"Borrower two writes","startTime":2958.6,"endTime":2962.56,"body":"there's a big problem here."},{"speaker":"Borrower two writes","startTime":2962.56,"endTime":2965.11,"body":"And the problem is that,"},{"speaker":"Borrower two writes","startTime":2962.56,"endTime":2965.11,"body":"at the end of the day,"},{"speaker":"Borrower two writes","startTime":2965.11,"endTime":2968.05,"body":"we don't know why these"},{"speaker":"Borrower two writes","startTime":2965.11,"endTime":2968.05,"body":"things are happening."},{"speaker":"Borrower two writes","startTime":2968.05,"endTime":2970.81,"body":"We are still measuring correlations."},{"speaker":"Borrower two writes","startTime":2970.81,"endTime":2973.96,"body":"This is not causation,"},{"speaker":"Borrower two writes","startTime":2970.81,"endTime":2973.96,"body":"these are correlations."},{"speaker":"Borrower two writes","startTime":2973.96,"endTime":2977.26,"body":"So what we are doing is"},{"speaker":"Borrower two writes","startTime":2973.96,"endTime":2977.26,"body":"putting people into groups"},{"speaker":"Borrower two writes","startTime":2977.26,"endTime":2978.97,"body":"and saying they are like this,"},{"speaker":"Borrower two writes","startTime":2978.97,"endTime":2982.06,"body":"using something like"},{"speaker":"Borrower two writes","startTime":2978.97,"endTime":2982.06,"body":"principal component analysis,"},{"speaker":"Borrower two writes","startTime":2982.06,"endTime":2983.29,"body":"discriminant analysis."},{"speaker":"Borrower two writes","startTime":2983.29,"endTime":2985.307,"body":"You put them into groups, and we say,"},{"speaker":"Borrower two writes","startTime":2985.307,"endTime":2988.24,"body":"\"You are like that because"},{"speaker":"Borrower two writes","startTime":2985.307,"endTime":2988.24,"body":"you are with that group.\""},{"speaker":"Borrower two writes","startTime":2988.24,"endTime":2989.83,"body":"But are you like that?"},{"speaker":"Borrower two writes","startTime":2989.83,"endTime":2991.36,"body":"It's unclear, right?"},{"speaker":"Borrower two writes","startTime":2991.36,"endTime":2993.55,"body":"Let's take a couple of examples."},{"speaker":"Borrower two writes","startTime":2993.55,"endTime":2997.03,"body":"Kaggle had a competition"},{"speaker":"Borrower two writes","startTime":2993.55,"endTime":2997.03,"body":"in 2012 to find out"},{"speaker":"Borrower two writes","startTime":2997.03,"endTime":2999.01,"body":"what are the best used cars,"},{"speaker":"Borrower two writes","startTime":2999.01,"endTime":3001.38,"body":"the ones with the highest"},{"speaker":"Borrower two writes","startTime":2999.01,"endTime":3001.38,"body":"quality on the market,"},{"speaker":"Borrower two writes","startTime":3001.38,"endTime":3004.02,"body":"the ones that don't get into accidents."},{"speaker":"Borrower two writes","startTime":3004.02,"endTime":3008.733,"body":"Turns out the answer is orange"},{"speaker":"Borrower two writes","startTime":3004.02,"endTime":3008.733,"body":"cars, bright orange cars."},{"speaker":"Borrower two writes","startTime":3009.81,"endTime":3011.85,"body":"Why? I don't know."},{"speaker":"Borrower two writes","startTime":3011.85,"endTime":3013.32,"body":"They didn't know, right?"},{"speaker":"Borrower two writes","startTime":3013.32,"endTime":3016.62,"body":"Maybe orange car drivers are more careful,"},{"speaker":"Borrower two writes","startTime":3016.62,"endTime":3017.91,"body":"maybe they're so eye-catching,"},{"speaker":"Borrower two writes","startTime":3017.91,"endTime":3020.4,"body":"other people keep away"},{"speaker":"Borrower two writes","startTime":3017.91,"endTime":3020.4,"body":"from them, nobody knows."},{"speaker":"Borrower two writes","startTime":3020.4,"endTime":3022.74,"body":"It just pops out of the system."},{"speaker":"Borrower two writes","startTime":3022.74,"endTime":3024.693,"body":"Or another example here in England,"},{"speaker":"Borrower two writes","startTime":3026.19,"endTime":3029.01,"body":"the famous example of cholera,"},{"speaker":"Borrower two writes","startTime":3029.01,"endTime":3032.007,"body":"which broke out here in England after 1839"},{"speaker":"Borrower two writes","startTime":3032.007,"endTime":3033.84,"body":"when it was brought in from India."},{"speaker":"Borrower two writes","startTime":3033.84,"endTime":3037.41,"body":"So people were dying all over London."},{"speaker":"Borrower two writes","startTime":3037.41,"endTime":3042.41,"body":"And there's a book called \"The"},{"speaker":"Borrower two writes","startTime":3037.41,"endTime":3042.41,"body":"Ghost Map\" by Steven Johnson,"},{"speaker":"Borrower two writes","startTime":3042.84,"endTime":3046.53,"body":"and he writes about the"},{"speaker":"Borrower two writes","startTime":3042.84,"endTime":3046.53,"body":"marvel of causal inference"},{"speaker":"Borrower two writes","startTime":3046.53,"endTime":3048.21,"body":"that John Snow did."},{"speaker":"Borrower two writes","startTime":3048.21,"endTime":3050.49,"body":"So John Snow basically thought"},{"speaker":"Borrower two writes","startTime":3050.49,"endTime":3052.56,"body":"that it was transmitted through the water,"},{"speaker":"Borrower two writes","startTime":3052.56,"endTime":3056.49,"body":"eventually narrowed down his"},{"speaker":"Borrower two writes","startTime":3052.56,"endTime":3056.49,"body":"investigation to one pump"},{"speaker":"Borrower two writes","startTime":3056.49,"endTime":3059.01,"body":"on Broad Street where"},{"speaker":"Borrower two writes","startTime":3056.49,"endTime":3059.01,"body":"everyone was drinking water"},{"speaker":"Borrower two writes","startTime":3059.01,"endTime":3061.53,"body":"and dying subsequently of cholera."},{"speaker":"Borrower two writes","startTime":3061.53,"endTime":3063.42,"body":"So he persuaded the authorities"},{"speaker":"Borrower two writes","startTime":3063.42,"endTime":3065.46,"body":"to remove the handle of the pump,"},{"speaker":"Borrower two writes","startTime":3065.46,"endTime":3068.7,"body":"and cholera stopped in that area."},{"speaker":"Borrower two writes","startTime":3068.7,"endTime":3070.71,"body":"And in fact today, every year,"},{"speaker":"Borrower two writes","startTime":3070.71,"endTime":3074.37,"body":"they have the Pumphandle Lecture,"},{"speaker":"Borrower two writes","startTime":3070.71,"endTime":3074.37,"body":"which is on public health."},{"speaker":"Borrower two writes","startTime":3074.37,"endTime":3078.15,"body":"And afterwards everybody goes"},{"speaker":"Borrower two writes","startTime":3074.37,"endTime":3078.15,"body":"to lunch at the John Snow pub."},{"speaker":"Borrower two writes","startTime":3078.15,"endTime":3080.76,"body":"So worth checking out as well."},{"speaker":"Borrower two writes","startTime":3080.76,"endTime":3084.51,"body":"But that's a causal inference story."},{"speaker":"Borrower two writes","startTime":3084.51,"endTime":3088.14,"body":"The correlation story was"},{"speaker":"Borrower two writes","startTime":3084.51,"endTime":3088.14,"body":"his colleague William Farr,"},{"speaker":"Borrower two writes","startTime":3088.14,"endTime":3089.97,"body":"when the prevailing theory at that time"},{"speaker":"Borrower two writes","startTime":3089.97,"endTime":3094.38,"body":"was this is caused by miasma, right?"},{"speaker":"Borrower two writes","startTime":3094.38,"endTime":3097.56,"body":"It's the bad air which is causing cholera."},{"speaker":"Borrower two writes","startTime":3097.56,"endTime":3100.17,"body":"And he did a beautiful"},{"speaker":"Borrower two writes","startTime":3097.56,"endTime":3100.17,"body":"analysis where he showed"},{"speaker":"Borrower two writes","startTime":3100.17,"endTime":3101.88,"body":"that the closer you were to the Thames,"},{"speaker":"Borrower two writes","startTime":3101.88,"endTime":3103.93,"body":"the closer you were living to the Thames,"},{"speaker":"Borrower two writes","startTime":3104.859,"endTime":3108.42,"body":"the higher the probability"},{"speaker":"Borrower two writes","startTime":3104.859,"endTime":3108.42,"body":"you're going to get cholera."},{"speaker":"Borrower two writes","startTime":3108.42,"endTime":3113.42,"body":"And unfortunately, you know, he was wrong,"},{"speaker":"Borrower two writes","startTime":3114.0,"endTime":3116.25,"body":"but the data was there, the correlation."},{"speaker":"Borrower two writes","startTime":3116.25,"endTime":3119.499,"body":"There was no causation,"},{"speaker":"Borrower two writes","startTime":3116.25,"endTime":3119.499,"body":"but he basically said"},{"speaker":"Borrower two writes","startTime":3119.499,"endTime":3122.227,"body":"it was actually caused by the fact that,"},{"speaker":"Borrower two writes","startTime":3122.227,"endTime":3124.35,"body":"you know, the closer you"},{"speaker":"Borrower two writes","startTime":3122.227,"endTime":3124.35,"body":"were living to the Thames,"},{"speaker":"Borrower two writes","startTime":3124.35,"endTime":3127.38,"body":"you were actually drawing"},{"speaker":"Borrower two writes","startTime":3124.35,"endTime":3127.38,"body":"water from that same pool,"},{"speaker":"Borrower two writes","startTime":3127.38,"endTime":3130.35,"body":"the same watershed, but"},{"speaker":"Borrower two writes","startTime":3127.38,"endTime":3130.35,"body":"people living further away"},{"speaker":"Borrower two writes","startTime":3130.35,"endTime":3132.72,"body":"were drawing from a different water body,"},{"speaker":"Borrower two writes","startTime":3132.72,"endTime":3134.733,"body":"and so that was what was causing it."},{"speaker":"Borrower two writes","startTime":3135.75,"endTime":3139.41,"body":"Big data says we can try"},{"speaker":"Borrower two writes","startTime":3135.75,"endTime":3139.41,"body":"to solve these problems"},{"speaker":"Borrower two writes","startTime":3139.41,"endTime":3142.113,"body":"by collecting more and more data."},{"speaker":"Borrower two writes","startTime":3143.25,"endTime":3145.323,"body":"But at the end, we don't know why."},{"speaker":"Borrower two writes","startTime":3146.19,"endTime":3147.78,"body":"And that's going to be a problem."},{"speaker":"Borrower two writes","startTime":3147.78,"endTime":3151.83,"body":"And I'm going to talk about"},{"speaker":"Borrower two writes","startTime":3147.78,"endTime":3151.83,"body":"that problem next time as well."},{"speaker":"Borrower two writes","startTime":3151.83,"endTime":3156.03,"body":"So next time it's going to be on AI,"},{"speaker":"Borrower two writes","startTime":3156.03,"endTime":3158.28,"body":"and how AI is working."},{"speaker":"Borrower two writes","startTime":3158.28,"endTime":3161.464,"body":"Things like ChatGPT,"},{"speaker":"Borrower two writes","startTime":3158.28,"endTime":3161.464,"body":"we'll talk about that."},{"speaker":"Borrower two writes","startTime":3161.464,"endTime":3164.28,"body":"And the question will be,"},{"speaker":"Borrower two writes","startTime":3161.464,"endTime":3164.28,"body":"you know, how do these things"},{"speaker":"Borrower two writes","startTime":3164.28,"endTime":3166.26,"body":"carry out a conversation with us,"},{"speaker":"Borrower two writes","startTime":3166.26,"endTime":3168.18,"body":"sometimes a very creepy conversation,"},{"speaker":"Borrower two writes","startTime":3168.18,"endTime":3169.35,"body":"but how does that work?"},{"speaker":"Borrower two writes","startTime":3169.35,"endTime":3171.51,"body":"How do these algorithms work?"},{"speaker":"Borrower two writes","startTime":3171.51,"endTime":3174.24,"body":"And this is an issue which"},{"speaker":"Borrower two writes","startTime":3171.51,"endTime":3174.24,"body":"I'm going to talk about"},{"speaker":"Borrower two writes","startTime":3174.24,"endTime":3177.06,"body":"again next time, all right?"},{"speaker":"Borrower two writes","startTime":3177.06,"endTime":3179.703,"body":"And I think we are, end of it."},{"speaker":"Borrower two writes","startTime":3181.05,"endTime":3182.071,"body":"Okay."},{"speaker":"Borrower two writes","startTime":3182.071,"endTime":3185.238,"body":"(audience applauding)"},{"speaker":"Borrower two writes","startTime":3186.24,"endTime":3188.55,"body":"- First question from Anna Lisa."},{"speaker":"Borrower two writes","startTime":3188.55,"endTime":3191.22,"body":"She says, \"From this lecture,"},{"speaker":"Borrower two writes","startTime":3188.55,"endTime":3191.22,"body":"it looks like big data"},{"speaker":"Borrower two writes","startTime":3191.22,"endTime":3194.19,"body":"heavily rely on behavioral information."},{"speaker":"Borrower two writes","startTime":3194.19,"endTime":3196.53,"body":"Do you think in the future"},{"speaker":"Borrower two writes","startTime":3194.19,"endTime":3196.53,"body":"new ways and metrics"},{"speaker":"Borrower two writes","startTime":3196.53,"endTime":3199.827,"body":"to capture users' behaviors"},{"speaker":"Borrower two writes","startTime":3196.53,"endTime":3199.827,"body":"will be used or developed?\""},{"speaker":"Borrower two writes","startTime":3201.36,"endTime":3202.82,"body":"- That's actually a"},{"speaker":"Borrower two writes","startTime":3201.36,"endTime":3202.82,"body":"very, very good question."},{"speaker":"Borrower two writes","startTime":3202.82,"endTime":3204.09,"body":"- It is a good question, yeah."},{"speaker":"Borrower two writes","startTime":3204.09,"endTime":3205.86,"body":"- Excellent question."},{"speaker":"Borrower two writes","startTime":3205.86,"endTime":3208.92,"body":"The thing though, there are"},{"speaker":"Borrower two writes","startTime":3205.86,"endTime":3208.92,"body":"two parts to this, right?"},{"speaker":"Borrower two writes","startTime":3208.92,"endTime":3210.48,"body":"We use behavioral data,"},{"speaker":"Borrower two writes","startTime":3210.48,"endTime":3213.69,"body":"we use how people are"},{"speaker":"Borrower two writes","startTime":3210.48,"endTime":3213.69,"body":"actually behaving in big data."},{"speaker":"Borrower two writes","startTime":3213.69,"endTime":3216.423,"body":"The point is, we don't know why, right?"},{"speaker":"Borrower two writes","startTime":3218.16,"endTime":3220.56,"body":"Does somebody, like, take another example,"},{"speaker":"Borrower two writes","startTime":3220.56,"endTime":3222.21,"body":"which I didn't mention here."},{"speaker":"Borrower two writes","startTime":3222.21,"endTime":3226.17,"body":"Turns out that one of"},{"speaker":"Borrower two writes","startTime":3222.21,"endTime":3226.17,"body":"the big patterns you see"},{"speaker":"Borrower two writes","startTime":3226.17,"endTime":3231.123,"body":"is the sale of nappies and"},{"speaker":"Borrower two writes","startTime":3226.17,"endTime":3231.123,"body":"beer are highly correlated."},{"speaker":"Borrower two writes","startTime":3232.14,"endTime":3232.973,"body":"Why?"},{"speaker":"Borrower two writes","startTime":3232.973,"endTime":3235.259,"body":"Is it that male, you know,"},{"speaker":"Borrower two writes","startTime":3235.259,"endTime":3237.87,"body":"when you're going shopping for the baby,"},{"speaker":"Borrower two writes","startTime":3237.87,"endTime":3240.51,"body":"you pick up a can of"},{"speaker":"Borrower two writes","startTime":3237.87,"endTime":3240.51,"body":"beer to keep you going"},{"speaker":"Borrower two writes","startTime":3240.51,"endTime":3242.97,"body":"while the baby is, you know,"},{"speaker":"Borrower two writes","startTime":3240.51,"endTime":3242.97,"body":"maybe women like drinking beer."},{"speaker":"Borrower two writes","startTime":3242.97,"endTime":3246.06,"body":"We don't know why, but that's"},{"speaker":"Borrower two writes","startTime":3242.97,"endTime":3246.06,"body":"a high correlation, right?"},{"speaker":"Borrower two writes","startTime":3246.06,"endTime":3248.85,"body":"So there is no behavioral story here."},{"speaker":"Borrower two writes","startTime":3248.85,"endTime":3252.03,"body":"Without the story, it's really"},{"speaker":"Borrower two writes","startTime":3248.85,"endTime":3252.03,"body":"difficult to come up and say,"},{"speaker":"Borrower two writes","startTime":3252.03,"endTime":3253.47,"body":"okay, this is what we're doing."},{"speaker":"Borrower two writes","startTime":3253.47,"endTime":3254.94,"body":"That's problem number one."},{"speaker":"Borrower two writes","startTime":3254.94,"endTime":3258.42,"body":"But problem number two is"},{"speaker":"Borrower two writes","startTime":3254.94,"endTime":3258.42,"body":"actually more insidious."},{"speaker":"Borrower two writes","startTime":3258.42,"endTime":3260.43,"body":"It is, I predict what you're going to do"},{"speaker":"Borrower two writes","startTime":3260.43,"endTime":3262.23,"body":"based on these behavioral patterns,"},{"speaker":"Borrower two writes","startTime":3262.23,"endTime":3264.13,"body":"and then I give you information"},{"speaker":"Borrower two writes","startTime":3265.26,"endTime":3269.07,"body":"about precisely the things which"},{"speaker":"Borrower two writes","startTime":3265.26,"endTime":3269.07,"body":"I think you're going to do."},{"speaker":"Borrower two writes","startTime":3269.07,"endTime":3272.19,"body":"But because you have no"},{"speaker":"Borrower two writes","startTime":3269.07,"endTime":3272.19,"body":"other sources of information,"},{"speaker":"Borrower two writes","startTime":3272.19,"endTime":3275.16,"body":"you behave differently from"},{"speaker":"Borrower two writes","startTime":3272.19,"endTime":3275.16,"body":"how you would do it otherwise."},{"speaker":"Borrower two writes","startTime":3275.16,"endTime":3277.44,"body":"So in a way it's like, you know,"},{"speaker":"Borrower two writes","startTime":3277.44,"endTime":3281.13,"body":"I give you information"},{"speaker":"Borrower two writes","startTime":3277.44,"endTime":3281.13,"body":"so that you do something,"},{"speaker":"Borrower two writes","startTime":3281.13,"endTime":3283.95,"body":"and you have nothing else"},{"speaker":"Borrower two writes","startTime":3281.13,"endTime":3283.95,"body":"to do, so you do that."},{"speaker":"Borrower two writes","startTime":3283.95,"endTime":3288.95,"body":"So it reinforces, it becomes"},{"speaker":"Borrower two writes","startTime":3283.95,"endTime":3288.95,"body":"a vicious circle, right?"},{"speaker":"Borrower two writes","startTime":3288.99,"endTime":3291.03,"body":"How much one is going to"},{"speaker":"Borrower two writes","startTime":3288.99,"endTime":3291.03,"body":"be happening to the other,"},{"speaker":"Borrower two writes","startTime":3291.03,"endTime":3291.863,"body":"I have no idea,"},{"speaker":"Borrower two writes","startTime":3291.863,"endTime":3294.75,"body":"but it's a very good and scary question."},{"speaker":"Borrower two writes","startTime":3294.75,"endTime":3296.16,"body":"- Yeah, that's a good question."},{"speaker":"Borrower two writes","startTime":3296.16,"endTime":3298.86,"body":"The second question"},{"speaker":"Borrower two writes","startTime":3296.16,"endTime":3298.86,"body":"actually is a bit related."},{"speaker":"Borrower two writes","startTime":3298.86,"endTime":3301.267,"body":"It's from Pedro C, and he says,"},{"speaker":"Borrower two writes","startTime":3301.267,"endTime":3304.41,"body":"\"What is your opinion on"},{"speaker":"Borrower two writes","startTime":3301.267,"endTime":3304.41,"body":"apps using users' data"},{"speaker":"Borrower two writes","startTime":3304.41,"endTime":3307.53,"body":"to purposefully make"},{"speaker":"Borrower two writes","startTime":3304.41,"endTime":3307.53,"body":"themselves more addictive?"},{"speaker":"Borrower two writes","startTime":3307.53,"endTime":3310.29,"body":"And at what point does"},{"speaker":"Borrower two writes","startTime":3307.53,"endTime":3310.29,"body":"it become unethical?\""},{"speaker":"Borrower two writes","startTime":3310.29,"endTime":3311.94,"body":"So I think that question sort of takes you"},{"speaker":"Borrower two writes","startTime":3311.94,"endTime":3314.09,"body":"in several possible"},{"speaker":"Borrower two writes","startTime":3311.94,"endTime":3314.09,"body":"directions, doesn't it?"},{"speaker":"Borrower two writes","startTime":3316.287,"endTime":3318.15,"body":"Are you affecting the"},{"speaker":"Borrower two writes","startTime":3316.287,"endTime":3318.15,"body":"thing you're measuring,"},{"speaker":"Borrower two writes","startTime":3318.15,"endTime":3323.15,"body":"or can you collect more data"},{"speaker":"Borrower two writes","startTime":3318.15,"endTime":3323.15,"body":"than is ethical by addiction?"},{"speaker":"Borrower two writes","startTime":3323.28,"endTime":3324.123,"body":"- That is true."},{"speaker":"Borrower two writes","startTime":3325.95,"endTime":3327.78,"body":"I personally have to say"},{"speaker":"Borrower two writes","startTime":3327.78,"endTime":3330.48,"body":"that I don't have any social"},{"speaker":"Borrower two writes","startTime":3327.78,"endTime":3330.48,"body":"media of any type, right?"},{"speaker":"Borrower two writes","startTime":3330.48,"endTime":3333.48,"body":"I don't use Facebook, I don't"},{"speaker":"Borrower two writes","startTime":3330.48,"endTime":3333.48,"body":"have LinkedIn, I have nothing."},{"speaker":"Borrower two writes","startTime":3336.63,"endTime":3338.73,"body":"And whenever I go to Amazon"},{"speaker":"Borrower two writes","startTime":3336.63,"endTime":3338.73,"body":"or anything like that,"},{"speaker":"Borrower two writes","startTime":3338.73,"endTime":3342.03,"body":"I'm always using a VPN or some"},{"speaker":"Borrower two writes","startTime":3338.73,"endTime":3342.03,"body":"other thing to say where..."},{"speaker":"Borrower two writes","startTime":3342.03,"endTime":3344.43,"body":"I probably go a little too far, right?"},{"speaker":"Borrower two writes","startTime":3344.43,"endTime":3348.51,"body":"But for most of us, the question is,"},{"speaker":"Borrower two writes","startTime":3348.51,"endTime":3350.07,"body":"these are addictive, right?"},{"speaker":"Borrower two writes","startTime":3350.07,"endTime":3352.14,"body":"I mean, if you use TikTok for example,"},{"speaker":"Borrower two writes","startTime":3352.14,"endTime":3354.48,"body":"people can spend hours just"},{"speaker":"Borrower two writes","startTime":3352.14,"endTime":3354.48,"body":"scrolling through TikTok."},{"speaker":"Borrower two writes","startTime":3354.48,"endTime":3356.7,"body":"I mean, it's a nice time-passer thing,"},{"speaker":"Borrower two writes","startTime":3356.7,"endTime":3358.95,"body":"and it keeps pushing stuff on you."},{"speaker":"Borrower two writes","startTime":3358.95,"endTime":3360.27,"body":"I'll spend a lot of time"},{"speaker":"Borrower two writes","startTime":3360.27,"endTime":3362.58,"body":"talking about that in my final lecture,"},{"speaker":"Borrower two writes","startTime":3362.58,"endTime":3365.55,"body":"which is about the dark side"},{"speaker":"Borrower two writes","startTime":3362.58,"endTime":3365.55,"body":"of all these things, right?"},{"speaker":"Borrower two writes","startTime":3365.55,"endTime":3370.55,"body":"At what point does this go"},{"speaker":"Borrower two writes","startTime":3365.55,"endTime":3370.55,"body":"beyond giving you what you want,"},{"speaker":"Borrower two writes","startTime":3370.68,"endTime":3375.68,"body":"to making you do what they want"},{"speaker":"Borrower two writes","startTime":3370.68,"endTime":3375.68,"body":"you to do, in a way, right?"},{"speaker":"Borrower two writes","startTime":3376.32,"endTime":3378.51,"body":"This sounds vaguely, \"they want you to,\""},{"speaker":"Borrower two writes","startTime":3378.51,"endTime":3380.64,"body":"it sound vaguely paranoid already, right?"},{"speaker":"Borrower two writes","startTime":3380.64,"endTime":3382.59,"body":"But wait till my sixth lecture."},{"speaker":"Borrower two writes","startTime":3382.59,"endTime":3383.61,"body":"- Great answer."},{"speaker":"Borrower two writes","startTime":3383.61,"endTime":3387.843,"body":"So I did have another question"},{"speaker":"Borrower two writes","startTime":3383.61,"endTime":3387.843,"body":"actually, if you don't mind."},{"speaker":"Borrower two writes","startTime":3388.844,"endTime":3391.02,"body":"So harbingers, I wanted to ask you about."},{"speaker":"Borrower two writes","startTime":3391.02,"endTime":3393.09,"body":"There's all this work"},{"speaker":"Borrower two writes","startTime":3391.02,"endTime":3393.09,"body":"on harbinger customers."},{"speaker":"Borrower two writes","startTime":3393.09,"endTime":3395.34,"body":"And so a Harbinger customer,"},{"speaker":"Borrower two writes","startTime":3393.09,"endTime":3395.34,"body":"if I've got this right,"},{"speaker":"Borrower two writes","startTime":3395.34,"endTime":3398.16,"body":"is, you know, let's say"},{"speaker":"Borrower two writes","startTime":3395.34,"endTime":3398.16,"body":"you are a supermarket"},{"speaker":"Borrower two writes","startTime":3398.16,"endTime":3400.8,"body":"and you decide to have turnip ice cream."},{"speaker":"Borrower two writes","startTime":3400.8,"endTime":3402.3,"body":"And it's not very popular."},{"speaker":"Borrower two writes","startTime":3402.3,"endTime":3406.05,"body":"But it turns out that certain"},{"speaker":"Borrower two writes","startTime":3402.3,"endTime":3406.05,"body":"customers are much more likely"},{"speaker":"Borrower two writes","startTime":3406.05,"endTime":3409.65,"body":"to buy these duff products,"},{"speaker":"Borrower two writes","startTime":3406.05,"endTime":3409.65,"body":"and they're called harbingers."},{"speaker":"Borrower two writes","startTime":3409.65,"endTime":3411.48,"body":"And the interesting thing"},{"speaker":"Borrower two writes","startTime":3409.65,"endTime":3411.48,"body":"about harbingers, I think,"},{"speaker":"Borrower two writes","startTime":3411.48,"endTime":3416.28,"body":"is that they all tend to live"},{"speaker":"Borrower two writes","startTime":3411.48,"endTime":3416.28,"body":"in the same sort of place."},{"speaker":"Borrower two writes","startTime":3416.28,"endTime":3420.524,"body":"And so they're called \"harbingers\""},{"speaker":"Borrower two writes","startTime":3416.28,"endTime":3420.524,"body":"because they spell doom."},{"speaker":"Borrower two writes","startTime":3420.524,"endTime":3422.25,"body":"So if you look at your customer base"},{"speaker":"Borrower two writes","startTime":3422.25,"endTime":3425.85,"body":"and harbingers are buying stuff,"},{"speaker":"Borrower two writes","startTime":3422.25,"endTime":3425.85,"body":"that product is a duff one."},{"speaker":"Borrower two writes","startTime":3425.85,"endTime":3427.71,"body":"You know, it's the turnip ice cream."},{"speaker":"Borrower two writes","startTime":3427.71,"endTime":3430.77,"body":"So my question, though,"},{"speaker":"Borrower two writes","startTime":3427.71,"endTime":3430.77,"body":"was there's an example"},{"speaker":"Borrower two writes","startTime":3430.77,"endTime":3434.76,"body":"of pulling data in from"},{"speaker":"Borrower two writes","startTime":3430.77,"endTime":3434.76,"body":"multiple different data sources,"},{"speaker":"Borrower two writes","startTime":3434.76,"endTime":3436.38,"body":"you know, where people live and where..."},{"speaker":"Borrower two writes","startTime":3436.38,"endTime":3438.12,"body":"and that seems to me the point where"},{"speaker":"Borrower two writes","startTime":3438.12,"endTime":3441.96,"body":"privacy really starts to"},{"speaker":"Borrower two writes","startTime":3438.12,"endTime":3441.96,"body":"become very challenging."},{"speaker":"Borrower two writes","startTime":3441.96,"endTime":3444.12,"body":"I wondered if you'd sort"},{"speaker":"Borrower two writes","startTime":3441.96,"endTime":3444.12,"body":"of given any thoughts"},{"speaker":"Borrower two writes","startTime":3444.12,"endTime":3445.8,"body":"to either the ethical position"},{"speaker":"Borrower two writes","startTime":3445.8,"endTime":3449.811,"body":"or the sort of practical"},{"speaker":"Borrower two writes","startTime":3445.8,"endTime":3449.811,"body":"consequences of this?"},{"speaker":"Borrower two writes","startTime":3449.811,"endTime":3451.47,"body":"- It goes back to that question"},{"speaker":"Borrower two writes","startTime":3451.47,"endTime":3453.45,"body":"of how much should you"},{"speaker":"Borrower two writes","startTime":3451.47,"endTime":3453.45,"body":"protect yourself, right?"},{"speaker":"Borrower two writes","startTime":3453.45,"endTime":3455.31,"body":"I mean, at the end of the day,"},{"speaker":"Borrower two writes","startTime":3455.31,"endTime":3458.04,"body":"and one of the topics I'm"},{"speaker":"Borrower two writes","startTime":3455.31,"endTime":3458.04,"body":"also going to touch upon"},{"speaker":"Borrower two writes","startTime":3458.04,"endTime":3458.97,"body":"in my sixth lecture"},{"speaker":"Borrower two writes","startTime":3458.97,"endTime":3461.61,"body":"is what happens if they"},{"speaker":"Borrower two writes","startTime":3458.97,"endTime":3461.61,"body":"make a mistake, right?"},{"speaker":"Borrower two writes","startTime":3461.61,"endTime":3465.24,"body":"So if the algorithm classifies"},{"speaker":"Borrower two writes","startTime":3461.61,"endTime":3465.24,"body":"you as something you are not,"},{"speaker":"Borrower two writes","startTime":3465.24,"endTime":3467.167,"body":"how are you going to tell the algorithm,"},{"speaker":"Borrower two writes","startTime":3467.167,"endTime":3468.48,"body":"\"I'm not that type of person.\""},{"speaker":"Borrower two writes","startTime":3468.48,"endTime":3470.703,"body":"For example, let's take a simple example."},{"speaker":"Borrower two writes","startTime":3471.63,"endTime":3473.49,"body":"Google Photos."},{"speaker":"Borrower two writes","startTime":3473.49,"endTime":3475.86,"body":"A lot of you may be using Google Photos."},{"speaker":"Borrower two writes","startTime":3475.86,"endTime":3478.68,"body":"It turns out that when"},{"speaker":"Borrower two writes","startTime":3475.86,"endTime":3478.68,"body":"people uploaded pictures"},{"speaker":"Borrower two writes","startTime":3478.68,"endTime":3481.47,"body":"of Black people, Google"},{"speaker":"Borrower two writes","startTime":3478.68,"endTime":3481.47,"body":"identifies everything, right?"},{"speaker":"Borrower two writes","startTime":3481.47,"endTime":3482.523,"body":"So it says this is a picture of a beach,"},{"speaker":"Borrower two writes","startTime":3482.523,"endTime":3483.96,"body":"this is a picture of a mountain."},{"speaker":"Borrower two writes","startTime":3483.96,"endTime":3485.67,"body":"So when you type in \"beach, mountain,\""},{"speaker":"Borrower two writes","startTime":3485.67,"endTime":3488.34,"body":"it'll give you a picture"},{"speaker":"Borrower two writes","startTime":3485.67,"endTime":3488.34,"body":"of a beach or a mountain."},{"speaker":"Borrower two writes","startTime":3488.34,"endTime":3491.01,"body":"It turns out that when"},{"speaker":"Borrower two writes","startTime":3488.34,"endTime":3491.01,"body":"people uploaded pictures"},{"speaker":"Borrower two writes","startTime":3491.01,"endTime":3492.0,"body":"of Black people,"},{"speaker":"Borrower two writes","startTime":3492.0,"endTime":3495.9,"body":"Google Photos identified"},{"speaker":"Borrower two writes","startTime":3492.0,"endTime":3495.9,"body":"them as gorillas, right?"},{"speaker":"Borrower two writes","startTime":3495.9,"endTime":3498.9,"body":"And so, obviously, you know,"},{"speaker":"Borrower two writes","startTime":3498.9,"endTime":3501.36,"body":"the major reason was"},{"speaker":"Borrower two writes","startTime":3498.9,"endTime":3501.36,"body":"that the training dataset"},{"speaker":"Borrower two writes","startTime":3501.36,"endTime":3502.953,"body":"were employees of Google."},{"speaker":"Borrower two writes","startTime":3503.82,"endTime":3504.99,"body":"Who works for Google?"},{"speaker":"Borrower two writes","startTime":3504.99,"endTime":3508.47,"body":"White people, Asians,"},{"speaker":"Borrower two writes","startTime":3504.99,"endTime":3508.47,"body":"Indians, people like that,"},{"speaker":"Borrower two writes","startTime":3508.47,"endTime":3510.96,"body":"but very few Black people."},{"speaker":"Borrower two writes","startTime":3510.96,"endTime":3513.213,"body":"So how did they solve this problem?"},{"speaker":"Borrower two writes","startTime":3516.72,"endTime":3518.67,"body":"Any wild guesses?"},{"speaker":"Borrower two writes","startTime":3518.67,"endTime":3519.66,"body":"- [Chairman] Better training data."},{"speaker":"Borrower two writes","startTime":3519.66,"endTime":3520.493,"body":"- Sorry?"},{"speaker":"Borrower two writes","startTime":3520.493,"endTime":3521.64,"body":"- [Chairman] Better training data."},{"speaker":"Borrower two writes","startTime":3521.64,"endTime":3522.473,"body":"- Well, no."},{"speaker":"Borrower two writes","startTime":3522.473,"endTime":3523.47,"body":"They couldn't solve it,"},{"speaker":"Borrower two writes","startTime":3523.47,"endTime":3526.26,"body":"because the algorithm had gone so complex."},{"speaker":"Borrower two writes","startTime":3526.26,"endTime":3528.75,"body":"So what they did was"},{"speaker":"Borrower two writes","startTime":3526.26,"endTime":3528.75,"body":"get rid of the ability"},{"speaker":"Borrower two writes","startTime":3528.75,"endTime":3531.96,"body":"to identify either Black"},{"speaker":"Borrower two writes","startTime":3528.75,"endTime":3531.96,"body":"people or gorillas."},{"speaker":"Borrower two writes","startTime":3531.96,"endTime":3533.49,"body":"That's the only way they could solve it."},{"speaker":"Borrower two writes","startTime":3533.49,"endTime":3534.323,"body":"They couldn't figure out"},{"speaker":"Borrower two writes","startTime":3534.323,"endTime":3536.34,"body":"what had gone wrong in the algorithm."},{"speaker":"Borrower two writes","startTime":3536.34,"endTime":3538.14,"body":"So the question there is what happens"},{"speaker":"Borrower two writes","startTime":3538.14,"endTime":3539.73,"body":"if the algorithm makes a mistake?"},{"speaker":"Borrower two writes","startTime":3539.73,"endTime":3542.97,"body":"You know, you don't even know"},{"speaker":"Borrower two writes","startTime":3539.73,"endTime":3542.97,"body":"why the mistake is happening,"},{"speaker":"Borrower two writes","startTime":3542.97,"endTime":3545.4,"body":"and that's the worry I have, right?"},{"speaker":"Borrower two writes","startTime":3545.4,"endTime":3548.76,"body":"If you are not let out on"},{"speaker":"Borrower two writes","startTime":3545.4,"endTime":3548.76,"body":"jail, because some algorithm"},{"speaker":"Borrower two writes","startTime":3548.76,"endTime":3551.97,"body":"says you're very likely"},{"speaker":"Borrower two writes","startTime":3548.76,"endTime":3551.97,"body":"to commit a crime again."},{"speaker":"Borrower two writes","startTime":3551.97,"endTime":3554.1,"body":"- Yeah, explainable AI."},{"speaker":"Borrower two writes","startTime":3554.1,"endTime":3554.933,"body":"- Exactly."},{"speaker":"Borrower two writes","startTime":3554.933,"endTime":3556.14,"body":"- Or unexplainable AI."},{"speaker":"Borrower two writes","startTime":3556.14,"endTime":3558.15,"body":"Now I'm just watching the time,"},{"speaker":"Borrower two writes","startTime":3558.15,"endTime":3560.34,"body":"because Gresham College being"},{"speaker":"Borrower two writes","startTime":3558.15,"endTime":3560.34,"body":"an academic institution,"},{"speaker":"Borrower two writes","startTime":3560.34,"endTime":3563.67,"body":"sort of runs on the clock,"},{"speaker":"Borrower two writes","startTime":3560.34,"endTime":3563.67,"body":"and we have run the clock,"},{"speaker":"Borrower two writes","startTime":3563.67,"endTime":3565.98,"body":"which is bad chairmanship on my part."},{"speaker":"Borrower two writes","startTime":3565.98,"endTime":3568.5,"body":"There's only one thing,"},{"speaker":"Borrower two writes","startTime":3565.98,"endTime":3568.5,"body":"time left to say really,"},{"speaker":"Borrower two writes","startTime":3568.5,"endTime":3571.29,"body":"which is Raghavendra, that"},{"speaker":"Borrower two writes","startTime":3568.5,"endTime":3571.29,"body":"was a really good lecture."},{"speaker":"Borrower two writes","startTime":3571.29,"endTime":3572.43,"body":"I mean, I really enjoyed it."},{"speaker":"Borrower two writes","startTime":3572.43,"endTime":3574.53,"body":"I mean, I felt it's an absolute treat"},{"speaker":"Borrower two writes","startTime":3574.53,"endTime":3576.63,"body":"listening to someone"},{"speaker":"Borrower two writes","startTime":3574.53,"endTime":3576.63,"body":"who's so distinguished"},{"speaker":"Borrower two writes","startTime":3576.63,"endTime":3578.55,"body":"sort of just talking about this stuff."},{"speaker":"Borrower two writes","startTime":3578.55,"endTime":3580.29,"body":"I mean, for me, it was a just a delight."},{"speaker":"Borrower two writes","startTime":3580.29,"endTime":3581.998,"body":"So, thank you very much."},{"speaker":"Borrower two writes","startTime":3581.998,"endTime":3583.373,"body":"- Thank you very much, I"},{"speaker":"Borrower two writes","startTime":3581.998,"endTime":3583.373,"body":"really appreciate that."},{"speaker":"Borrower two writes","startTime":3583.373,"endTime":3584.244,"body":"Thank you."},{"speaker":"Borrower two writes","startTime":3584.244,"endTime":3587.494,"body":"(audience applauding)"}]}