Today we are joined by Torfi Olafsson, the game director for Minecraft Earth. Listen as he shares and talks about his experiences working on online and MMO games since 1999.
Today we are joined by Torfi Olafsson, the game director for Minecraft Earth. Listen as he shares and talks about his experiences working on online and MMO games since 1999.
James Gwertzman (00:05):
Hello. I'm James Gwertzman.
Crystin Cox (00:06):
I'm Crystin Cox. Welcome to the Art of LiveOps Podcast.
James Gwertzman (00:15):
Hey. So, this was a funny one today, because I actually ... You flew this one solo.
Crystin Cox (00:20):
Yeah. I know you couldn't join us, but I had the opportunity to have Torfi Olafsson sit down with me, and so I ended up taking it. And Torfi is the game director for Minecraft Earth, which is the, as of this recording, it just went into closed beta, so very exciting.
James Gwertzman (00:39):
Yeah, getting a lot of press. This is like Minecraft in the real world. It's location-based. It's augmented reality. It's a pretty hotly anticipated new project.
Crystin Cox (00:48):
Yeah. So really exciting. It was really awesome that he was able to spend some time with me. We got to talk.
Torfi Olafsson (00:54):
My name is Torfi Olafsson, and I'm the game director on Minecraft Earth, a new mobile AR game that's just hit beta. Before, I worked at CCP Games, which make EVE Online, and I was part of that team from [inaudible 00:01:09] guess I started working in the industry in 1999, and then mostly focused on online and MMO games, and dabbled a little bit in VR as well.
Crystin Cox (01:18):
Also, Torfi comes from a long history in MMOs. He worked at CCP working on EVE Online for many years.
James Gwertzman (01:25):
Wow. Okay, so I can't wait to hear how it went.
Crystin Cox (01:28):
Yeah, you'll have to listen to it later.
Crystin Cox (01:29):
Nice. Yeah, I mean I think that's something that I hear a lot from people who've been mostly in MMOs, is you get years and years of experience on one game.
Torfi Olafsson (01:45):
True.
Crystin Cox (01:46):
And so your time at EVE, you probably saw things change quite a bit. When we talk about LiveOps, starting back in '98, we're sort of talking about the very beginning of LiveOps, especially in the western world. Can you talk a little bit about how you saw the perception of operating a live game change from the time you guys were launching EVE, to the time when you were stepping away just a couple years ago?
Torfi Olafsson (02:11):
Okay, well this is [inaudible 00:02:13] history, and I'm sure this is very skewed. This isn't accurate data. But yeah, so on EVE we started developing it in 1999, and the original founders of the company, they were heavy [inaudible 00:02:28] online players, and of course we were inspired by other games, but I worked at a VR company back then. In '98 we were trying to do VR. It was kind of the ... everybody was inspired by Neal Stephenson's Snow Crash, and William Gibson, and kind of the cyberpunk literature of the era, so we were trying to build a virtual world, a virtual universe. It was the same seeds that rather then created places like worlds, after worlds, and later Second Life. I wasn't involved in those projects, but the thinking was the same. We had live avatars on the internet, and people, we were using motion capture and 3D graphics, and it was all very fancy. But back then most people were using dial-up modems, so you could only have like five people in a very small room that was made out of 300 polygons. And basically the technology in our thinking was a bit too early. So the company I was with kind of abandoned that VR strategy.
Torfi Olafsson (03:32):
But the technology was there to have servers connected, to have people connect from all over the world, have things moving around, have some game rules, and so the founders of CCP took that technology, or basically didn't take the technology, because they weren't allowed to, but they took the engineers that knew how to build the technology, and the designers, and made EVE. Now we were of course going to ship it a year-and-a-half later or something. That's what we told the investors. It was like a complete total lie, because it didn't ship until four years later, but ... or three-and-a-half years later, but the thinking was always, let's work on this game, let's ship it, and then let's move on and make a sequel. And what's the next game going to be? I remember all these conversations like, "What's the next game going to be?" during development. And as it turns out, there was no next game, at least for like 10 years or something like that in the history of that company. But a lot of the developers that we had came from a business background, had been writing business applications, and online applications, so I think we adopted the LiveOps mentality that you could never actually ... you'd always have to test your hypothesis. Like you have an hypothesis about the behavior, but that hypothesis basically goes out the window the minute you go live.
Torfi Olafsson (04:56):
So we built a number of systems, thinking that they would work in a particular way, but we always at least strive to have instrumentation, and be able to watch our metrics. Back in the early 2000s, and sorry, I'm nostalgic and reminiscing here, storage was expensive, and CPU was expensive. And the concept of the cloud wasn't there. So you had to build your own data centers, and fast hard drives for servers were expensive. So you would just throw away your data. After like 30 days, you threw away all your user data. And imagine the tragedy when we kind of discovered that, when we wanted to start digging into the history of what had happened, both the behavioral history and also to let our players know what, since we had players that had been playing for 10 years, what their journey had been. We just didn't have a record of it. Some of it had gone on tapes, and those tapes have gone into vaults, and those vault were thrown away, or something like that. That has changed a lot.
Torfi Olafsson (06:07):
But I would, though, say that our approach to LiveOps, and in particular ... so our definition of LiveOps was more like service-oriented than design-oriented. LiveOps was what we would call the people that basically ran the servers and made sure that the CPU graph wasn't at 100%, other than the engagement metrics. But we had a very naïve way of looking at it, simply just how many people online, and how many people logged in yesterday. Watch those charts. So a lot of our game systems, so a lot of our actual ... the mechanics of the game, were completely opaque to us. And as a result, people had all these theories about what people were doing in the game. EVE Online in particular is kind of famed for its player versus player mechanics, and its kind of anarchist landscape, where people build empires and go to large wars, and that's the stories that would emerge from the game. But the sad truth was that most people were soloing the game, as most MMOs do. The people prefer to play alone together. And the content they were engaged in was the mission content and killing NPCs and grinding, basically.
Torfi Olafsson (07:34):
And then there were these apex players, these apex predators, at the top of the food chain, who were organizing these large player alliances and large corporations, and those were also the most vocal ones on our forums and in the media, and media would write about them. So that when we would read the forums, or we would read our own press, that was our kind of source of data, like in the beginning. So we would make design decisions and focus our design efforts to kind of appease or react to that signal. Whereas we didn't have a signal of the actual gameplay yet, not strong enough. And by only serving the 1%, I know this is ... yeah, how could that ever happen, that you would only serve-
Crystin Cox (08:28):
What? Yeah.
Torfi Olafsson (08:29):
... a narrow minority of people who sit at the top of the food chain? It's crazy, right? I think we really suffered and had a lot of churn. Meaning we lost a lot of users, and because we weren't serving the kind of mid-range and entry-level users, because we didn't focus our efforts on those.
Crystin Cox (08:53):
Right, yeah. I mean I don't think a lot of people talk that much about that era, and how much data storage and access to data really shaped a lot of the way MMOS were ran at the time. We went through similar things on MapleStory, where that game was very much built in an era where, yeah, very quickly all of your data needs to be moved to tape, and then you don't really have access to it anymore, and maybe you never get access to it again. And how much that did shape how we were able to make decisions, and have to really prioritize picking and choosing what data can we actually look at, what data can we actually care about. And I think it's actually a really common story that you default to qualitative, sort of ad hoc feedback on forums, or social media. And I still see people do it today. What was the transition like when you guys started going, "You know what, we see the results of only serving the very vocal minority. How are we going to actually get better about that sort of early game retention?" Because I actually think ... I know there must be something of a success story in there, because EVE sort of famously had a lot of growth in the middle of its life cycle.
Torfi Olafsson (10:11):
And I cannot remember the percentages, but the retention numbers, like the ... for these highly engaged players, they would play for, I think it celebrated 16 years the other day, like they ... When I left there, a couple of years ago, I think 20% of the people who had signed up in the first month were still playing. So you had a huge amount of people that would keep their account active for, what, 15 years or 14 years, so that's great. So that's the argument for serving the 1%, it's like, you will keep them. But your churn is going to be much ... you're paying all this marketing money and you're getting all these people into your funnel, and you're just churning them out like crazy, because they're not being served, and their needs are not being met. Now your question was about what the kind of transition, when we started getting data. And looking back, I think it was kind of after the company had had an explosive growth, and was able to, of course, hire more staff, and storage was starting to get more affordable, so we started bringing in people who were doing analytics, and were looking at databases, and they started presenting the data to the designers. And of course as you know, game designers are infallible priests, whose truth is-
Crystin Cox (11:43):
Absolutely.
Torfi Olafsson (11:44):
... absolute. Yeah.
Crystin Cox (11:44):
They never make any mistakes at all.
Torfi Olafsson (11:47):
Unquestionable. And they know their experience. But game designers are very commonly people who enjoy playing games. And not only do they play games, they play a lot of games, and for the game that they play, they are very engaged. Like people talking, or they're chatting, "Oh that's a fun game. I did a couple hundred hours in that game," and like this many hours on that game, so their opinions are very biased towards high-end players. So when they were presented with the data, they would ... the first reaction of course would be to challenge it. Because when some data guy shows up, doesn't know anything about design, shows up and starts showing you all these charts and all these numbers, and telling you you're wrong, your first reaction is of course to dismiss that data person as like some sort of coldblooded number cruncher, bean counter, and who just completely doesn't understand the true art of game design. So I seem to remember there were a lot of debates, and people were quite unwilling to accept, whereas some designers would say like, "We should just follow the data entirely," and I think that's dangerous as well.
Torfi Olafsson (13:08):
Because then you get into a situation much like a news website, that would follow the data, and which we observed of course, the internet would create click bait and headlines which were just based for clicks and likes, and click to get the next page. And if you design a game like that, yes you will get a lot of engagement, and yes you might get a lot of ad views, and yes there are very successful games that use machine learning to optimize games for that, but they are perhaps not the most fun games.
Crystin Cox (13:46):
Right. And I think you trade ... a lot of times it's a very simple equation of trading short-term engagement for long-term engagement, right. It's, to me, when I think about really successful LiveOps, and really successful LiveOps design, it's a marriage between data and psychology, and you have to have that psychology side of it as well, so that you don't just blindly follow trends. Also recognizing blindly following data down rabbit holes can lead you somewhere you never intended to be, which can be very dangerous. So you have to have the psychology side, and I think mostly designers, and when we think about whatever the skills designers have, one of the biggest ones is being a practical applicant of human psychology.
Torfi Olafsson (14:35):
Absolutely. And also understanding the wide spectrum of emotions that you as a game designer are supposed to play. Commonly, that's actually ... armchair game designers will commonly come into the game design room and tell you stories like, "My cousin was playing this game, and something happened to him, and he felt bad. We should never allow people to feel bad playing the game. Games should always be fun." Now that was a notion that was like, as you can imagine, highly challenged on EVE Online. Like if you compare ... At the end of the day we're not creating bits and bytes. As a game designer I'm not trying to create bits and bytes, and we're not creating binaries, and we're actually not, yes, money is a side effect of what we build, but my goal as a designer is to create emotion. And it's to create, like at the core neuro-biological level it's to control the secretion of dopamine and serotonin in the player's brain, and to just, to have a player experience joy, anticipation, loss, regret, anger. You have to play all these notes, and if you're just hammering the single note of joy, constantly, you will not ... first of all that's not, you're going to churn out very fast. That's not a fun game. You have to have anticipation, you have to have drama, you have to have a twist. So I mean just like a storyteller has to play all the emotions.
Torfi Olafsson (16:21):
Now you have different types of stories. You have romantic comedies, where they go super light on the negatives, and it's mostly about joy, and maybe he gets the girl, oh he gets her, that's great. But you still, even writing the worst corny rom com ever, there is some loss, there is some tragedy, and there's some-
Crystin Cox (16:39):
There's some conflict.
Torfi Olafsson (16:40):
... conflict.
Crystin Cox (16:41):
Yeah.
Torfi Olafsson (16:44):
So in a game, designing only for joy and happiness and fun, that game's going to die very soon. But if you have tragedy, if you have loss, if you have tremendous regret, but if you then ... and then empower the user with a sense of mastery, so that their loss and their conflict and their defeats are not random. It's not like just being randomly attacked in the street. So you give them a sense of mastery to overcome these challenges, and make it their own, but not just a random dice roll, you have created a very engaging and entertaining game.
Crystin Cox (17:25):
And I mean I think that to me there's a big difference between entertainment that's meant to be consumed, and is sort of this static thing that often does lean a little bit more towards just joy and light, because it's meant to be consumed, like a rom com, as you say. They're relatively short, you're sort of meant to consume that, have this sort of escapist feeling, be sort of teleported into this state, but you're not really meant to live there. And you're not really meant to engage with that.
Torfi Olafsson (17:56):
Exactly.
Crystin Cox (17:56):
Whereas when we're talking about LiveOps games, we are talking about games that are really centered around community. They're centered around human interactions, usually with each other. The human drama that happens there. That is only worthwhile and interesting because there is some risk. Because, as you said, there's this drama and this tragedy that comes with it. And actually, molding that kind of experience for players is incredibly challenging, but I do see ... recently, Rod Chang, who was the lead on Minecraft China, and I think he recently got promoted into managing more games in the Southeast Asia area. But he gave me this very interesting analogy, where he said that a lot of game designers want to be movie directors, but a lot of game designers actually need to be nightclub managers.
Torfi Olafsson (18:58):
Absolutely. I couldn't agree more.
Crystin Cox (19:00):
Right, I really love the analogy. I think it's great.
Torfi Olafsson (19:02):
Yeah. And that's exactly what I'm describing there. So I have friends down in California, and I'm often at some dinner party or something with people that work in that industry, like television or in film, and like "Oh, you are working on a game, so what's it about?" And then there is no story. Like, "What's the ending, what's the goal?" There is no goal. "And how do you win?" You do not win. It's like life. It is impossible to win. And furthermore, they ask, "So you're a game designer. So you write the story?"
Crystin Cox (19:36):
Right.
Torfi Olafsson (19:39):
And the concept of it being nonlinear is completely alien to them. Also, I've been lucky enough for the past few years to consult at the ... There's a lab that the Sundance Institute runs called New Frontiers Story Lab, which is an amazing lab where people who are designing interactive experiences, such as VR experiences or gamified experiences, which still have a strong narrative element, and are being brought kind of offsite with game designers and screenwriters and other people from various industries, to consult with them on their projects. So these people are at Sundance, they have a tremendously important topic or story, usually, they want to tell, and they want to tell it through interactive media. And the first day is always the shattering of dreams. Because they will usually lay out an incredibly detailed story. "So what's this game about?" "Oh, you start here, you go into this room, and then you pick up this object, and this person comes and talks to you, and then you go through this door," and my question is always like, "Where's the agency in that?" "What do you mean?" "Where are the choices?" "Oh, you can go through this door and that door," but you always go through like ... because the ending ... because they want to be an auteur.
Torfi Olafsson (20:51):
And I would actually admit that a lot of us who were working on and designing EVE Online, we wanted it be like that. We wanted it to be way more auteur, and we wanted to control the experience [inaudible 00:21:04] and I'm sure you experienced the same, as you launch a game, this ... It's like when your children become teenagers. You have no control anymore, and they've just become something you didn't even make, and you have no way. You just have some vague form of carrot and stick, but they're mostly on their own. And now your duty is perhaps just to kind of nourish and serve, rather than to guide and direct.
Crystin Cox (21:31):
I mean that's very true. I feel like it's hard, I think, for people who've never had a live game to kind of understand how much loss of control happens once players take over your world. And you can't really win that. I mean you can try to win that battle, you can try to continue to control it. I always used to tell my teams, "They outnumber us. We'll never make it." Right?
Torfi Olafsson (21:54):
No. Absolutely.
Crystin Cox (21:55):
They're always going to win.
Torfi Olafsson (21:56):
Their sum IQ is-
Crystin Cox (21:56):
Absolutely right.
Torfi Olafsson (21:56):
... supersedes ours.
Crystin Cox (22:00):
We can't really win this battle. And so you have to let go of that, and you have to really get, not only okay with, but hopefully you come to really appreciate the beautiful sort of challenge of crafting a world for people to live inside of. And I worked a lot on MMOs that are aimed at kids, so people sort of preteen and teen age. For us a lot of it really came down to finding a lot of meaning in understanding why players wanted to be in this world, and why they wanted to have agency and control inside of this world, and what that was doing for them, that they weren't necessarily getting. Especially for minors, it's very much ... The real world is, there's not a lot of respect for them out there. There's not a lot of opportunity for them to feel like they're valuable and they're contributing, and they really have something to offer to the world. So they're sort of permanently in the state of, well no, you're just learning. You can't really give anything back.
Crystin Cox (23:01):
But when they start making that transition, what we saw a lot, especially in MapleStory, was this great desire for them to come into this space and be important, and be skilled, and be able to express themselves in a way that other people respected, to have knowledge and information, and so that was really, ultimately we were like, "Wow, we're creating this opportunity for them to have this thing that they really can't get anywhere else."
Torfi Olafsson (23:24):
Well they also don't, like if you think of this generation, and compare it to when we were growing up, when I was growing up, which was very long time ago, and the agency that we had over ourselves, and our environment, and our schedules, like being on a bicycle the entire summer, and just riding around town and exploring abandoned buildings-
Crystin Cox (23:46):
Totally, significantly more-
Torfi Olafsson (23:46):
... and construction sites.
Crystin Cox (23:48):
... I think, than the subsequent generations.
Torfi Olafsson (23:52):
Exactly. So they're very much boxed in. They don't get to play with scissors or fire, or let alone gasoline, or experiment with pyrotechnics, or high voltage electricity, with all these things [crosstalk 00:24:05] that I had fun with, like as a child. So their only place to experiment with these things is inside virtual worlds. Their only place to have agency over their environment, to have freedom to explore, and to have fun with each other, like they're not even allowed to have conflict anymore. They can't fight anywhere. So the only place where they can fight is within virtual worlds. So they have to, and I think experimenting with all these things, with causality, and setting things on fire and pyrotechnics and gasoline, and getting hurt, and getting into arguments, and dealing with group dynamics and conflict, is a necessary part of growing up. Because just like any other animal, we have to go through these things. We watch kittens and they're fighting the entire time. They're preparing for hunting. We have to get that preparation somewhere. So what I'm trying to say is, perhaps we're providing the last refuge for them to develop these skills, because their parents have taken that away from them, because they're always in violin class and-
Crystin Cox (25:11):
And yeah, very controlled environments. And you really spoke to me with your ... it's providing a space to have tremendous loss, and have that kind of risk, was something we saw over and over and over again. That kids would come in and they would spend hours and hours and hours building up an item or something, and then they would just risk it all. They would just do the most insane sort of risky things, because they were just experimenting. They wanted to push the boundaries, they wanted to know what would happen. And it was fascinating, because they would do that in a kind of public space as well, and they would get reinforcement from their peers. And sometimes it would be a, "Why did you do that? That was stupid. That was so silly of you do that," and sometimes it would be, "No, no, yeah. You should do these things," and you really did see that kind of development around group dynamics and what it means to be inside of a culture, especially a culture of your own choosing, not just a sort of culture of geography.
Torfi Olafsson (26:10):
And that's actually a really good point, because I mean that's what the internet did. And you remember the early days of the internet, it was all about tribes, lost tribes finding each other. Because that single goth person somewhere in the Midwest could finally find all the other goths on Usenet, and be super active on a Usenet group, and the internet was much more of a fringe place than a usual place, because it allowed these lost tribes. And I observed that super strongly in EVE Online. I remember being so surprised I was making all these friends within the community. I'm like, "Why do I like these people? This is some random guy from New Jersey, why are we connecting so strongly?" We have exactly the same interests. Ah, the game is, like the theme and the setting, and is bringing us together. And we don't have the boundary of geographical boundaries anymore.
Crystin Cox (27:06):
Mm-hmm (affirmative). So now though, you're working on a game that is a mobile game, and a very sort of modern take on LiveOps. How, I mean, you've hinted at it a little bit, that definitely access to data has changed things, but how is this experience different? I mean probably a million ways it's different, but what are some of things that really stand out to you, now making a new game in 2019, as opposed to what it was like building a game in 2000.
Torfi Olafsson (27:36):
Right, well so we went into closed beta, meaning we had our first batch of invites sent out, two days ago, so I cannot speak with the same depth as I can speak after working on EVE for 18 years or something like that.
Crystin Cox (27:50):
It's not that fast yet.
Torfi Olafsson (27:51):
Right.
Crystin Cox (27:52):
You can't know everything in a day.
Torfi Olafsson (27:53):
Exactly. Now that being said, this game is entitled being, of course, Minecraft Earth, it's a fully online game. There's not offline mode to it. People, just to kind of recap just a little bit, for the audience out there, Minecraft Earth is an AR game, which actually uses augmented reality, actual augmented reality, not geocaching a lot, but it also uses geocaching. People go out, they have a map, they collect items, and also find these small adventures, which are geo-placed, and the items are Minecraft blocks, and they use these blocks to build up their own worlds in augmented reality. And then they go into these adventures, these small Minecraft worlds, in augmented reality, meaning they always have their camera active, and they have to move around and move their phones, and fight monsters and collect blocks, and build up their collection of blocks and animals and mobs.
Torfi Olafsson (28:49):
And it's built from the ground up as a social game, so all the game balancing is very much skewed towards collaboration, and kind of like a board game would be designed, where it's you versus the environment. This is very much PvE, and it's also designed so that a player who is very advanced, playing with a player who is at a much lower level will have a great time. And the player at the lower level will actually be lifted up to the level of the advanced player during their play session together. So a parent can play with a child, and so nobody feels left out, or has to feel bad about they're not good enough. So that's the very short description of the game.
Torfi Olafsson (29:37):
And now we have amazing data and amazing metrics. The only issue with launching a Minecraft title is scalability. So we can't, if I go into like deep technical stuff, we can't just throw it all on a single SQL database. MMOs of the old would be able to run on a 486 in a corner just running a single instance of a single database. EVE Online runs on Microsoft SQL. It's on a cluster, but the core SQL database, I think still to this day is a single computer.
Crystin Cox (30:14):
Wow.
Torfi Olafsson (30:14):
But with super fast SSDs, and it's all overclocked and water cooled and rack mounted in some amazing hosting center, but that's a single SQL database. Meaning that all the data is relational and easily indexable and accessible. However, Minecraft Earth had to be built up to be hyper-scalable across the globe, across Azure data centers, in a lot of different regions, because we're going to be supporting a lot of countries, so we weren't able to host it all on a single SQL shard, meaning that user data ... It's much harder to just run a query like, "Give me all the people who have an obsidian block." Run that query, that would just tank the Azure. Once the game, and hopefully as it becomes as successful as we're hoping, that would just completely kill everything. So everything is stored in blobs, in some data streams, and in some sort of big data databases. So it's a bit different. So we have to build up ... so we can't query the actual game data, so we built up a number of data warehouses, and stream data, like player activity data, to our data warehouses, and then we can query that data in separate tools like Kusto and other tools, and then stream that to Power BI.
Torfi Olafsson (31:33):
And as we went live a couple of days ago, I've been playing with the data. Some unfortunate developer accidentally gave me full access to everything. And with my poor old SQL skills, I've been digging into it and it is amazing. It's just like night and day. It's so easy to build a chart of like random things, and run queries random things, and bring them into Kusto Explorer and have pivot table ... I'm going super technical, pivot table in Excel and just play with the data, completely live, you just right click and refresh, and you get the live player data, and you have charts on your spreadsheet, something that would have taken days or not weeks to compile and ask for, and then you publish that to Power BI, and then you install the Power BI app on your phone, and like I was yesterday buying pizza, and I was watching like tens of thousands or hundreds of thousands of samples of data moving across a global map on my phone. I was just watching player behavior in London as I was standing on my pizza, waiting for a pizza. So the access to data and the ease of use with tools is tremendous.
Torfi Olafsson (32:43):
The only ... our biggest challenge is dealing with scale. And then of course there's the usual issues of knowing which questions to ask. What to record. What are you tracking, and is your data solid, and is it correct?
Crystin Cox (33:02):
Yeah. I mean that's something we hear a lot. A lot of teams, especially in this world, where it is so much easier to get access to data, will get kind of ... go a little bit data mad, and they'll just telemetry everything, right? They'll just instrument absolutely everything, and they'll just be drowning in data. And then sort of run into this wall of going, "Okay, I know everything, and yet I don't know anything, because I cannot look and absorb all of this data," so we have to figure out what is actually meaningful, what data matters to us. And I think it goes back to something you said at the beginning, which is it's really about hypotheses, forming hypotheses and experimenting to see if you're right.
Torfi Olafsson (33:45):
Exactly, and it was useful. For example, we're looking at ... People have experience points in our game, and they collect the experience points, and when they have enough experience points, they reach a level, and the level unlocks some content. And we've designed a number of levels, and we're expecting people to go at a particular pace. During our play testing we've installed the client and we walked around Redmond Town Center, and walked at [Redmount 00:34:04] and tracked our own behavior, and then we created the hypothesis around that.
Crystin Cox (34:08):
Right.
Torfi Olafsson (34:10):
And then we had spreadsheets for this. And what's really cool, once the minute we went live, or actually a little bit before, because a number of people had the game before, within the team, I was able to extract all the data, and simply chart it exactly. We had our spreadsheet of the hypothesis, and then I had a scatterplot of the exact data, so I was able to correlate the exact data, and I showed there was [inaudible 00:34:30] there were some things that it hadn't anticipated before. So I was able to go in, and we were able to tweak the experience point values and the levels like almost live, and basically adapt the branding model to our hypothesis model, and deploy it just in a matter of hours. And that basically fixed progression in the game. At least for now. We're probably going to have some issues later.
Torfi Olafsson (34:58):
And now I'm observing other behavioral data, I can't actually ... I think I shouldn't be talking about exactly what it is, but there's a major issue in the game, which I'm observing in behavioral data, which is streaming in, and we're addressing it very quickly, and it was an issue we couldn't have seen until we hit some scale. And that's the more important thing, that a lot of issues, especially because if you think about games, it's kind of cynical, but I think of games just like combustion engines, where fuel goes in, it goes through like a turbine, and there's ... this thing spins and it propels this thing, and it's like ... or like a chemical factory, where all these things go in and there are certain reactions. And one reaction feeds another reaction, which feeds another reaction, and some of them have like a flywheel effect, and turbine effect.
Torfi Olafsson (35:51):
So they're completely nonlinear, and there are certain patterns that emerge when they hit certain numbers. And if the engine isn't spinning fast enough, that pattern will not emerge. Because there not enough heat in the system for the secondary effect. What's the point I'm trying to make. Yeah. So as you go live, I think that you have to ... you're talking about drowning in data, you actually have to watch all the data, because you don't know what you want.
Crystin Cox (36:21):
Don't know.
Torfi Olafsson (36:22):
That's my approach right now, is to just stream all of it. Look at all of it. And some of our specs, as we were planning for data, a lot of people wanted to show us averages. "Oh, we can show you the average actions done by a player per month, or per day." It's like, "That's not valuable. Averages tell me nothing." "Oh okay, the sum of all players who do ..." No, no sense. Actual instances of data. I want a scatterplot with 50,000 samples across like in five dimensions.
Crystin Cox (36:54):
You want to see the deviations, right?
Torfi Olafsson (36:56):
Right.
Crystin Cox (36:56):
You want to know how, yeah, just an average is, who knows how much a standard deviation is in that average?
Torfi Olafsson (37:02):
Yeah, yeah, like otherwise we would have 1.5 children. It doesn't tell you ... There's a tremendous amount of data that can be hidden in averages and medians and so on. So if anything, my proposal for anybody thinking about launching a game or going live, is to just fire ... spray your systems with data, try to look at it in multidimensional ways. Like look double charts and scatterplot and three-dimensional plots, and then you will start seeing correlations. Once you start seeing correlations, you can converge, you know what data is important and you know what's going to stay constant, and what data is just noise. But then you're ... then it becomes super valuable to know the averages and know ... but just starting with averages is, you're going to be flying blind.
Crystin Cox (37:54):
Right. So sort of let the game teach you what it's about.
Torfi Olafsson (37:58):
Yeah.
Crystin Cox (37:59):
While it's live.
Torfi Olafsson (38:00):
And just as a side note, Tufte ... I can't remember exactly, there's a scholar on infographics, which is my favorite scholar, his name is Tufte. He's written a number of books. He, and in one of his books he write about how the Challenger disaster could have been averted with better infographics. You may have read-
Crystin Cox (38:21):
Yeah, yeah. I-
Torfi Olafsson (38:23):
And then people are kind of just looking at numbers and averages, and they weren't actually looking at deviations. In the Challenger disaster example it was the temperature, the outside temperature, which made some rubber O-rings that were holding fuel tanks together become too hard, so they were leaking.
Crystin Cox (38:41):
Yeah, but if looking at the averages it was like, "Oh, well we're well below the limit on the average." But unfortunately, there were spikes in those numbers that were way above where they could handle.
Torfi Olafsson (38:51):
Exactly.
Crystin Cox (38:52):
So thank you very much for this. We're just about out of time. I have one more question, because we like to do this with everybody. Do you have any LiveOps disasters that you'd like to share with us?
Torfi Olafsson (39:03):
Well, our biggest disaster was when, on EVE Online I think, we had a file in the C program files, it wasn't called 86 back then, because we were all 32 bit, EVE Online directory called boot.ini, which was like a short ini file that described what the app was supposed to do when booting. Incidentally, Windows has a file in its C directory called boot.ini which tells Windows what to do when booting up.
Crystin Cox (39:36):
Mm-hmm (affirmative).
Torfi Olafsson (39:36):
So we released an update to the game, which took the boot.ini and updated with new data. And of course it just went to your C directory, and it didn't go to your program files, EVE directory.
Crystin Cox (39:47):
Right.
Torfi Olafsson (39:48):
And we destroyed the Windows install on people's machines, and we had like 20,000 players affected or something like that. It was like the biggest disaster, and we had no way of telling them, because they didn't have their machine. They couldn't go on the internet.
Crystin Cox (40:02):
Right, you had actually destroyed their Windows install.
Torfi Olafsson (40:03):
Yeah, and so we had to send out all these emergency emails, and-
Crystin Cox (40:09):
Oh no.
Torfi Olafsson (40:09):
... messages and websites, and it was so hard to reach people. Like of course, I mean their email goes to their spam, they're not on Twitter. That was a major disaster day. I mean we eventually recovered, and we posted the instructions on how to fix it, and I think we gave people some stuff, but it was one of the worst days in the history of the game, where basically we didn't just destroy the install of the game, we wiped their ... they had to reinstall Windows to stop ... yeah.
Crystin Cox (40:33):
Wow. That's a great one. I think also because it really speaks to, it's just a naming convention problem. I mean it's literally just, we gave this file the most obvious name, not really thinking that, yes, a lower level function would probably also have used that very obvious name for this file. I think it's a good thing to remember. Well, that was great. Thank you very much. This has been awesome. Thank you so much for coming.
Torfi Olafsson (40:59):
Thank you. Thank you for having me.
Crystin Cox (41:06):
Thanks for listening to the Art of LiveOps Podcast.
James Gwertzman (41:08):
If you liked what you heard, remember to rate, review, and subscribe, so others can find us.
Crystin Cox (41:14):
And visit playfab.com for more information on solutions for all your LiveOps needs.
James Gwertzman (41:19):
Thanks for tuning in.