What do you think is the #1 predictor for retention? In our first episode, we talk with Raph Koster, a legendary game designer who has worked in online games since 1993 and was the lead designer behind Ultima Online as well as the creative director for Star Wars Galaxies. Raph was there at the birth of modern LiveOps in early MUDs and MMORPG games, so what better guest to answer these and other burning questions for our first episode?
What do you think is the #1 predictor for retention? In our first episode, we talk with Raph Koster, a legendary game designer who has worked in online games since 1993 and was the lead designer behind Ultima Online as well as the creative director for Star Wars Galaxies. Raph was there at the birth of modern LiveOps in early MUDs and MMORPG games, so what better guest to answer these and other burning questions for our first episode?
Hello, I'm James Gwertzman. I founded PlayFab five years ago because I saw a huge gap in the kinds of access to live ops technologies game studios needed to be successful. I'm Crystin Cox. I've been making games for 16 years as a designer and a game director focused on LiveOps. We've put this podcast together because a lot of the information on how to do LiveOps effectively just isn't out there. There's way more information about how to optimize your graphics pipeline or how to put together effective game design than there is on how to do effective live ops and since it's pretty tough to go around yourself and just find everyone that does live ops and ask them questions. We did it for you. We've picked together a list of some of the top practitioners in the industry, folks who we think are really pushing the boundaries and doing some cutting edge work and we interview a different one every week and ask them about their experiences running live games, doing live ops and even having live ops disasters. Yeah, train wrecks are the best. That's where you really learn how to do things effectively. So tune in. We have another interview for you today and be sure to stay tuned after we're finished with the interview. If you want to learn more about live ops academy and PlayFab. On with the show. Hi James. How's it going? Hey Crystin, g ood. So who do we have today? Today we are talking to R aph Koster. R aph Koster I'm just kidding. Everyone knows who Raph Koster is. Yeah, no, it's, this one's v ery exciting. Raph has, you know, been sort of a fixture in the live g ame space for almost, you know, 25 years. I don't know, maybe h e'd be like, don't make me sound so old. U m, we were lucky enough to corner him here at GDC. Y up. U h, where h e's speaking as he has it. Probably every GDC going back as far as I can remember. Right. And we're going to chat with him about, u h, of how live ops i n games and, and probably a whole lot more. Yeah. Especially, I think history a nd his, you know, he's got a lot of cool insights on community and the way live ops has changed over time. All right, well let's, u h, let's go right to it.
Speaker 1:[ Music]
Raph Koster:So thanks for joining us. Why don't you, uh, introduce yourself? Sure. My name's Ralph Koster and I'm a game designer who has been working in online games since about 1993, um, in one way or another, um, ranging from the old text MUDs to massively multiplayer online RPGs like Ultima Online, Star Wars Galaxies to Facebook and social games and mobile games, and even multiplayer augmented reality.
Crystin Cox:Nice. So you've been working in the industry while we've seen it change quite a lot, especially the online space. When you hear the term live ops, what does that mean?
Raph Koster:I flashed back on a day in a hallway in origin where Rich Vogel, who was the producer on Ultima online and I, he stopped me in the hallway and said, hey, I think we need a name for the team that's going to stay and keep running the game. And we kept batting around something. And finally we said, why don't we call it the live team. Right. And so I flashed back on that. Yes.
Crystin Cox:In such, there has been now dozens of live teams on MMOs ever since. Well there've been live team
Raph Koster:Well there have been live teams before that, right? I think people tend to forget that online games, you know, the earliest, the earliest computer games were multiplayer, not single player cause we couldn't do AI. And networking came along really early. Right. So even though it wasn't available in homes, online games were around throughout the 70s on networks like Plato and the first text MUD and shortly thereafter the birth of the commercial online game industry. That was 1978. So there's really been live teams since then, although we may not have called them that.
Crystin Cox:I know for me personally, actually when I look back on that era, the MUD era, I actually think there's some things that we were doing then as an industry that were really good as far as social structures and social design. They sort of lost along the way.
Raph Koster:I could not agree more. Um, yeah, I could certainly tell stories. I don't know which ones you want to hear but yes.
Crystin Cox:Well, I mean I think that everyone likes to hear mostly the disaster stories as we've found. We probably were, I was asking people for a live ops disaster.
Raph Koster:But origin stories are good. You know, early on in online games, the line between the team that kept the game running and the people who played was really, really thin and that was an intentional design choice, right. In the early games, uh, they were running on code written by students mostly, uh, on university computers and therefore they were volunteer run, which meant if you got good at the game or advanced really high and you happen to know how to delete files from a file system or reboot an executable, great, you could join the team. Right. Um, that meant that an awful lot of things that we would consider about professionalism weren't a thing. So I still vividly remember the first person I ever fired from an online game was actually an admin on a MUD that I was working on who thought it was a perk of the job. And this was a common view, a perk of the job to invisibly lurk in the game, hovering over people invisibly and watching them do things including, you know, personal things. Um, and uh, you know, just basically invade people's privacy. And when I went to him and said, what are you doing? He said, it's a perk of the job. Why else would we have the power to be invisible? And back then we had tools that actually, some of which I miss today. One of them was a command called"snoop" terrible name. But what it meant was have your view of the game exactly match that of another player. Right. Get literally directed and cloned to their network stream so that you could see exactly what was having them, which is a phenomenally useful tool if you're trying to debug a problem. But oh my gosh, is it invasive? Yes, because you could see their private messages. You can, it's everything. So, so yeah, that was the first guy I fired. And those sorts of things gradually led particularly over on the commercial side, but even on the text MUDs that gradually led to, um, the notion that we really needed to develop codes of conduct, not just for players, which were common but codes of conduct for admins. And that was sort of a radical thought. Um, I mean early admins were called Gods and wizards. Right? Right. So they had, you know, one of the most famous admin manuals was about how to turn players into toads.
James Gwertzman:I think that we've lost a lot as we've, as we've gone along. Um, because one of the themes that we've been developing over the interviews we've been doing this week alone really has been the focus in live ops and how that has shifted game design around the player and how, you know, the, the players really reemerging now as the center of the design, not necessarily a designers grand vision, but how do you design for that continuous engagement and social structures I think are really key to that continual engagement. And, and you know, now the peaks seems to be lots of players running around shooting each other, but it seems like there's a lot of those more richer interactions that maybe are coming back.
Raph Koster:I think. So we ended up back then, uh, in order to deal with those sorts of issues, what we ended up doing was taking the admin staff and splitting it in four, um, which today I think this will probably sound more familiar, but we split it into sort of the team of programmers who today we might call either live or Dev ops. Right? Um, so these folks were, we had their home phone numbers so he could call them if the MUD went down, you know, they are the ones who prompt changes, right? And all of that. And so that's a team that more or less exists today. They were also though the Games Dev team for code, but, um, the games back then were arguably more data driven than they are now. Right, right. Which was another thing that I think is coming back. And, um, and that meant they weren't necessarily needed. Features didn't actually come along that often, so they weren't continuously pushing that sort of changes. It was mostly bug fixes. Then a second group, um, was admin, an admin actually, um, the cops know, right? Um, and we intentionally split that role from community, right? Which actually we call PR back then, um, player relations not public relations. And so admin were the ones who dealt with infractions, who provided assistance as well, right? Whereas player relations was about cruise directing. It was about, you know, how do we engage just so they, how do we engage the users? So super common in those days to do things like, hey everybody, it's Thursday night, 8:00 PM, let's do a trivia contest on live channels. Or Hey everybody, we're going to play tag. You know, there were, you could play tag with like these magic wands and stuff. Um, and so on. Right? Uh, they'd build custom areas that could be tag arenas and things like that. Um, and then the last was content creators or builders, right? And those folks were the ones who are constantly adding to and updating the game. So, uh, segmented admin powers. So that you know, different people have different permissions and all of that sort of thing. And I think a lot of those practices, and particularly I think of the sharp division between the Admin and the player relations was particularly powerful and something that isn't always as clear cut today but I think is a best practice for player engagement.
Crystin Cox:I definitely agree. We had a really similar split on Maple Story and it was very helpful not just in managing the way people interacted with the game, but also managing the perception of the player base. And you know, I think they were more comfortable knowing that there was some kind of balance of power that there was someone they could appeal to. It was kind of on their side and we never asked the community team to come down hard, you know, that was separated out, GMs handle that and that's something that they're going to do. I think that actually did really help. So do you, you also worked in the social space specifically and in social games. How did you see the earlier sort of MUDs, MMOs impact how we approached that sort of new platform?
Raph Koster:Not nearly enough is basically the bottom line. And I look back on the social gaming boom as possibly the greatest missed opportunity in video game history. Um, there we had the single largest connected platform of people in history that we ever, ever had the potential to make back then the first million player concurrent game, you know, that kind of thing. And we didn't[ laugh] just didn't even, we didn't even take a serious try. It is the way I'd put it. Um, so we ended up making games that were not actually that social of course. I think it's fair to say that we eventually discovered the networks themselves are not necessarily that socially in some of their intrinsic design elements. I think it turned out there was a lot for everybody to learn. Right. But, um, yeah, so I see it as a, as a huge missed opportunity. I think a lot of the lessons around, uh, so many lessons, um, around emerging content about building live community, like very few social games actually supported players talking to one another.
Crystin Cox:Which seemed crazy at the time. Right. Like for me, I was, when the boom was just starting, I was working on Maple Story and I'd come from working on like Toontown online. So mostly with kids, but those games are all so focused or I mean they were practically just 3d chat rooms and they were really focused on like one on one chatting and physically seeing each other. It was kind of strange to see social games lag so far behind getting just chatting going.
Raph Koster:Yeah. And that's, I mean, fundamentally it's because their architecture was completely web based and so, uh, basically, you know, a Farmville client or one of these other things is really, even though it was in flash, it was really just a display of static data in a database. So there wasn't really a game server there. Uh, so there's no way for people to talk to one another. There was nothing in the way or nothing in between. Literally you played your game, you just updated the database directly and got back results. And just when we updated a row or column it, you know, it happened to look like a cabbage or whatever. Uh, we actually, I work, um, I was doing social games at a studio of mine called Metta Place, which was acquired by Playdom in Disney and uh, Metta Place prior to that had been a web based virtual world engine. So our farming games actually you could log in, see other people, chat with them, hang out and we had things like professors running classes in the vineyard around the farming game we had um, wine tasting parties and you know, all of these social things and it dramatically increased retention because community is the number one predictor of retention.
Crystin Cox:And that has been something I've noticed very interesting, especially having spent a lot of my years more focused on Asia that we were really interested in relog in, retention numbers, how long the relationship was with players. And it seems like it took the western free to play mobile and social, uh, communities a little longer to get to that point. They were more interested in acquisition it was really, really a big key KPI. And Monetization was a big, uh, sort of leading KPI. Why do you think that took longer for us to sort of come around on?
Raph Koster:Yeah, I think that's a complex story and a lot of it is actually tied in with metrics, right? So you know, there's the whole, you become, what you measure is, is sort of an a classic lesson. Um, there were early metric systems, even in the text MUDs, like I even have a slide, I pull up some times and talks showing graphs of frequency of kills and game balance. You who managed to take out this monster, is it under or overpowered for Legend MUD that dates back to 1994. Right? A graph, by the way, that far too few modern games could pull up to be honest. All right. Because they don't necessarily see,
Crystin Cox:Game designers love numbers. You know you cannot stop us from looking at that stuff.
:Yeah. And then the answer is it depends on what you're choosing to log and what you choose to dashboard.
Raph Koster:So MMO is because of their business model, it was a monthly recurring subscription that meant you lived and died on your churn rate. Um, it was all about retention. And even once the games went to free to play, which not all of them did, but most of them did. It was still about retention because there was a very clear sense of the revenue is the area under the curve. Right. And so the longer we make the curve the better. And that was just a, it was just a mentality. Yeah. We paid attention to activity numbers, but not closely. So I mean we would have like a brag wall with an led display or something, like how many people who are online, but we weren't obsessively, like, we weren't dashboarding that at five minute intervals or anything because it just wasn't relevant. We mostly wanted to know are we hitting our 10% of the user base logged on at peak concurrent time kinds of questions. Right. Yeah. Because those would be leading indicators, but you only need to measure that basically once a day. It's the rest of the time it's not as critical. It's not as crucial. Um, so our real time monitoring tended to be for emergency things. Oh, something's crashed. Yeah. In a social games, the mentality was very different. And the realization that I eventually came to after talking to a bunch of the analysts was that a big part of it was how the database queries and how the databases were even literally structured. You know, depending on how you arrange a database, a query can either be really cheap or really expensive. Right. And sort of, you know, think querying and Excel along a row or a column. Right. And basically the analogy I tend to use is that most of that industry was really, really good at building dashboards that were the equivalent of a traffic camera sitting above an intersection. At any given moment. They could tell you how many cars are in their intersection, how many people are crossing the street, how many red cars are there. Right. They could tell you everything you wanted to know about that instant. And they had no idea where t he car came from, where it's going, how long it had been there. No clue. H alf shot a t it was all snapshotting, which made for wonderful, cool looking dashboards. S ure. But if you started trying to ask a question, hey, I'd like to run a query on how, u h, how many o f this cohort who are regular chatters with players, how much is that impacting A. Their lifetime in the game and B. Their lifetime value versus players who don't have friends? I got back. Wow, we'll get back to you. Was that in like three weeks? And on an MMO, we could run that analysis relatively easily. It's like, oh well let's pull up their sheet. How many friends do they have? You know, that all were done. It was super easy. Yeah. So that was a lot of it. It was, you know, if you become what you measured to an even deeper degrees and then it seems your tools end up shaping your practices.
James Gwertzman:What I've also noticed a lot of the early mobile social games when you know, there is a, they were hiring a lot of folks that didn't have to have game backgrounds per se, but more kind of analyst systems backgrounds, right? So they would take a problem like retention and say, well, what are we measuring? Okay, day one, our attention our day, setting our attention. Okay, let's put in systems that reward you for logging every day and treat it with this insurance bond, little escalating jackpot type type mechanic as opposed to an underlying social mechanics is way harder to measure. Way Harder to design, but maybe it is ultimately more, more sticky.
Raph Koster:Yeah. We actually, in the MMO business, there was no such thing as day one. retention. We actually use the term conversion to discuss pretty much everything in the entire first month because player's got a free month and early on. I mean that conditions, everything you do, you're already thinking one day who cares about one day? If I don't make it to day 32 right, I've wasted$15 and you know that, I mean that was our, that was what we were doing, right? So for us, it was all about building an emotional connection with the player in the span of 30 days in order to convert them to a paying customer and during the social, and even actually, even today, you know, when I've worked with mobile clients, I've urged them to think similarly and say, look, this is an ongoing relationship. So, you know, in the first however long you're dating, right? And there is no not yet a conversion to going steady. There's no put a ring on it yet. Right? That's not what's happened yet. And so you have to actually get the consumer to convert to a customer rather than basically a trial. Right. And that's an emotional journey on the part of the player, right? Not a, yeah, not a day. One thing. Very rare. You know, that that would be love at first sight. It happens, but we tell fairytales about it.
Crystin Cox:Yeah. I mean, I think it's also challenging because it is much easier. And if you're not someone who's done a lot of sort of, uh, you've not spent a lot of cycles on learning to do really
Raph Koster:deeply engaging game design. It's very simple to, and I still hear this all the time from game teams where they're like, well, I want people to log in so I'm going to reward them. I'm just going to reward them for doing the thing that I want them to do. And that is so, so quickly doesn't work anymore. Right? We, we're humans, just our baselines get reset all the time. Like we don't really that motivated by extrinsic value. Um, how do we deal with that in a world where, I mean, cause I mean sometimes I like to look back at, at building MMOs 10, 15 years ago and I'm sort of nostalgic because it felt like we had more time. It felt like we, we had more opportunity to build these deeper relationships with players. How do we deal with it now? There are, there are thousands of games and you do need to make an impression quickly, but you don't want to just be what I call is like paying players to pay your, to play your game. Yeah. You know, there's a, are the techniques that Walmart uses that, uh, you know, a lot of people make fun of, but they pay people to say hello. Right. And, and it turns out that the bottom line impact of having a human greeter in that, in that service role is massive double digit returns on, on effectively conversion to being a Walmart shopper. And, uh, we saw the same in online games, uh, that, that having people, uh, you know, and at first it was the player relations folks would show up and say, hi, welcome. You know, what, are you confused or do you need any help? Or, you know, now obviously that doesn't scale very well today. Um, it didn't scale back then. Yeah. So, um, on the very firstMUD I played, uh, I was asked by one of the admins there who, uh, has also ended up in the video game industry. Uh, he ended up as working on, uh, designing on Meridian 59, Shadow Bain, Star Wars, the old republic fellow named Damian Schubert. Um, he said, hey, we need a, you know, I've been playing for a while. He said, we need a, the guild leader for one of the Guilds in our game. Back then, guilds were hardcoded players did not create them. And it was called the Golden Pyramid. I don't remember why, but the guilds charter was newbie helping. Right? And the, the deal was, hey, you guys are a guild just like any other guild. But when a new player shows up, the admins are going to tell you and tell you their name and you can go find them and offer to help. Right? And so our entire group was sort of like a lion's club or something. Right? And, and people joined the guild who enjoyed doing that later on, much later when we were doing, um, star wars galaxies, there was simply a command you could type in slash helper, and it popped up a flag over your head. And the people who chose to do it, we're advertising players could look around, see, oh, that person, somebody who's volunteered, right. To help. And you know what, it made a massive difference, right. To the community. And that scaled because it was free. Yes. And it we had actually almost no issues. There was, it was completely voluntary as players who enjoyed doing it because they liked often gilts would do it because they needed to recruit members. Right? So it was in the too often to their benefit, um, That this is such a simple technique.
Crystin Cox:I think that speaks to something t oo that I've seen a lot working on guild wars t wo where we were really adamant that we wanted our community to be friendly. And I think people were always surprised when we t alked to them about i t i s w e c an say actually we have very few problems. Basically we set expectations. We built the game around the idea of you're always happy to see another player. We tell people to be friendly and then they mostly are, I mean t his not that hard. I mean, I don't want to downplay i t b ecause it can be very challenging to deal with toxic behavior online, but I think we s ometimes shy away from just setting the expectation of offering people the opportunity to be helpful.
Raph Koster:The most important thing you can do in return to reduce toxic behavior is expectation set right to behavior gets the most toxic when no expectations are set of any sort and people come in with a mentality of therefore, anything goes right. The more you can kind of provide a framework for people to operate in, the more likely they are. There's peer pressure to do it. There's, you know, social standing reasons. There's lots of reasons why. One of the results out of psychology for a long time now, and we'd probably didn't need the results from psychology because it's, it's such a common practice it's actually in shrined in law, in most cultures, right? When you are about to take on something important, where you're making a commitment, say to tell the truth or to deliver something or whatever, you have to take an affirmative stance, sign your name or state and oath, right? This is a piece of human culture going back thousands of years. We have no idea where it started, but we know it works like it works. Well. Guess what? You can very easily ask people to make an affirmative statement and I don't clicking through terms of service is not hey, yeah, right. But it was don't fit that Eulas don't fit that. But if you, when you logged in, um, message like stark message popped up on your screen that said, hey, this is a friendly community, not going to burry you in rules but be nice to one another, be excellent to one another. Excellent to one another. And then when you click i t's s igns your name. Right. Yeah. That's been shown to noticeably reduce the i ncidents of bad behavior. Y eah. I t's human psychology. Yeah.
Crystin Cox:Switching gears a little bit, you've been doing some form of live ops for most of your career, but there are a lot of people in the industry today who have spent a decade or more in the industry and not ever have been touched LiveOps, but they're starting to have to dabble in it or go full on indeed. Do you have advice for them?
Raph Koster:Patience. Um, my experiences that an organization learning to run a game as a service is typically a three to five year long learning curve. It is not something you can just flip a switch. Um, there are temperament issues around it demands, um, marathons, not sprints and all these other cliches. Uh, it demands a certain level of emotional investment in gardening and tending a user base, which is a very different mentality from making cool stuff and making new stuff in particular so it isn't actually suited for everybody. Some people prefer making lots of new stuff and constantly doing something different. This involves the, that sort of slow effort of okay, it's going to take 20 years to shape this into a proper English garden. Let's start. Right? And that's a very different mindset and it can be very, very challenging because studio cultures are not necessarily aligned and you start getting developers who want to move off. You start getting, when are we making another game? And if business pressures are when we're making another game as a service, then there's no escape game as a service eats your studio, right? Like you don't end up with spare people afterwards. In fact, you end up at people deficits. And so that's a, a structural thing about them. Right. Um, so yeah, it's, it's a very, very challenging shift to make. And I think probably the biggest shift, and I don't want to sound too forceful about it, but the biggest shift is that the customer, the player, the, your guest becomes the king in a very particular way. It doesn't mean they're always right. It doesn't mean the perfect, but the orientation around serving. Yeah. Right. I mean the word has a root[laugh] is absolutely crucial and that is not the root of no packaged goods, that that's not at the root of that experience. And that's a difficult fundamental mindset shift for people.
Crystin Cox:Yeah. As I've been talking to teams that are about to take the plunge, a couple of them asked me, what's the thing I don't know, like I've heard a bunch of stuff, but what am I, what is it a blindside me? And my answer is always customer service. You're not thinking enough about it. You don't have enough people. And you probably haven't trained them correctly because if you've come from packaged goods, what it means to do customer support is totally different. It's not, I had a problem answer it. It's not a lot of form responses. It is a lot of individual personal experiences. It's going to be a lot of volume and you have to have that mentality of whatever I need to do to take care of you.
Raph Koster:Yeah. I and it cuts across, I mean, even if we just get super concrete on numbers, right? Even if you're, you're trying to get away with, you know, minimum wage frontline people, which I don't recommend, but let's say you're trying to get away with that. You're still in a position where you know, you're wiping away, taking one call might erase your entire revenue from that user. Yeah. So one problem that they had means they now got to play for free. And so it starts creating a very different mentality around, okay, what level of bug free can I handle? Right? Um, if I'm losing a month revenue from talking to this person, then I better get two months or three months lifetime out of the interaction. Right. And that means the interaction can't be go away, obviously. No. Um, yeah, I think, I think you're absolutely right. There's an enormous amount of things around customer service people need to learn. And one of the fastest ways that I've always seen to teach it is make absolutely everybody in the org sit at a customer service desk and have them do that periodically. Like, Hey, every quarter you gotta do your shift. Yeah.
Crystin Cox:You can't run away from the players. That's right.
James Gwertzman:I want to, I want us just to talk a little bit about, uh, the comment you made earlier about being cruise director. You know, that notion of, you know, a team is really thinking about programming the content, programming the events, um, what, especially in the world is in a mobile and social, where maybe the community structures are, the social structures are weaker than a classic, you know? And then though, what, what advice do you have of us as designing, thinking about your live events? And those sorts of games.
Raph Koster:Mobile is particularly challenging. We're walking around with, you know, the most powerful communications device ever invented. And there are virtually no channels for developer to talk to a player. It's actually appalling how bad the infrastructure there is, no offense to all giant hardware companies. But, um, the fact that in the mobile world we primarily communicate our users via reviews and, and comments on reviews and maybe sometimes patch notes that aren't even visible to the user most of the time is dreadful. Like actively dreadful. I, I honestly believe as big as mobile hits have been, they have probably left an additional digits worth of revenue on the floor because we can't even talk to the users. Uh, and that's, I mean, some of that's on the storefronts. It's on the carriers, it's on the operators. Some of it is on the developers as well, right? Because in theory, you can build those channels into your game, right? It's additional expense. It's hard, but it can be very, very worth it, right. As we've seen with those mobile games that then do end up doing periodic events and so on. So, yeah, mobile, they're actually, it's a, it's a bad place. It's very hard. Um, so I would love to recommend that people, you know, change that practice, but I also recognize that it's actually probably too big a hill to climb for quite a lot of studios because platform support for doing it is basically absent. Yeah. Tough, right?
Crystin Cox:Yeah. I want to see if you can hear that. Let me see though. The uh, the lovely, that's the noises of San Francisco. This is the ambiance of where this has happened to. We're fine. Yeah. It's not that big a deal. Um, so I think that is also something that's interesting about the era that we're in right now, which is that it used to be in a lot of ways, a lot more work to get a live game off the ground because you did have to build a lot of infrastructure. Now we are relying more and more on platform holders to support games. Do you think that that's going to stay true? I mean, I think there's some interesting things happening, you know, with, with epic specifically in Fortnite and pulling away from the Google play store and things like that. Do you think we actually will start to see, and this has been true in Asia for basically ever right, that there are these really well developed platforms specifically for games. Do you think we'll see that over here?
Raph Koster:I think so. I think, uh, you know, certainly, um, steam for has an, I've actually seen the complaint from independent developers that one of the reasons they don't want to be on steam is because team forces them into doing a bunch of community work. Yeah. Right. Suddenly, wait, I just wanted to sell my game and now I have to moderate forums. Yup. Or my or my sales will suffer because my front page will look awful. Right. Things like that. So yes, I think we are going to see more awareness of that. Um, and I think there's a lot infrastructure popping up all over the place. Um, not just storefronts, but also, you know, I mean, well, I mean there's what Microsoft is doing, you know, PlayFab and interesting that it comes b ack. Yeah. But I m ean, but you know, as you know, you're not the only ones. I mean there's, there's a lot of stuff like that that's out there now. So in a lot of ways I think the soft expertise is more important. But, u h, that said, I also think, and this is just my general advice to developers, is that they need to not be beholden to any given platform. Don't hand your community over to a storefront. You know, you need to own your own community, particularly if you're one of the smaller developers who, uh, moves from platform to platform when there's opportunity, right. You know, it has to go look for the best development deal and so on. Right. There are a lot of reasons why a developer like that will not, should not want to have their community tied down. So I think it's important for them to think in terms of owning their relationship and not letting it get, um, disintermediated or whatever. Right. But by somebody else and I, one practice that we've lost over time, I think it's actually a really long term, not a great idea to outsource your community to Twitter or reddit. Um, because you, you know, let players have a community on Reddit, that's fantastic. But if you don't have a means of communication and a bridge to the users and the only one belongs to third parties, I mean not only might reddit, go away, theoretically it can happen. We've seen that sort of thing happen. But also you can't moderate reddit. It, you can't, you know, that's the front page for your community then you don't control it as a developer.
Crystin Cox:Absolutely. There's actually been a lot of debate around this in the video industry in the last couple of years. So I think that there is some real concerns from developers being asked to wait to like sort of wade in to the community. And I think it's been interesting to see a lot of developers push back on the idea of actually getting out there and having one on one relationships with players because like the feeling from a lot of developers is that the players are too needy or they're too entitled or they're too demanding. And I tend to think of it as a, we are not doing a great job of training and supporting our devs in doing community management. What do you think about those, those concerns?
Raph Koster:Yeah, it's, it's sort of one of those situations where everything can be true at once, right? Like it is true there in any given population of players, there's going to be some that are demanding or expensive or trolls or whatever. That's just statistical. It's going to happen. Um, it is true that it is a ton of work and expensive small studio may not be able to afford to allocate the headcount. It is true that, um, you'd need to do it from a business expense. Like all of these things can be true at once and there is no easy way to cut through it. Right. I completely agree that developer, I mean everybody is increasingly in the public eye. I'm not even sure it's safe to say developers. I kind of feel like people need a license to Twitter where it's like, okay, let's put you through some basic training on how not to end up getting flashed mobbed on Twitter. Right? Um, because it's very easy to fall into those traps. One unwary remark and you might suddenly be trending worldwide for no good reason and destroy your life. Right? And that's pretty terrifying. So I think there's plenty of blame to chalk up on the platforms themselves. And again, there's lots of social design there that I don't think is optimal. I've given talks about that. So people who want to can go look on my website or whatever and find them. But I think it is also in common, particularly on companies to take care of their employees and protect them. And that means helping the employee by giving them the tools to cope, which means things like a certain amount of social media training, you know, um, when and how to use conciliatory language, how not to escalate, you know, how to deescalate tension. Um, you know, how to respond to criticisms in a way that doesn't inflame things.
Crystin Cox:It's hard. I mean, I know me personally, when I started getting death threats, like our studio just didn't really know what to do. Right. I mean, you know, like they were like, well, we'd love to help, but what do we do about this? And I think that it's a new world. Like we don't always know what to do about it.
Raph Koster:Yeah. I remember it was clear back on you. Oh, that I got my first anonymous package in brown paper wrapping at to my Home address, right? Yeah, it was, it was back on you. Oh, the first time that some podcast called my home phone number in the middle of the x- files. Actually, I was really mad at them. Um, who would you call[ laugh] please. Exactly. Um, yeah, and I, and I've been unlisted ever since. Right. But it's, uh, yeah, it's, it's a, it's a challenge and I think it comes down to the players see the developers out there. Developers are answering and engaging. Yes. Twitter is not a private platform in any way. Everything you do is public. There's no way to avoid that. You might think you skate with a security through obscurity for, you know, when you have 500 users or a thousand showers or whatever. But you know, yeah, you're, you're in public and that's it. Deal with it. Which is unfortunate, and I hate to say it that way, but then it also means that the, the company, the employer has a couple obligations. Training is one, right? But also if they're employee missteps and the community goes after the employee, part of it is the company saying, we have community standards. If our employee said or did something wrong, that is an internal disciplinary matter for us to deal with. And our community values are respect, civility, politeness. Therefore, even if the employee did something wrong, which we're not going to discuss, don't. And that's a community standards thing. And I think it's super important that anybody in a service organization understand that community standards like that have to be reinforced daily in advance so that when you need them, the culture is there already. And that's a difficult lesson.
Crystin Cox:It is. I had some of it's framing. I used to tell my team that we should think of ourselves more as a professional sports team. Um, and that one of the services we offer to our fans is a safe thing to be angry about. So they were going to get angry. Right. They were getting mad at us. Right. They were going to get on the forums. A bsolutely. R ight. B ut it's not personal. It really, that's really what it's about. It's not that they hate you or they're really out for blood. This is part of a hobby. It's a safe thing t oo for a lot of people just debate some people to yell and scream and get very fiery about.
Raph Koster:Yeah. It's a thing to get emotionally invested in. Right, right. Passion is actually the good sign. Yes. Right. What your enemy is indifference. Indifference is death for a certain right.
Crystin Cox:People on the forums screaming every day are playing in the most, a lot of times we're spending the most too. They're very invested. They're very invested.
James Gwertzman:I would love take this in another direction. So let me throw this out there and we, we had a very interesting business conversation yesterday with a, a Korean developer and they have been making huge investments in machine learning, AI on the live ops side. And they were very proudly saying that, you know, cause we're talking about costs and they were very proud to saying that they'd gotten in one of their games or live ops and this may be hyperbole, their live ops team from a hundred people down the four people. Wow. Thanks to you know, really rigorous use of essential optimization techniques. Very curious to hear your thoughts. You know, either a, are you seeing or hearing similar stories and be, you know, where you think this could go and you know is a good thing or bad thing. And I think it's an inevitable thing. I'm curious to see how we, we, cause we have, we have some role to help in guiding where that could go.
Raph Koster:Of course. Um, uh, this is certainly the first time I've ever heard of a 96% reduction. So that sounds like a, an enormous outlier to me. And my gut reaction is unwise, but, uh, is machine learning assistive? Absolutely. And um, it's really alright. It's important that everybody know machine learning is a really nice buzz word right now. But I mean, and it's really cool and awesome and it is not nearly as fancy as people think. Like it is not magic. It is not even AI. Okay. Right. And I think, you know, probably your listeners are sophisticated enough to know that, but I think it's really important that people understand machine learning is essentially, it's, it's a fancy pattern recognizer engine and um, or alternatively it is a fancy pattern variation generation engine and that's about it. Now it's a giant brute` forced one of those. But once you frame it in that context, you start realizing that, well, but particularly for something like this, pattern recognition is probably what we're talking about more, you know, generation, there's tons of interesting things down that way, probably not live ops directly live ops related. Um, as soon as you frame it that way, then you go, okay, what are things that live ops has used pattern recognition engines for before? We just have a better one now. And suddenly it's like, oh, okay, chat filtering who jumps out at you? Um, you know, recognizing troublesome, uh, service histories over time jumps out at you. You know, there's a whole bunch of things like that where machine learning could immediately apply value. Not just reactive value like chat filtering, but uh, because it can detect subtler patterns it can offer predictive value. Like you can, you can analyze'em, you can analyze chat in order to detect this person is going to explode in profanity soon rather than they already did. Right? That sort of thing is of course, incredibly valuable. Um, uh, churn. You know, churn is a great example. We have been able to predict churn to very high degrees of accuracy for like a decade and a half. Right. And machine learning is going to let us do it marginally better basically. Yeah. And we'll be able to respond faster to it, you know, so the efficiency gained will be consequential, but we're still talking about going from like even a crude model of how often is this user chatted with other people. Right. That by itself doing anything else will get you about 80%. Yeah. Like literally 80% accuracy on predictions. So we have a pretty good starting point, right. So there's enormous value in going from 97 to 98. Don't get me wrong, but, uh, but I think it's super important that people realize, you know, this is another step along the way. Um, it cannot replace human connection. Yeah. That would be the generative side. And we're still not good enough at the generative side. And if anything, I'd be actively concerned. You know, we've all had this experience. We, especially the spam calls these days, you know, he gets a phone and somebody who, you know, it rings and hello is Frank there, wait no. So there's no like, oh, well I'm glad I got you. I'm here to talk to you about this. Click right here. Right. Um, and I, and I think most people are angrier at that than I am at an actual person calling saying, are you interested in solar? Right. Um, and the same has been true. Like the more impersonal the contact is, and part of what we resent is the fact that it feels like fakery and trickys And so there's this temptation to go use, let's use the generative side and procedurally answer all questions. Yeah. That actually really worries me. I think if any valley exists for more than just a visual, exactly. You know, it will come across as they don't actually care. Right. Right. And so that's why a 96% reduction would worry me. Sure. But is there enormous value? Absolutely. There's a number of things out there. We are, we are seeing entire companies forming about middleware for doing things like, um, uh, AI based moderation, right. With machine learning of various sorts, often in really clever ways. Right. Um, you know, having NPCs react to bad behavior, things like that. Cool. But, um, you know, there was a bartender in Celtic Ireland on legend MUD that when a player was in the tavern and cursed, she would march over frog marched him out of the tavern and wash their mouth out with soap. Right? So yeah, you know, simple rejects can go a long way and it's the power of that is actually in the frog marching and the soap, right. Not the detection. Right. I think that's super important. Right? It's really important and machine learning isn't going to provide that kind of magical spark, right. That that made that moment where like everybody laughs and stops cursing. Yeah.
James Gwertzman:Well this looks like it did is really important. Yeah. And I think, you know, tools like the ones that we provide allow games and capture a ridiculous amount of data if they choose to. And yet we, one of the common things again from these we've had this week is it's hard. Analytics is hard and it's hard to know what to capture. It's hard to know if it's right. It's hard to know to do with it. Love to hear again. You're kind of just musings on, you know, analytics and data, both. Either advice for developers just getting started or lessons you've learned about, you know, using it effectively.
Raph Koster:I think the first and biggest thing is people need to think about what are their true KPIs. Right? And you know, I mean that's kind of an old business school lesson, but, um, but I think people still get it wrong all the time. You know, when they, when they start thinking about, let's say it's gameplay analytics, it's still about, okay, what are the things I actually care about? Like, it can be awesome to measure how many player weddings were held in my shooter or whatever, but there's not going to be a KPI probably. U m, it's really easy to fall into the trap of activity versus retention or you know, what one of those really big ones. And picking the wrong one of those, r ight. That is a really easy trap. U m, the, the, the, the thing I always tell people is that, look, if you really take the step back, I've had access to an awful lot of granular data about usage, acquisition and churn for a lot of online games. And the shape of the curve on all of them is exactly the same. They all have exactly the same curve. A J curve and an S curve are just slices of the same curve. And if you run enough regression analysis on the curve, you can actually predict your peak usage from the first like eight hours of data. It's actually shocking how accurate it is. Um, and, uh, if you, you know, you can go look up things like the best diffusion curve and you know, all of his stuff and it really makes you realize that the, the, the thing that separates the fad from the viable service is whether or not the curve stretches out. Right. And again, it's area under the curve and the area under the curve is going to be driven by what are your key metrics. There are business situations. Absolutely. If your key performance metric is I need to get this much money in this quarter, okay, that is fine. Right? Like some businesses are like that. Um, let's say it's a, even in the game as a service context, maybe it's a DLC or an expansion, in which case you don't need the curve to be long. You just need it to peak high. That is okay. Even in a game as a service context, sometimes that is the business need, right? Uh, you know, we'll run short on cash. Okay. I mean, you've got a need. Yeah. You need to work against it. Um, that is the kind of of careful thinking, right? Cause we might easily fall into the trap of it always has to be stretched out as long as possible. Right. Or we've, we had just as often, it's gotta be tall. And No, it doesn't have to be tall. If it's really long. Yeah. There's, uh, you know, what are you trying to do, right? What do you, what is are you trying to do? That's the key to data. What are you trying to do? And actually thinking about it because I think we go on autopilot a lot. That should tell you then what is it that I, what are the questions I should be asking? How do I dashboard and then be very careful about what you dashboard. Don't, don't dashboard too much and don't dashboard things that don't serve the point.
James Gwertzman:Yeah. So back in back in the days in, you were in a, in live ops or guiding live ops the tools, almost got like a pilots skin rotation on a dashboard. What are the things that you would look at each day? You'd come in the morning and what were the things you'd want to see to sort of check the health of your game?
Raph Koster:Uh, it varied by era for early MMOs, the, the things that were most important. Uptime, um, lag latency performance, right? Because both of those then drove population and for synchronous participation games, if you concurrency falls low, the whole thing falls apart. So population concurrency, but uh, sentiment. And honestly the first thing I would check every day when I sat down was not the dashboards it was the forums. Yeah. Sentiment was more important. Right. And sentiment is the slowest to move. Well rather it's the slowest to move up. It can go down really fast. And so it because of that it's, it's extremely sensitive. It's a predictor of all the others. So if something starts to go south, it's the first thing you have to work on, right? The server will come back up. You know, people are going to get that back up and it's not going to take a week, but it might very well take you a week to fix the sentiment. Like a week might be quick actually. And so I'm back in those days where you were in it very clearly for the long haul. Um, you know, an an MMO kind of thing where that the moment your, your sentiment, uh, predictor is running on a month out because it's, what's the resubscribe moment, right. You know, so there were a shelf moments before that, but the true decision point is when they cancel, right? So that's, we had to think of it that way. As we move forward into more modern days, some of those are always there. Performance doesn't go away. Um, concurrency. So on, uh, those, those don't change. Um, there's uh, measuring engagement with other users becomes even more important because many of the games are not structured around it. Right? And in an MMO, you could pretty much assume players are getting together and chatting. It was so intrinsic to the genre. If you're making a game where that's not the case, then it's something you need to actively tend and actively garden. And even in the Facebook games, this was around things like how many friends, how many active friends are, uh, you know, are the little viral requests for assistance? Are they getting listened to? Are they responding right? That was our proxy for is there actually a community of people actively engaged here. Uh, one of the common early mistakes in social games was to build giant friend libraries and it turned the process of doing the invites into a chore, which led to less engagement, which led to the game failing. And we found not that bar cannot be too big. Right? Right. It needs to be something people can use. So it shifts over time. It shifts with, with the genres. Um, but you know, there's always the truisms of uptime, community sentiment, performance.
Crystin Cox:So as we wrap up, do you have any disaster stories you want to share with us? All kinds? Uh, probably one of the most famous in my case was Christmas and Ultima Online. We always gave away Christmas presents to the players every year and early on one year we wanted to give them a present. We hadn't budgeted development time are very, our time for a new piece of art. And we were, you know, we just misjudged, you know, there was a cycle for patching art into the game back then. And it's like, oh no, we forgot to have something in there. So, um, I took a pine tree and we already had, we had pine trees and we also already had a little colored rocks and you know, in Ultima online we had a s an extremely powerful server side game scripting system, uh, which an enormous amounts of the entire game were built in that it was, by the way, something I highly recommend for all lives off teams code as data server side updates, not client updates you shouldn't need to do builds to keep the game fresh.
Raph Koster:Um, so what I quickly hacked together was a little, you got a little deet in your backpack and you could take it out and place a Christmas tree in your house or outside or whatever and it would place the pine tree and then spawn all the little rocks and each of the rocks had a little callback timer to turn it to go and come back. So they were all blinking. It was a really pretty little Christmas tree. Yeah. Beautiful. Until I overloaded the message callback queue for the entire service. And because of course, everybody got the deed on Christmas morning. I brought down the entire service on Christmas morning. Yeah. Completely. U m, fortunately because of that style of b ackend, it was a one character fix. Right. Like literally comment out the blink and we were back up. Y up. But boy was my producer angry boy were the e ngineer's mad because it took them a while to figure out what it was. U m, I have many, many more embarrassing stories like that. O h, we c ould go an hour on just those. Absolutely. We could, yeah.
Crystin Cox:Everyone loves to hear them. I think it, it makes people feel better knowing we all make mistakes.
Raph Koster:Yeah. It's life. It's life. I mean, also, I'm not sure why it was always Christmas. Uh, but, um, we spawned Santa clauses in the towns and they sat there and they would periodically say, oh, Ho, Ho, merry Christmas. Wonderful. We, we forgot to make it. They, we've remembered to make them invulnerable. That was good. Nice. We forgot to make it so that players couldn't steal the clothing off them. And, um, you know, Santa hats and Santa's outfits were very quickly a hot commodity, which meant we were faced with two simultaneous problems, right? The first was you now log into the game and there were naked kids and naked men in the game standing in front of you going, oh, Ho Ho, merry Christmas. That was not so good. And then what was perhaps just as bad is that we also had roving bands of homicidal Santa clauses who would rush at you, kill you, and take your stuff chanting Ho Ho, Ho as their battle cry. So well, and I think that's the, that's one of the beauties of live ops. Okay. Um, is, you know, sometimes this crazy shit happens and it's actually, it can generate the best stories, the things that people remember that happened 20 years ago. People still tell these stories. And so, you know, to some degree you gotta roll with the punches. You gotta lean in on him sometimes, like, okay, why not? Um, uh, one of my favorites is we allowed people to build a furniture and ultima online. So, uh, guild decided to block all the entrances and exits to a city by piling up armoires and desks and chairs and they prevented people from getting out the main gate. And there was another gate that was really narrow so players would walk out and get bopped on the back of the head and robbed the minute they came out. That's classic. That's actually like a medieval technically. It is. It is. So our live ops solution was to lean in and the next patch note, we just quietly mentioned that axes could now chop up furniture and yes, game play evolved it turned into drama. Right. And, and I still favor that, you know, that sort of approach rather than the approach of having somebody just delete all the furniture from the database. Right.
James Gwertzman:That's not how memories prompt drama. It's like yes and no.
Raph Koster:The genesis of live was, well, it's kind of like we've been in rehearsal, right, the shows live now. So yes, it is improv drama. That was literally where we were trying to draw the, the word from.
James & Crystin:That's awesome. Well thank you so much. Yeah. That's a really, really good, good, good stories and great lessons.
Raph Koster:Yeah, happy to. That's great.
Speaker 3:[ music]
Raph Koster:The LiveOps podcast is brought to you by Microsoft Azure. PlayFab is a complete life app solution with everything you need to iterate, optimize, and grow your live game. Learn more and sign up for a free account@playfab.com that's p l a y f a b.com. Thanks for tuning in.