Have you ever invited your players out to lunch? If not, Scott Koenigsberg has some great advice for you in the third episode of The Art of LiveOps podcast. Scott is the head of product and ad monetization at Zygna where he has worked for over 10 years. In this episode, he talks about Zynga's culture of experimentation and shares his experiences on how games are more about the players than ever before.
Have you ever invited your players out to lunch? If not, Scott Koenigsberg has some great advice for you in the third episode of The Art of LiveOps podcast. Scott is the head of product and ad monetization at Zygna where he has worked for over 10 years. In this episode, he talks about Zynga's culture of experimentation and shares his experiences on how games are more about the players than ever before.
Hello, I'm James Gwertzman. I'm Crystin Cox. Welcome to the Art of LiveOps podcast.
Crystin Cox:I've been making games for 16 years as a designer and a game director focused on live ops.
James Gwertzman:I founded PlayFab five years ago because I saw a huge gap in the kinds of access to live ops technologies game studios needed to be successful. We've put this podcast together because a lot of the information on how to do live ops effectively just isn't out there. There's way more information about how to optimize your graphics pipeline or how to put together effective game design than there is on how to do effective live ops.
Crystin Cox:And since it's pretty tough to go around yourself and just find everyone that does live ops and ask them questions. We did it for you.
James Gwertzman:We've picked together a list of some of the top practitioners in the industry, folks who we think are really pushing the boundaries and doing some cutting edge work.
Crystin Cox:And we interview a different one every week and ask them about their experiences running live games, doing live ops and even having live ops disasters.
James Gwertzman:Yeah, train wrecks are the best. That's where you really learn how to do things effectively.
Crystin Cox:So tune in. We have another interview for you today.
James Gwertzman:On with the show.
Speaker 1:Hey James. Hey Crystin. So we have a really cool guest today. We have, we're lucky enough to have, I should say Scott Koenigsberg.
Scott K.:My name is Scott Koenigsberg. I am currently the head of product and head of ad monetization at Zynga. I've been at Zynga for for almost 10 years.
James Gwertzman:And has been doing live ops and running live ops there for a very long time across a lot of titles.[inaudible].
Crystin Cox:Yeah, I mean I think he's going to be able to give us some really great insight into the way specifically Zynga, which is a company that everyone is familiar with that had a lot of influence on live games in the west. He's going to give us some real insight into the ways Zynga has changed.
James Gwertzman:He was there in the Farmville days, which is really what put zinc on the map. And if you're anyone who is old enough to remember, you know, black sheep wandering onto your Facebook page, that was Farmville and they really, I would say were almost single handedly responsible for bringing free to play from Asian markets where it had being for awhile were, had been big for awhile into the Western markets.
Crystin Cox:Yeah. At least certainly for popularizing it. And you know, he's been there for a long time, so I think he's going to be able to give us some really interesting insights.
James Gwertzman:Yeah. And Zynga itself has been on a real a tear recently. They've been growing quickly. They had ups and downs. But I think it's, it's neat to see how that company itself has also had their own sort of live ops journey.
Crystin Cox:Yeah. So let's dive right in. Cool.
Speaker 2:[inaudible]
Crystin Cox:Zynga is really well known for having sort of a culture of experimentation. Can you talk about what that means for you guys?
Scott K.:Um, yeah, so we, you know, we've been experimenting really since day one. And I, you know, comes really from Mark Picus, who was the founder who, um, definitely had this philosophy of we should test more than everyone else in our industry. And it gives us a competitive advantages of do so-and-so always been the lifeblood of the PM Culture? And now the data and analytics culture at Zynga. And so when we think about a feature, we start with a hypothesis and then we think about how do we collect right amount of data to prove or disprove our hypothesis. Um, what we found is we certainly shouldn't find the data first because that's a bad approach. You, you usually end up going in the wrong direction. Um, so everything before we put out a feature, we, we define the principles, uh, like the KPIs that we want to move or the player engagement metrics or the player experience that we want to change. And then we try to design tests that allow us to accurately measure that. And so literally every feature that goes into every one of our games is experimented on before it goes live. And usually we'll go out to a small cohort of people, um, and then we'll ramp it up over time. We'll measure the effects of obviously some features take longer to measure than others. If you're trying to move 30 day retention, you probably need to test it for close to 30 days before you make a call. Um, and over time we've developed really sophisticated systems around, um, our experimentation platform and it kind of put out there against, you know, most of the, the most advanced experimentation platforms out there. And it allows us to really focus on designing the experiments that focusing on the outcomes rather than, than doing the analytics posts, post the results. Um, and also has allowed us to do things like making sure our variants aren't overweighted to certain, uh, biases that we have that the test actually have statistical significance. Um, and I think a key part of the testing is really knowing what KPIs you want to move it. Right. I'm doing a test to move ARPDAU. Just, it's not a good test because ARPDAU was a very trailing metric. So you want to focus on really more of the leading metrics, um, that you wanna move that are indicative of what will move retention or ARPDAU or LTV. Um, so you know, in, in a game like words with friends can be moves per day, you DAU, you or in, uh, poker can be hands played, but there are even more base metrics of every part of the funnel that you need to focus on. And I think in the early days of Zygna, we were much more focused on the trailing metrics rather than kind of these leading indicators.
Crystin Cox:I think that's a, that's a great thing to dig into working with a lot of game devs. I find one of the things they struggle the most with is you'll tell them, we're going to have this experimentation mindset, right? We're going to do the scientific method. Right? And they really struggle with, okay, how do we know what we want to have happen? Right? Right. How do we actually get to this place? And how do you work with teams? You know, they're, they're working on a game, they're building a game. How do you help them drill down and actually understand what should we be testing for?
Scott K.:Well, I think that goes back to where design, product management really meets, um, is when designers come up with a game, they, I think they tend to think of it more of fun and this, the experience I want to deliver and even those experiences can be measured by, I expect my play the players in our game to do this action more than any other action because this is the most fun action. So, you know, we have empires and puzzles now at Sega and you know, I, I love the game. I played a ton and I play the match three aspect of the game a lot. And sometimes I do it on auto play, but you know, if I were to run an experiment, I might look at how many people are doing autoplay versus doing manual play. And is that based on their love of the game or the progression in the game or what, what changed that? Because if I want to move that metric, I need to understand the core motivations. And that's where you start to, you know, you bring in your, your consumer insights, you bring in your design, you say, okay, what are the, what are the, how am I designing the player's experience in the game and how am I focused on moving that? Because what do I think is most accretive to the players' enjoyment? And so that's how you start to thinking about the right KPIs, right? So that's why I'm, you know, words. It's easy to say it was for Dau. If you're not taking a move against an opponent, you're probably not enjoying the game very much. Right? So that's an easy one. But I think some games are more, uh, more finesse, uh, and trying to figure out what that right thing is, you know, and some were crafting games. Do you want people crafting more? Like, you know, we have merge dragons. You want people doing the challenge is more, do you want them spending time on the, on the base camp? Um, you know, in Zynga poker it's kind of easy, but in our match three games we have now, um, you know, an invest express match three where you can build the gameboard assets and, and customize them as well as play them at street. Where do we want people spending their time and what's best for the enjoyment. And so you start to figure out those, that's where experimentation again comes into play. So if you can define what you want them to do and say, Hey, I'm gonna measure this based on how many days a week they play or how many turns they take, or sometimes how much they spend that will tell you what the right KPIs are. So you can go back and forth and test those things. Yeah.
James Gwertzman:And you mentioned y our, you mentioned your platform i s fairly sophisticated. Are you testing only with new players coming in for the first time o r are you running tests for t he existing player?
Scott K.:We tested with everyone. And again, it depends on, and so the most important thing about the testing aspect is the cohort and the segmentation. So understanding who your players are, segmenting them upfront and then putting them through the experiments and understand the impact of each one of those experiments on different cohorts are players. Right.
James Gwertzman:So you're obviously running a whole bunch of experiments at the same time.
Scott K.:Yes. Yeah, we probably run, you know, it's hard to count, but it's probably hundreds of experiments on a given game. And in a week I'd say, um, could be more. Um, and you know, there's a danger of over experimentation and so we also have to be mindful of that we have a really strong, you know, data science team that helps kind of advise the pms who are, you know, if you give us a PM product manager a tool, just use it and we'll use it until it kind of breaks or we break it. So we have people also advise, you know, some of what the right approach to experimentation is. Right. That's cool because if you have, you know, the same person in 10 different experiments, then you have to start thinking about how those experiments impact each other. Right.
Crystin Cox:Yeah. And I mean, I think that as a data has become more and more available to developers. We've also had to run into this question that I hear a lot, especially from smaller developers, is, okay, now that I have all this data, how do I know when it's the right time to look at the data? And when is, when is it too much data? And I know you guys have a whole team full of people whose job it is just to really like data scientists to understand data. But even without that, if you, if you sorta had advice even for people who are just starting out, what, how do you, how do you get people to first approach data?
Scott K.:Um, so I, I think, uh, going back to the kind of the beginning, I think, um, first you have to have a hypothesis and an instinct, right? You can't just start with the data. And so, you know, I've had, uh, experiences in my career where quite simply I had an instinct on something and I knew it was the right instinct and I knew it should be working, but the data said it wasn't. Um, and in some of those cases we just went forward with the instinct, right? Because I knew that maybe I was measuring the wrong things or over time we shot. So I think you have to find that healthy balance, but you had just, you should be open about why you're making the decision because you're right, sometimes the data isn't always going to get you to what you want, but sometimes you just have to have conviction.
Speaker 3:And I think that's the art and science of making products that data isn't always going to get you the right answer. Right. Um, and it's interesting as we work with other game companies, as we acquire companies and bring them into Zynga is culture. We're learning a lot from them as well as they're learning from us because sometimes are they don't have a sophisticated, uh, data and analytics as do. And you know what, they're making great decisions. Right? Um, so, so I really think it's important not to hone in too much on the data to use it, uh, as appropriate. And then use your instinct where you also have to, right. Cause it's hard. Like you said, it's hard to always know, do I have enough or do I have too much?
Crystin Cox:Yeah. I always like to say data's not predictive as well. Right? Like you could collect all the data in the world. It'll never tell you the future. Right. It can tell you what has happened, but it can't really tell the future.
James Gwertzman:One thing I'm going to bring up from those questions are, one thing that we saw a shift happening in gaming as we've gone from games is going package goods to on the game industry to kind of games and services now to going to games community where the Games are becoming these community platforms. And you've seen a lot of players in, there really engaging. But you guys were s you started with social games, right? And so from the very beginning your games were communities, you know, a think back to like a Farmville. And that was, that was a really kind of very defining moment in a game space as kind of community elements that have been popular and MMOs may be for a niche audience went mainstream for the first time. How have you seen those social elements now coming to mobile and you know, the strengthening or weakening and where do you see games like fortnight diet now factory into that sort of social element? Yeah,
Scott K.:I think it's, it's interesting because, um, w w in the early, you know, Farmville, Mafia wars and Cityville and I was, I was the GM of Mafia and the GM of city. Um, and they were incredibly social games, but they were social on the platform, right? They weren't social outside of the platform. And now we have a mobile space where we don't have the same availability channels, we can't dominate the feed and requests and notifications. So, um, and quite honestly, viral, uh, organic virality is nowhere where it used to be. You know, the social graph of l from Facebook is limited because of data and privacy issues. Um, people don't necessarily want to send text messages or emails yet, so you have to create graphs truly of, of, of strangers in these games. Um, and so I think there is, but what's interesting is there still really strong social, it's the out of the game social that has now gone to discord and Reddit and other, they've self formed communities. Back in the day on Facebook. It used to be, you know, we used to laugh on Mavi wears. We used to have, you know, an ad add me thread of millions of people who just wanted people to be added to and we never brought it into the game, which was a total shame that we didn't bring that functionality into the game. Um, so it's gotten much harder. Um, and so this is again, I think where our data comes into play is that we can start to identify social graphs and tools and match making of this where can actually becomes really important because, um, in a lot of games you have alliances and then as it was aligned to degrade over time, how do you refill them? How do you create the right matchmaking? And so we've spent a lot of time, especially on games like words with friends where it's, you know, turn-based pvp. I need to offer you the right people in the right context to play a match against. If you don't have a friend available. And it becomes an all like Fortnite is interesting because Fortnite isn't actually inherently social. Like there is no real social design in it, but it is social because it's such a fun game. Um, so, and you know, the question is should they build social structures on top of that and they, maybe they should, I don't know. Um, I would say that it'll probably be creative if they did. Social generally works if you do it correctly. Um, whether it be, you know, a light, a light crew, light friend factor where you're doing a raid mechanic as a group and you all get a reward or it's really deep where you're like a leaderboard and you're competing tournament's or your true pvp. Um, social really works. It's just the matter of how do you surface it and how do you do the matchmaking break, but it's become a lot harder.
Crystin Cox:Yeah, it's such as social either. Right. Live ops has the entire package of live ops has become much more ubiquitous across the industry. You have been doing it for a while and you've been at Zynga for 10 years. How has live ops changed at Zynga in those 10 years?
Scott K.:Um, we have to be a lot more, uh, we have to plan a lot farther in advance. You know, back in the web days it was pretty awesome. You could throw two, five release a week. If one didn't work, you just pulled it back right. Or you did a hot fix. Um, that doesn't work on mobile anymore, so you have to be a lot. You need a lot more tools you need, you need to plan a lot more in advance. I think at a very high level, I kind of look at it as if we spend about 10 to 20% of our time on live ops. Not, because it's not an[inaudible], it's not 80% of the game, but because we can build the tools in the systems and tune the events and all the other things the right way, then we can focus on developing big features. Whereas back in the day, I think we spent more time doing live ops because we were creating content on the fly. We were doing so many releases and we didn't have the systems and tools and a lot of it wasn't predictive and we didn't segment right. So now we can do things like can we predict likely to churn and likely to pay. And that gives us more leverage to target people with the right content over the course of their lives, whereas we didn't have those before. So we're much more efficient in what we do. Um, but again, I think events have also become, have replaced the quests of old in the task list and players love events. But the other cool thing into your point about social games have become inherently more social where, um, and I'll call it Pvp, but whatever was some sort of, the players have become more of a content than ever before, back, back in the day. And that's allowed developers to actually spend less time on live on traditional live ops and more time on, on developing features. Yeah.
James Gwertzman:Well, one thing we've seen for sure is that the, you're right to get these games are way less about an epic story arc. And you know, it's way more about the player itself. Any clear interactions are the meat of the game right on. And that really is the heart of it. Oh. And
Scott K.:which also makes it fun, right. We always talked about in any PvP became that the players create the content and that's what allows for Evergreen Games. And when you look at Zynga poker and words with friends, those are evergreen games that have been, are 10 years old because the players are the content. Right?
Crystin Cox:The poker has been around a while. Yeah.[inaudible] a lot longer than 10 years, right? Yeah.
James Gwertzman:So I'm going to switch gears for a second. Let's talk about tools cause I, I'm a total tools guy. I love tools, you know, and uh, and you guys have alluded, you know, you alluded already to tools you built for experimentation and that sounds awesome. Uh, what else goes in your tool stack, you know, either on the support side or on the designs. I mean it starts all the way at the design when you are correct thinking cause when did we've also found is live ops starts in design. You can't build a game in an add live off like absolutely. Yes. I would love to hear this. Your thoughts on what tools you use at each stage of the game.
Scott K.:So we've built a series of central services that a lot of our teams use, like a guild service for example, scenes that are a bit commoditized that everybody's going to use and everybody can take advantage of. But then you know, one, one of the other things we have is an event framework. So we know what events look like and we can repeat them, not necessarily across all of our games, but certain high certain games that are very similar. You can use a similar event framework. The other thing that Zynga does is, um, because we have such a wide portfolio of game, we have a culture in our product management teams of sharing. So we send out readouts to each other, we learn from each other. So w if you put out a feature, one of our slots games and we, you know, does great and then we take him and say, Oh wow, we should apply this to words with friends. We often do that. And so because we have that combined learning pool and sharing, it makes us much more efficient in finding these things. I'm not, that's not really a tool, it's more of a process, but it's incredibly helpful because we're all sort of using the same KPIs and the same measurements to understand what success look like and how do we apply that. Um, from a tools perspective, we have our experimentation platform. Um, we have some central services that allow for fast deployment of certain services. Um, but events are still done on a, on an ad hoc basis. I think over time we'll probably get more sophisticated in that. Uh, but it's hard to do because events can be so game specific and the economy tuning has to be so specific to the game that we haven't gotten to the point where game teams are willing to necessarily release that from control, which a I kind of get. Yeah. And they're doing a lot of experimentation anyhow. So, um, but I think over time we'll see more of that. I think some of the, uh, matchmaking services that we're offering are becoming more generalized and we're becoming better about finding the right things because, uh, as games become more social, more pvp, what identifies a good player versus a bad player for a match is becoming more similar. Same thing with services around guild guilds and matchmaking alliance matchmaking.
Crystin Cox:I'd love to dig in into matchmaking if you don't mind. Speaking is such a fascinating topic, right? I think for, it's one of those great systems that from the outside, from the player's perspective seems very simple, just find me the right person to play. But from the inside is so complex and we've been struggling with it for like decades. As an industry, what are you guys using? Like what are your favorite of, um, vectors that you're using to do matchmaking?
Scott K.:So I, it's funny, I, so I, I went on like a matchmaking craziness two years ago. I was just said our matchmaking is not sufficient enough. Right. Um, so we just basically took everything we could think of and any, again, as game makers, we don't have a lot of first party data. Right. Because it's unnatural for you to give me, you know, your, your, your sex, your gender, your age. Like you're related. Like why should you just want to play a game?
Crystin Cox:So I so different in the east though. Oh is it? Yeah, that's, yeah.[inaudible] profiles. Like Twitter, I did most of my training and it's just crazy cause you get the social security number and you type, you get every, yeah, absolutely. Be like, were they born with[inaudible]? Yeah, it's, it's amazing. And then there's like this other side of it, what you step away, where you're like, it's a little creepy, but it is a, it was, it's funny cause on the West, yeah. You can't trust any of that.
Scott K.:So we spent a lot of time using inference, right? And so you play this way, you chat like you like to check, you liked to play at night in words with friends, maybe you, you take a move every four hours. So I should not match with somebody who takes move every 10 minutes. We have things like average word score. So we try to use in-game actions because we don't have first party data. You know, in some cases we can infer for, uh, we can make the inferences of how old you might be, um, potentially, you know, by, uh, the way you behave or your first name or things like that. So, um, so we use everything we can and then we try to test which one has the highest quality result. Right? So we may look at game completes or moves taken. Right? And so we look at all the things, I think signal this is a good game verses bad game. And we, you know, sometimes we would just run some manual tests among our team to try to figure out what works rather than rolling out to the public because we can control it.
Crystin Cox:Do you ever take qualitative data in? Do you do surveys?
Scott K.:Yes, I'll absolutely. We have a really a really great consumer insights team now who helps us kind of figure out what are players, motivations, how should we stack against them? And it really drives a lot of our roadmap. And that's another big part of live ops is figuring out, you know, if you're going to really do live ops well I think you said it, we need to plan at least three quarters out, you know what we're doing and six quarters out have an idea.
Speaker 3:Right? And that means that we have to have a very clear perspective on our roadmap from a live ops perspective of how we're going to get from a to B. Um, but the other thing that often gets missed in the live ops space I think is people plan for a future, but they don't plan for version two, three, and four, that feature. Right? And so then they have to go back and take something off the roadmap. It's going to be perfect. If you're successful, you can probably do better. So planned that you're going to spend more time on it. Don't spend think that you're just going to get it right the first time because you never do. Yeah. This is what we make software.
Crystin Cox:We talked about in one of the earlier is I do you always plan to be successful? Don't back yourself into a quarter plan to be like, it's going to be so great that a million people are gonna show up and that I'm gonna want to put all the effort into exact. Exactly.
Scott K.:And often, and teams don't plan for that. And that's when you're like, oh, now I have to make a tradeoff on my roadmap. Right. And you shouldn't be in that position because you should plan for success. Um, it also allows you to kind of free up your, you if you think without constraints or like do something better than if you say, I only have 40 dev days and so what can I get done in 40 Dev days? Right?
Crystin Cox:So you hit on one thing. Um, ray surveying is something I, I think not enough people are doing really. It's fairly lightweight and survey, like velocity is actually really great way to measure sentiment. But are there other things that you, you see the industry is starting to adopt more and more of these techniques across a wider part of the industry, right? It used to be there was a time where MMOs did this and then it was, okay, MMOs and social and then it was, MMOs and social and mobile and now it's becoming everybody polls and some popups or, oh yeah. I mean just talking about survey, survey velocity. Like if you can, uh, serve the same survey over and over again, you can start to see sentiment trends. And even though individual answers to surveys might never be all that valuable. Right? Like one person, you can say whatever, like right at any moment in time. But if you can get trends and it's fairly easy to send like a one question survey out like over and over and over and over again. But not a lot of people do it. No, I don't get asked my games that much. What I think,
Scott K.:I totally agree. Well, I think there's two parts. One is, um, players don't always know exactly what they want. So, you know, we've all seen it. I'm sure that, you know, if you ask the player, do you want to be more social? Nobody ever says yes. They never say yes. It's always, no, I want to be on my own. But then you say, would you like a feature that connection with your friends? I go, yes, that sounds great. So you have, again, it's the art of wording. Um, one of the tools that we've used in it sort of clicked testing within the game. You know, putting it in the context of the feature that you want to build and knowing how people react, right? And telling them, this is the way that we're understanding how to, because at you asking you a question about, hey, do you want a feature that allows you to rate a boss? And you're like, well, I don't even know what that is, but if you put that button in a game, you show a picture of a boss and so you could attack this and get a huge treasure. People go, that sounds fun. Right? But enough people do it. Then you know, you have some heat around it. Right? Um, so I think, you know, again, um, in terms of surveys, I like them as directional to start, but I also like to put the, the feature in the context in which we're going to build it, right. And then understand from there, communities can be a great source for that too, where you let the queue, you feed the community a question, let them kind of iterate on it and then see what they come back with. Yeah.
Crystin Cox:And you can, if you get really good at pulling those diamonds out of the rough too, you can get really fancy.
Scott K.:And that's again, a great consumer insights team and is worth its weight in gold because it'll PR. It really does, uh, prevent you from, uh, making really long forward-looking mistakes, so to speak. Um, and, and they, they kind of give you as developers that some, when you're in the weeds, it's sometimes, uh, it's hard. Yeah. And so one of the things we've always done is I always try to have put myself in those shoes as consumer. We used to just invite players in for lunch, just sit around and talk to them and ask them what they wanted. We got some of the most amazing feedback and a lot of it became features in our games because we're in an industry where we have access to our players who would just say, you're in San Francisco and you're like, come to lunch. Absolutely.
Crystin Cox:Some of my favorite insights we ever got was we used to do live chats when, um, whenever maple story was down for, uh, for maintenance, we would just open up the channel channel and be like, Hey, you guys aren't playing right now. Do you just want to talk? That's awesome. You would just get, you know, some, I mean a lot of it would just be fluff, right? A lot of it would just be like whatever's happening, but every once in a while people would really hit on something and you'd hear the whole community, oh yeah, yeah, we'd really like that. Or Wow that we did this thing and it was the most fun we've had, you know, all year or something like that.
Scott K.:It's such a powerful tool and I think all consumer product companies would love to have the access that we do to our players to be able to have a live chat.
Crystin Cox:They want to talk to you. I mean, they're dying to talk to you.
Scott K.:You just have to be able to find the diamonds in the rough. Right? Yeah.
Crystin Cox:But you know, I think a lot of Devs don't actually necessarily have the patience for it, which I get and I've definitely not sitting here saying, oh, as a, dev should just be available 24 second time.
Scott K.:So I, you need people on your team. We talked about live ops. I think having a good community manager so you can understand that, you know, um, VIP, like these are important parts, a live ops that are gonna make or break your game in the long run because fostering that community and fostering vips is, is something that's hard to do and it's really special and has longstanding rewards.
Crystin Cox:Is there anything else that you are starting to see the rest of the industry dabble with that you're like, yes, I want to see more of this or maybe, oh, we learned that lesson 10 years ago. What are you guys doing?
Speaker 3:You know, I think everybody is looking to do more personalization, right? Um, and because we now start to have the tools and the data science available to do so, uh, and these predictive things, I think that there'll be more personalization in games. And, um, and you know, your experience and my experience made may actually be different over the course of time because we played different styles. Not that the value of what we're offered or any of that will be different. It just, it'll be customized more to how we play and our enjoyment level. Because, you know, if I design a game that in essence, in order to be successful, you need to play six days a week and you love the game, but you only have three days a week to play. You shouldn't be penalized for a game that you love, that was designed not necessarily for how you want to play.
Scott K.:So what can we do to help alleviate that stress or that's still enjoying them for you? Because I would like you to play the three days a week that you have available. So I think we're gonna see more personalization. I think we're gonna see more predictive modeling, so I'm likely to churn likely to pay and then potentially evolving in, uh, your experience in the game wall Vav as a result of those predictive models. Um, I think obviously I'm a big fan of ads. I have run as a Zynga. So, uh, I think rewarded video is becoming more and more part of the core loop of many games. It's no longer an add on and that's making it much more successful. Um, I think again, Pvp is a continuing trend that you should have seen in almost every genre and every type of game. There's some type of competition that exists and even the players that tell you they don't want to be competitive are always open to somewhat competition. Um, those are kind of the main things I think.
Crystin Cox:Yeah. I mean player to player engagement I think always comes about you. Even if you're talking, I do have a group of people that are like we have, we're not interested in head to head competition, engagement player player engagement I think always has value.
Scott K.:Absolutely.
James Gwertzman:When you'd mentioned earlier on, you mentioned guilds as an example of a system and I love guilds. I actually think you guys did a really good job with them and I think in games don't use them enough. So I'd like to learn more about, you know, where w what, what role does a guild play in not just the design of game but that longterm retention and what are some examples of ways more games could be using guilds?
Scott K.:Well, you said it earlier I think is a, I think you said you can't, I forget where we were talking about, but you can't tack it on at the end. Social is one of those things that most games actually do try to tack on at the end. Like, oh, we got our core loop, we got our core game. Oh we should put some social in here. And then it never works and it's always broken. Um, and it's a hard thing to do. Social is really, really hard to get. Right. Um, so I think, you know, I even look at some of our own games and you know, we're, we're always looking to make things better. And I think one of the things that I look at is how do I make, how do I provide you the best experience, not only within your guild, but when your guild is gone all at some point. Guilds fade away. So when your guild, we used to be 10 people and now as five people, what happens? How do I match you with the right people? How do I, again, it comes down to matchmaking, but it's making sure that the right type of people are filling the right type of guilds and they have the right aspiration together as a group. And it's a hard thing to get right. And the problem with it is you can't test it unless you have players in the game. It's a hard thing to test and simulate one-on-one. Um, so I think that's really the struggle with it. Um, so I, you know, for me the, the right experience is saying, okay, where am I, where are people like me playing? Can I serve as that? Are My friends playing the game? Can I surface that? Um, what is my guild aspiration and why is it a creative for me to be in a guild or an alliance? Uh, if I can do all three of those things, I think it can be successful. If I can only do one or two of those, it gets harder and harder over time because people always lapse out of the game. We have too many choices in mobile gaming.
James Gwertzman:I've got another question, which is, is viral debt? I mean, you mentioned the virals way harder now than it used to be. And we see a lot of games continuing to try to add viral acquisition. I'm obviously UAE, you know, we paid user acquisition is a big topic and we could probably go very deep in that. Um, but I think organics are so clearly better if you can pull it off, you know, and, and know, curious to know what you, what you design in to try and encourage people to add friends and make it more, more viral and, and where you thinksthat's going.
Scott K.:I don't think it's dead, but it's really, really hard. Um, and um, but we have many features in our games that essentially are the, uh, effectively referrals, right? If you, if you do this and refer a friend, uh, you'll get some benefit in your friend will get a benefit and we're still seeing those things pay off. Um, it just harder and harder. And so, but if you don't have a social game where there's no benefit for you bringing in a friend that you can play with your friend and it's really hard to do. So, you know, if I have a single player match three game and I say invite a friend and you get a life, well why should I bet a friend just to get a life? What's the added benefit? How do I make that better? And I think, you know, we're even seeing some of the old gifting systems that worked on the web, we're seeing those work. So there's still a very big opportunity for it. Um, and I think as we have more, uh, social channels available, so like can we do something with discord someday? Like, you know, use that as a viral channel where there are gamers on that channel. Um, so I think it's finding the right mediums and then the re right reward vectors for people, but then ensuring that I can, you know, if I invite you to come play with me in the game, I should be able to play with you and find you in the game, not just you get a gift. I get a gift. Right?
Crystin Cox:Yeah. Not so transactional. I'm really fascinated by what we might see happen with gifting and streaming. Um, because I think it is bringing back some of what was there early on in the social networking[inaudible] group support. Yeah. Yeah. Like I can give, if I can give a gift to a big group and it's very much out there in a public arena because we used to see that a little bit and guild wars to where we have a lot of items that you could buy that then you would use and then everyone on the map would get.
Scott K.:I think that that is a very successful mechanism that we see over and over again. Repeat and we saw, you know back in the machine zone, days of game of war, it was if, you know, if you bought a gift your whole alliance got it or you bought something paid everybody. I think that's a super effective mechanism that the I, the key thing I think in that is to make sure that the rewards justify the purchase. I think one of the things that we often do in games and we've said is you forget how much, how much of value to put against somebody's time. So if you spend an hour grinding on a quest and I give you 10 coins and you have a balance of a thousand coins, it's like what. You know, why are we nickel and dime? Let's make sure we award people's time appropriately. And that's where that, that value exchange, that right. When you make it non transactional, how is it something that it feels special and unique to me every time?
Crystin Cox:I think we also don't really account for the, he we're talking about personalization. The longer you play in a game, the value of things changes to you. We don't do a great job of addressing that. Like we want to believe. I think as game developers we were kind of gods, we set this value this way and it's going to be like that for you forever.
Scott K.:And, and that's again where the sort of the, the systems that we have now make it much easier to say, hey, I want to tune these rewards on this arc for this player as they progress in their journey rather than every player gets three points. Right.
Crystin Cox:Um, if you have one, we, everybody loves here the train wrecks. Everyone wants to hear, everyone wants to hear about the live ops disasters. So do you have any hope, maybe they're instructive or informative or just interesting stories that a good one, not a disaster. Oh well we can switch it around if you just want to be super policy.
Scott K.:I have disasters. Don't worry. Everyone loves a disaster. I mean this goes way back and, but it sticks out of my mind because I did learn a valuable lesson. Um, we had a feature in Mafia wars called top mafia and essentially it was a, you know, you assign different mafia members to your different people of your mafia to different roles and you got bonuses, right? Um, and the way the original design, it was just over-weighted the bonuses were way too overpowering and it was sort of breaking the game a little bit. Um, and so, you know, we took a long time, we were like, we really need to correct this, but we want to make sure it's, we know we're taking something away from players, but so we have to give something back if we're taking something away. And so we designed some other bonuses and some other things that we did and they're all excited that we're going to fix the game. And you know, we, I remember we launched it like 1159 at night. Um, cause we were all still in the office. Those were the old days of, you know, pushed it in the middle of the night. And I mean, it was incredible. We had, I forget the tool that we had, like a buzz meter. So anyhow, that showed player sentiment. We watched players sentiment go from the high eighties to like negative 40 in less than half an hour ago. And you know, then when you're on the Facebook threads that people were just outraged. Um, and you know, I was on the forums kind of responding to people. Uh, like the lead product guy was like, you know, like, oh my God, what are we at? I was like, we have to just pull the plug, revert it. Just sometimes it's a roll back. We gotta roll this back because it's so bad. We thought we had it figured out, but it's so bad. And the, you know, the big thing was when you take something away, you have to overweight so much what you're giving back, even if it's only temporary, because it's a hard thing for players. Players invest their time and their energy and their money into these games. And when you take something away from them, it, it hurts that loss aversion is real, it's real. And so, uh, it was a good learning for us and we didn't we were much more careful about future changes.
Crystin Cox:Well, I think, I think every MMO probably has a similar[inaudible] better from that era. So yeah, that's very true.
James Gwertzman:But now you've earned your favorite story. So once, like, what's a great story?
Crystin Cox:Here's a wonderful, a wonderful success story from live ops.
Scott K.:I mean this is, I mean it's uh, you know, I think, no, I'll just stick with the disaster story fairly.
Crystin Cox:You know, it is like people listen it, I think it makes them feel better when they're like, oh, I, everyone's, nobody talks about all the times that you want. It's much easier to talk about the near misses and anything was so close. GDC, I wish, I wish we talked about failure more.
Scott K.:I agree with that, that that's actually great feedback to them. I have watched a lot of these presentations.
James Gwertzman:It's always a good stuff. It's always like, here's how I made them. Yeah. Here's why. Here's how it was so easy for me to be successful. Well, I give a talk back in, Gosh, what year was it? 2003 called, uh, less what to do when all goes to hell. Lessons learned at shutting out a game studio. That's awesome. And I talked about how I learned more in the act of shutting on my game studio that I learned in creating a first place.
Scott K.:Yeah. I, and I did. I will say that, you know, I launched, uh, over the years have to go. Obviously, we've, uh, worked on a lot of games that haven't launched or haven't been successful, and I've learned so much from those experiences. You know, thing, easy things like, um, you know, if you don't satisfy your core, you're not going to satisfy anyone. But yet, you know, we've tried to build games that satisfy everyone and we're like, oh, the core will get it eventually.
Crystin Cox:Yeah. Cool. Thank you very much. Thank you for coming to you both. This is great.[inaudible]
James & Crystin:thanks for listening to the art of live ops podcast. If you'd like to what you heard, remember to rate, review and subscribe so others can find us and visit[inaudible] PLAYFAB.COM for more information on solutions for all your live ops needs. Thanks for tuning in.