Kongregate: Peter Eykemans and Tammy Levy

Is there a template for free-to-play game success? In this two-for-one episode, Peter Eykemans and Tammy Levy, who both work in monetization at Kongregate, share the blueprint they use to help their entire portfolio of games engage, retain, and monetize players. 

Transcript

Speaker 1: 0:05

Hello, I'm James Gwertzman. I'm Crystin Cox. Welcome to the art of live ops podcast. I've been making games for 16 years as a designer and a game director focused on live ops. I founded PlayFab five years ago because I saw a huge gap in the kinds of access to live ops technologies game studios needed to be successful. We've put this podcast together because a lot of the information on how to do live ops effectively just isn't out there. There's way more information about how to optimize your graphics pipeline or how to put together effective game design than there is on how to do effective live ops and since it's pretty tough to go around yourself and just find everyone that does live ops and ask them questions. We did it for you. We've picked together a list of some of the top practitioners in the industry, folks who we think are really pushing the boundaries and doing some cutting edge work and we interview a different one every week and ask them about their experiences running live games, doing live ops and even having live ops disasters. Trademarks are the best. That's where you really learn how to do things effectively. So tune in. We have another interview for you today.

Music: 1:10

[inaudible]

James & Crystin: 1:10

Hey, so Crystin, I understand today we have two guests on our podcast. Yes, it's our first time and that's a, it's a double interview. Oh yes. We've got Peter Eychmans and Tammy Levy and they're both from congregate. Right and I know they work on monetization there. I've largely handled the production side and the logistics and how to do this and sharing information and Tammy's largely handled the analytics solid working and so as kind of the the combo of leadership from congregate, we provide kind of the best possible services to our development partners based on what we see elsewhere and all of that element. We're on one hand kind of middlemen and consultants, but on the other like we're in deep and we see these things happen and we see what works and what doesn't. Yeah, they think they're going to be mostly talking with us about things like the way gay monetization has changed a bit, especially when it comes to the sort of rise of ads and like ad viewing and ad revenue and also then the move to mobile for congregate. Right. Congregates been interesting because they started originally doing free web games and flash and[inaudible] transition now doing a full on mobile game publisher and they've worked in some really high profile mobile games like the hyper hippo games, a venture capitalist, a whole bunch of others. So we should have some really good insights from them on how to make those games successful and how to monetize. Well let's get them in here.

Music: 2:32

[inaudible]

James: 2:32

So you're in a really unique position because you have a chance to look across a lot of different games in different stages of sophistication and life cycle and understanding. Whereas I think a lot of other developers have, you know, if they're lucky, one game they're really focusing on. So maybe you can share sort of based on your experience, what does sort of the past evolution of, of live ops sophistication look like when you're starting with a developer who's maybe only ever made premium games and they're tiptoeing into the waters of trying to add live ops? Like do you say, okay, start here, like do this, and then you know, as you see more and more sophistication, what's the kind of the leader later stages look like?

Peter: 3:08

It's a, it's very dependent on each developer. Like you said, there's teams coming out of other aspects from console development, from premium development, premium mobile and elsewhere that they see these popular free games on the app stores, um, and download them and they're like, Oh, this worked, this worked, this worked. And so they check a few boxes of what other teams are doing and have these assumptions that they're immediately going to have a bunch of installs and it just will work. There's, there's users and free to play. I know this automatically happened, but I think the case is understanding that you really free to play starts at the fundamental design of a game from from balance flow progression and how you're being rewarding players, what you're doing to get them to come back later and make them want to come back later. And it's, I always kind of call it the most democratic method of gaming because a player votes with their, their interests, their, their eyes, their time and eventually their money. If you retain them and they like what you're doing and they don't, if you make a bad game, they don't have to stick around with it. It took five minutes of their life to download this thing, check it out and walk away cause you weren't not doing something of interest to them or giving them a reason to stick around. And so you have to find these elements to make a quality game, to keep people sticking around and coming back day after day to really move into the long tail of these games. Cause the free to play game isn't just a product launch. It's, that's just the start of a race. Everything before the start is just training, prep, all of that. But the race is a, you know, we're looking at games on our mobile publishing that are six years old now and still driving meaningful revenue. They're still up and running based on these fundamental fundamental live servicing pieces.

Tammy: 4:45

Yeah. And I think that something that's very interesting when you said like, uh, you know, levels of sort of sophistication with, uh, you know, some developers coming from premium wanting to kind of get started on on Frito play. Think of would what Peter mentioned is really important. Like it comes from the ground up and how you think about the game. Many times we've seen and we've launched some games where, um, they, they design it as a payday and the, the, they're like, Oh, premium games are no longer working. Let's just slap some free to play mechanics on it and launch it and get a bunch of installs and make a bunch of games, a bunch a bunch of money. And that doesn't work. Like, it's not at the core of the game how you design it. You, you designed it to be a game that you, uh, pay for and then play and you're adding kind of all this stuff on top of it to have people kind of pay as they go and it just doesn't work. It feels kind of a little bit disconnected. So it's really, um, many times it's just letting a developers really internalize that and think about the game as they're designing it. It's a free to play game. What does that mean for the game? Yeah, maybe the mechanic is the same as if it was a premium game, but everything after that, it really has to speak to that free to play mentality and the live liveops you're going to do afterwards.

James: 6:11

Yeah. And that's one thing that we've, that's one theme we've been talking a lot about is this theme of how company cultures and mindsets have to change to be successful in this new world of live ops and, and, and just how the core of game design is changing. There's more about creating these meaningful, interesting player engagements to keep them coming back as opposed to I as a designer, I have this massive vision for an experience I'm going to create. And maybe players like it or not. It's a very different mindset. Um,

Tammy: 6:35

yeah, one thing that I really liked that you mentioned is that it's those mentality of uh, engaging players and having them come back and engage with your game. And that's part of the free to play mentality. And I think that that's where the industry has evolved because, you know, 10 years ago when, uh, quote unquote freemium games came out or like, uh, you know, like virtual currencies was like a new thing and we, we still cold these players that were doing micro transactions that are not really micro. Um, you know, it's like, it's evolved over 10 years. And uh, back then the mentality was just like kind of just tried to get players to spend, right. It was all, um, kind of thought about from like the point of view of like spending. And now it's really, it's much more than that. It's an experience. It's a full experience that you want to provide to your players so that they keep coming back, like in and that's going to encourage them to pay. But first and foremost they need to come back to your game.

Crystin: 7:34

Can you talk a little bit too about how that shift over to designing games? Thinking from the beginning, this is going to be free to play, has changed the idea of who the player is and it has changed like what we think of as the addressable market.

Tammy: 7:49

Um, yeah, I think one, one of my favorite, uh, talks from our, our CEO Emily is, uh, she talks about people, we spend a lot in games not as whales but as hobbyists. And I think that's very important cause you know, like people are like, Oh yeah, just go find the whales and like get the money out of them. Right? And no, it's people who are really, uh, playing a game as a hobby. And I think that the market is now more and more open to that. And you see it with a lot of the games that have captured out audiences for you. New as Peter was saying, five, six, seven years and people are really like just, it's part of that they're there every day and they're engaging with this game on an everyday basis. I think that's how it's helped evolve the market and I think it's come with time and seeing that kind of crystallize, uh, that it's no longer like attached to a console or something like that. It's like it's on your pugging and you're going with this game and it's coming along with you and you're playing it for many, many years.

Peter: 8:53

Similarly, I think that's why we've seen a lot of success with kind of cross platform games where, you know, we, we've had this web platform all along and as we brought games from web to mobile and also back from mobile to web, we've had these unique links where we're giving players more opportunities to kind of play at their leisure and how they might like to play. Like maybe they're playing a game on the bus on the way to work. Maybe they open up a congregate tab in their browser at work and hide it when their boss's nearby. But they keep playing and they'd bring their account up and they're able to kind of experience it at their leisure. And you know, you see things like Fortnite that are across all these platforms now. It's kind of that similar experience. It just fits into your life wherever you are. And it really ties into this hobby aspect. So there's, it doesn't mean that if you go to a new platform, you're suddenly going to have a better game or more successful game. But it's about providing the opportunity for players to engage and interact and, and really do it their way.

James: 9:48

I'll just say, let's talk about data because I'm really in need. You mentioned you use yourself as head of insights and data for, for, for congregate. And I think, I think for a lot of companies that is actually where they start. They start with, okay, what do live ops games? Give me the, give me an opportunity to, to learn about players in real time. So I'm gonna start with the analytics. Um, but gosh, that's a big field. Where do you, where do you, how do you, where do you start with a new game? Like if you're in, if you're advising a game on Han instrument or game to be useful, what do you tell them? Where do you start?

Tammy: 10:16

Uh, so one, one thing that we've learned over the years is that most people are really bad at analytics and that's, that's not to to this anyone, but it really is, uh, because it sounds easy and it's not easy in the sense that when you are starting you might think you need something and in reality you might need something else or you might realize that you have instrumented something, uh, poorly. So one of the things that we've done as a publisher is we've tried to, it's actually what call blueprint, set out a blueprint for all our games to kind of give every game about the same level of information. And the way we think about it is in tiers. So, and it's, it's very much like your game funnel, right? You started at the very top. Our tier zero is all the basic information that you want. And if nothing else, you at least know of the health of your game by looking at that. So all the analytics tracking that you need around, um, session behavior and spending behavior for your players, but at the very high level and from there we start to drill down a would we go tier one, tier two, we basically just go deeper into the game. So your tutorial, um, high level interactions with the game, like the players play a battle in a car battler uh, that they win or lose kind of very basic. And then from there we go deeper into economies and all those pieces I think we're a lot of teams get lost is that they want to capture everything at the very beginning and from the very start of, of when they launch their game. Um, and what test markets are for. Yes, when you launch your game, you want to be there, but give yourself time to instrument that information as your game starts solidifying because your game is going to change and test markets, parolee and in development as well. So, um, have the very basic so you know, you know, are my players coming back? Are they finishing the tutorial? Are they understanding what the game's about? That's the most important thing that you need to answer and that's needs to be there when you go into test markets. But you do need to that time in test markets to refine not only your game, but also your analytics around the game and be ready to live, operate your game when you launch the game globally,

Peter: 12:35

There's a, an element of kind of speaking a common language too. Oftentimes when a developer comes to us with a game they maybe had in test markets and they'd been working on, they want to partner to help make it bigger and, and grow in launch. Yet they'll present us with data and say, Hey, this is how our game retains and all this stuff. And if it's abnormally high, the first thing we do is ask how they're calculating it and try to figure out if we're on the same page there. And oftentimes I'm a developer, we'll bring us what's called rolling retention, which is kind of an average of everyone who ever comes back versus hard retention, which is looking at specific days as a more realistic measurement of how often people are playing the game. And if it's too high, it's almost always because they're taught. Speaking of a version of this we don't use, so we always have to double check that. And there's this great moment where early in our publishing days, this game came to us called little Alchemist and they showed us these numbers. We're like hat right? Okay, yeah, you're using the the soft one that makes you look better. And they got us the actual raw data that we started digging our way through it. We're like, wait, no, this is real. We are on the same page. They are talking the same language. And we partnered with them to, you know, bring an incredible game to a more broad audience. And it became part of the team that built animation throw down one of our biggest hits to date. So it's these unique ways of learning to speak to each other. With the right type of data and the right type of understanding how things are going.

Crystin: 13:53

So you guys really came up as a company in on the internet, right as co connected kind of from the beginning. Um, and now your, you're really a publisher both for web and for mobile. I think that um, the kind of services you guys offer are very different in a lot of ways than what game developers traditionally think of as what publishers do. Can you talk a little bit about like what you guys see the role of publishers being and especially as we move into a world where games are more and more focused on live ops?

Peter: 14:26

Yeah. I think it's a case of our whole kind of history of understanding free to play. It came from the web before we ever started on mobile. I think in 2013 when we launched our first game, we had, we'd had data from over 400 free to play games on web. So we already knew this works, this doesn't work. And that gave us this kind of baseline to, to enter the market there. And so immediately from game number one, we were working with a partner on, we had all these insights into what the game might look like and what's working and what isn't. And we've kind of built upon that ever since. And our teams have grown in different ways to understand, um, kind of what we're good at. And we've had some stumbles along the way. We've launched some games that just haven't hit, I mean, in games I think most games actually fail or just aren't huge hits. And so we've seen a lot of that ourselves and we'd done what we can to try and grow them and kind of make them work. But in the end, what we come with is this, you know, years of experience in free to play that is just compounded by this portfolio and not just one or two or three hits that, you know, maybe some of our competitors have. But this kind of more broad multi-platform approach. Um, that gives us this really good insight into what's going to work and what isn't. And we try to match that up to a developer's needs cause we're working with teams from, uh, one of our games is basically made by two guys in Reno and artist and an engineer. And we have other ones that are studios of 30 or 40 people who have data scientists on staff and are sometimes ahead of us on our analysis. And so it's this combo of finding our strengths and a developer's strengths and how we kind of pair together like a puzzle piece. So ours, I don't want to say that our services differ, but the importance of what we do differs on kind of a need by need basis so that we can best serve what the developer is missing based on what we know and what we do. So one theme that we've, um, that's been coming out, a lot of this conversation so far is the importance of focusing on engagement and retention versus a monetization at least initially. Um, and one of the big tools that engage in there are live events and games. You know, we were just chatting with someone who's talking about how the day they started adding events, their numbers really kind of took off. So it looks like what events, what, what?

James: 16:32

What do you advise games? Maybe I've never added events their game. I get them. Where do you start? What are the key ingredients of an effective live events strategy?

Peter: 16:41

I think it's a matter of understanding what's fun about your game. Like what's that loop? What's the core kind of goal? What are you trying to earn or build upon? And then you find an offshoot of that that you can k ind o f encapsulate in a finite amount of time and you, you can toy around with that t ime f rame. We, for every game i t's a little bit different. Is it a two day event, a five day event, a week long, or even a month long? I n some cases, these ongoing pieces. And then ensure that players are clear o n what t hey're supposed to be doing, earning or fighting for that there's l eaderboards to kind of measure where you're going. And then that for being on the leaderboard, there's rewards for doing well and they're just these elements to keep pushing things a little bit further. I was walking through, um, I don't want to be in the bars pocket politics that did all of these things post-launch. So we have this baseline from launch of what the game looked like and then they added events without leaderboards. So just, Hey, do this thing for two days. And players would, they got him to go aged and things got a little bit better on the monetization side. Then they added a leaderboard but no rewards and it went up like another stair in a staircase. And then finally they added rewards for the top of leaderboards and it just went up again. So it's a staircase of improvements that really pushed that element of human nature to kind of compete and get engaged and be involved. And so a good event just has, you know, um, a limited timeframe rewards that make sense. And oftentimes developers will run into this issue where, Hey, we don't have enough to give away in the game any more. Like we've given away too much of this high powered thing or too much of this. And so that's why you see in a lot of these top games like Oh, there's a thousand, you know, a hundred different crafting materials or all these different things to kind of ensure you can always give out meaningful content. And so that's part of an effective event is having the economy and the resources and game to give away to do it. And then yeah, leaderboards and if you have the ability to do guilds or clans to make it more social, great. But you can also just do solo events. So people kind of competing individually, but it's just about the time, the rewards and the visualization of competition is really what puts the pressure on a good event.

Tammy: 18:49

Yeah. And I think that um, a lot of what happens is that developers might get scared. Like open my game is not about competition. Why? Why would I put some, an element of like a leaderboard on the, on a non competitive game. It's a single player game. Right. And it really, we've seen it time and time again where as soon as you add a leaderboard on a game, even if it's a single player game, now you start seeing your progress against other people. They might be complete stranger, you could even be fake and people now though we do fake ones but people could be right and people all of a sudden start engaging more in is this human nature of like now I have kind of like this uh, kind of guide of how I'm doing. So I engage more with it. And obviously as Peter was saying, if you get rewarded for it is even better. You kind of get that extra boost from being, uh, having a meaningful reward from going up that leaderboard and the other pieces. Tying the events to the main game in some way or form is really, really important. When the game is kind of like on one side and the vet digital's like extra work, it doesn't tend to work out as well as if it's, it kind of ties in some way. And pocket politics is a game that Peter does mentioned eventually did that word. There's a really good loop from the main game to the events to the main game, uh, based on the rewards that you're getting. But it took time because it was a very simple game. It was an idle game and for those familiar with without games it's you have very few resources and it's just, it's just about like this very specific type of progress and they had it too, like they had to like add these layers to the game to make sure that the main game was still cohesive to those layers but to also be able to Lear the event progression into the main game. Cool.

Peter: 20:42

It's interesting if you boil it down kind of to those basic elements. One thing I was kind of fascinated about that some other games we're doing we, we keep close tabs on what competitors are doing and what other games in the market are doing. That's interesting. Or try to kind of backtrack and figure out why they're making certain decisions and just keeping, you know, a finger on the pulse of all this. And there's games like um, war dragons that had this really interesting element I was looking at where they really did just distill it down to time limit and rewards and they basically make an event out of like upgrade your dragons just as something you're going to do anyways, but for the next 24 hours, whoever does the most upgrades is gonna make this leaderboard and you get more extra rewards for doing that. And so it kind of builds this, the idea of anything can be an event, anything in the game timebox it, reward it. There you go. You have something more engaging that you'd be doing anyways. But here you are getting just that extra bonus for taking part.

Speaker 1: 21:36

Do you worry at all about um, resetting baselines with players when you do things like that? Like is there a point at which you can do that too much and then play with, you're desensitized to it?

Peter: 21:46

There's definitely a limit in Oh like exhaustion when you have too many modes, too many boxes to check every single day. Cause you know we were kind of talking about earlier, the biggest battle to win is keeping players in the game, retaining them and giving them a reason to come back tomorrow and the day after in a week later. But if every session you're spending two hours just checking boxes at for a mobile and something that's more kind of on the go in general, it's just that much harder to do. So I think there is a limit you can push that you realize there's an exhaustion in a certain point. And what you'll see happen then is on the forums and chat or wherever else, players will basically prioritize what's important and tell each other. That'd be like, Oh skip this today. Rewards aren't worth it. Make sure you double down with your energy in this mode or whatever it's going to be. And they very quickly figure out how to kind of mid max getting the most out of their game sessions.

Tammy: 22:38

Yeah. And that's a, that's very a good example of a data that is not like hard data. Uh, cause your, your community is also data for, uh, live operating your game and you have to be listening to your community and understanding what they're talking about in it. That doesn't mean react on everything they say, but many times they'll have a much better um, kind of turnaround you really something and they'll just immediately deconstruct it and start talking about it and evaluate whether it was a good change or bad change. And their evaluation might not be bagged up in the data once you actually look into it. But it actually gives you, uh, an insight into how players are receiving the, the, the content that you're providing.

Crystin: 23:28

Yeah. One of the things that I've had just to close this thought out, one of the things I have had teams really struggle with is satisfaction metrics, right? How do we actually measure whether or not players are satisfied? Do you guys have some favorite stats you'd like to use to give you indications?

Tammy: 23:45

Yeah, I think, um, for, for us as a measuring return rates of, of players. Um, and the thing is that if you're, if you're looking at like your forums and your community, you might get a lot of like you're gonna get the loudest voices, right? So if a little change really upset a handful of very vocal players, you're gonna think it's like this, the world is falling apart when it isn't. So you have have to actually look at the data and see are players returning on a regular basis to your game and that kind of, that triangulation is what really helps you understand. Are the changes actually keeping players engaged or are they events that I'm putting out actually keeping players engaged? Um, you know, everything we look engagement at engagement rates if we were at least in event, we're always looking at that. Um, just to make sure that players are actually going engaging with a game, engaging with the modes that we're releasing and with events that we're really saying.

Peter: 24:45

And coming back to the game, one of the scariest stats for kind of live servicing games as a business is keeping tabs on elder's spend kind of players who have been around a long time. Who are your most engaged players, some of your most invested players who are happily spending week to week and investing in their hobby, their experience here. But if you release an update or content or something and that number goes down and they start, they turn their investment down and start to change their behavior because of where the game's going. It can often be a telltale sign that the balance in the game is going off. The rewards aren't as meaningful as they once were and you really need to start to look at how you can change course to make it meaningful for your most dedicated players. So the early engagement pieces interesting for kind of the heartbeat of the the main game. But that's really when you're looking at this as a business and driving revenue and growing this, like those are the pieces that are like, well something's really wrong here. They might be not be talking about it too much, but these, the people are quietly pulling down their investments and not spending as much and these things are starting to turn.

Tammy: 25:47

And that's a really good point. Um, we, it's something that we really, really try to, um, you know, get ingrained in, in game in teams and games. The idea of looking at, uh, just data and metrics and especially monetization by player age. So a lot of people look at all of these things as an aggregate of the game, but the, the game has players that started playing yesterday and[inaudible] start playing two years ago. And if you average those out, you're not going to learn exactly what's happening with your game. So you really have to understand, you know, what are my new players doing? What are my mid players doing? What are my elder players doing? And split those. Um, however it's meaningful for your game to make sure that you're actually keeping a pulse on how, what you're really saying is impacting all those different groups of players because you're early players are eventually and hopefully going to become elder players. But you can't forget about your elders players and vice versa.

James: 26:46

What, um, I don't remember what analytics on this stuff because when you're, when you're trying to measure it, it wasn't an event successful, you know? What are those key that you mentioned to now engage in retention? You talked about participation, you talked about how they're, do I have to spend those, those important, other, other, um, I don't know, maybe more deeper insights that you get into to really measure events. Are, is that, or is that, is that, you know, the total of it?

Tammy: 27:10

Uh, I think it really depends on the type of the event than the game. Um, most of the times it's a kind of like a, an event focused version of the KPIs that you're looking at for the overall game. So if you're looking at, you know, your, uh, how many players are spending, so you're a percent payers, uh, how much are they spending spending? So your average revenue per paying user, um, how many of them are engaging? So you're, you start looking at all those pieces, but you try to also segmented by your event. And just that event and, uh, many times what helps there is if you're tracking things properly, you can actually, um, kind of split out your analytics between main game engagement, uh, event engagement, main game on station engagement and monetization and be able to see, you know, is, uh, our players investing in the event. And one really, really important thing is, is that additive to what they're spending in terms of time or money to the main game or the cannibalize it, cause that's, uh, that that could also be problematic, right? If players are abandoning the main game to go to the event.

James: 28:25

So how do you tease out event monetization versus main game monetization? Are you looking for specific items that are only used in the event and that's tied to event monetization? Are you looking at simply all players who chose to participate in events that's any monetization they generate? Is, is tied to the event versus main game? Like how do you, how do you make that split?

Tammy: 28:44

Uh, so there's two types of, um, you know, things that we're selling on a game, um, things that you're buying with real money. And, uh, many times, a lot of our events have bundles that are, you know, real money bundles that are event specific. So that is a very easy way to measure it as our players actually buying those, uh, game specific bundles. The other pieces with the, um, you know, the items and bundles and things that you're buying for in game currency and many times a, we also try to split that out. Um, whether it's, you know, the way we're tracking it or the currencies that you're using to be able to, um, keep those things separate. So some games when the economies get, uh, complex enough, you actually have currencies that are specific to your events so you can track, you know, are players buying those currencies are in, are they engaging with the game in that way? Uh, but if not, you can still to separated based on the items that they're buying. So in most of our games, you know, you can, you can get things are in our specific clique going to help you for that event. Um, so being able to track that separately, make sure that all your tracking kind of can give you an insight into that is very important.

Crystin: 30:00

You talked earlier about the dangers of cannibalization, right? Like you don't want to create live events to the point where now it's not additive anymore. It's just changing player behavior. When that happens, what do you do? What, what is your advice to two games that end up in a situation where they realize, okay, I've got a huge cannibalization problem either in engagement or in monetization?

Tammy: 30:26

Uh, I think a great thing about, uh, events is that there are limited time, right? So you can just finish the, yeah, it's, it is actually, um, you know, if it's a very long event, then you have to figure out how to fix it on the fly. Right? But, uh, the great thing about events is that they're limited time. So the, the key I think is to be evaluating the performance of every event. And it's just like this, this iteration loop of I release an event, I measure how it did, and I change how I do things. Um, and eventually you maybe change too much that you run into that can of low station problem and you have to rethink what you're doing. So, um, one, one great thing that uh, one of our teams, the adventure capitalist team is doing right now is uh, they've identified, you know, intuitively we kind of all knew that there were these archetypes of, of event balancing and now they're iterating on those archetypes and understanding which ones are not as engaging and therefore they're not performing as well and figuring out, okay, how do we come up with New York types? They're not the same cadence over and over again as the other that we know already work cause you have to keep, keep it interesting for players. Um, but that, you know, get from those, learn from the ones that are doing well and learn from the ones that haven't done well as well. Um, to come up with new types of events and balancing and cadence.

Peter: 32:00

There's a lot to that that we see in some of our card games that we do with our, our partners over at synapse where they use kind of this, this feature that um, I think largely we call it a battleground effect. Meaning we have all these different card games, uh, all of these different cards in game and they're all thematically different and are kind of split out into categories if you know to to compare it to say like an RPG that has fire, water, earth, that kind of thing. And so basically you have these interactions where during the period of this battleground effect, certain types of cards will get a boost and any event running will take advantage of that. So you want those cards perform the best you can in that event, but they actually have the battleground effect carry out to the rest of the game. So everything else you're doing during that period, your normal like campaign progress, just normal PV, day to day player versus player kind of battles. It all gets the same impact of the stuff you're also investing in to do better in the events. So it gives it this way to kind of stretch out the content and make it valuable for everybody. So all the investing you're doing and all the pieces you're really buying into help you out throughout the game. And then again, the piece we mentioned earlier where the rewards from the event work best when they tie back to your core game as well. So the reason you're fighting for the event is still to do well in the main elements of the game.

James: 33:23

So I haven't gotten an AB testing because this comes up a lot. I mean a lot of people especially, you know, PlayFab provides tools, a lot of the, you know, we look a lot of tools like AB testing and that's uh, that's a hard system to build and do it. Um, do you, do you advise your teams do lot of AB testing and if so, where, where, where is even testing useful? Where is it frankly not useful? Uh, and like what does that successful AB test look like?

Tammy: 33:43

We don't, no, I'm kidding. But it's hard. It is really hard. Um, I think when you, what you just said is, is spot on. So we, we actually don't recommend, uh, AB testing for the majority of the teams because it's hard to do it well. And you also need a big user base, which means either you have a lot of players or you have to buy a lot of players. So it becomes very, very expensive. So it's, it's a fantastic tool that you have to be very careful about kind of jumping in and using, cause you have to have enough players, you have to understand how to design it. You have to have the resources to analyze the results. So it's, it's a very, I think it's a very sophisticated tool in that sense that um, a lot of people kind of just thing it's like, Oh everyone does AB testing, so I should do AB testing. Uh, most tests in that sense will not give you meaningful results. Um, so you have to be very mindful of how you use the tool of AB testing and understand, um, kind of how to set it up to actually get meaningful results.

Peter: 34:54

You also worry, I can remember from the[inaudible] background and so we were always very nervous about AB testing because of crosstalk. Oh absolutely. That's a huge consideration for our stronger monetizing games. They're all very social. You have guilds that talk to each other, you have world chat, you have these elements where people know you. When people are telling them, telling each other what to invest in, like, Oh, buy this pack this weekend, it's great deal. That kind of thing. It's 10 bucks in somebodies like, no, it's 20 bucks, but then you know you have this explosion. Right. And so because of the social nature of it in general, we also kind of stray away from regional pricing too because that becomes an issue where we've actually seen players go through extreme lengths to, you know, find their ways to that app store through VPNs or whatever going to be, to find their way to buy currency for other players and basically create this fraud market of it. It's a whole, you know, cascading set of issues. But yeah, you love waking up in the morning and finding out the Poland is suddenly your largest spending in the world. How did that happen? And so yeah, we're, we're very sensitive to that. If we're going to do any testing like that, it's either going to be in a more single player kind of environment or it's going to be early and test markets when we can, we're, it's okay to make broad changes and it's not as social as it's going to be kind of later on in its life cycle. Right. If there are teams that really do want to try to get into AB tests, even though it is a fairly sophisticated tool, is there a suggestion you guys have for the kinds of things that'd be good for them to try it out on?

Tammy: 36:21

Yeah, I think for me, I always tell people I'm the most important thing about AB test is a test. Big changes and that's, we're kind of cross dog is also problematic. But if you want to get meaning for results, you want to be a Tez big changes. So you don't want to change your balance from like, um, you know, the, the drugs are 1% versus 1.05%, because you're not going to get a meaningful result unless you have millions and millions and millions of players. So you want to be able to make these big, big changes in test big changes. So started to think about that from the very top that gives you an idea of which types of AB tests you should run. So if you're doing timer AB tests, you want to test, um, you know, one hour versus eight hours, right? I mean that, that's, that's a little bit of a big jump, but things, things like that or um, you know, uh, animation through, around random fantastic AB Tez during test markets where it was actually how, uh, the combos and the game play works. So it was a very big change. The game was the same, but how we presented the mechanic was different, was completely different, um, to see if one was more engaging than the other. Right. Um, so that's, that's a huge change. Um, so that's, that's a type of AB tests that I would recommend, uh, people really thinking about.

Peter: 37:50

Yes. And not to run multiple tests at the same time, especially if they're small because you multiply these cohorts against each other and suddenly you do the math and you're like, Oh, we need 100,000 people to test these three minute things, right. Where you're going to get a lot better results if you just do big changes at one at a time to figure out where to go from there and kind of incrementally try to change things as you go. We've had teams try to try to run, you know, four to six things at a time and ESU started sub segmenting who goes where after the first kind of gate, it gets pretty weird.

Tammy: 38:22

Yeah. Or are they tried to do an a B, C D E F G test. Yeah. So that, that's also another thing that I would recommend against, uh, is tried to do an AB test, like actually one thing versus another or in most three, three versions. But that's about it. Because again, you started multiplying the number of, of people that, um, you need the other pieces. What are you measuring? How are you going to measure the success of your AB test? What's the one KPI that you care about for this AB test that's also going to tell you how complex your AB test is going to be. So if you're going after spenders and your spender rate is 1%, now you're talking about like a very small subset of your players that you need to let them mature to get a meaningful result out of them. So maybe, maybe it is a good AB test, but maybe it's going to take you a month to get meaningful results.

James: 39:22

So I want to switch gears completely for a second. Let's talk about globalization and targeting and global audiences. So, you know, um, you're in North American company, I'm guessing the majority of your players and traffic is North America. But yeah, I think you're also a global company and I think you've got players all around the world. And so how do you, again, how do you advise studios to think about going global? Uh, how extensive do localized games, for example, how extends would you localize, uh, uh, real time events, communications, maybe emails, push notifications, all the kinds of of mechanisms that might affect a global global issue.

Peter: 39:55

It's an interesting process. That is one of the hardest to tell if you're doing a good job because you have to count on external vendors to do the translations and the QA and see if it makes sense. And then no matter how well they do, your players are gonna be like, this doesn't make any sense at all cause they start to know the game even as well as you do or better. And so between the platform, Apple and Google putting a high value on localization so they can promote the game more widely. Um, it's something we try to do just to cast a wider net as well because our user acquisition teams have started broadening their scope and, and going to new territories around the world. And so we put a pretty high priority on it. We do probably the best pass around global launch when the game is, you know, complete quote unquote I say because life servicing is, that's just the start. Um, some teams will then we kind of decided on a team that team basis like what's meaning, what's not going to completely drown them in their week to week operations. If it's, you know, I say that to mem team trying to get all the texts out and launch the event and get these pieces together and they're scratching, you know, they're finishing at 4:30 PM it goes out at five o'clock, like they're down to the wire kind of team. And so sometimes you know, we'll lean into Google, translate for a few things and update it maybe later or we save it for kind of a major content update that really needs more explanation to do the more fine tooth comb localization elements as well. It's something we're constantly iterating on because there's all, it's, it's so hard to get right. Um, when you're relying on kind of external vendors and other teams to help get this together. We have you, we have bilingual teams who are in Montreal and they, they do the game in French and English and it's solid. But everything beyond that gets a little dicey. And so it's something that we're always working on and thinking not about, but in general we do try to hit all the main points of our local area, our notifications localized are the events at least basically explained with you know, the core elements of what's going on and just try to keep up with it. But you run into amazing problems like um, when you tried to translate something like animation Throwdown which leans very heavily into potty humor and elsewhere and teams across the globe trying to be like, what does that mean? And having to very like straight faced explain very matter of factly what the joke is a reference to in all of these things. And like trying to find a way to both capture the spirit of the game in its original language while maybe doing something that would be funny or in another language, things like that. So I have a lot of respect for localization teams, especially on big projects that have 20, 30, 40,000 words on mobile. And that's not even to speak to the, probably hundreds of thousands of words going into like triple a big, big premium releases. It's, there's an art to it, to Bose cap capture the spirit of the game and what makes sense. And so we do our best and the operate live operations in general make it pretty tough, but we try to keep up.

Tammy: 42:55

Yeah. And I think that, um, for teens in general, especially for small teams, one thing it's important to think about is, um, you know, mobile games, uh, you can get away without ports. So try to use more, uh, images, uh, icons, kind of visual language instead of words whenever possible. That's gonna make your life way easier. If you want to localize your game and the other pieces from their beginning, think about, uh, even if you don't have a plan on localizing your game, think as if you are going to localize it and don't them bet single string into the code of the game because that's just going to make your life really, really hard. If a year from now you decide, you know what, I should localize my game.

Speaker 3: 43:41

Yeah. Hard-coded strings are not that great.

Crystin: 43:45

I think about localizing content or localizing store content and localizing prices. Um, I feel like for me in the last 15 years, the thoughts around this, I've changed a lot in the industry.

Peter: 43:58

I think the pricing element as mentioned earlier is the dicier one just because the regional ability to try and trick stores for the best deals, right? But the content in general, we want to be able to capture having players understand what is on offer, right? So we do a lot of things kind of like Tammy mentioned that are heavy on iconography. You have kind of a few core core categories of reward or item or resource or currency or what you're buying. And so we basically build out these templates that allow you to almost list these things. And so it's more the quantities that are changing then the offer itself. And so because we have these baseline localization translated items that players can understand in multiple languages, you're just by knowing it, bundling up a variety of these and using really heavy imagery to ensure folks are knowing what's available.

Tammy: 44:48

Yeah. And then the, the piece that uh, Peter was mentioning of like big changes, big releases, the release, a whole new type of event or mode into the game, that's when we would actually really be careful about the localization. Like we're introducing new items. So now we need to have a kind of a new dictionary of localized items to make sure that it's approachable for everyone playing the game.

Peter: 45:11

The unique thing too about localization is it can almost act as a game update itself. Where say we launched globally in E figs and then we realize there's opportunity in Russia. Our stats are lower in Russia. There's good interest, good conversion and installing the game. This is actually something Google sent us information on. They're like, Hey, you're doing great here, here, here and here. Look at how low Russia is. Why don't we take a look at the, you should consider updating it and so we, this is forbid here I was actually and we localize the game to Russian. We found a new low team that was super hungry to do content. They're like, we're going to, we're going to do great. We were, we'd done all these games in the past. We really want to prove ourselves. I was like God, like their spirit. And so they did the game in Russia and they had great questions about what things meant and understanding the tone of the game. This is an, you know, a pixel RPG where you can get a pizza costume to wear to show yourself off in the town square and so sure enough, we put in Russia and we let the platforms know. We were kind of relaunching in these with this update and it gave us this opportunity where install started. To tick up and then new language and then performance actually followed along. You could see retention rising day by day as these cohorts hit the newly translated game. So it acts as like its own content update in a way cause you're just casting a net to this new group of players that didn't really have access to it before.

Tammy: 46:30

Yeah. And have you follow like, um, you know, big studios, uh, they're, they're doing very similar things with, with the games like you see like they're just hitting like one new country or one new language every update. Right. Um, so it's, I think it's hard for everyone. Uh, and it's just a matter of kind of taking in one step at a time. Yeah.

Crystin: 46:54

I mean I think we're starting to have those questions. Like, can we ever get to the point where Vishy translation is good enough because that would just open things up so much because right now it's so labor intensive, you have to localize.

Peter: 47:07

It's kind of a funny story, but one of our early teams after getting frustrated with the back and forth of like fixing this and updating this and they were doing such regular cadence and updates, they actually self built to kind of plug in for Google translate. That would translate any new word in the game on the fly via Google translate, which led to some funny results but did keep it up. At least it was almost like we know this is very painful to be this active and this and this, so let's just do the simplest way we can and have a machine do it.

Crystin: 47:38

Yeah. We experimented a little bit back in the like the mid two thousands with a crowd funding localization and definitely found the quality was lower. But if we were really open with the player race to say, do you want to opt into this language knowing that it was done by not, you know, not us. Right. But it was done by, I wonder if you could use the same thing for machine learning. Like, cause when I go to websites and I know it is a machine translation, I'm like, Oh I understand what I'm getting into.

Peter: 48:09

Yeah, we've um, kind of like that and kind of speaking to the visual language of games. I know when we've vetted projects from say Eastern teams coming out of China to try to come West or in the browser days when we were getting a lot of Chinese MMOs, um, oftentimes when we were looking at early ones, they weren't localized yet. So we're poking our way through, you know, I don't speak a word of Chinese or can't read a character and still poking my way through the game with a lot of support from, you know, heavy arrows and this kind of thing. But you start to pick it up, you know, so there's, you can take it kind of with a grain of salt and it all just goes back to leaning into the visuals as much as possible cause it's just universal language at that point when you train them what the elements of the game are and pictures.

James: 48:51

So you've been operating games now for quite awhile. Uh, do you have any Trainwreck stories you can share or situations where you, you know, something happened to a game, maybe you, you know, made a change in the economy, let's say that really, really had a had, uh, you know, we want to hear the process.

Tammy: 49:07

Everyone wants to hear the disaster story. Yeah. Oh man. Everything, everything always worked great. Yeah. Perfect. Thanks are always perfect. Uh, no, I think so. One, one, and this is not specific, but I think one of the most frustrating things that you can run into, uh, is when you, you know, you see softness in the game, you're like, okay, we have all these theories of why the games not doing well, go back to the drawing board, make a bunch of changes, release the game again, or, or change in, release an update and nothing moves. Yeah. And in your like, but, but why? Like I already like I change all these things. It should be working better. And we actually had a game that we did not release, um, in Tez markets last year that had great ratings. Reviews were all positive. There were, there was no kind of crash reports that were like, Oh my God, the game's not working. It's broken. But it just had the most horrendous retention. Just please, we're like, great game, not coming back, never to play it again. Never coming. It was one of those things where I'm here and now we're like, why? And then we, we went back to the drawing board, came back with a game three months later, I think, same year, salt or like these, this is just why there's part of me that's like, you don't want to give up

Peter: 50:46

cause you want a break dancer, right? Like what is that game that just isn't, it doesn't rotate. And it's this, this really tough balance of time and money at that point to where you know, it, it costs money to run a business and keep a studio going in life service that, so you need to see some level of return. But when something breaks so fundamentally without a clear answer, it's much harder to say, Oh, let's keep going. Let's put X amount of dollars more in. Let's keep this up and running. There's one case, um, of a total flip and expectations where we were attempting to learn something about our testing to try and grasp and manage marketing CPIs cost per install. And so we had an, an idle game engine, um, that our team was working on that had all the right elements in place, progression, good monetization, live events already ready to go, all built out. But we wanted to be able to cast the broadest net possible. So we were like, we need to find an art style that's going to have appeal, get players involved, get clickthroughs and kind of suit, get players in the door. Because one of the hardest things that's the hardest thing to do now is just get users. And so we ran, um, a marketing test that looked at four different art styles and we had maybe half a dozen variations of advertisement for each style and they were all pointing to one version of the game. So we knew that as soon as they click through the ad, the, the test kind of breaks. You can't actually measure the cost per install and the other factors involved in marketing. But we said, okay, if one ad gets more clicks than another and at least shows initial player interest in that style and it shouldn't directionally point us to say, Hey this is going to work, people will click on this. And so we ran one test and one style one and we ran another test to verify it. Same results like we were getting a lot more clicks on this one style and so grab the style game, went into production, attached all the assets and all of this pieces to it went to test markets. Click-throughs came down and cost started to rise and we're like, wait a minute. I'm like, God, it's still viable. It's going to be more lifting cause LTVs hard in the idle space to hit this, but I think we can get there. We got to a point where like we launched the game CPI suddenly or three X what they were. So it's just like drowning the lifetime value of a user in the game against something we thought we had a little bit of control over at least directionally, but we just couldn't grow the game because it just outpaced it on costs for marketing and you know, the, the market could have changed from pre production to launch or it's, I think the end result is you just can't control that.

Crystin: 53:19

You need to, did you guys end up thinking that it was actually possible that this art style great for getting you the click through put it, it actually maybe hurts retention potentially.

Peter: 53:32

I mean, again, that goes back to one of those questions where it's like this is what it did. Like the actual game performance for players who found it was actually pretty strong. It was on the higher end of performance for idle games, which is what we were going for. But there's that breaking point where your marketing dollars or marketing costs just get too high that you can't catch up. It just outpaces it forever. So you're always losing money when you're trying to grow the game. Pardon me? And, uh, I think our director of marketing every once in a while the conversation, like what if we just tried one of those other art styles that we think will work better in this Mark this market? And just again redo it but then watch us in a year being like, well, we learned a new lesson happened again, you just can't control these things.

Crystin: 54:13

It reminds me of the Coke and Pepsi story right w hen, when they did t he, y eah, when they did the taste test for Pepsi and it outperformed c ode blind t aste a s o utperformed Coke by a lot. And so Coke w as like, we need to change our, our formula to be more like Pepsi because people clearly like it better. And then long story short, and it's almost a p ocket o f g old at this point, come to find out, u m, people really like one sip of Pepsi, they don't like a whole can of Pepsi. That's hilarious. Y eah. C oke like they really almost ruined themselves. Trying to appeal to one sip instead of understanding that actually satisfaction after the whole c an was probably more important. I do like I've, who knows if that's true, but you wonder if that sort of speaks to how complicated it can be to really know what's driving players to come back.

Peter: 55:03

Yeah. Cause there's definitely interesting elements of markets and we kind of straddle, a lot of them were kind of our bread and butter games come from the mid to hardcore space. And even our idle games are becoming more core as they get more evolve and get more complex and get more competitive. And so it might've ended up being that the style was too casual, which gets clicks from a broad audience. But once they get into the game and they're like, Oh wait, there's this event in the S these equivocable survivors on this Island and all this stuff, it's like, Whoa, Whoa. Like this is a little too much. I didn't sign up for this. I clicked on the Tiki drinks. You know what I mean? So I know we've just a few months left to kind of wrap up, I'm curious, uh, thoughts in a feature of live ops, you know, where do you, each of you, just quick thoughts. Where do you think this is all going? You know, where do you think the trends are leading us in the next six to 12 months? I think you're going to see a lot more out of subscriptions and ways to do the more retention and monetization based tie-ins where the, the kind of heartbeat of a game is going to be on these monthly signups where you're getting boosts, bonuses, rewards for being a subscriber that all feed into a greater economy as well. You see some games doing it quite well. We're starting to get our feet wet with it. We'd done some[inaudible] non system level ones, but basically in game subscriptions that we kept tabs on. Now that the platforms are doing such strong renewable subscriptions that you're actually signing up for real money. I think we're going to see a lot of interests going in that direction. You see it, you know the the kinda S time Lim time box subscriptions. You see them in Fortnite and games like that. I'm not saying that's why it works, but there's an interesting nature to that that works in say legendary game of heroes. That's been going really well for them and ways to tie it all together to make kind of the invested players experience that much better.

Tammy: 56:52

Yeah. And I think there's also going to be a, and you're, you're already starting to see that the evolution of, um, how we categorize games. Um, you know, up to this point we talk about like casual games, mid core games, core games, and I think that, uh, you know, there you started to see this thing and we're doing this as well of cross learning from one to another. So kind of more hardcore mid-core games who are trying to, uh, reach broader audiences, which at that point they're not really hardcore games anymore. And casual games are kind of bringing all these learnings from mid-core games and getting like these very complex mechanics and systems and economies into games that were quote unquote casual. And I think that at that point they're just games that are reaching different audiences and that are leveraging different core mechanics, but they're learning from each other. And I think there's, it's this rethinking of how to categorize games and what can I even implement into my game if my game is a casual game or a match three, does it really have to just stick to that? It doesn't, in my game is an idle game. This would have to stick to just this very specific type of progression. It doesn't, and you can learn from other types of genres and other types of games and kind of make a much more engaging experience for your game. And you see that every time that happens within a genre, there's like a new, a new type of genre that you got to happen with the match. Three spays were kind of a mitigating was incorporated and now like a bunch of games came out with a million. But I think it's just gonna happen more and more and, uh, cross genres. Um, cause right now it's, it was, it had been kind of very encapsulated within its own, like little ecosystems. And you're starting to see that just cross pollination. Yeah. Cool.

Crystin: 58:52

Well thank you guys so much for coming. Thanks for having a conversation. Yeah. You guys are really hitting all your stuff. This is great. This is really great. Thank you. Yeah, absolutely. People were really enjoying your

Speaker 2: 59:02

cool[inaudible].

Speaker 6: 59:08

Thanks for listening to the art of live ops podcast. If you'd like to, you heard, remember to rate, review and subscribe so others can find us and visit playfab.com for more information on solutions for all your live ops needs. Thanks for tuning in.