Are you nourishing innovation? Or are you killing creativity by shutting down out-there ideas? Brett Nowak, creator of Liquid and Grit, the leading product research company in mobile gaming, is here to inspire you to up-level your studio through a balance of risk-taking and solid product planning.
Are you nourishing innovation? Or are you killing creativity by shutting down out-there ideas? Brett Nowak, creator of Liquid and Grit, the leading product research company in mobile gaming, is here to inspire you to up-level your studio through a balance of risk-taking and solid product planning.
James Gwertzman (00:05):
Hello, I'm James Gwertzman.
Crystin (00:06):
I'm Crystin Cox. Welcome to the Art of LiveOps Podcast. Hey James.
James Gwertzman (00:16):
Hey, Crystin.
Crystin (00:17):
So this one is a little different. I, unfortunately, could not join you but you are going to be interviewing someone that I believe you know?
James Gwertzman (00:29):
Yes, in fact, Brett Nowak and I actually go pretty far back.
Brett (00:33):
I graduated from Harvard in '03 and then went and actually played professional hockey for a couple of years. And when I got out and moved to Seattle, I actually met with you a long time ago. And you talk to me about just combining different things that you did in your past, I remember you were in theater and that's led you into games. And I thought that was a really great way to think about your next career path. I had grew up playing cards and games and loved poker and all those kinds of stuff. And then based on our conversation ended up at a small gaming company with Pete Parsons who's the CEO of Bungie Studios now.
Brett (01:11):
And then I went to business school and went back into games, landed at Zynga, loved it, and decided to go a little bit smaller company, Blue Shell Games. And when I was there, I felt like product managers really lacked the support that other roles had. And so I ended up starting Liquid and Grit, which is a market research company that really provides monthly reports and tools for product managers, game designers, product owners who are trying to figure out what is the next revenue driving feature mechanic event that they're going to put into their roadmap.
James Gwertzman (01:45):
Brett's interesting because he currently is actually at sort of almost a consulting firm called Liquid and Grit that actually puts out a regular report for Liveops professionals, called product managers. And that was based off of his experience at Zynga where he was a monetization PM for a long time. So he was really in there in a lot of their things, their casino games and others, thinking about how do you monetize and build-out and have effective Liveops. And now he's off kind of providing almost education and ongoing advice for folks doing this for a living. So sort of continuing the education process with it, which I think is fascinating, because this is one of those fields that as we know, changes so quickly, it's great to see someone building sort of a business around that education. And so, I think he's got a lot of very contemporary insights for us and I think it'll be a fun interview.
Crystin (02:36):
Well, cool. I'm excited to hear it since I wasn't there. I'm going to be experiencing it for the first time too.
James Gwertzman (02:41):
Awesome, all right, let's go to it. Because the whole reason we started this podcast is because of what you just said, which is we recognize that there are so many resources out there for pretty much every other discipline of game production, art, game design, programming, engine development, client development, there's a lot of resources, but for some reason, maybe because it's new, maybe because it's sort of this new hybrid discipline, Liveops or product management just hasn't gotten the same level of attention maybe it's because teams try to keep it secretive and see that as a secret sauce that they wouldn't want to share, I'm not sure what it is, but for whatever the reason, there's not a lot of information out there on doing effective Liveops and so that's really the motivation of this podcast and I think it sounds like that's also the motivation of you starting your company.
Brett (03:32):
Yeah, I definitely think that's a difficulty of product management. And one of the things that was somewhat, I thought, a little bit sad about Zynga. I mean, it's doing well now, but not becoming maybe as big as it was projected to be when an IPO it was that, it was a great place to develop and train product managers. And it was like one of those companies, and still is, but I think that time it was almost coming to it's really just like trained what I felt was the definition of a strong product manager. And there's been companies like that in the past in Silicon Valley and whatnot, they sort of cranked out really strong certain of roles in different companies.
Brett (04:17):
And then once Zynga didn't become like as big as it really projected to be, then that that role, it was harder to train people. And the hardest part about being a product manager in many ways is to get the skill because everyone's looking for people who have the skill. But it's sort of a chicken and egg problem is if you don't have the skill, you can't get a job. And if you don't have people getting the jobs and developing, then you don't have more product managers. So I think it is a real difficulty of product management in general.
James Gwertzman (04:45):
So what are the profiles of the kind of studios that come to you to Liquid and Grit? Is there a common pattern of a certain type of studio that comes to you or is it really across the board?
Brett (04:58):
We generally work with the largest mobile gaming companies in our genres. So we cover puzzle casino and RPG, and we actually have a slots support as well, and we're subscription business. So we generally work with the top-grossing companies in those spaces. So, I mean, if you think of any of the top-grossing gaming companies in the world, mobile gaming companies in the world, those are the ones that subscribed to our reports.
James Gwertzman (05:24):
And tell us about what goes into a report, what's an example of the kind of material you share out.
Brett (05:30):
Sure, we track the market both with our own primary marketing analyst team and then also third-party tools, and basically combine all that information to surface what is driving revenue in these specific genres? So each month in our puzzle report, for example, we're finding new mechanics like, let's see what did we just write about in casino, Oh, we're just surfacing casino that we're actually seeing mechanics from puzzle and RPG getting released in casino, one being custom design and mechanics, which is really big in puzzle and we're now seeing that casino.
Brett (06:11):
And we just reported it on the fact that that's new in that we actually aren't seeing it drive revenue yet, but typically we do highlight features that are driving revenue. In this case, we just highlighted that it's a new mechanic that product managers should be aware of that is coming into casino and where it's TBD, whether it's actually working for those type of players.
Brett (06:32):
So that's the type of information we typically focus on. We also do teardowns of breakout apps, teardowns of soft launch apps, generally what's happening in the market. Any platform changes, the information that product managers need to know to manage their games properly.
James Gwertzman (06:47):
Cool, and so when you're talking to product managers and you're cataloging sort of the mechanics that are emerging and what's driving revenue, do you almost have for new subscribers, like a getting started, you're the top. If you're not doing the start here or maybe if you're going to give a talk at a conference, let's say, what would be, I don't know, top five mechanics you would encourage studios to start with and saying, "Look, if you're going to do nothing else, do these things first. And then you can start to add more complexity on top of that."
Brett (07:19):
Well, I mean, I think it just really depends on what market you're going after. Unfortunately, I don't like the depends answer, but it really does. I mean, because if you look at just one John Ryan mobile gaming, let's just take casino. I mean, there's just definitely subsets of casino apps and subsets of casino players. We actually create a slots persona, a puzzle persona, we're actually working on an RPG persona report where we work with a team in Finland to create the personas and match our data with their ability to make the personas and figure out the different player types that are in market. And based on that, you want to change your Liveops strategy and what you're executing against to fit what type of player you're targeting.
James Gwertzman (08:18):
Let's try and make it concrete. So for example, take puzzle games. I mean, I was PopCap for eight years, so I know puzzle games really well. So I guess, how do you break down the player types in the puzzle genre, for example? There are certain categories or segments that you break out and say, "These are the segments we see in public gaming and here's how to attract each one?"
Brett (08:39):
Yeah, I mean, let's just say we were going to enter the puzzle market, is that sort of a way that you want it?
James Gwertzman (08:47):
Sure.
Brett (08:47):
Yeah, I mean, if we were going to enter the puzzle market, if we were building our own game or we were working with some team, we would probably segment the market based on what apps are currently in the market. I mean, the first thing I would say if I was recommending what to do is you want to align your product team with the user acquisition team from day one. And I think that's probably one of the biggest mistakes we do see when we see new game developers jump in the space. They have like, "I have a great idea," or "I really like this," or "I think this would be a good combination," or whatever, and that might be the case, but the problem with, well, the difficulty with today is that you have to have a really strong user acquisition model that matches the product model.
Brett (09:29):
So to do that, in the beginning, you want to decide whether you're going to create an app to grow the market or create an app to sort of in red ocean, blue ocean, red ocean being sort of take users from other apps and blue ocean being sort of create this new market. And then once you've decided that approach, you want to go after a specific subset of apps. And from there, I think what we would do is we would analyze those apps.
Brett (09:59):
I mean, I can give a very concrete example of that, but in puzzle, for example, Homescapes and Gardenscapes were very similar apps doing very well by the same app company and Matching Mansion came in and built a very similar app to those apps, they did very similar economy but a little bit tighter and they looked at the Liveops, they look at the level design and then they adjusted based on what their hypothesis was that those players wanted, that they weren't getting from Homescapes and Gardenscapes.
Brett (10:29):
And what that allows the UA and product team to do is really be aligned. The UA teams clearly targeting Gardenscapes, Homescapes players, or players who play those apps and similar apps to that. And the product team is building that product that the marketing team is saying they have, that this game is better than Homescapes because it has more Liveops events that are based around collections as opposed to custom designs. And when they have that target, that group of players, they have the nice marketing message and the product really backs up what they're saying to the player when they download that app, then you have the ability to enter the market, which Matching Mansion successfully did not too long ago because they positioned themselves against those two apps, so.
James Gwertzman (11:21):
Got it, I think one of the things that I find interesting when we talk to studios who are new to Liveops and new to gaming services is something you just talked about, which is that notion of thinking about the entire business of the game sort of from the very beginning. We typically see a lot of game studios, they have an idea for a game and they just go and build it and then trust that the players will come. And I think typically, with Liveops, the more advanced studios have a sort of built-in model of the business in the beginning, they're thinking about their acquisition strategy, they're thinking of their marketing strategy, their monetization strategy and the game strategy, the design strategy all integrated together.
James Gwertzman (12:01):
And obviously, you produce a report for those kinds of studios, the ones that want to see the sort of all-inclusive view, do you see or do you do consulting work with companies that sort of are looking for help figuring out sort of, how to stage it out or how to make that transition from being a traditional game studio to a more of a Liveops games or service model?
Brett (12:26):
A traditional, so more of a console business, like they're getting into mobile gaming, is that-
James Gwertzman (12:30):
Sure, yeah, console's a good examples, and a studio that's traditionally used to a waterfall model of build a game and ship it every... build a sequel every few years. And it does now moving to the Liveops model and sort of, because we've seen that there's a big culture shift for those kinds of studios.
Brett (12:44):
Yeah, that's a huge culture shift. And I saw that when I was at Zynga, when we had people from EA join Zynga, I mean, it's a very different model. I mean, you have sort of an iterative approach versus that that waterfall is what you're describing. I think that maybe the importance of production, these games are more so like a sitcom. I mean, these players need content constantly and if you don't have that content coming out, then you're going to fall behind.
Brett (13:17):
And that does a lot of things. I mean, one, it just feeds the players enough new stuff that's going to keep them excited and engaged to come back. Because the Liveops is really, I mean, it's core sort of like a bar model, like Thirsty Thursdays and Half Off Fridays, and that, that's just something new that you can go into bar and experience, but you need to change that up, so you need to have that content cadence. It also allows you to innovate because if you try something new and miss, then your next Liveops event or next feature is a much shorter time period. So you're not going to get hit with as much of a negative impact for that innovation.
Brett (14:01):
So you're allowed to also iterate and try new things because you know, hey, the next thing is going to come up to three days from now, so we're only going to have two or three days of weakness. So I think, really the importance of figuring out a strong production model and in that way, it's sort of a much more decentralized control system. I didn't work in console, but my understanding is it's a little bit more centralized, meaning you have to have someone at the top really understanding what that whole game experience is, that whole story and you have... and then everyone else is executing this vision that you're going to release at some point, you're going to release it March, 2020.
Brett (14:48):
So you kind of have to be on the same page. It's much more centralized control for good reason, because you don't want level 100 to be so different from level one, this is one story, it's a book that someone wrote. Whereas in game product management, it's much more decentralized where you want to have the sub teams in your department. You might even have a Liveops team just for engagement, one for revenue, one for whatever, custom designer, it needs to be much more decentralized. And allowing that team to iterate and try new things very quickly, and not have to check in with that, whoever point person is.
Brett (15:29):
Because what that point person is doing is they're slowing down production to the point where the ideas have to carry so much value because they have to overcome so much loss of time that, I mean, you just have to make these amazing ideas. And the thing about great innovation is that you have to understand that we don't really know what's going to work. And so you have to act and build on the premise that you're wrong.
Brett (15:59):
And that's a really weird thing to think about, but you basically have to have a culture where if you come up with an idea and it's totally ridiculous, and everyone thinks it's totally ridiculous, you still have a way of shipping it or testing it. Because it's totally ridiculous ideas and ideas that 90% of the people think is stupid that often end up being the most innovative ideas because at their very core, innovative ideas are contrarian. So that's why decentralized control is so important because if you're not decentralized, you're going to have these people who have the authority to basically kill all these ideas that do sound ridiculous.
James Gwertzman (16:37):
Too easy to say no, the committee will say no, or the designer will say no, and then you don't get that chance to innovate.
Brett (16:43):
Right, because an innovative idea is going to have a large percentage of people think it's ridiculous. I mean, just think about all the innovative ideas in the past. I mean, think about COVID right now. It's like, if someone told you three months ago, if we were all going to be isolated in our homes, you would say you're being ridiculous. So that's the same premise in game design, and really strong Liveops, and really strong mobile games in general is you have to have people on your team say, "Hey, tomorrow, this idea is going to work." And a bunch of people say, "That's ridiculous," and still go through with it.
James Gwertzman (17:21):
So let's talk about the makeup of good Liveops organization because you mentioned that already two, you've talked about a product team, you've talked about a customer acquisition team, what are the other typical ingredients of a successful organization?
Brett (17:35):
Well, I mean, like I said, I think, I was just interviewing a really strong team CMO the other day, and he talked about a lot of them. It's not any different from making a strong team in your company. I mean, it's having a strong system to allow people to operate within a framework that they're going to be able to operate quickly, but also within the sort of the guidelines that doesn't put the company at risk and to try out new ideas and new things quickly and iteratively and see with some metrics, what is working, what isn't. So not just to be totally out of left field.
Brett (18:19):
And then, within that decentralized structure, have it aligned with the company strategy, the product strategy that we talked about. You can't just have people being rogue out there. And I mean, if you find something that works, but it's totally unrelated to the core brand and strategy of the product, that's also useless in itself too, it has to be aligned with that centralized ideology and sort of a roadmap. So yeah, and I think that's maybe too philosophical.
James Gwertzman (18:53):
No, it's okay [inaudible 00:18:54].
Brett (18:54):
Yeah, it can be more technical, I guess.
James Gwertzman (18:56):
Yeah, what would be the functions within effective Liveops? So we have acquisition, we have game design. I often see teams, monetization teams whose primary focus are OKR is measured by sort of pumping revenue. I guess, how would you break down to the different goals? So maybe retention is one, monetization is another, that you typically have different teams owning and then both, what are those, but also where do you see sometimes the conflicts emerging? Because I know, I've heard, for example, back in the day, that Zynga often had tension between the retention teams and monetization teams. There's always a tension between, do you try to take a lot of money out of old someone's wallet early and potentially risk burning that player and them not coming back? Or do you sort of put the monetization later in the life cycle, but then you potentially have players who never monetize and how do you balance those different goals?
Brett (19:52):
Yeah, that's a difficult one. I think goals can be inherently a little too simplistic in terms of using with these teams. So I think guidelines are probably a better way. I mean, people really get angry listening to this, I know OKR as a super popular, but I don't know, I think they're oversimplified because exactly, if you just assign somebody in OKR, it's sort of like a game, you can gamify anything and humans are generally apt to do that. So if you just say you're in charge of revenue, I was a revenue PM and we had that struggle. If you don't have guidelines around that, then you're going to have people do ridiculous stuff like launch sales for a quarter and then have it totally ruin the game, which has happened many times in mobile gaming.
Brett (20:44):
Generally, what you want to do is have a diversity of people in a team, and that works really well. So generally, the product manager, and this is how I would set it up, the product manager is really the money person. That's why I always tell people who are getting into product manager, being product managers, don't be another opinion, be the person that basically understands how money is generated in the game and is constantly improving the odds of making more money. And it's sort of like playing poker, it's like you always want to be playing the odds. You need to be odds person and say, "Hey, that bet right there, it's statistically not the right bet to do." Or like, "If you did this right now, that would increase your odds of winning this hand." That's the role of the PM.
Brett (21:29):
The game designer is really the voice of the player. And the game designer is someone who's going to love that the game, love playing the game, love the players a lot. Well, I mean, not so much love the players but love the game itself, really know that this is what a collection system is supposed to be, this is how it's supposed to be designed. I don't care that we're going to drive revenue or not. And then you do want to have a community voice, whether that's the game designer, or someone in community, or someone in customer service, or something like that, that's really focused on the player.
Brett (21:59):
And then when you get a bunch of those different voices who know they're playing that role together, then you can, I think, create great products and great features that are going to be able to do both, they're going to be able to drive revenue in the short and long term without cannibalizing players and forgetting the longterm vision of the company, which is, to have short term and longterm revenue.
Brett (22:23):
More tactically speaking, I would say that one of the things that we learned at Zynga, and one of the things that we do see now with really successful games is that games, I think a couple of years ago, people thought that you had to pinch players to drive revenue. You had to have them lose any ability to play the game for you to make money. So the equation was you run out of coins and slots, and then you're going to pay, you run out of chips and poker and you're going to pay.
Brett (22:55):
For the most part, I think is, I wouldn't say it's incorrect, it's less correct than we originally thought. Meaning, what a lot of these games are doing is actually allowing players to play indefinitely and they can get free coins forever. I mean, if you go open up Slotomania, for example, you can play at 0.01 cent bet amount, so you can play that game forever. And what they're doing is they allow you to do that and then they're creating Liveops and features to get you excited about the game. And then you're electing to increase your bet. And when you elect to increase your bet, you're going to have more volatility. And then when you have more volatility, you're going to lose potentially all of your coins or not be able to progress as fast as you'd like.
Brett (23:43):
And in that moment, you're the most excited about the game because whatever they released was something that actually got you excited to increase your bet. If you run out of coins or aren't progressing fast enough, you're going to choose to make a purchase so that you can do one of those two things, continue to play or progress faster to get whatever you want. If you don't want to, you can always go back down and lower your bet and play indefinitely. And I generally feel like that is a very strong way to think about mobile games.
Brett (24:13):
Having said that there's always the alternative, which is some people have a desire to be gambling, and gambling and excitement really only is produced when there's potential to lose. And in those games, you do want to have an economy that's much more tightly tuned so that people are going to lose their chips, or coins, or whatever it is all the time, because that's the excitement of gambling, which is like, you're going to lose all this money. I mean, you can win big, but really gambling is no fun if you never going to lose as well. So that's a little more tactical.
James Gwertzman (24:49):
Well, okay, you mentioned that you were at the monetization PM for a while, so let's drill in on that. So what tools do you consider essential in your toolbox or your Swiss army knife as a monetization PM, to be able to do your job effectively? What do you need coming in and then what levers do you need to have at your disposal to do your job effectively?
Brett (25:12):
That's a great question, a couple of different things. The biggest thing about a revenue PM that you have to think about is that your funnel is much steeper than the DAU, or retention, or engagement PM. Meaning that you're only going to be working with 5% of the player base that you have. So it's a much smaller group of people, and therefore, you can't afford to release some less effective steps in the funnel, you have to have a large percentage of people to progress from one step to the next. And so you have to spend a lot more time thinking about how you convert from just entering into the game, down to the bottom.
Brett (25:54):
And when you do that, you also have to get them to, I mean, not literally anymore, but I used to always say, reach in their pocket and get their credit card out of their wallet. It requires a lot more thought in terms of getting. Whereas DAU engagement, it's a different muscle. Obviously, I'm not saying it's better or worse, but you're generally thinking of everybody entering the game, 70% of people are going to see this pop-up, they're going to go into this pop-up and most of those people are going to go in, and that's going to drive engagement and retention because those are just like the masses, and then you're focused on that. So when you're a revenue PM, you're focused on a smaller group.
Brett (26:35):
The other thing that you want to do is stop thinking about yourself. I mean, unless you're fortunate enough to be on an RPG game and you're an RPG hardcore player. And even then, even then I wouldn't spend too much time thinking about yourself. I mean, that's probably one of the hardest things for a revenue PM or just a PM in mobile gaming to really do in general, which is, I'll give you a good story. People used to join Zynga Poker, and it was kind of a training ground for PMs and GMs.
Brett (27:07):
And they would always come over to me like, "Oh, I have this great idea, I got this great idea." And I'm like, "Really, what is it?" And they like, "Have you tried to sell real world prizes? You give them a house or a car." And I'd be like, "Yeah, we did." "Oh, what did it do? Did it drive tons of revenue?" "No, it didn't." "Why?" "Because people would say, I have a car, I want chips. I have a house, I want chips." And we did it and I mean, we did the real world prize and it took all these logistics, all these stuff, and the bump was minimal compared to just releasing virtual currency, which is what they wanted.
Brett (27:43):
So that's a good example. I mean, another one is thinking bigger. You know there's people like, "Well, have you released a $1 package?" And I'd be like, "Do you know how many $1 packages I needed to sell to put a bump in our revenue?" And people be like, "No, how many?" And it'd be a massive amount, it'd be like it had to be 10,000 a day. I mean, 75,000 a day. Okay, so those two are important. The third skill is the one I'm leaning towards you have to be that numbers guy all the time. Don't come into a meeting and if you say, I think or I like or any opinion based thing first, then you've just described yourself as a value to the team.
Brett (28:31):
They don't need a 25, 28, 32-year-old person who generally likes this or likes that, what they need is someone to say, I built a funnel, I built a spreadsheet, it generally says that if we invest two more days, it's going to cost this much more. This is what the funnel shows, these are the numbers, this is how we can increase those odds. That's a really important skill that I think a revenue PM really needs to bring to the table. And then probably the last part is, I find, for a product manager, the heart and soul of product managers is really the spec or the definition of the feature that guides the development, guides the artists, guides, whomever, to create that product.
Brett (29:10):
And that's really your development code for PM, the spreadsheet and the spec, and you have to earn respect with the team because you're not really creating anything, you're not writing code, you're not designing art, they could generally probably do it without you. And that's why you need to add value with that spreadsheet and that spec because they're not going to be able to do that or they don't want to do that and that adds a lot of value to the team.
James Gwertzman (29:43):
Well, I was going to say, I remember when I was working for PopCap over in Asia after EA bought PopCap, I had a chance to spend a lot of time with the FIFA team in Japan which is shipping a very specific type of FIFA game built in Japan for Japan market. One of the highest-grossing games that EA had at the time. And I remember the event model that team always had, at any given time, there were three or four live events running simultaneously. They had daily events, weekly events, they had tournaments, challenges, and they had these very complicated spreadsheets showing their predictions as to what exactly the revenue would do each day over the course of a month.
James Gwertzman (30:16):
I was amazed at how accurate they were. They were within 5% or 10% of their numbers two weeks out. And they did that because it had such a good understanding of how each event would impact the numbers and how they overlapped and the impact of the day of the week because certain people got certain credits in days of the week. And it was just a very interesting model. To your point about not ever having it be someone's gut, it was really the sense of, okay, here's what the numbers show. And then you had other people in the team whose job was to have the gut and the instincts and there was some friction there sometimes but also a lot of good synergy.
Brett (30:51):
Yeah, I mean, that was the heart and soul of product management at Zynga at the time when I was there. I mean, you would basically think of an idea and then you would create a funnel estimate, meaning 100% of people are going to come to the game on Tuesday, I'm going to do a popup and that's going to get 70% of the people and from that popup because I see this other feature that was a popup last week and it got 70% so that's where I got 70%. And then this other feature a year ago was similar in that it targeted, it was a collection feature and that got a 30% conversion off of that 70%. So now you're down to whatever that number is 70-30.
Brett (31:28):
And then from that 30, we're going to charge $8 and last year we did a feature that we charge $8 and we got a conversion of 2% from that. And I think most of the people can do $8, some people will do the 15 and boom, boom, boom from there. And then you talk, so you get some estimate like, "Hey, I think it's going to be 2%." You would build that funnel, give it to your boss, he'd save it. You would put that 2% in a roadmap, you would present that you're going to build it. And it would get sent all the way up to the top of the organization and it would have your name on it and 2%. You would go release that feature, you would put together a presentation on how it did, you would pull all the data on all the funnels, you would compare your predicted funnel to the actual funnel and you would present it to the VP and then you would send that report out to the entire company.
Brett (32:18):
I mean, the entire product or they would, it was people who had access to the money. So you're basically the product org and the design org all the way up to the top. And I'm talking like Pincus, he would be reviewing it. So it'd be Brett Nowak said he was going to drive 2% and his feature drove 0%, or 2% or 5%. And in fact, if you drove 10% and you said it was 2% they would be kind of, not annoyed, I mean, obviously happy but it had to have been something really unexpected but it was sort of almost why did you not predict that? I mean, it was really about this-
James Gwertzman (32:57):
You almost got beat up just as much or being under than over because your job is to be accurate.
Brett (33:02):
Yeah, and so that's how we did it. And so that's, again, a good example of how the PM can bring a lot of value to the company by decreasing their risk. And that's really kind of how Liquid and Grit helps them out a little bit is, or not a little bit, a lot is like, "Hey, this is the feature that drove revenue in another games, this is a feature that didn't drive revenue in another game, you should consider these things when you do this feature in that funnel step." But yeah, that's really a super important role. Now, having said that the thing that I think we lacked or we didn't lack, we could have improved in our process which I think we've talked about a little bit is you don't want to put so much of an emphasis on predictability.
Brett (33:47):
So the column, I think in that equation that we didn't put enough emphasis on is likelihood to succeed. And you want to have just like a good financial manager, you want to have Liveops events that are predictable that are secure, you're going to have a good confidence on it. Just like you were saying in Japan where you're saying, "Hey, we're 95%, 90% confident that this feature is going to drive a 1% bump or 8% here." That you want to have a decent amount of your portfolio but you do want to have 20%, 30% dependent on your, again, product strategy of your portfolio that's going to have a much lower likelihood to success but higher potential output.
Brett (34:28):
Those are the ideas that somebody is going to come along and they're going to say, an example is like, Oh, they're going to say, "What about tipping the dealers in Zynga Poker?" We'll let them tip the dealer and five or six people, saying that, "This is so stupid, why would you tip a digital dealer? That it makes no sense." And that person to say, "Well, I really believe in this." And you're going to say, "Okay, well then put the likelihood of success lower but it better be a really good feature because it has to compensate for that low likelihood of success." That's an actual example of a feature that went live in Zynga Poker and was very successful. And a lot of people in the room were like, "That's the most ridiculous thing in the world? Why would anyone do it?"
Brett (35:08):
Mobile gaming in general? The whole business is borderline ridiculous at the start. It's like put money down for virtual currency. So you have to have that in that model and a roadmap where if you're building a strong Liveops, you do have 70%, 65%, whatever percentage you on around of confident events, stuff that you're going to read about in our reports or whatnot, and then have other that are just wild ideas that someone have come up with the game designer, whomever, the designer, it doesn't matter that really origin of it. And then now you're testing them, you're trying them, why not? It could work, who knows?
James Gwertzman (35:46):
Let's talk about testing, because I remember Zynga was famous or Mark Pincus was famous for saying something to the effect of the game company that does the most testing wins and sort of always running tests. And I don't know if that was a reference to sort of formal multivariate or AB testing, or sort of more hails to try this thing, apply to everyone and see what happens. What your own kind of both experience with testing and experimentation in games and any tips on, for teams you don't do a lot of experimentation today, tips for sort of beginning to do AB testing or sort of more formal testing methodologies in games.
Brett (36:20):
Yeah, well, first of all, one tactical recommendation is Product Development Flow is a book that I read that. I mean, it's my favorite product management book and it actually outlines a lot of these mathematical processes for product managers. And it's an excellent, excellent book.
James Gwertzman (36:37):
Product Management Flow?
Brett (36:38):
Yeah, Product Management Flow, I can look it up in a second, well, I'll look it up after, but it's just... do you want me to look at now? I can.
James Gwertzman (36:50):
No, we'll check it afterward, we'll find the link for our listeners.
Brett (36:54):
It's a little bit dense, I mean, it's could put you to sleep a couple of nights but if you're in product management, you'll love it. I loved it, I've read it a bunch of different times but they talk about testing. And testing, again, I think there was a strength of Zynga's, I think it was, again, a little simplified where everyone was just like, "You have to test everything." Everyone just came away with Zynga tests, everyone needs to test. Testing, like anything else in product management, is a tool that you should use when it's appropriate. And it's most valuable when the likelihood to fail is closest to 50-50, because then the test is going to tell you a lot of information. If you have a 99% confidence level that, that feature is going to be successful, then testing it is going to be very costly. You're just affirming information that you already knew.
Brett (37:44):
So testing is another part of product management that you really want to think about, particularly when you're in a Liveops team because your potential impact of a feature is much smaller if you're releasing it for a day or two days, or whatever your releasing now, and we're actually seeing a lot of Liveops getting extended and having be more developed or we'll call it, we actually find things called temp... we're categorizing things called temp features because we're seeing these events go live for four to six weeks so there's sort of this middle area as well.
Brett (38:17):
But if you have an event that's only a day or two live, I mean, testing it, the impact of it is pretty restricted already. So testing it, it's not going to save you a lot, so it could cost you a lot for not saving a lot. So again, I do think, if Mark said that, I don't remember him saying that but testing was certainly part of it. Is Mark was generally understanding that there are inexpensive ways to create a lot of value and you can create a lot of value by finding information. And at the time Zynga had nine, 10 games. So if we learn something, we would get 10X the value. And that's what we were doing, we were sharing those reports with each other, as we were saying like, "Hey, this worked in this game, you should try it if you build a leaderboard, this is what you should do, this is what I learned."
Brett (39:11):
If you're a game company and you have one game and you find out the leaderboard worked really well, it's not that valuable to you because you just release a leaderboard and it's probably pretty obvious because the revenue increased. So how are you going to share that with? And what value you're going to create from it. And so that's why testing, just like anything else like I said, it's a tool that you need to think about, how am I going to create this value? How much cost am I going to incur? And what value am I going to find out? But I generally feel like you can find out a lot. And so it's just a matter of understanding what that is going to be and how much value you're going to create from it.
James Gwertzman (39:53):
Very cool, one of the things we always talk about on the show, we love to ask people for their Liveops horror stories, their train wreck stories, you have any stories you can share from games you've worked on or games your friends have worked on where you experienced just a train wreck, you did something. And it really had an unintended consequence or hurt the game in some way.
Brett (40:17):
I think one that I've actually seen a bunch of different times, although we're seeing it differently now, which is really interesting. And this is probably, this was back in, this was earlier when Liveops wasn't actually as much part of the game, but I think a couple of train wrecks that I've seen is investments in UI UX. I think that's a good example of what I talked about earlier is like, you don't want to think about yourself. In Silicon Valley, San Francisco, Seattle, these sort of tech hubs, we like our apps and tools to be pretty, and that's something that we generally value, I think, disproportionately relative to a lot of our players.
Brett (41:02):
And so we have updated our whole poker table UI, we were going to make it just so much easier for people to just enter a table and that was just going to be so pretty. And we were going to design the heck out of it and a lot of top-down ownership of what looked nice and what didn't look nice and what they liked and what color they liked, and purple and yellow, and round corners are square corners, or three options or two options and we released it and it just was a total bomb. I mean, it just was like the total bomb because you're changing every little pixel in the most important element of the feature, I mean, the likelihood to succeed is just so low.
Brett (41:44):
I mean, every single person has their own preference about how they're using the ugly data table selection screen with a million different options. Everyone uses differently and it just totally tanked and it was also like a huge investment. And I've seen that happen a bunch of different times. The funny thing is now that we have Liveops become so important in mobile games as it has lots of years, those UI feature updates we've actually noticed have been doing very well for mobile games. But for years, we would report on not doing well. So they're actually it's interesting now that events are so much more important that those are becoming more and more important to surface to players. So but that was a horror story that I saw, fortunately, I was a spectator and that wasn't-
James Gwertzman (42:40):
We've talked all a lot, special events, live events and you talked about long ones, short ones, limited-time ones, my favorite aspect of Liveops is the events because I think there's a such a critical piece of the retention engagement and they're fun because they're different. For someone that's a short attention span like me, you get to do something a little different all the time, any tips on sort of an effective event strategy on how to layer your events.
Brett (43:08):
I mean, I think you want to be thinking about them holistically. And that's where you want to have both a team that's decentralized and that strong central group as well. Because you don't want to have a franking game where you have all these PMs running around or game designers or wherever you were structuring it and releasing these individual events that are effective on their own. They're just like on Tuesday you have that really cool event that, so and so released then on Thursday, that really, really cool event that so and so released. And one drives a ton of engagement, the other one drives it on a revenue and they don't really fit together.
Brett (43:48):
And that's where you have to have, you have to have those decentralized teams that are finding that information but you kind of have to have that central voice that's matching those together so that when the players playing every single day and I think that's probably something I didn't bring up when you're thinking about being a really strong revenue PM, you want to think about someone who's playing your game 50 times a day, that person is super important and or eight hours a day, that person, what do they experience and understanding that the event on Monday does affect the event on Tuesday that does even affect the Wednesday and each of those have to fit together in a cohesive experience.
Brett (44:27):
And I think a good way to think about that is the restaurant menu. I mean, I think in general you want to have your staples, you want to have your items on the menu that people come in and always can kind of understand and want, you see a lot of event calendars, complex event calendars have that they'll have a certain amount of them that are 24 hours but they're every day or release stuff for a certain amount of time and then they go away, then you do want to have your specials. You want to have your items on the menu that are just here today and they're just going to be here today. And if you don't come on today, then you're not getting it or you'll have to wait until next week today or you're going to have to... And those also fit into whatever theme you're doing.
Brett (45:07):
And we're seeing really good teams combine those Liveops events with the features that they're releasing or the temp features as we mentioned. And those are the really strong teams. So they have the features of the releasing the game that are permanent. Then they have the temp features, which are the seasons, which is sort of the classic, but we're seeing different versions of seasons. A lot of seasons could also be a content release, where you're going to collect this set of cards and you have two months to do that. And then on top of those seasons are temp features. Then you have the Liveops that are feeding into the temp feature that's feeding into the feature itself, that's feeding into the game. And so, yeah, it's not easy to do and then you want to also have people experimenting with it and then having a strong, confident roadmap as well. So yeah, that's how I would build it.
James Gwertzman (46:02):
Very cool, well, I know we're getting close to time here. I want to take a minute to just really thank you for coming on and talking about your experience. I'm happy to give you a chance to plug Liquid and Grit if you'd like and talk a little bit of what you guys do. If I'm a product manager, listening to this podcast is intrigued by what you guys do, how do I learn more about what you guys do?
Brett (46:22):
Yeah, I mean, you can go to liquidandgrit.com and check us out and download a sample report. If you want to check one out, you can email me at brett.nowak N-O-W-A-K @liquidandgrid.com. I'm very active on LinkedIn that's probably a great way to connect with me. And like I said, we really focus on the PM, the game designer, the game owner in a product, mobile gaming org. I mean, if you read our reports and you're not that person, you're going to know it's not for you, that's really who we serve, that's who we focus on. It's not that we're going to say, no, you can't read it but that's really our target audience, that's who I was, that's who I really enjoy, as you can probably tell, I love this stuff. And it's been cool talking to you about this.
James Gwertzman (47:14):
All right, well, thank you very much, I appreciate it and good luck.
Brett (47:17):
All right, thanks.
James Gwertzman (47:24):
Thanks for listening to the art of Liveops podcast, if you'd like to what you heard, remember to rate, review and subscribe so others can find us.
Crystin (47:31):
And visit playfab.com for more information on solutions for all your Liveops needs.
James Gwertzman (47:36):
Thanks for tuning in.