Today we are joined by Jason Coleman, founder and CEO of Sparkypants, specializing in Long-Play, Cross-Platform Strategy Games. Listen as Jason explains how the roles of Developers, Players and Publishers have changed in this brave new LiveOps world.
Today we are joined by Jason Coleman, founder and CEO of Sparkypants, specializing in Long-Play, Cross-Platform Strategy Games. Listen as Jason explains how the roles of Developers, Players and Publishers have changed in this brave new LiveOps world.
James Gwertzman:
Hello, I'm James Gwertzman.
Crystin Cox:
I'm Crystin Cox. Welcome to the Art of LiveOps Podcast.
James Gwertzman:
Hey Crystin.
Crystin Cox:
Hey James. So today we are going to be talking to Jason Coleman. He's the president of Sparkypants.
James Gwertzman:
Right, and also a pretty technical guy.
Jason Coleman:
Yeah. I'm the President of Sparkypants Studios. My primary role as developer, is lead backend programmer. So the first was really with Drop Zone, which was the first largest title that we did for Sparkypants. There really wasn't anything that existed to do what we wanted to do, so we built it all. We used Node and we use pretty much straight JS for that. Did our whole backend for that. And then we basically had to do version two of it, or had an opportunity to do version two of it for TESL, The Elder Scrolls game.
Crystin Cox:
Yeah. I mean, I think he comes from more of a technical background, and he's been helping, been very involved. And Sparkypants has been making this transition to really live game. They're working on The Elder Scrolls online card game. And so, he's been super involved in how they've built out that backend. And he's going to have an interesting perspective because, we're from PlayFab and-
James Gwertzman:
We know LiveOps tools, yeah.
Crystin Cox:
We know a little bit about LiveOps tools, but I think they've home grown pretty much all their tools. I think he's going to have a really interesting perspective.
James Gwertzman:
Cool, awesome. Let's go chat with Jason. How much of your game logic is living in TypeScript? And are you using TypeScript because it lets you more flexibly change and update, make changes to your game systems?
Jason Coleman:
Yeah. I mean, our basic setup these days, we moved to Docker, and so basically, Kubernetes is wrapping a bunch of Docker containers. And we have 26 services that are running on those, and they're pretty much micro service, maybe not micro, but mini services, and they have their own responsibilities. So TypeScript is really the go between whether it's the client or the game servers, and the DBs, and sort of tying all that together.
Jason Coleman:
So it doesn't have a lot of work to do, we don't like to do a lot there. The actual game logic in terms of running a game, for this game is done with .NET, and that's, again, wrapped in a container and then we're connecting those.
James Gwertzman:
Go it, so those mini services, those are things like storing player profiles, and...
Jason Coleman:
Profile, inventory. I think we have three that do matchmaking for us, four that sort of have different responsibilities for that. Yeah, events, account, so authentication. So in the night, the awesome thing I think, is that if something goes wrong, we pretty much know, "Okay, that's the service that's responsible for it." If we needed to do an update, we can roll that out. It can come online, and then just roll over to that new version.
James Gwertzman:
So one of my personal hobbies, and everyone has their own hobbies. I'm fascinated by tools, and the tools people build for themselves and top of services like that. So I'd love to hear a little more about... And maybe this is one of things that you've evolved over time, because typically these do evolve. Is what sort of other web pages, or apps, or UI did you build on top of those services, to help your team do the actual work of running, and maintaining and monitoring, and operating your game?
Jason Coleman:
So the biggest one that's just a straight up tool for the backend services, is called the admin tool, because it's super creative.
James Gwertzman:
Creative, yeah.
Jason Coleman:
And so that's another thing we switched over. So we're using Vue for that, in terms of the frontend. And then it talks to a different service, which is the admin service, in terms of actually doing any kind of work. And that has essentially everything that you'd want to look at, in terms of the database, in terms of player data, or game state, that's not the actual game server. Anything that we store allows you to make any sort of changes. In fact, a lot of the developer workflow, is we'll go and we'll create the version of some kind of action, there first, a view of it, before we go and actually use it in the game. And so we know it's there the whole time.
Jason Coleman:
And one of the super cool things that actually made us move to Docker, is that we can run a full local development pretty easily. So we can have a stack that includes setting up the DBs, a [inaudible 00:04:22] version, all that kind of stuff locally. So with one command, any developer can set up a whole backend stack, so that part of development can flow the same way. And they have their own admin tool for local and all that.
Crystin Cox:
So a lot has changed in the past 10 or 15 years, in the way that a game development team interacts with platform, and the way it interacts with technology. As you're doing that work today, and you're interacting with developers, what's the biggest changes that you've seen, as far as the things that they're concerned about, or their needs?
Jason Coleman:
Probably the biggest one for us, is moving from back when we were starting, we were doing RTS games, and we were doing simultaneous SIM where we had to have same code running, because it was all peer-to-peer. So the biggest change was moving from that to doing client server, and doing a server authoritative. And that has affected kind of the flow, in terms of game design and all of that, of what data is stored where? How do we have access to it? And I think that's probably been the biggest transition.
Jason Coleman:
And then there's just, how do we set up a system that allows us to have one source of truth for all the data, and how does that flow through? And how are the backend systems aware of it and all of that? And everyone has to have kind of a baseline knowledge of what's going on there, to make it work really well. And so that's probably the biggest.
Crystin Cox:
Yeah. And I know you mentioned a little bit, there's sort of a data pipeline that [inaudible 00:05:44] has to get built up. I know when I was starting in the industry, that wasn't necessarily a given, right? People who were building games didn't necessarily think, "How am I going to handle my data pipeline?" How have you seen developer's relationship with data change, as that has become just an essentially, "Oh, we have to have an engine, and we have to have a network into the middle of our pipeline." We also now have to have a data pipeline.
Jason Coleman:
I think mostly just as there's become more to do, there's the expectations on having it all automated. It'll become a lot greater in terms of developers. So I mean actually, it's a piece that we're pretty proud of in terms of how we sort of flow through it, of they know they have one source of truth, and for this game, it's Unity. For drop zone, It was our Sparkle engine. And this is the one place where you need to set up all the data, and we have tools to do all that. And then it flows, so we use TeamCity for automation.
Jason Coleman:
So there's one button that people can push, to push a build, and so everything kind of gets crunched through there. And then so we have transformations of that data, and all the different forms that needs to be. So whether it's for the game server for the client, or for the backend systems. And so I don't think that's affected the developers, in terms of designers say, or artists so much, other than that there's a lot more validation that goes on, and they need to pay attention to it. So we try to catch it obviously early on their client, but they also, all the CI is running, and they need to pay attention and say, "Oh, I actually broke the build, and I need to go and fix it." [crosstalk 00:07:14].
James Gwertzman:
And what's the origin? What's the authoring tools, like Google Sheets or Excel? How do your designers actually author that type of data?
Jason Coleman:
So we definitely love Google sheets, but actually I shouldn't say that, we have a whole shop catalog. The source of truth was Google sheets for a while, just because it was kind of this dynamic data, and now it's moved into the admin tool. But primarily, the source for a Unity game will be all the Unity tooling, and they're basically modifying JS on files. And so some of those were split up into reasonable chunks. For a card game, it can be by season, or it can be by collection, that kind of thing, but that's what they're modifying. And then all of that gets crunched and merged, whether it's into binary or into SQL script, or anything like that going through.
Crystin Cox:
Nice. So you've made this transition, and a lot of other teams are making this transition. As you said, not that long ago, you guys were making RTSs that were probably pretty client authoritative. You've now made this transition to a LiveOps world, do you have advice for people that are going through this transition now, as a lot of developers are staring down this transition?
Jason Coleman:
I'd say that the biggest things, are the things that have evolved to work really well, and that I'm grateful for, had to do with just understanding what the state of the game is. And so that is in our DB, for example, we hardly ever basically never delete anything, we're just adding. So that's one place that we can always go and look and say, "Okay, we understand what transpired here." We have a lot of transaction rows and things like that, that we're adding. And then we do archiving every now and then, for the tables to get really big.
Jason Coleman:
And the other is logging, in terms of you want to kind of log everything, but you don't want to log everything, because if you log too much, you can't really understand what's going on. And then there's Graphon, in terms of whatever you're using for all your graphing. And so I think the thing that's that's happened over time, is you get woken up at 2:00 AM, multiple weekends in a row, and we have a NOC that we work with, with Bethesda. So Network Operations Center. And the more they're able to handle everything, you don't actually get woken up. And so you start thinking through that mentality, of, "What do they actually need to be able to diagnose, and what tools do they need to go and fix it, without making that call?" And that's been the biggest change.
Crystin Cox:
Yeah, that's so true. I mean, I think for me, the thing that's been one of the most interesting things about running live games, has been not only, okay yeah, get into a place where you're not babysitting it personally 24/7, but I'm constantly surprised at what it is. What is the data point that's actually the useful data point to watch? Because I think people think that it's just like, "Yeah," you're just like, "are the servers up and running, or are there a bunch of errors being spun up?"
Crystin Cox:
My experience is actually, there's other really interesting key indicators that happen first. So I wonder if you have interesting stories about strange graphs, or maybe counterintuitive numbers that you're actually watching, that really tell you if things are going well?
Jason Coleman:
I don't think that they're probably not counterintuitive, but I mean, we learned a lot from the NOC with Bethesda, in terms of what they had seen before, and the biggest one was just population. So if you have a population change, or population drop that's really too big, that's always the key indicator of something amiss.
Jason Coleman:
Although, it was interesting when Game of Thrones, when the next to last episode came out, we saw this 10% drop in our CCUs, right when at 9:00 PM on EST. And clearly there's a big crossover with that and Elder Scrolls Legends. That was what was causing that.
Crystin Cox:
That stuff is always fascinating.
Jason Coleman:
Set off all sorts of alarms.
Crystin Cox:
Oh yeah. This was always fascinated. I always tell people too, if they're getting started in DevOps, velocity. Velocity is actually the thing you want to track, you never want to really look at total numbers. Because as you said, yeah, it's as simple as maybe looking at how many players are logging in, but player logins should follow a fairly predictable curve. And once you set that velocity change, it's like, "Oh, suddenly?" Even if it's a positive one, you want to ask, "Why suddenly are people logging in at five times the rate they usually log in?" That's probably, something's happening here.
Jason Coleman:
No, yeah. Population is definitely the key one. And then there's a lot of, "What do we need to look at, in order to tell, who to go and blame, and whether it's us or somebody else?" Which is a big thing that we have, we hook into Bethesda services, we hook in obviously steam and all these different authentications and everything. So just figuring out, who do we actually need to wake up? Or is there something that we can do?
Crystin Cox:
Totally. Yeah, we used to look at chat volume really closely. But you know something's up if chat velocity changes significantly. Although interestingly enough, as you said with the Game of Thrones example, you get false positives. You get like, "Oh, actually what's happening is the World Cup is going on."
Jason Coleman:
And that's totally where you need someone with experience, with a brain who can come in and say, "Oh yeah, it's probably for this. Let's watch it for a little bit before we get too antsy."
Crystin Cox:
I think that leads us down an interesting road. As you said, there is this idea, and we all get caught up in it, where we're like, "LiveOps is delivering this future, where machine learning is going to do all of this stuff, and AI is going to do this, and automation is going to happen." And that stuff is fantastic, but at some point, where are those places where no, we need someone with a brain to come in and say, "Oh, actually it's the World Cup."
Jason Coleman:
Yeah. And there's a lot of discussion, they're hard decisions, particularly when we first launched. I mean, TESL was kind of a crazy thing. We had to totally redo the game in nine months, all the systems and everything from scratch. And so it was definitely rough in terms of why. We didn't have all the tooling, all the things that we wanted to have. And so when Reddit is your main kind of source, of what's going on with the game, that's a really bad place to be.
Jason Coleman:
And on the other end, is when someone can come in and say, "Okay, I pretty much have the data right in front of me." But we still have these tough choices of, "Okay, we have a way of fixing it, maybe we can fix that." And the nice thing, is often we can fix it on the backend. But when we do that, what's the player impact actually going to be like? What's the experience going to be like? Is this worth doing?
Jason Coleman:
The gut reaction as a developer is often, "I wish you'd just fix it." And someone who's really thinking about LiveOps, is like, "Well, is the risk worth it to fix it right now, or should we wait till tomorrow? Or when should we actually do this?"
Crystin Cox:
Yeah. I mean, I think you're hitting on something there too, and that so often in LiveOps, Yeah, there's a lot of things that we do over and over and over again, but also it's live. Right? I mean, it's like a rock concert or a theater performance, and there's one-offs that happen, and you don't necessarily have a playbook that says, "Oh yeah, of course do this," because you're only going to experience it once.
Crystin Cox:
We love also to hear, we ask everyone on the podcast, "Tell us your horror stories. Tell us the disasters."
Jason Coleman:
Oh, there's probably been too many. We have, I mean, lots of games have a daily reward system. So your log-in reward for the day. And there's always this one system that's sort of your Achilles, that shouldn't be that difficult to get right. But we had all sorts of issues on launch, and they just seemed to sort of keep going. And it was a timing issue, because it happens at a certain time. And so now you're trying to to fix it for the future, but also retro fix players. And so there's fixing the code, and there's fixing the DB without making it worse. And you got people logging in while you're doing that in different times.
Jason Coleman:
And so it just kept going and going, and it was super embarrassing, because you think, "Okay, we're going to totally solve this." And we had this time zone offset that was causing weird issues. So yeah, that was probably the nightmare that wouldn't go away.
James Gwertzman:
Well that's actually a really good segue into testing, because I'm actually very fascinated also by how we test live games. because a lot of times, the things you want to test are time dependent, and you can't just roll the clock forward on your local client to test something, because it's all a bunch of interactive servers. So when you're planning maybe a time-based event, that's going to have a certain stop date, or maybe you're planning a new version update that requires a bunch of changes in technology, how do you test that kind of stuff?
Jason Coleman:
So it took us a while, and it seems like a no brainer after the fact that I say it, all these things do. But there's two things. One is the admin tool for anything that's actually got start and stop times, those are settable. And we're on a different environment obviously for tests. So QA can go and set those. The one though, for something like anything that has a roll over season, the season rolls over like our monthly season. Or the daily rewards, that kind of log in time, we have an offset.
Jason Coleman:
And so again, that's settable in the admin tool, we generalize this, so there's basically a bunch of strings or inch. You can go and set in the admin tool, and then you can go and fetch those on the service side when you want to. And they get cashed.
James Gwertzman:
And the offset's per player? Or that offset is per-
Jason Coleman:
That's for everybody. And so what QA will do, is they'll basically set. I think 0600 UTC is our normal time, so they'll just set it in the middle of their afternoon, so they're actually there. Because this was a huge problem, is all of the testing was taking place at a time, when wasn't going to be impacted by daily rewards at all. And so we'd only find out about it when players were seeing weird stuff at 1:00 AM EST. So...
Crystin Cox:
Do you guys do a lot of automated testing? Do you guys do a lot of scripting based...
Jason Coleman:
We don't do a ton for this one. We did more for Drop Zone, a little bit the nature of the game. Because there, we had lots of different types of replays that we could run through, and see how things were stacking up. So we don't do a lot of automation. We do a lot of CI, so there's a lot of data validation, and then QA's job each morning, their first job is to smoke tests. So we have automated builds that come on, and they'll go and test that one, and they'll promote it from smoke test to QA, and then it'll start getting hit from there.
Crystin Cox:
Yeah. So I find that a lot of the challenge for a lot of traditional sort of game developers, is games as product, was really focused on black box testing. Just go get a bunch of players, and have them play a lot. And then you'll sort of find all the defects, and that isn't really the way LiveOps works.
Jason Coleman:
Yeah. I mean, it would be nice to do more. In fact, once we again, get more levels of replay, that's a really nice thing to be able to run through. So on Drop Zone, what we would do, is we would essentially take here's 10,000 games, and we would run them through the game servers to be sure we got the same output on this before, unless we actually changed something that we knew would change that. So we knew all that was working well.
Jason Coleman:
I mean, it gets more complicated on the client side obviously, and with Unity and all the visual effects and everything, in terms of verifying all that. But yeah, we try to do as much as we can.
Crystin Cox:
How often do you guys update the game?
Jason Coleman:
Right now, we're on a fairly decent new version every month. And most of that's actually, it's on sort of four platforms right now. So it's Steam, there's the Bnet launch, or there's iOS and Android. And a lot of that's kind of based around mobile, in terms of that cycle. So when we first launched, we did, I think one a week, an hour, more than one month. And pretty much a major version every three months.
Crystin Cox:
Dude, what made you guys sort of settle on that cadence?
Jason Coleman:
Predictability. A lot of it internally, of just saying, "Okay, we're doing it once a month, and that's what we're doing." There are a lot of expectations that we're working with Bethesda, we have our own internal staff to kind of get on board, and everyone to understand what's going on. And so the more we could just say, "Yes, once a month," the easier it was going to be to align everything from a production standpoint.
Crystin Cox:
I think predictability is super important for live updates, so I completely understand that. Another thing that a lot of developers really struggle with when they moved to LiveOps, is build test, deploy pipelines, especially console developers. Those pipelines can actually... Even today, I hear teams say, "Oh, it takes us about a week to produce a good build." Can you talk a little bit about what you guys' build test and deploy pipeline looks like? And that you are capable of doing once a week live builds?
Jason Coleman:
Yeah. So essentially, we would go through TeamCity, there's the button you push, and that will take whatever version that you specify, in terms of a change list on Perforce. And that'll send it all the way through, that'll create all the clients, it'll deploy them so they're available. One of the things that gets generated is a big SQL script, that makes the DB content aware of that version. And so that gets pushed.
Jason Coleman:
One of the things we like to do, is to be able to run multiple versions at once in the DB. And so all of that's segmented in a pseudo-clever way. Clever in a good way, not in a bad way of software development, in terms of the content. So all that gets pushed, and you can do that all locally as well.
Jason Coleman:
So there's multiple stages. You can test all this locally, if you're going to push it, that can push everything. It's automatically live. When we go to production for the services, we're taking the same containers that were tested on integration, and we're just moving them over into production. So we basically want to have the fewest steps when we actually go live, and so mostly that's just setting the clients to be downloadable.
Jason Coleman:
So yeah, when we're actually in our final step, we have integration, which is where we're doing all the QA testing. When we're ready to go and deploy to live, we move them over to production servers, but they're on what we call a different channel, sort of like Steam Beta Groups kind of thing. And so all of that can be tested there. And then when it's 10:00 AM and we're going live, all we're doing is setting those clients to available, and they start downloading.
Jason Coleman:
Another big one, was things like shop catalog, which we make that. So whenever we're doing a new change to the items, we essentially copy it and make the changes, that's a new shop catalog. And we can set that either on a channel, or on a group. So we can have the group QA, in which you can set on a particular account, so they can do all that testing. So your final step when you want to make it live, because we may do that more often than once a month, is all you're doing is changing one number in the DB. You're just saying like, "Okay, now you're pointing to that shop catalog, instead of a previous one.
James Gwertzman:
And one thing you mentioned a couple of times now, is you're working with Bethesda as your publisher. And so I think back in the old days of premium games, it was very clear what developers did, and what publishers did. Developers made the game, they threw it over the wall. Publishers took it and went to market, and did all the... They got on the shelves of Walmart or wherever. How have you defined your role vis-a-vis Bethesda, in [inaudible 00:22:04] of a LiveOps game?
Jason Coleman:
So we ended up doing more operations, I think any of us, including Bethesda thought we'd ended up doing. So we essentially set all that up and all the systems. The things that are now stable, are they provide the NOC, so they're, the ones that manning the wall. And then there's some content pipeline, in terms of for CCG, in terms of things like card art that they really manage that piece. And then marketing and QA. So we have some very small QA in house, so most of the QA is with Bethesda.
James Gwertzman:
Interesting. Yeah. Do you do a lot of paid user acquisition? Or is it free to play game? It's a free to play game, yeah? Or is it freemium?
Jason Coleman:
It's a free to play game, yeah.
James Gwertzman:
So one of the things we've seen a lot of, we haven't really talked about this yet, is how games have gone from... We talk about packaged goods and premium, we talked about games and services. We have this belief that games are now in this new era of community, where games are actually communities, where the community starts to become way bigger than just the game itself. And maybe it's also the streamers, and maybe it's also blog posters, or forum posters, or whatever else it is.
James Gwertzman:
And in that world, where it's community, oftentimes the acquisition becomes more and more organic. Players are coming in on their own accord, because it's way stickier and there's more friends, but even at that level, often there's a lot of paid acquisition. And so I'm curious, how have you thought about this notion of paid acquisition campaigns, or running ads, versus more of the organic stuff? And do you do specific activities, specifically drive up the organic acquisition?
Jason Coleman:
We do. And that's actually, I totally over the part that probably is most shared between Bethesda and us, and the thing I think we need to figure out the most, even now. And that is in terms of business intelligence, where's it going? Who's doing the reports? How are we dealing with that? All UA and marketing is handled through Bethesda, so there's even a question of, how much are we working together on that and everything? Is sort of ongoing, figuring out how to deal with that.
Jason Coleman:
So yeah. And I think absolutely, we think of the community piece is the key marketing element. And so it is a weird thing to try to figure out in this kind of relationship, in terms of we as the developer, they're the publisher, we're co-developing this game in certain ways. And where does community come and go, and who are they interacting with? And so it's definitely a blend. And I think it's one of the tougher pieces still for us to figure out, even as a developer.
James Gwertzman:
You just mentioned marketing, I want to jump on that. I think marketing itself has changed a lot in the last decade, as long as the shifts. What is State of Art of Marketing in games? When you think of marketing, how do you think about that? And what part again, is your responsibility versus your publishers?
Jason Coleman:
Well, I'm going to veer from what we're doing on TESL, and more what I think should be the case.
James Gwertzman:
Yeah, that'd be great, please.
Jason Coleman:
And I think it's becoming more and more clear, that overall, there needs to be this deep link to the developer, in terms of how you're building community, and how you're communicating. There's just this sort of sense of authenticity that needs to come through, that is there. And so that relationship needs to be nurtured and grown over time.
Jason Coleman:
And so it's very difficult to disconnect those two, and I think that's one of the big transitions for anyone doing Triple-A development over the last 20 years, where you had this clear delineation between, "Here's the publisher, they're going to handle marketing." They will talk about the developer, but they're really pushing it from that side, versus actually, mostly the interface is going to be things like Reddit, and Twitch and Twitter, and the developer talking about the game and dev diaries. That's going to become more and more the mainline way to market. I mean, they've sort of blended marketing and community.
James Gwertzman:
Would you actually take this to the next? We actually just had this fascinating conversation with another person, about the nature of the game development of cells being part of the story of the game. And so to the point where marketing is dev diaries and stories, yes, you have to be authentic, but there almost comes a part where you start to... I don't know, if that's your marketing, how much is that truly authentic? It's almost like reality TV, is it really real? Or is it crafted and scripted?
Jason Coleman:
I think as player, I mean, a lot of the reason we as players play games, maybe less so as devs, but in general, is you get into the idea that you're part of the development of the game, and you really are. You were there, that's part of why early access kind of works.
James Gwertzman:
For starters.
Jason Coleman:
Yeah, yeah. I was part of the reason this came into being and why it exists, my feedback was incorporated in some way. And so that happens to kickstart it all through early access, in terms of just your attachment to the game over time. And so actually, the players kind of become part of the story as well. And there's this sort of feedback loop that grows and grows.
Crystin Cox:
Yeah. I mean, I think that that's one of the things that we've seen a little bit of a tension, as we've moved into a world where LiveOps is becoming more prevalent. Is that there's an older way of thinking about game development, or more traditional way of thinking about game development, which is very developer centric, right? It's very much like, "What game does the developer want to make? What story does developer want to tell?" And they're going to make this thing for you, and they're going to just give it to you and you're going to consume it. Into this world where the player is really part of development, and the game sort of belongs to the community, and they have a big influence on what happens with the game.
Crystin Cox:
So you guys, it sounds like, are much more in a world of, "Yeah, we do it for the player. The player is a part of this," but has that been a tough transition? And especially with that relationship with the publisher, can that be hard to manage?
Jason Coleman:
I mean, I think that that tension's always been there, in terms of players say they want this, and part of the designer's job in particular, is to figure out what do they actually want. We sort of talk about the best gameplay programmers are. They don't just implement what's requested, they go and figure out what does a designer actually want to happen, because they can figure out the best way. So it sounds almost paternalistic, but it's not, it's just, you need to figure out how to actually manifest it in the game.
Jason Coleman:
So that tension's always been there. Now the feedback, there's a lot more feedback coming from the players, and that loop is tighter. So I think that's in some ways, increase the tension, because you still need designers to be able to say... I mean again, we're making a CCG, and there's going to be strong opinions in every which way, about what direction the game should take. I mean, the designers have to figure out, "Okay, who are we trying to serve here? How do we serve them best? How do we serve a broad audience?" So yeah, I think that's always kind of been there.
Crystin Cox:
And what kind of tools do you guys use now? Because you've mentioned Reddit a couple of times, and I know as a game developer, I have a real love-hate relationship with Reddit, right? It can be an amazing source of temperature taking for the community, but we all know that it's a tiny percentage of your community that's very, very loud. How do you navigate that? I mean, data is part of it, right?
Jason Coleman:
Yeah. So we're still trying to figure it out. I mean, that was one of the early questions, is how do we beyond Reddit in terms of understanding our audience? Because it is, and most of our audience is mobile, for example, and that's not necessarily well-represented, or really well represented by Reddit. And so that was one of the early questions, is how do we kind of do that? And there are various social tools, some of which I can't think of the names of that we're trying out. But mostly yeah, it's just actually looking at the data.
Jason Coleman:
And it's interesting, again, that tension of, we have all the data in terms of win rates for cards and things like that, obviously. And we're using that to try to understand the development. The community feels a certain way, and would like to see that data. And so that's even a fundamental tension of, is that data you should share? Because then everyone's... Are they just playing the stats then? And really changes the nature of playing the game. And so yeah, that's a really tough one for us.
Crystin Cox:
Yeah. That's a [inaudible 00:30:34]. We talk about this a lot when we talk, I've worked on a lot of RPGs, talk a lot about drop rates. And there's this constant debate, should we have the drop rates be public? And it's really challenging, because we know humans are terrible at math. We're really bad at understanding math. It's not like, you know if you put it out there, people are going to look at it, and they're not going to come to the same conclusions. Multiple people are going to look and they come to different conclusions. They're not necessarily going to be representative of reality. And so you do end up in this situation, where you're sort of in a governance situation. What information should I put out there? I want to be transparent, but maybe there's too much transparency.
Jason Coleman:
Yeah. I mean, one that keeps coming up for weird reasons is, is random really random? And players, I mean, they really go and mine the data, and try to really understand it. And I think we had it on... May have been Civilization II or something, where people were convinced it wasn't random. And I think Brian Reynolds went and actually posted the code for a random number generator, so everyone could check it out. And we've had that come up in elder scrolls as well. It's [crosstalk 00:31:47], yeah.
Crystin Cox:
It'll keep coming up, right? I mean, it's this classic problem that our brains are not wired to actually understand random. It is never going to seem random.
Jason Coleman:
Although I mean, one of the transitions that I'm amazed about, is how much players now talk the language of design. And in a legitimate, they really understand it. And so I mean, what we'll see is people will go and they'll do experiments, and they'll get a lot of data. It's not enough, and it's just their one account. But they will go and actually mine real data, and try to figure out what's going on.
Crystin Cox:
If I could just teach everyone in the world what's statistically significant, it would make my whole life a lot better.
James Gwertzman:
So I've got a question. You were talking about interpreting data from players and studying what changes to make. And one thing we saw, especially on smaller teams, and oftentimes in the data's maybe not so clear, or actually maybe even when the data's super clear, is sort of internal dissension, or a discussion what changes to make next.
James Gwertzman:
There's a sense of you have a limited bandwidth of how much work you can accomplish every week, and choosing how to use those precious resources, what changes to make, can often be really divisive for the team. Have you seen much of that? I mean, how do you resolve it, through the dictatorship model? Do you resolve it through the committee model? How do you decide what updates you'll make, and when?
Jason Coleman:
We're on a pretty clear cadence in terms of the content. So we have, again, quarterly, we're doing big releases, and then we have our monthly release. Our monthly releases are generally quality of life improvements, small features. There may be some balance changes. Mostly we do that by we trust the group that's responsible for that, so the card design team's responsible for... And they'll get feedback from all of us, and I'll get feedback, and maybe mine carries a little more weight sometimes, but mostly it's for them to decide. And then we see what the results are like. That's basically it.
Crystin Cox:
Cool.
James Gwertzman:
Yeah, I think this has been a really wonderful interview.
Crystin Cox:
Yeah, thank you very much for coming, this has been really interesting. I think one of our more, which has been nice, been one of our more technical interviews, because we've had a lot of product managers, and designers and producers on. So this has been really fascinating to get a little bit more nuts and boltsy, with pipelines and backend stuff.
James Gwertzman:
Yeah. I guess why don't we close with some prognostication, some predictions. Where do you see the technology of live games going over the next couple of years? I mean, what do you think about? What new tools do you see coming out on either the open source communities, or maybe the Cloud providers that you think are going to be making big differences for you and other developers?
Jason Coleman:
That's a tough one, because this one's so fresh. We started this a little over a year and a half ago, and then had to quickly make the whole game. And so at that time, we were evaluating what's the current state and sort of looking into the future. And I think we kind of used the pieces that we thought were where things were going. So I think mostly for us, it's being independent of a particular platform, in terms of whether it's AWS, or whatever we're actually going to put the hardware on, and being able to move about. But yeah, I think we're using most of where we thought things were going in the near term.
James Gwertzman:
Cool.
Crystin Cox:
Nice. Well, thank you very much for joining us.
Jason Coleman:
All right.
James Gwertzman:
Yep. And yeah, good luck with everything, and thanks again.
Jason Coleman:
Thanks guys.
Crystin Cox:
Thanks for listening to the Art Of LiveOps podcast.
James Gwertzman:
If you liked what you heard, remember to rate, review and subscribe, so others can find us.
Crystin Cox:
And visit playfab.com, for more information on solutions for all your LiveOps needs.
James Gwertzman:
Thanks for tuning in.