As the general manager for Destiny 2, Justin Truman is responsible for the overall success and health of the Destiny product, business, and team. He spends his days helping to empower Bungie’s teams to make Destiny 2 the best it can be, and to have the best possible working experience while doing so.
Since joining Bungie in 2010, Justin has influenced the evolution of the Destiny universe for a decade from a variety of roles throughout the studio. On the original Destiny he helped build the Destiny engine as an engineering lead, and on Destiny 2 he served both as a design director and then later as a production director.
Justin discusses the challenges of moving from a box released game to a LiveOps game; how Bungie organizes their teams to address live feedback and how to be more comfortable with transparency and less with always being right.
This episode is brought to you by Azure PlayFab: https://www.playfab.com
As the general manager for Destiny 2, Justin Truman is responsible for the overall success and health of the Destiny product, business, and team. He spends his days helping to empower Bungie’s teams to make Destiny 2 the best it can be, and to have the best possible working experience while doing so.
Since joining Bungie in 2010, Justin has influenced the evolution of the Destiny universe for a decade from a variety of roles throughout the studio. On the original Destiny he helped build the Destiny engine as an engineering lead, and on Destiny 2 he served both as a design director and then later as a production director.
Justin discusses the challenges of moving from a box released game to a LiveOps game; how Bungie organizes their teams to address live feedback and how to be more comfortable with transparency and less with always being right.
This episode is brought to you by Azure PlayFab: https://www.playfab.com
00;00;05;00 - 00;00;06;14
James Gwertzman
Hello. I'm James Gwertzman.
00;00;06;14 - 00;00;09;23
Crystin Cox
I'm Crystin Cox. Welcome to The Art of LiveOps podcast.
00;00;15;04 - 00;00;22;21
Crystin Cox
Hey, Crystin here. I'm on my own for this one, but I am joined today by Justin Truman.
00;00;22;21 - 00;01;02;25
James Gwertzman
I'm Justin Truman. I'm the general manager of "Destiny 2". I kind of holistically oversee and support the business, the team, the organization, the live service of the "Destiny 2" game. I've been in the game industry for just about 20 years. Started in engineering, spent most of my career in gameplay engineering or systems engineering, did a brief detour into design for a while, was a design director on some of the Destiny releases and "Destiny 2" when it launched and moved over into production for a bit and kind of built our current incarnation of our live service team, what we call our Vanguard team, and then more recently moved into a more holistic kind of business and team role.
00;01;02;25 - 00;01;27;06
Crystin Cox
And Justin has been at Bungie for quite a while of the journey pretty much the entire journey that Bungie's been on for "Destiny 1" and "Destiny 2". And he has a lot of great insights into the way that Bungie has transitioned to being a truly liveops focused team in all of the trials and tribulations and big success that they've had with the "Destiny" franchise.
00;01;27;06 - 00;01;54;19
Crystin Cox
So really interesting conversation, Justin has a lot of varied backgrounds. He's been an engineer, he's been a designer and now he's looking out for and sort of running the live operations for "Destiny 2". So let's dive into this really interesting conversation with Justin. So you've been with Bungie through pretty much this entire LiveOps journey.
00;01;54;26 - 00;02;04;16
Justin Truman
I have spent 12 years at Bungie and every one of those days working on "Destiny", which was longer than I expected to work on a franchise ever. Yeah.
00;02;04;16 - 00;02;30;21
Crystin Cox
That's awesome though. That's sort of the liveops dream, right? Ten years on one thing. So I'd love to hear you talk a little bit about that journey, right? Like a lot of teams...on this show, we've talked to a lot of teams who are very much in that. Like they come from a triple-A background. They have a lot of history as a studio before they sort of make their first true LiveOps title, and it's always an interesting journey for them.
00;02;30;21 - 00;02;36;23
Crystin Cox
It's a bit of a shift. So I love to hear you talk a little bit about what that's been like for Bungie.
00;02;36;29 - 00;03;08;00
Justin Truman
Yeah, I know it was interesting because like Bungie, when I joined, it was on the tail end. they were still working on "Halo Reach", and it was "Destiny" was like a small incubation team of about 15 people or something like that. And Bungie definitely already had very robust like multiplayer and live game operations experience. So like will we be able to launch a game and have the servers work on day one?
00;03;08;00 - 00;03;36;01
Justin Truman
Like all of that experience was already baked in. The stuff that wasn't there, that we all had to kind of figure out, was we were trying to take sensibilities from at the time our best analog was MMOs on the PC space. There were also some like mobile free to play games that had some of these live game sensibilities, but nothing really existed like that on in like the console or the big budget Triple-A action space.
00;03;36;13 - 00;04;07;06
Justin Truman
And so that was where it felt like we were kind of forging new ground of how do you take these ideas and then turn them into a mass market triple-A PlayStation or Xbox game was interesting. Like I remember when we were working on "Destiny 1" and we watched the launch of I distinctly remember "Diablo 3" launch and then one of the "SimCity" games launches, I can't remember "SimCity online"
00;04;07;06 - 00;04;08;07
Crystin Cox
oh yeah probably the reboot.
00;04;08;22 - 00;04;25;08
Justin Truman
Yeah, they launched with basically just like kind of standard online game service issues, but they were games that players expected. Like I didn't choose to play with other players. Why am I suddenly being disconnected from my like normal go out and punch monsters or my normal, like, build my roads?
00;04;25;08 - 00;04;25;17
Crystin Cox
Right.
00;04;25;17 - 00;04;37;09
Justin Truman
And so we were very scared about that when we were working on "Destiny 1", which is like, hey, we want to make a game that kind of feels like "Halo" but if we force you, if we even just force you to be online, is that going to be something console players will accept?
00;04;37;16 - 00;05;01;00
Crystin Cox
Yeah, I remember hearing, you know, some of the early discussion coming from I was in the MMO space. I started hearing in the early discussion about what Bungie was maybe up to. And I was always a little surprised at how much fun you did lean into and maybe in a way that the community maybe got you guys in a little bit of trouble.
00;05;01;00 - 00;05;22;04
Crystin Cox
At some point, like really being like we're trying to kind of make something like an MMO, like we're really looking more at MMO than we maybe are looking at some of these other things. Was that really the prevailing thought when you guys were heading up to shipping "Destiny 1", that you were like, We're kind of feel like we're making an MMO?
00;05;22;04 - 00;05;49;22
Justin Truman
There wasn't what folks were saying. Like a thing that I think is part of Bungie's success is the design culture of Bungie is really good at. It's like those like experimental, like more chemistry based chefs where they're not talking about like, oh, I'm making this type of traditional Italian dish. Instead they're breaking it down to the component ingredients and talking about the ingredients and the impact on your palette and stuff like that.
00;05;49;26 - 00;06;16;00
Justin Truman
Oh, my gosh. That sounds horribly pretentious, but just go with it. But what I saw and so this was me as an engineer. This is not me taking credit for what I saw as an engineer working with our design team was a lot of kind of playing the forefront of action games, playing the forefront of MMOs, playing the forefront of like the kind of like nascent live service on mobile and then pulling individual components.
00;06;16;00 - 00;06;37;29
Justin Truman
So there would be something like, hey, MMOs do this really cool thing. Where there's like actually impact with calendar time and like things will happen on a weekly cadence or like Christmas things will happen at Christmas and we want to take that. And so we're going to grab that ingredient and put it in our dish. And so we were intentionally grabbing ingredients from those spaces, but the dish we were making, we weren't looking
00;06;38;01 - 00;06;54;22
Justin Truman
at that and be like, Oh, we're cooking an MMO. Because that wasn't really how like and in a way that like it would have been easier for us to get to some of the answers we eventually got to if we were more willing to label what we were doing. But I think it also freed us up to try some things that were kind of different than what had been done.
00;06;55;00 - 00;07;15;23
Crystin Cox
Yeah. No, it's so fascinating. "Destiny" is fascinating. And it's an amazing game. And I'm huge fan of the original game as well. And "Destiny 2" is obviously a big evolution of it. It is that it does sit at such an interesting intersection of taking a lot of different influences and a lot of from a lot of different places.
00;07;15;23 - 00;07;40;17
Crystin Cox
I am curious to hear a little bit more about that journey, because when I look at "Destiny 2" now, I do see so much more. I get I agree with you. I don't know that I'd say like it's an MMO but I would say it is certainly as robust as an MMO. Like it certainly is something that you're sort of logging in every day to check stuff out and you're having a community and there's a lot of different activities going on.
00;07;41;02 - 00;08;11;16
Justin Truman
Yeah. And today we talk about it internally. We're more willing to use labels, so we'll call it like an action MMO because I feel like that space is also been expanded out a little bit. And you can talk about the three or four players in that space and stuff like that. But, but like to give some examples. So "Destiny", when "Destiny 1" originally launched, we had the idea of just a place, a world that would last for years and years, but a bunch of stuff was still up in the air.
00;08;11;16 - 00;08;31;21
Justin Truman
Like our initial live support plan was releasing DLC content packs like that was kind of all we understood. And then there was an open debate leading into 2016 2017 about do you sequel the game or not. And we ultimately sequelled it and then afterwards we're like, oh, I don't think we do that again.
00;08;33;01 - 00;08;50;21
Justin Truman
We potentially do something as big as a "Destiny 2" launch just within the base game. But, that like the concept of seasons was four or five years down the horizon. We didn't have any microtransactions or any of that kind of like free-to... the game wasn't free to play until just a couple of years ago.
00;08;51;00 - 00;08;58;07
Justin Truman
So we like, we kind of I feel like we started in the shallow end of the pool and then have been like slowly walking into the deep end over the years.
00;08;58;16 - 00;09;23;22
Crystin Cox
Yeah. And this is something that when I work with teams now who are a lot of the teams I work with now are traditional triple-A teams who are moving into more live services space. One of the things that can be tough is this understanding of there isn't going to be this moment of launch where you're at the highest point that you're ever going to be at, and then there's a long, slow tail down.
00;09;23;22 - 00;09;57;05
Crystin Cox
It's actually more you're going to be at this beginning point and then hopefully you're growing over time. But that's not an idea that is even now, all that well accepted in the triple-A space What was that like for you guys? Because you really I mean, Bungie such a huge name in the triple-A space, expectations not only high, but very specific, but you guys did make that transition to a game that I'd say now, looking back, when I think of "Destiny," I think of something that started from a great place but then grew to more, not something that started big and then long tail.
00;09;57;26 - 00;10;27;13
Justin Truman
Yeah. And it it's funny because it's somehow managed to kind of be both like I think we launched with that kind of insane standard-like, tentpole Activision budget, live action trailers with like Giancarlo Esposito in them and stuff like that and all that jazz. But then you had that standard, like when you push things up that high at launch, you get the standard like entropic decay.
00;10;27;19 - 00;10;51;19
Justin Truman
And then we kind of like leveled out and then started on the growth phase after that. And we there was definitely some big learnings there. And I feel like now that we've become our own independent like developer and publisher, we're like fully in the. Yeah, I no, it is not the standard splashy box producty release with the curve, the trends down.
00;10;51;26 - 00;11;09;17
Justin Truman
It is the launch and develop and like nourish and watch it grow. And we're seeing that success. And so even when we talk about like future games, the Bungie you might make, it's like we don't ask the question like what is the game like at launch? Like even at the very beginning of the work, it's like what's the game like two years after launch?
00;11;09;23 - 00;11;17;21
Justin Truman
Whereas that's actually I think the interesting question about the health of a live service, not did you survive launch day or make it to launch day or whatever. [ad break music]
00;11;23;04 - 00;11;44;10
Crystin Cox
[ad break] The Art of LiveOps is presented by Azure PlayFab. PlayFab is a full featured backend solution that powers LiveOps for games of all sizes. Whether you need multiplayer services, analytics or LiveOps tools, PlayFab can help you reach your retention engagement and monetization goals. Learn more and you can get started for free at PlayFab.com
00;11;45;07 - 00;11;58;14
Crystin Cox
That's PlayFab.com P-L-A-Y-F-A-B dot com and you can get started for free. [ad break music]
00;11;58;14 - 00;12;18;18
Crystin Cox
You're listening to The Art of LiveOps podcast, and today we are joined by Justin Truman who is a LiveOps manager for "Destiny 2". And we are talking about the really interesting journey that Bungie has been on across the entire Destiny franchise.
00;12;18;25 - 00;12;55;16
Crystin Cox
He's given us lots of interesting stories about the way that they've grown and changed over the years. So let's dive back in and keep digging into this story of "Destiny" and "Destiny 2." We've talked to a couple teams on this show about that challenge of when you launch, you go through this period of stabilization. You know, you make all these plans like, yeah, we're going to launch if we're going to hit this cadence, you know, then players happen and then like all kinds of stuff goes on.
00;12;56;03 - 00;13;05;09
Crystin Cox
And for you guys, how long did that take? How long do you think it took before "Destiny" was a real live game, LiveOps game.
00;13;05;22 - 00;13;44;19
Justin Truman
So. So "Destiny 1", we launched very much just like a box product that had a bunch of MMO ingredients we launched a series of DLCs that were like increasing the amount of MMO in it, but they were like boxes that you could buy of MMO stuff. At the very end of "Destiny 1", we had this kind of small, scrappy live team when a bunch of folks had moved on to "Destiny 2" that actually got really far on that journey and they like introduced the first free live events, introduce some like really player friendly micro-transactions had these like cool things going on and regular updates.
00;13;44;23 - 00;14;05;27
Justin Truman
And then we kind of like forgot that because they were, they were over here and we were like, we're working on "Destiny 2" and then had to relearn it, but we relearned it. We had to relearn it very fast when "Destiny 2" launched because "Destiny 2" launch was pretty rocky and the game was we made some big pivots that weren't what our players wanted.
00;14;06;06 - 00;14;26;14
Justin Truman
And so we had to react at a live service speed before the like plane hit the side of the mountain. We had to, like, pull up on the stick. And so it forced us to like develop a bunch of those muscles in the span of like eight or nine months. And then I feel like by the time "Forsaken" launched one year after "Destiny 2" launched, at that point, we were like full fledged live service.
00;14;26;14 - 00;14;51;11
Justin Truman
Like after "Forsaken", we were like releasing seasons. We had a bunch of these, we were releasing four or five patches after every major content release before we were trying like we don't want to like good work, doesn't need a patch. Was kind of like the mindset leading into "Destiny 1" and like all of that mindset shift, I think really solidified in the foraging fire after "Destiny 2" launched.
00;14;51;20 - 00;15;13;02
Crystin Cox
Right. It is really interesting to hear you talk about that story because I will say like sort of from the outside, especially coming from a LiveOps background, it was interesting to see "Destiny 1" go on that journey and as you said, like really gain big ground and so sort of at the end of it's cycle it was like yes we're hitting on these cylinders.
00;15;13;02 - 00;15;30;23
Crystin Cox
We're doing... This is the kind of thing I kind of expect from a live game. It's like keeping the player base happy. But then as you said, this is the challenge of a team had broken off much earlier to go make "Destiny 2". They're not actually doing that work and then having to kind of relearn that work.
00;15;31;19 - 00;15;51;14
Crystin Cox
And I'm kind of leads into another question I have about I think some of that lag was, as you said, there was a group of people still playing "Destiny 1." This was a very dedicated player base they had been iterated on. And then "Destiny 2" launched, it felt a little bit like, well, this is feedback from three years ago, really.
00;15;51;14 - 00;16;02;05
Crystin Cox
This is addressing feedback from three years ago instead of where we are today. How has your guys's relationship with feedback changed? Like how does your relationship to that community iterative cycle changed?
00;16;02;15 - 00;16;22;04
Justin Truman
Yeah, no. And there's definitely stuff about feedback that I get that I'll get to in a second. But I think one of the most powerful changes we made was actually in the org design. So and I love org design. I'm one of the few nerds who have just like I love thinking about talking about how our teams organized and stuff like that.
00;16;22;04 - 00;16;41;27
Justin Truman
And we had kind of the standard game development project based model until a little while after "Destiny 2" launched. And so what that meant was you would pick a team and you'd be like, Hey, so we've got the team that's working on the "Taken King" and that's the "Taken King" project team, and then we've got another team that can have an extra year.
00;16;42;04 - 00;16;44;29
Justin Truman
So in parallel, they're going to be building "Destiny 2",
00;16;44;29 - 00;16;45;08
Crystin Cox
Right.
00;16;45;08 - 00;16;56;14
Justin Truman
And that gave you the like amount of time you needed to develop these larger beats, but it meant that you were pretty much guaranteed to give your players whiplash,
00;16;56;14 - 00;16;56;23
Crystin Cox
Right.
00;16;56;23 - 00;17;05;18
Justin Truman
Because whatever came out was a team that had been in a different room not learning the lessons of the immediate prior releases.
00;17;05;18 - 00;17;05;27
Crystin Cox
Right.
00;17;05;27 - 00;17;30;27
Justin Truman
And so we had to and it's tough because we're like, we're a pretty big team, like hundreds of people. And so we have to have a fairly complex structure, but we had to figure out how do you have that structure in a way that doesn't silo off the releases and that allows like if someone is responsible for the economy or someone who's responsible for the tuning of the combatants, they're monitoring that and shepherding that across all the releases.
00;17;31;08 - 00;17;44;13
Justin Truman
So you're feeling a sense of like continuous improvement rather than, oh, the like Red team loves to make things fast. And then the blue team wants to make things like cover base shooting and all of a sudden we're just like flopping back and forth between these different sensibilities.
00;17;45;09 - 00;18;03;00
Crystin Cox
Yeah. And that's, that's tough. But I'd love to hear more about it, you know? What structure did you guys land on? I am so curious to hear, does it look a lot like sort of a standard MMO team? Or does it look more, more like smaller, interconnected pods?
00;18;03;28 - 00;18;28;10
Justin Truman
Yeah. So we have at the team level we went through kind of like the typical game dev agile transformation of like interconnected pods of experience base, like interdisciplinary teams and stuff like that at the large scale. I think it has a lot of similarity to MMOs but were a fair bit larger than most MMO teams.
00;18;28;10 - 00;19;01;02
Justin Truman
And so you have some there's some like challenges of scale that don't manifest at the like 150 but then start to manifest at the like 400 or whatever And so we have to have some of these super structures we call these development groups. So there's like the Vanguard Development Group that does our seasons and our live services. We have a systems development group, we have a what we call scenario development group that does the like big kind of bombastic stuff that tends to be in expansions like brand new raids or destinations or campaigns and stuff like that.
00;19;02;13 - 00;19;21;12
Justin Truman
And it's like sort of siloed off per release so like the Vanguard team is generally doing the seasons and the scenario teams generally doing the expansions. But there is like an important philosophical difference that they own types of work. And so if we want a thing like a holiday event to be in one of our expansions, Vanguard is going to help like that.
00;19;21;18 - 00;19;42;27
Justin Truman
If we if we want a raid, like when we rereleased "Vault of Glass" to occur alongside a season scenario helps with that. And so what you get is these interdisciplinary teams own a type of experience across every release that might release that type of experience. And so that means each type of experience is just steadily getting better.
00;19;43;11 - 00;19;58;01
Justin Truman
And once we structured like that, that's when we started getting the wins of like every year we were 20% better at most of the stuff we were doing in the previous year versus the like three steps forward, two steps back that we've done for years.
00;19;58;24 - 00;20;11;08
Crystin Cox
And yeah, circling back around to feedback, what does that feedback pipeline look like? Right because like scale aside, even small teams struggle with the community feedback pipeline.
00;20;11;17 - 00;20;46;06
Justin Truman
Yeah. Oh, man. There's so much there like the simplest like overarching lesson for me was that it's more important to be transparent and to be active in the conversation than it is to be right. And when you think like for me, more traditional communication, especially when you're building a triple-A game in secret and then releasing it all at once and stuff like that, you're very careful about always being right.
00;20;46;11 - 00;20;59;28
Justin Truman
Like you're not going to go out and talk about a feature that may or may not end up in the game. You're not going to, like show off something that could radically change. In fact, like, you'll get like excoriated by players and stuff like this is what they showed at E3, like this is what they gave us.
00;21;00;11 - 00;21;21;29
Justin Truman
And so you hold everything until you know that what you're saying is always going to be correct. And we started with that mindset. We kind of kept it through "Destiny 1" and then had it at the very beginning of the challenges of "Destiny 2" and it just didn't work, like in the modern Internet landscape, especially for a live service game.
00;21;22;19 - 00;21;51;13
Justin Truman
Players want to be like co-creators of the experience. They view themselves as collaborators and they want to have a direct two way communication and we found like it is much better to just tell them what they're interested in knowing about and tell them stuff that will help guide the conversation and will provide rapid reactivity, even if you have to then 3 months later be like, Oh yeah, that thing we said, we're actually not releasing it next year.
00;21;51;15 - 00;22;12;06
Justin Truman
We've like changed our minds. And so we are much more comfortable making tentative or future facing statements and then apologizing for them or undoing them or changing them. And that was a that was a really hard thing for a lot of people to get used to because you have like a decade of experience before that of like never do that.
00;22;12;10 - 00;22;13;06
Crystin Cox
Right.
00;22;13;06 - 00;22;18;06
Justin Truman
That is when you've gone when your PR person is like pulling you away from the microphone because you've gone off the edge.
00;22;18;14 - 00;22;37;23
Crystin Cox
Right. But that is so important, right? Because building a liveops game, it's an iterative process that happens with your community. And you have to be comfortable being wrong because no matter what, you're going to be wrong. Sometimes right. And I agree, like a lot of times it's better to get comfortable with that and to get your community used to.
00;22;38;00 - 00;22;55;20
Crystin Cox
Hey, we're going to be wrong sometimes you have to hear apologies from us. Sometimes we're going to change direction. Sometimes we're going to talk about something and then it won't show up for five years. Like that's just a normal part of this. But you know, the community, as you said, really wants to be involved in that way.
00;22;55;20 - 00;23;02;23
Crystin Cox
And you can't have it both ways. You can't be both perfectly correct and polished and also incredibly responsive and iterative.
00;23;03;06 - 00;23;25;10
Justin Truman
Yeah. And to me, what's interesting is that I think this lesson, while it's most strongly felt in live service games right now, like I think is just happening with people's relationship to brands and entertainment in general. And so you could be like you could be working on a single game that's only ever going to release once and maybe you're doing some early access or something like that.
00;23;25;10 - 00;23;36;02
Justin Truman
And I think you still have to have that kind of the player expectation is baked in for you to have that rapid back and forth, relatively transparent relationship.
00;23;36;07 - 00;23;36;20
Crystin Cox
Yeah.
00;23;36;20 - 00;23;58;09
Justin Truman
and I think the to me, the strength that we get from that besides just how do we make our players happy? Like that's ultimately why we're doing any of this, right? Yeah. Is that it's kind of recognizing that if you're trying to make entertainment, like if you're measuring your success, as did a bunch of the people that I wanted to like my thing, like my thing.
00;23;58;09 - 00;24;18;17
Justin Truman
It's not just like the auteur like I think it was it was my vision. And so I don't care if no one likes it, but if you're actually trying to get that, that positive response, being able to have that iterative conversation with your audience, we found like avoids so many landmines. There's so many things that, like, inside the studio can feel like a great idea.
00;24;18;18 - 00;24;35;23
Justin Truman
Everybody's high fiving like, we've got this awesome idea, and then we hold like a community summit and bring out some of our players and we show them the idea. And immediately everyone in the room is like this terrible never do this, and we're like, What? And it's just it can be it can be much more nuanced feedback as well.
00;24;35;23 - 00;24;44;12
Justin Truman
But in general, you just like you kind of relish that iterative process because it's another way to make your game a little bit better month over month.
00;24;44;22 - 00;25;16;02
Crystin Cox
Right? And, you know, it's not just, you know, direct community feedback, right? Because obviously that would leave you listening to only people who are willing to go post things on your forums or send you messages. There's also data, and I'm always interested to hear how a team's relationship with data evolves over time. You know, I came from a design background and I think design and data should be best friends, but that's not always the way that it works for everybody.
00;25;16;02 - 00;25;24;03
Crystin Cox
So I'm really curious to hear, you know, what do you guys use data for? How has your relationship data changed over time?
00;25;24;19 - 00;25;44;02
Justin Truman
Yeah, that's been that's been another interesting one with really big transformations over the years because, yeah. So like early on, I think we got receptive to sentiment very early. But sentiment like to your point it's like it's the vocal sentiment. And so there's a certain type of player that's going to be really loud online about their feelings.
00;25;44;09 - 00;26;05;25
Justin Truman
And then there's a ton of people who maybe either loving the game or leaving in droves, but they're doing it quietly. And so you like so you miss that if you're just paying attention to the community as it like manifests vocally, We also had like we built up a bunch of analytics for, for our live game, but then we had that hurdle that
00;26;05;25 - 00;26;48;18
Justin Truman
you were describing in design where it's hard for some people to still feel creative ownership and willingness to experiment and to do radical things based purely on their like, artistic guts when we're then also providing them with like reams of data. And so some folks felt that was like directly intrusive, like, no, this is like polluting. This is a bunch of suits telling me how to like make my experience but I think overall the team pretty quickly started to realize that we're not going to use data like we talked about data inform design instead of data driven design.
00;26;48;18 - 00;27;05;19
Justin Truman
Like we're not going to use data to decide what we should do. Because I think one of the pitfalls that can create is that you aren't able to do things that you don't have data for, like you're not able to do things that you haven't done before if you're only willing to do it, if you have data to support it.
00;27;05;26 - 00;27;25;19
Justin Truman
And instead we're totally willing to like have our creators and our artists and our designers come up with something that's totally different than we've ever done before, goes against the grain of what our data says. And then what we want to do is just we're going to we're investing the experience, we're going to trust you. We now want to make some hypotheses about what we think this will do.
00;27;25;20 - 00;27;57;28
Justin Truman
Like we think that this will be an experience that brings a bunch of players in, gets them really happy, but then they probably leave the experience sooner than other ones because it's intended to be bursty. So that's like a play experience design that we can now turn into a hypothesis and then we can just test the data and we found that like for example, we know a lot more about our seasons right now than our expansions because you can isolate the variables better.
00;27;58;07 - 00;28;27;07
Justin Truman
Like when you release, we do these like secret exotic quests sometimes where we'll release like a secret of the game and there's like a cool like derelict shift you can go to and then play a standalone mission and then get an exotic and release things like that for free every now and then to surprise players. But because every one of those is released completely on their own, we have this ton of like this awesome dataset of isolated variables of like, oh, we tried this one and it was like a really deep secret that people had to find on Reddit.
00;28;27;07 - 00;28;46;01
Justin Truman
And we had this one, and it was at the top level of the director. And we know exactly how that influenced different performance metrics. We released an expansion and we're firing all our guns at the same time. We're releasing 25, and we're going to keep doing that because we like, we want that impact. But that means it's much harder to tell.
00;28;46;08 - 00;28;49;05
Justin Truman
Oh, we did this one cool experimental thing in the campaign.
00;28;49;16 - 00;28;49;27
Crystin Cox
Right?
00;28;49;28 - 00;28;55;29
Justin Truman
Is that what caused this number to go up, or was it the 20 other things that we did on the same day?
00;28;56;08 - 00;29;19;13
Crystin Cox
Right. And with a game as complicated as "Destiny," that's always going to be somewhat true, right? But I think that that's what you're describing to you is such a common experience, right? Like especially for teams that maybe haven't worked with data as much data is... I this a lot to teams, data is not predictive. It only tells you what has happened.
00;29;19;14 - 00;29;19;26
Justin Truman
Yeah, I like that.
00;29;19;26 - 00;29;39;09
Crystin Cox
Right. So the idea that like you could make design decisions with data is not quite true anyway. They can't really tell you what's going to happen. You can do a bunch of analysis and make forecasts and predictions, but it's ultimately looking backwards. That's all data can really do for you. So it can be hard to know what to do with it.
00;29;39;09 - 00;29;54;01
Crystin Cox
I work with a lot of teams now who are like, we're instrumenting everything and it's like, awesome. Like, what are you going to do with all that data? And a lot of times I think teams are like, I don't know, I'm just supposed to have it right. Like, I'm supposed to have all this data.
00;29;54;23 - 00;30;19;02
Justin Truman
Yeah. And so we I love that how it's entirely backward facing. I completely agree. Because where we've gotten a lot of our value is like, so. So when you release like a "Halo 3" or something back in the day, your measurements of success were basically: I sit at home and I play it myself. Do I think it's good? Because like I trust my own judgment.
00;30;19;22 - 00;30;41;23
Justin Truman
I look at some aggregate like Metacritic or something, and I look at what people are saying on social media, if that existed yet. And then I look at how well did it sell. And that's basically the extent of your measurements of success and the thing and people still do that. Like people would look at that data like religiously back then, even if they were not like, I don't want to get a bunch of metrics right.
00;30;41;26 - 00;31;09;23
Justin Truman
And so what we've done since then is we've just had a much more robust, much more robust set of data that we can use to measure our success. So when we release any product, an expansion, a season or holiday event, at some point after the window of that release, we'll do a full teardown and kind of like performance review where our team will spend like an hour to 2 hours kind of going through a presentation of like across every axis of success.
00;31;09;23 - 00;31;35;22
Justin Truman
Like how did this how did this respond to engagement? How did this attract players who weren't currently playing the game? How did people say in like word clouds that they felt about the game? What happened after they reached certain completion points? Did they stick around or did they leave? And we found those types of like backward facing data analyses we've gotten everybody on our creative teams and our leadership teams to really, really value them.
00;31;35;22 - 00;31;51;15
Justin Truman
And so we get we don't just present those to like the like suits or whatever. It's like our experience owners and our creative leads and our design leads who are all sitting in the room drinking up all that data and then using it as kind of inputs into whatever wacky idea they want to try
00;31;51;15 - 00;31;52;29
Justin Truman
in the next release.
00;31;53;14 - 00;32;12;12
Crystin Cox
Yeah, I mean, it's so funny because design sometimes will be like data kind of scary. But the truth is designers are huge data nerds. Like, we want to know, of course, do you want to know what people said about yours? Of course I want to know what people said about my, my design. Do you want to know, like, how many people got to this step?
00;32;12;12 - 00;32;18;01
Crystin Cox
Of course I want that. That's just interesting and see if you can sort of present it that way. You get a lot more buy in there.
00;32;18;10 - 00;32;31;00
Justin Truman
Yeah. And then it's also created a vocabulary, not that we're using the data to drive the decisions, but we're using the vocabulary that the data is given us to you talk about the decisions that we're making.
00;32;31;09 - 00;32;51;07
Crystin Cox
Right? And I think a lot of it comes down to just defining what good is. You said earlier where you're talking about the traditional boxed product cycle. You know, so much of design is taste like you want to have taste and you want your desires to have great taste. It's part of why they're professional designers, right? And you want their judgment to come into play.
00;32;51;17 - 00;33;16;14
Crystin Cox
But for me, liveops, you know, really a lot of it because a lot of liveops games are so broad in their player base is building a skill, not only having my own taste as a designer but to be getting good at delivering good for other players who maybe don't have my taste. And I do think, you know, "Destiny" got a huge, broad audience.
00;33;16;21 - 00;33;41;07
Crystin Cox
Early in my career. I worked on kids' MMOs, which was very helpful. I'm never going to be the audience there, so like, how do I learn how to judge whether or not something is good based on something other than I like it or I don't like it? And it sounds like you guys have gotten a little more sophisticated in being like, We can break it up and talk about like how it's making different kinds of players happy or whether it's succeeding in these different ways.
00;33;41;27 - 00;34;11;24
Justin Truman
Yeah. In fact, there's a very tangible example of this, which is that there used to be our company motto, like In Stone on the front of our building. That was we make games we want to play. And we actually took that down and got rid of that motto. And instead it's we create worlds that inspire friendship which besides, I think, getting better to the spirit of like, what are the types of games we're trying to create?
00;34;12;00 - 00;34;23;09
Justin Truman
It was a stepping away of like success is not we think it's great, because there's a ton of things that we might think it's great. And that doesn't actually mean that the world is going to think it's great.
00;34;23;09 - 00;34;23;17
Crystin Cox
Right.
00;34;23;17 - 00;34;33;12
Justin Truman
And so it yeah, like that got into our bones and our DNA down to the level of the words etched in front of the entrance of the building.
00;34;33;16 - 00;34;53;20
Crystin Cox
Nice. So we're just we're starting to actually get up on time here. This has been a great conversation, but there's one question that we ask everybody who is on this show. So we'd love to hear you give a personal experience of a LiveOps disaster.
00;34;53;25 - 00;34;55;22
Justin Truman
Oh, man, there's so many to choose from.
00;34;56;24 - 00;35;11;28
Crystin Cox
That's the beauty of it. Everyone says that because that's sort of why we do it is, you know what? It's good perspective. If you're listening to this and you're a veteran you know it. But if you're just starting out, everyone's got a disaster. Everyone's got a lot of disasters under their belt.
00;35;12;15 - 00;35;30;14
Justin Truman
I will say while I'm coming through my filing cabinet, like one of the things we have really learned over the last five years or so is really actually defining what hits the bar of a crisis. Like when you're doing triage early on, it can feel like almost anything is a crisis. Oh, there's this terrible player experience.
00;35;30;14 - 00;35;59;17
Justin Truman
And then you realize like, no, that's not that's urgent. That's important. But, let me tell you what a crisis is, right? So here's one. So we had a moment I think this was a couple of years ago, probably in 2019, I think in 2019 where we released a patch and discovered when the patch went live that it was deleting player data off of our investment servers.
00;35;59;29 - 00;36;06;01
Justin Truman
That's a that's a big no no. You're running a game for like five years and people have accumulated all sorts of things.
00;36;06;01 - 00;36;06;11
Crystin Cox
Yeah.
00;36;06;23 - 00;36;36;09
Justin Truman
You cannot ever break that contract. And we discovered it relatively quickly. So within a couple of hours we realized, oh God, people are getting stuff deleted from their accounts. We pulled the giant like emergency lever, shut everything down and it took us, it took us like 12 hours or so to, to find and fix the issue. So the game was down suddenly for 12 hours, which is pretty rare for us.
00;36;36;09 - 00;36;37;06
Crystin Cox
That's intense.
00;36;37;06 - 00;36;55;25
Justin Truman
Yeah, it was, it was definitely intense during a weekly reset when people really want to log in and bang out their weekly stuff and it also meant that anybody who had been playing during that two hour window, we had to revert their progress because it was like we no longer trusted the data. Yeah. So people still lost some progress. we did...
00;36;55;25 - 00;37;14;10
Justin Truman
Like that was very bad. That was the first time that it ever happened in the live game. We did exhaustive post-mortems and like technical investigation of like, how did this happen? What are the, what are the ways in which our technical APIs can even allow this to happen? How do we make sure this can never possibly happen again?
00;37;14;10 - 00;37;24;20
Justin Truman
And did a ton of diligence deployed multiple like safeguard patches, all the standard stuff. Then about a month later, we deployed another patch and it happened again.
00;37;25;03 - 00;37;26;04
Crystin Cox
Oh my gosh.
00;37;26;10 - 00;37;37;17
Justin Truman
Yeah. And so that that second one was just like kind of staring into the abyss of like, oh my God, like, how, how could this possibly happen a second time?
00;37;37;17 - 00;37;38;05
Crystin Cox
Yeah.
00;37;38;19 - 00;38;05;09
Justin Truman
And it wasn't anyone's fault. Like, nobody made a mistake. It was like, if you ever watch that, did you ever watch like Chernobyl? Where they, like, go through this is what it takes for a nuclear reactor to break down and like 17 independent accidental things happened to all coincide and we had that weird like basically just the right set of lock picks happen to be in the system to subvert all of the safeguards we had made and revert us back to that old buggy state.
00;38;05;13 - 00;38;05;24
Crystin Cox
Right.
00;38;07;04 - 00;38;27;07
Justin Truman
Fortunately, we were able to fix it a little faster this time because we're like the team in safety spent 2 hours fixing it was like, Okay, we know what to do, so we could do it in like 6 hours. But we also just like felt terrible because we're like, we do not want to be a game that every now and then we accidentally start deleting people's progress.
00;38;27;08 - 00;38;27;21
Crystin Cox
Yeah.
00;38;28;09 - 00;38;48;07
Justin Truman
And so and so this goes back to that, that communication thing about being transparent rather than being right. What we did was we did a second postmortem, but we then posted it on our blog. Like we basically used on Bungie net. We were like, hey, you're probably wondering why the server went down twice and we had to revert progress.
00;38;48;07 - 00;39;14;22
Justin Truman
If you logged in during this narrow window, we're actually going to go through in detail what happened, how it could have happened, how it could have happened twice. And it's the sort of thing that like six years ago me would have never done or never thought would be like, there's no way it's good for you to just like open yourself up on the operating table and expose all of the ways in which bugs manifest.
00;39;14;22 - 00;39;30;19
Justin Truman
But what we actually saw, which I think present day me can predict a little better is like you had a bunch of people who are like their day job is they're an IT professional or they work on some other form of a live service and they're weighing in in our community because they play "Destiny" as a hobby and they're like oh yeah.
00;39;31;00 - 00;40;01;24
Justin Truman
This is reminds me a lot. Let me tell you about the thing that happened at my job and it became relatable and made us like ultimately became a positive sentiment left when we released that rather than a negative one. And it was yeah to me being able like the postmortem process but then being willing to just like own that publicly was something that I think we could only do after the many years of journey of understanding of what good is that got us there now.
00;40;01;24 - 00;40;21;23
Crystin Cox
But I will say rollback is one of the scariest things in live games. It's one of those things that you prepare for. You hope you don't have to do it, but you and there's always a question in your mind like, would we be able to do it if we had to? And you guys did do it, like you guys got two successful rollbacks.
00;40;21;24 - 00;40;23;24
Crystin Cox
So like I will say, congratulations there.
00;40;24;02 - 00;40;30;21
Justin Truman
I remember sitting in the war room where we said, okay, we have this operation that will roll back all the data.
00;40;30;27 - 00;40;31;06
Crystin Cox
Yeah.
00;40;31;15 - 00;40;50;22
Justin Truman
We've literally never performed it before in the live game, and it's fully destructive. So there's like a small chance that we're gonna press this rollback button and find out 4 hours later after all the servers are updated, that we ruined everything for everyone forever. Yeah. And we're like, Okay, well, let's just hold our noses and press the button and hope it works.
00;40;50;22 - 00;41;09;27
Crystin Cox
And that's exactly the scary part. The last time I did a rollback in a live game, it took long, not the rollback was actually very fast. The thing that was long was because of exactly we're talking about is we had to make a full, complete backup of everything just in case. What if we perform this operation and it completely destroy?
00;41;09;28 - 00;41;27;08
Crystin Cox
We have to be able to like to do a full, full back up stored off somewhere. And that took quite a while to build. So it's fascinating what has to happen in there. It's one of those funny Catch 22s. You prepare for it, but you can't really test it and then you kind of hope you never have to
00;41;27;09 - 00;41;28;23
Crystin Cox
And then someday you do.
00;41;28;23 - 00;41;36;00
Justin Truman
Yeah, and we've gone years with this functionality and never having to test it. And then we did. And then we did again immediately afterwards.
00;41;36;09 - 00;41;49;08
Crystin Cox
Yeah. Now that's what's really nuts. It's funny that it took half the time. Like we don't want to get good at this but it's nice that it's taking less time. Oh man. Well, thank you so much for coming out and doing this has been super interesting
00;41;49;08 - 00;41;50;03
Justin Truman
Yeah, thank you Crystin. This was awesome.
00;41;53;17 - 00;41;56;02
Crystin Cox
Thanks for listening to The Art of LiveOps podcast
00;41;56;02 - 00;42;01;10
James Gwertzman
if you liked what you heard, remember to rate, review, and subscribe so others can find us
00;42;01;10 - 00;42;06;08
Crystin Cox
and visit PlayFab.com for more information on solutions for all your LiveOps needs.
00;42;06;08 - 00;42;07;04
James Gwertzman
Thanks for tuning in.