Turn 10: Jennifer Lane

Join James and Crystin as they talk with Jennifer Lane, a design director at Turn 10.  Hear about her experiences working on the console focused franchise, Forza, and how she helped the team make the transition from traditional boxed products to live games. 

Transcript

Copy Transcript Copied

James Gwertzman (00:05):

Hello, I'm James Gwertzman.

Crystin Cox (00:06):

I'm Crystin Cox. Welcome to the Art of LiveOps podcast.

James Gwertzman (00:15):

Hey, Crystin, today's guest is very close to home.

Crystin Cox (00:17):

Yes. Today, we have Jennifer Lane. She is a Design Director at Turn 10, which is one of the Xbox Studios.

Jennifer Lane (00:26):

I've been in gaming for around 18 years now. It's a second career for me. I used to be a molecular biologist research scientist and wound up turning my passion into gaming, and here I've been.

James Gwertzman (00:43):

and making Forza for a long time, I think.

Crystin Cox (00:45):

Yeah, they've been making Forza for a while, and Jennifer, I think, joined them three or four years ago in order to help them make this transition from really traditional box product to something a little bit more live.

James Gwertzman (00:57):

I think, if I'm not mistaken, this maybe the first guest we've had for a studio focusing almost exclusively on console games.

Crystin Cox (01:03):

Yes.

James Gwertzman (01:03):

First with mobile or PC or MMOs.

Crystin Cox (01:05):

Uh-huh (affirmative). I think famously they've been pushing into the PC space quite a bit recently and they're putting a lot of focus in PC but really, Forza is a console franchise.

James Gwertzman (01:16):

Yeah, and a big one for Microsoft, so it's exciting to talk to her and talk about how live ops and live content, because you said about content and really driving a lot of their live events.

Crystin Cox (01:25):

Yes. Let's dive into it.

James Gwertzman (01:27):

Awesome.

Jennifer Lane (01:33):

I've been really fortunate that I've had a lot of different roles along the way and a lot of unusual roles that is now becoming something that a lot of people are doing because I've been so focused on player experiences and live gaming through a multitude of different games.

James Gwertzman (01:54):

I can't wait to dive into the molecular biology background and talk about how that-

Crystin Cox (01:57):

Because that's exciting.

James Gwertzman (01:58):

[crosstalk 00:01:58] was it about data? I'm curious, what's the thread? When you think [inaudible 00:02:03], what's the thread between those two careers?

Jennifer Lane (02:06):

There isn't but there is. I wanted to be, from the time I was three years old I wanted to be a surgeon, so I was going to be a surgeon and then that didn't happen because life happens. On the side, I was very, I would say, close to being addicted to gaming and computers and tech, and I started playing a lot of video games, especially online in the early MMOs. Once I decided that I wasn't going to medical school and research was just a stopover, I'm like, "What do I do with my life? I've never been in that position."

Jennifer Lane (02:39):

I started to, in fact, at one point I started getting training to become a paper MCSE and it was right around the time and NT 2000 was going away, and so that didn't work out and that wouldn't have been the right thing, but I was also with friends basically spending another 40 hours a week building fan sites, building websites, hosting developer chats. At one point, one of the people that I became friends with said, "We have a position here," it was at Westwood Studios, which is now closed, "You're already doing this for free, why don't you come work for us?" At that point, I'm like, "Okay."

Jennifer Lane (03:18):

I went and I interviewed and I got the job. I told my husband I'm like, "Okay, so I'm moving to Vegas. I think you're going to have to stay back and sell the house," which he did and then we got sick of that. He moved and the first day I was like, "I am with my people," and I felt comfortable and I also found out that my gaming was moderate compared to the gaming industry. That felt really good because I always felt like an outsider with these very serious intellectual scientists, remember this person's name in this paper.

Jennifer Lane (03:53):

To answer your question about the thread, when I first went into gaming, I was a community coordinator and it was focused on design. I never wanted to be a producer, because it was like, "I did all that organization and tracking and all that management stuff," and I'm like, "I'm done with that." Over the years and eventually going into design and lead design, my producer who I'm still friends with to this day, said, "You know? You're really a producer. You should look at going into production," and I finally said I'm like, "I'm tired of fighting it." I went into production, and I was a producer for about 12 years until I came back to Turn 10 and I got a design director role. I wasn't even applying there, I had applied to something else, and they're like, "We need you to talk to these guys."

Jennifer Lane (04:47):

I'm listening to the role and it was focused on community and design and some production and strategy, and I'm like, "These are the things I've been running from all these years, why don't I just give up and running and embrace it and take the role?" and here I am.

James Gwertzman (05:02):

Wow!

Crystin Cox (05:03):

You talked about being involved in that early MMO community, and that story is one that I am familiar with. I've actually worked with a lot of people in my career who were amazing game developers who came from a place of "I was really, really a great community member and then I just started doing the work."

Jennifer Lane (05:26):

Yeah. It's great that you're here now too because you're like my doppelgänger when I'm talking about something and it seems really new I'm like, "And here's Crystin who's done the same types of things that I've done," so it's really nice to have someone else to relate to on that level. Not everyone that comes from the community makes it but a lot of them do. Sometimes one of the big changes of coming from the community is understanding things from the developer side. One of my very first lessons was that the developers often, when there's something going wrong or there needs to be changing or tuning or balancing, they're fighting for it just as much as the people that are the players that are screaming for it, "And why don't you just do this and it's easy." I had an idea already, but I really learned just how difficult sometimes changes are and as much as people want them, because they're not looking at that one change that's impacting you, they're looking at a hundred changes or a hundred new things and it all has to be prioritized.

Jennifer Lane (06:31):

Sometimes, and this is the hardest thing about working with players is, sometimes it seems like the teams aren't doing anything when they're trying so hard to get something done. Frankly, it's not fair to the players. The players have paid for this game or if it's a free to play, they've joined this game, and they have an expectation set, we can't always meet that, and I think that's one of the biggest challenges I've learned about being in a live environment.

Crystin Cox (06:55):

Uh huh (Affirmative). You've actually gotten to watch live games grow up a little bit.

Jennifer Lane (07:01):

Yes.

Crystin Cox (07:02):

Right? I think because we entered the industry in a very similar time period, as I remember also what it was like working on MMOs in the very early 2000s. How have you really seen, I know we could probably spend hours talking about this-

James Gwertzman (07:17):

Yes, we can.

Crystin Cox (07:18):

We could sum up a little bit, what are the biggest changes you've seen happen as live gaming has gone from a weird niche curiosity to if you're not doing it what are you even doing?

Jennifer Lane (07:32):

Wow, that's a really great question. I want to say it's less on the development side because there's still a lot of teams and a lot of studios that this is still very new to them and they're having to learn sometimes the same experiences that you and I have learned 18 years ago. For us, sometimes it's hard when we're like we see the path but you do have to help lead those teams through it. The biggest change I think is the access the player community has to the developers.

Jennifer Lane (08:02):

When I first started, there was no Facebook, there was no Twitter, you relied on your forums, you relied on IRC and ICQ chat. But now, access to feedback is extremely quick and also we have a social responsibility to acknowledge that, embrace that, and I think more than ever the players have a bigger voice in the development of the games. I think you saw that with loop boxes over the past two years, where this was a mechanic that a lot of free to play games used, a lot of AAA started taking that on and the players reacted to it very strongly, and now these companies are all, they're pulling them out.

Crystin Cox (08:49):

Yeah. I think that we've taught a lot with people about how much that relationship with players has changed. Really, it's both that obviously the Internet has changed.

James Gwertzman (09:02):

Tremendously.

Crystin Cox (09:03):

It's not the same Internet it was 20 years ago, and so there's been cultural shifts. But then also design has changed, because having access to that information has changed the way people approach design. As you're having this point in your career where you've come back to that design responsibility, how have you seen the relationship change between designers and the feedback that comes from players?

Jennifer Lane (09:31):

I think, overall, and I think this is a positive, is developers and designers that are building something new, they're starting to, really, from the ground up building the features with the players in mind and ultimately, that's where everyone needs to be. First, you want to build a really great feature or a great design, or a great system, but that's not enough, the early MMOs especially because it was the wild West. When I was a designer I'd be like, "This is a great feature, and how are people going to destroy it for others?"

Jennifer Lane (10:06):

I think things that we had to guess and implement are baseline now, so people get that, but they're still, people play things in an alternative way, they have a lot of emergent gameplay, and sometimes that's really good and sometimes that's really bad, and it still, it's really difficult for a designer to try and think through all the possibilities of unintended play, but they're getting better at it.

Jennifer Lane (10:33):

Another thing to is people are now starting to approach the players and get feedback from them before they even start designing. What are you interested in? What do you like? What don't you like? It's up to developers to take that in, and really take that in, and try and understand the players, but they also have to be careful not to focus too much on the one voice because they still have to focus on the million voices and not everybody plays the same way.

James Gwertzman (11:00):

Do the YouTuber or the streamers or the influencers that you work with got a bigger voice? Do they do often pull those, the ones, the millions of followers, do they get a louder say?

Jennifer Lane (11:13):

I want to say yes and no, and the reason I'm saying that is because they're so visible and present, we often have an easier time contacting them. We try not to rely just on our top players or our most invested players because we do want to represent everybody, but a lot of times they're representing a large number of people. It's not just the streamer and listening just to what that person is saying, but also the people that are watching the streamer and what they say. They help build a pathway, I think, the larger player, so sometimes they have a bigger voice, but hopefully that's through a lot more players at same time.

James Gwertzman (11:51):

Yeah. I'm really curious what impact this shift has had on us to do like Turn 10. You've been at Turn 10 three years, I think.

Jennifer Lane (11:57):

Yes.

James Gwertzman (11:58):

This is a studio that I think has a story could've done very classic premium AAA content and you release it, you ship and then you've sequel a year later or sometime later. Now, moving to this notion of live content, I imagine, we've seen these many, many studios, it's a real massive culture shift, it's a huge sea change internally, and you're in the middle of that. I'd love to hear what are, a, how's the studio dealing with this culturally, and b, how are people shifting their practices accordingly?

Jennifer Lane (12:27):

I think that's one of the more positive things. It is a big shift for the studio and they've recognized it, and they've embraced it, they've taken it in, and they're taking it in at every level. We have made a commitment to our players with Forza Motorsport 7. The past year, we have very much worked with our community to try and make things better, try and correct things, put in the updates that people are asking for. Obviously, we can't do everything and we want to do so much more, but we've shared those experiences and for the first time, we've been sharing what were doing not what we've done, and that's a really big change and giving the players a chance to react to it.

Jennifer Lane (13:14):

One of the things we've been working on for quite some time now is our Forza race regulations, which is a way of making sure people are driving clean and trying to not have such a brutal race in a multiplayer experience, because it's really easy to do, those cards are hard to control, and we announced it well before he released it. We got feedback from them. We kept updating the community as things were coming along. Then we released it as a beta, so we had a special multiplayer event that was specifically labeled as beta. We have a discord channel where we're listening to people. Internally, there's also, we get routine reports of what people are saying, what people are thinking, and we include that as part of the next content release, which is usually monthly, of what of those changes can we take in, what's going to be better, how our players going to get better feedback.

Jennifer Lane (14:09):

We're also really focusing on the player experience and having player empathy, looking deeply at our players, and seeing what the different types of play are, and doing that from our monthly streams, and we're also streaming a lot more as well to have a closer connection with our players, for future designs.

James Gwertzman (14:30):

You're seeing the impact to that, is that showing up in your bottom line? Is it showing up in your key numbers like retention, engagement and besides the more qualitative like, "Hey, our fans are happier," or we feel if we're getting better design out of it?

Jennifer Lane (14:44):

We have been very much more qualitative about it at this point. We're not looking very strictly at the numbers. First, we're just like, "Let's make a good experience. Let's try and give them what they want." We've done made some things that are actually opposite of what we traditionally do because of that, so we're not looking at a, we have a 3% increase in MAU or mau as sometimes people call it. No, I've lost my train of thought, of course. Yeah.

James Gwertzman (15:22):

No, great.

Crystin Cox (15:24):

It's more qualitative, which is honestly a lot of times so much harder to deal with, like the challenge of understanding how do you measure satisfaction, how do you actually measure whether or not players are happy, especially when there's an incredible amount of asymmetry in ... there's always going to be an incredible amount of asymmetry in your relationship with the player base. I used to tell my team, "There's more of them than there is of us," and that'll hopefully never change or we're in a lot of trouble, but that makes that such a challenging job. I think I've seen the industry get more disciplined, more scientific over the years in the way that we approach that.

Jennifer Lane (16:10):

I think there are also community and community roles, community management. Customer service, I think, have also changed as a reflection of that. One of the things that we've seen specifically over the past year is people are still upset about certain things, and they deserve to be upset about certain things, frankly. But there's also some of the feedback that we get is they're listening, they're doing these things, thank you, and that's really nice to hear.

Jennifer Lane (16:39):

Our franchises are particularly challenging because there is a real hardcore element to it where you have to be the lead or you're competing with tens of a second difference. But there's also the more casual base out there too and we're really trying to focus on both.

Crystin Cox (17:01):

I know that you guys have done a lot of telemetry though as well inside of the game. When you're talking about are we doing right thing, are we in the right direction, it's more qualitative than it is quantitative. But I know that as you actually dig in, I can imagine to making changes, especially balance adjustments, what are the kinds of things you are collecting, it has to be a sea of data at this point.

Jennifer Lane (17:25):

It's absolutely a sea of data. I am one of the people that's deep in the data quite a bit. One of the things, my team is the engagement team and they're responsible for building all the live events in the game. Turn 10 has always had live events, but we put some strategy behind it, and simple things to ... if we're releasing this type of event, why don't we do it the same time every week, so people learn to expected and then they can come and know my event is there.

Jennifer Lane (17:53):

The other thing too is we developed a set of guidelines of what makes a good event, and then we would use the data from as we release things to see if we were correct. One of the things I am very, very passionate about is giving people some success. It's great for the really hardcore audience that's trying to be the top 1%, but sometimes the casual people can become those people if you help them along the way. One of the changes I made was we have these things called Forza Thongs, which is a series of challenges, and there's always one challenge that is fairly easy, it doesn't take that much time or it doesn't use a hard element of the game, and we try and do that to bring people in to give them some fun experiences and then along with that we have some of the tougher challenges.

Jennifer Lane (18:45):

What we're doing with Motorsport 7 is we have three of them a week. One of them is just a single go and do this thing, it doesn't take a huge amount of effort. Then we have our drifting community or we have certain types of cars that people like, so we'll do something special for them as part of it too, and having the ability to release a variety of events at one time helps us get something to all the players. As we pull in the data to see the success of how people are participating, sometimes we've seeing a 500% increase in participation because we were putting some rigor on ourselves. It doesn't take away from the design, it just helps mold the design and do things that we expect and hope that players will enjoy and hopefully keep coming back, and we have seen some of those results in our data.

James Gwertzman (19:38):

How do you think about the pacing of your live events in terms of big versus small or intense versus easy back a little bit, because with live content it's possible to just put so much stuff in there the people are just going to get exhaust it. At the same time, you want to have changes over times, so it's not always the same thing. Yes, rigorous schedules are important, but then within that having ebbs and flows in terms of building to a climax or taking a little bit of a step back.

Jennifer Lane (20:05):

That's absolutely correct, and we do look at that. One of the things we look at is, how many times in a week or a month does a player login? How much time did they spend in the game on average? While we have a wealth of data, there is still, I wish we had more so we could really hone in, but we can do a lot with what we have. We try and pace things. One of the things we're doing when we started doing, putting in strategy for events, is asking our test team, how long does it take to complete? How hard does it feel? We would do a weekly survey so we could address just exactly what you're talking about.

Jennifer Lane (20:40):

Some things might take hours or days. We started putting in monthly Forza Thongs where it was over a series of a month and based on the average number of races that someone completes, that's one example. We'll do some things that maybe it takes 10 minutes to complete. I think that helps. It's just so you can grab a variety of, make sure there's something for, maybe not every single person, but most people. Then, for some events, we don't look at that at all. We just have a ... because there's so many different types of events we can put in, we can't put them in all at one time because it would become so overwhelming. We put in a rotating calendar that events will rotate into a new one every X number of days or X number of weeks.

Jennifer Lane (21:29):

While that's great because we have a variety, it does take away sometimes from the players that only want that one type of event. It's playing like Tetris and trying to get lots of stuff out there, make things not feel stale, keep things fresh, and accommodate everybody. You can't do all of it, so you try and do the best what you can do.

Crystin Cox (21:56):

You talked a little bit about this idea of making it so, okay, the race is going to happen at the same time every week. That to me I think of is, that is something that is very much from the MMO world.

James Gwertzman (22:11):

Absolutely.

Crystin Cox (22:12):

We have this idea, it's really important that the players can schedule and go like, "Oh yes, Tuesday at noon. That's when my thing happens." Are there other examples of things you're seeing come from that MMO world and seeping now into this AAA world?

Jennifer Lane (22:29):

Sure. That is one of the big ones. You referred to trying to find the right level, the right amount of things, so there always feels like there's something to do and there is something next but there's not too much. One of the biggest things too is, and we're just at the tip of this and I hope over the years we're going to improve is getting the messaging out. Someone once asked me, "If you could do one thing to improve our retention and improve our engagement, what would you do? I'm like, "Be better at telling people that our events are there."

Jennifer Lane (23:04):

We followed up with a survey with some of our players, and it was like, "Why don't you participate in X a bit?" They're like, "Because I didn't know about it." We started trying to message our players more. We don't have the exceptional tools because this is something relatively new, so it's something that we're going to grow and develop over time, but getting messages out going, "Hey, there's an event. Do you want to participate in it?", and then providing links so you're taken directly to the event. You can press one button instead of trying to press 10 buttons and try to remember what event you are trying to do in the first place.

Crystin Cox (23:39):

That's such an interesting challenge on console because I know coming from mostly MMOs and PC space, you have some different tools available to you there, you can have an expectation of the kinds of communication the player will have access to. Then also when we talk to mobile developers, it's very similar. "Oh, you're on a phone. I know you're going to have access to a certain amount of communication pathways." That's a really interesting challenge on console.

Jennifer Lane (24:09):

So much so that that has been one of the biggest learnings coming because coming to Microsoft is, I've done a little bit of console work before, but primarily I was on PC for many, many years, and then also a few years with mobile. Console is a whole ... I had to change how I thought about things, when to surface things, what is important, and what it's going to replace, because on PC it's a lot easier. In console, you're very limited by that and you want to do it in a way that's not spamming the player as well.

Crystin Cox (24:43):

Yeah.

James Gwertzman (24:43):

I would love to drill in on that, because I think a couple years ago when live content was, obviously, it's free to play that's critical, and with mobile phones especially, it was critical. But I think there's a time when people who are in consoles said, "No, no, that's not coming to my world. Consoles are so going to be very different." Now, we're seeing live content on consoles and we see games like Fortnite now with full cross-playing, cross device where it's almost no difference. I'm actually very curious how, driving it a little bit closer, what are some of those ways in which live on console is different, besides this communication channels are being more complicated. You can't push notifications so easily on a console. What are some of the other differences?

Jennifer Lane (25:21):

I think from the start is a lot of the games just aren't developed to be able to do this easily. The biggest thing also is you have a very limited set of controls. On PC, you can map whatever key you want to do anything, you have a lot more options, where on a console, you have a couple of buttons and a couple of thumb sticks, and they also need to run your game. You don't want take away from the gameplay, so where do you put a message of, "Hey, there is a new event," that's not in the middle of your UI while you're driving a car at 200 mph, and you need button a and button b and a couple of thumb sticks to do that. That's still a challenge. I think it's going to be a challenge, but I think the difference is, and you mentioned crossplay, is developers now, and I think it's going to be much larger in the next few years, are going to have to develop with multiple types of devices in mind.

James Gwertzman (26:25):

One of the thing I want to go into a little bit more is, you mentioned data missing, you mentioned ... actually, I was going all the way back a couple of minutes ago, you're talking about you can measure retention, you can measure session length, but there is so much more you'd love to see. What? What else would you love to see?

Jennifer Lane (26:40):

It depends. It's really situational. We have so many hooks in the games that are tracking things, you need a team to be able to build a dashboard or people that are data analyst to be able to extract that, and you also need the expertise of looking at data, and this is where my being an ex-scientist comes in. If you see a big spike in our chart, people get really, really excited, but sometimes that doesn't mean anything if you don't know the history, so people come to me like, "Look at that event," and it was huge, and there was 80% of the people, I'm like, "No, don't pay attention to that," and they're like, "Why?" I'm like, "Well, because it was something passive that if you went into and did this thing you automatically got a completion," which we do sometimes to give, "Hey guys, here's a freebie."

Jennifer Lane (27:33):

Being able through the data to explain that to someone that's just looking at a dashboard, we need some better ways of being able to look at the history and the bias parts of that because you can make a number be whatever you really want if you're not careful. You have to be incredibly unbiased. I know that's not exactly what you're asking, but I think that's really important. Just being able to easily collect the things in a way that tells a story is what makes sense. It's not just numbers floating around but you're putting a participation against how many times someone logged in, so you can not overwhelm them because they participate an average of two events every week and they login three times. That's not, this person has completed an event is, "Hey, maybe they would like an extra event because they login more times than events that they play."

Crystin Cox (28:33):

It's about context, and also correlation, and that is what's so challenging. I think you're pointing out a problem that we're starting to have across the industry, which is that, it is much cheaper now to extract telemetry than it ever was before. I remember being in conversations with people where we're like, "We can only store so much and we can't put all of these event. We just can't capture all these logs," because we're literally, we can't grab them. Now, this one was never a conversation, it's like track everything. But there is a consequence of tracking everything, which is that now that you have all this data, how do you create context around that data, and how do you actually know when something is correlated or when something is actually a causation as opposed to just correlation?

Jennifer Lane (29:24):

One of the things you've mentioned in the past is being ethical. You have to make sure that the data is not just telling the story that you want to be told because people can very easily, and this happens a lot with free to play, if say someone is they do an event and it means that they have an extra 10% revenue, people are going to hook into that and they might want to maximize that thing but it may be at risk of long-term retention for a player because it's really hard. After a couple times they're just going to give up. You need to have a group of people that are looking at this and what are the ultimate goals for every event, that's even more important than the data sometimes, so when you get a result and it's promising, it's promising for the right reasons. You have a balance of we're trying to have retention, we're trying to have revenue, we're trying to have engagement, we're just testing something.

Crystin Cox (30:20):

That's one of the things I found has been really complicated around a/b and multivariate testing because any kind of multivariate testing is usually, you're really having to narrow down to I want to find out which one of these options is best for X. It can be challenging when after you find out that this option is best for X, it turns out it's really bad for Y, and maybe you also need Y. This has been something that I've seen, it's been harder for more complex games to adopt, a lot of a/b or multivariate testing.

Jennifer Lane (30:58):

Absolutely. They're looking for the one result and that's easy. I would love to see in the future there is some kind of role where a person is a data adolescent, or also maybe a sociologist or psychologist, so they understand the meaning behind the numbers even better from a human perspective. I think, I don't know exactly how we do that, but I could see, I've met a very few people that are like that, but I would love to see us grow that area over time.

James Gwertzman (31:26):

I loved the way you put it that digging for data is like telling a story. I'm trying to understand what story the data is telling you. What I've seen more and more is the hardest part is assessing our question. You got all this data and it's the data analyst who can actually follow a very thin lead, almost, like a detective hunt. You have a lead and you follow this thread and it grows into a whole new line of inquiry and next you know, then you're starting to discover real insights into your own game, but that's hard sometimes, and if you just look at, does X correlate with Y, you often have results that are not actionable, they're hard to figure what to actually do with them.

Jennifer Lane (32:05):

You're also hitting a really important area. This is something we talked earlier about changes on teams as part of live. A lot of times developers and designers aren't sure the right question to ask. They know what they're looking for but they don't know how to ask it in a way that's going to be informed by the data, and that's when you you need the data people that also have an understanding of game design, which is also somewhat rare, to be able to work with them and help teams learn what types of things to ask when. I think we're getting a lot better overall in the industry. I think dramatically better, but I think we still have a long way to go.

James Gwertzman (32:44):

Some of the inquiries in data that I find most exciting is when you're using data like a time machine, where you can basically take a player who's, let's say level 80 or something. Okay, I've got a player level 80, and I want more players get to level 80, so how do we go back in time for this player back when this player was level 1 or level 2, and start asking how is this player's behavior over 1 or 2 different from other players at level 1 or level 2, who don't get to level 80? Who maybe tap out at level 10 or something and start to understand going back in time, what differences emerge and then start to ask how can I either influence that behavior or shape it to remove barriers or friction or try to figure what those blockages are, because that's I think where it really gets fascinating.

Jennifer Lane (33:27):

I think, really, the only way you can do it without having a fleet of scientists with you is to start looking at trends and sometimes looking at the anomalies like you're talking about, why is this person standing out, what did they do different, and are other people attempting to do that thing, and they just didn't get very far.

James Gwertzman (33:47):

Back when I was working for PopCap over in Japan or in Asia and working closely with the EA team, they were masters of live ops, so some of the games are running in Japan. Those are teams who, they would look at data every day. They also had a sixth sense what was going on the data, just looking at all the different trends and graphs. I think any live team over time starts to build for themselves a set of [notifications 00:34:14] or a picture of what to expect, so when the anomalies come up they're going to go, "Wait a minute, something's really wrong here."

Jennifer Lane (34:19):

That's true. I also, I worked at Zynga for three years, and they were very skilled at the data, probably the most skilled I've seen, and they had a lot of tools that showed it. It was as designers and producers helping them understand the game side of the data, that was the hardest thing, but I was on call 24/7, most people were. If something was off, you'd get a phone call at 3:00 in the morning because some graph dipped 10%.

James Gwertzman (34:52):

Yeah.

Crystin Cox (34:53):

To me, a lot of times, that relationship between design and data really comes down to ... a lot of times when you're looking at data, what you're looking at is you're trying to look at averages or you're looking at trends, you're looking at the whole population or some subset of the population, but it's a lot of people. As a designer, you're always trying to think about what is the individual experience of the player and marrying those things. You're talking about like, can we get a person that does both? I don't know. A lot of times I think I've seen you really need one person who's got the big picture data point of view, who's looking at the averages and the numbers and the trends and another person who's bringing that personal point of view.

Jennifer Lane (35:45):

I think you're always going to have people that are going to be experts in either the data or the game design, but I have seen a few people that can look at it both ways and sometimes they have a big challenge of trying to get teams to understand why they're doing it. I've sat down with designers before and the data frightens them because they don't want it to take away from making a good design, and one of things that I try and get across is, the data is not there to replace your design, it's there to help inform your design and you may learn some things about it or how your players are interacting for the future, if nothing else, to see what they're really gravitating to or what they're not enjoying so much.

Jennifer Lane (36:30):

I don't want anyone to think that only data-driven design is bad design. It's just like, you have to be careful, you don't just look at that one data point, you look at a lot of things and then you look at what the history was behind it. It's the same thing with game design, you don't just look at data and go, "Well, this works, so we're going to implement it." You have to look at what your overall goals are.

James Gwertzman (36:52):

I have another a whole different line of inquiry, which is I want to talk about of tools and content tools and content pipeline. You're in charge in live events live content, what's your content pipeline look like? What kind of tools do you have at your disposal to craft the live content you're doing?

Jennifer Lane (37:06):

That's something that we're actually trying to really improve on the future. The tools that we're using weren't really designed for what we're building. They were designed for a very occasional event by one person, and now we've built a whole strategy and an entire team around that, and so the scalability isn't quite there, but the team is looking at future of, they understand now what it means or what even could mean. You have to build tools for what it could mean, and so they're trying to build that in line.

Jennifer Lane (37:42):

One of the things that we talked about is, especially it's harder on a console than a PC that we talked about, it's making sure that we can tweak things without having to do a content update. I know that might be shocking but sometimes designers get things wrong, and you want to re-balance something so players can access it better and being able to tweak those numbers live so you get the update to them faster is really critically important.

Jennifer Lane (38:12):

I'm saying that, but I've been involved with a lot of live games, mostly actually on the development side rather than the live side. Your tools are never enough. I don't care if you're building a box product or building a life product, a designer or a developer is never going to be happy with their tools. It's one of things just as trying to show the importance of live and that it needs support is tools development is usually a thing that you have to prioritize. Do you want to get the game out? Or do you want to have this tool that's going to make it a little bit easier for you? But I do think, as a whole, developers are understanding the longterm benefit of investing in tools and our team certainly is right now.

Crystin Cox (38:53):

Yeah, I think it's funny to hear they'll never be happy with their tools, that's so true. I think there's an idea that someday we'll emerge in this place where it's all perfect, but it's probably never going to happen.

James Gwertzman (39:07):

I'm assuming Excel is probably still one of the top tools for some of this.

Jennifer Lane (39:12):

Yes, it is. Yes, it is. Less so on this project. I have been on projects where Excel was actually ingested directly into the tools, which was actually, excuse me. It was actually a benefit because you can make changes really quickly.

Crystin Cox (39:30):

Yeah. It is. I think I've said many times it is the most ubiquitous design tool in the world is Excel, and maybe the most sophisticated one.

Jennifer Lane (39:40):

I know, but it makes me happy too.

James Gwertzman (39:42):

One segment of this interview we always do we call the trainwrecks. Can you tell us of any live ops gone bad stories from any of your experiences? Not necessarily at Turn 10.

Jennifer Lane (39:54):

I'm going to use something that's probably not my own experience. In fact, I'm working with Crystin on for the next live ops, and I just came up with my title, which is going to be, "Mistakes I've Made so You Don't Have To."

James Gwertzman (40:10):

Yeah.

Jennifer Lane (40:13):

One of the early ... I'll use this because it's like almost 20 years ago. One of the early products that I worked on, and this was my mistake. I was a designer. I had a perforce, I had a bug in it and nobody else had this version, and so I was building a quest and it was a space game, and I had to add some place some characters to the map and I did that, so you could have the quest, and I turned it in. We checked it to make sure they were there, and they were, and went live with it, and the characters were there, the quest was there, and everything else in that sector was gone.

Jennifer Lane (41:02):

I didn't understand how that could happen, but really ... and it wound up we had to do a hot fix then everything was good. But that was my bad because I was testing, and as a designer, I feel that you have a responsibility to test your own stuff. It's really important having give good test plans to your QA or testing. I didn't do enough of checking things that were already there. That's a big part of doing your testing is making sure, destructive testing, making sure that the stuff that was originally there are still there. I just went in and checked to make sure my stuff was working correctly not knowing that the rest of the game was gone.

Jennifer Lane (41:46):

I needed to have more accountability, more responsibility I learned from that experience to always check some of the other stuff and let, if you're adding something to a world make sure the rest of the world doesn't go away. I took a lot of accountability for that because I'm the one that implemented it. I could say, "well, it's my fault, and it had a bug, dadda, dadda, dadda." But really it was still my job to make sure things were working okay.

Crystin Cox (42:12):

That's such a common thing in live games that I think is something new for a lot of box product developers, is the game keeps changing and while it's live, we keep adding things to it, so you end up with this world where, "Wow! How do you deal with testing six years into a game's lifecycle? How much content is there? How many things are there to remember to take a look at?" As you have many, many people adding, "I'm going to add an event here, I'm going to add a quest here, I'm going to add a feature here," it gets harder and harder.

James Gwertzman (42:47):

Am I testing for my newbie player? Am I testing for my level 100 player? Or both? The range of scenarios just keeps exploding.

Jennifer Lane (42:55):

One of the games I worked on was Ultima online. I was a major player from when it first released. I joined the team, I think, five or six years in. That was a game that things in the events in the systems, they even have a clue that that's what they were going to do five years. Every time you added something you made something else more fragile.

James Gwertzman (43:22):

Yeah.

Jennifer Lane (43:23):

Because the game, I think, they originally started development in the mid-90s. There weren't really a lot of advances there and it was crazy to test things on the ... we're fortunate that we can make changes pretty quickly, but sometimes it was guesswork, is this going to work?

Crystin Cox (43:48):

On that note, live games often are guesswork.

Jennifer Lane (43:52):

Yes, they are.

Crystin Cox (43:53):

I think we're just going to wrap it up because we're just about out of time. Thank you so much for coming, this was awesome.

Jennifer Lane (43:59):

Thank you so much for having me, I had fun. We can do this again.

Crystin Cox (44:01):

Yeah,

James Gwertzman (44:03):

Good, yeah, yeah. This is fun. We love doing these, and again, thanks again for joining us today.

Crystin Cox (44:06):

See, it's not torture like it's-

Jennifer Lane (44:08):

No.

Crystin Cox (44:09):

We're nice.

Jennifer Lane (44:10):

Yes, very. Thanks.

Crystin Cox (44:11):

Awesome. Bye everyone.

James Gwertzman (44:16):

Thank you.

Crystin Cox (44:17):

Thanks for listening to the Art of LiveOps podcast.

James Gwertzman (44:20):

If you liked what you heard, remember to rate, review and subscribe so others can find us.

Crystin Cox (44:25):

Visit playfab.com, for more information on solutions for all your live ops needs.

James Gwertzman (44:30):

Thanks for turning in.