Oscar Clark is Chief Strategy Officer at Fundamentally Games where he helps game teams understand what LiveOps is and how to make LiveOps work for them. He has a long history in LiveOps dating back to the late 90s. He's worked for the online games service, Wireplay; the UK mobile operator, Three; and for PlayStation Home, an early virtual world project. He also authored the book "Games As A Service: How Free to Play Design Can Make Better Games."
Oscar will discuss production pipelines and how to keep your game dev teams healthy by focusing more on iteration and less on content and by discerning the best data to collect. Crystin and Oscar also chat about the art of monitoring data; the days of dial up modems; sittable chairs and flushable toilets in games; and the TV series "Columbo" as a metaphor for predictability with your players.
This episode is brought to you by Azure PlayFab.
Visit https://www.playfab.com for all your LiveOps needs.
Oscar Clark is Chief Strategy Officer at Fundamentally Games where he helps game teams understand what LiveOps is and how to make LiveOps work for them. He has a long history in LiveOps dating back to the late 90s. He's worked for the online games service, Wireplay; the UK mobile operator, Three; and for PlayStation Home, an early virtual world project. He also authored the book "Games As A Service: How Free to Play Design Can Make Better Games."
Oscar will discuss production pipelines and how to keep your game dev teams healthy by focusing more on iteration and less on content and by discerning the best data to collect. Crystin and Oscar also chat about the art of monitoring data; the days of dial up modems; sittable chairs and flushable toilets in games; and the TV series "Columbo" as a metaphor for predictability with your players.
This episode is brought to you by Azure PlayFab.
Visit https://www.playfab.com for all your LiveOps needs.
00;00;05;01 - 00;00;06;16
James Gwertzman
Hello. I'm James Gwertzman.
00;00;06;16 - 00;00;09;24
Crystin Cox
I'm Crystin Cox. Welcome to the Art of LiveOps podcast.
00;00;14;12 - 00;00;22;28
Crystin Cox
Hello and welcome to this episode of The Art of LiveOps Podcast today. I am so happy to be joined by Oscar Clark.
00;00;23;01 - 00;00;45;12
Oscar Clark
I'm Oscar Clark. I'm Chief Strategy Officer at Fundamentally Games. Fundamentally Games is specifically a living game publisher. So we're literally going to be applying all these like two decades plus of experience to trying and help game teams understand what LiveOps means and how they can make it really work for them. I've been involved in LiveOps since basically it started like 1998.
00;00;45;13 - 00;01;04;27
Oscar Clark
I was one of the people running a thing called Wireplay. Wireplay was the second online game service on the planet by about two weeks. I was also at Three, the mobile operator, and that was fun because we were doing mobile Java games. I was home architect on a thing called PlayStation Home, a very early virtual world.
00;01;04;27 - 00;01;14;23
Oscar Clark
It's suddenly become very popular. I'd been a Unity evangelist particularly on the world of ads. I've been consulting on games as a service, literally. Literally wrote the book on the topic.
00;01;15;03 - 00;01;40;04
Crystin Cox
He is a mainstay of the LiveOps community he's worked in a wide variety of capacity over the years, from the early days of LiveOps to the telecoms in the UK. To PlayStation and PlayStation Home Project. And now he's done a ton of work more recently in social and mobile and indie spaces. We're going to have a great conversation.
00;01;40;04 - 00;02;12;00
Crystin Cox
I really want to pick Oscar's brain about the history and the things that he's seen change in the industry over the years. And also just his thoughts about production and production pipelines. Because he has such a wealth of knowledge and experience there. So let's dive right into it and get the conversation. I think that a lot of times when we talk about the history of LiveOps, there's there tends to be a pretty big focus on Asia.
00;02;12;00 - 00;02;12;09
Oscar Clark
Yep.
00;02;12;09 - 00;02;33;00
Crystin Cox
Understandably, a lot of great things were done there, and there is a lot of high speed Internet access, you know, that was ubiquitous. But I actually think there's such a deep history for LiveOps in the UK. There's actually so much of that, like really I think foundational work actually got done there because when we do trace it back.
00;02;33;00 - 00;02;37;15
Crystin Cox
You get to things like MUDs and you get to things like the.
00;02;37;27 - 00;02;39;16
Oscar Clark
Oh, Richard. Richard Bartle.
00;02;39;16 - 00;03;02;24
Crystin Cox
Yeah, yeah, exactly. And the work that you guys were doing really early in when yes, it was dial up, but you were still pushing that kind of boundary on, well, what can we do if people are connected? So I'm curious to hear you talk a little bit more about that really early time. And what you guys were looking at was, was it mostly influenced by MUDs and message board... you know, bulletin board posting games and things like that?
00;03;03;01 - 00;03;23;20
Oscar Clark
Yeah, I mean, it was a lot of that bulletin board. It's that GeoCities style era where people were creating websites. So we did have websites. We were, we had hit the WWW era just only just so we're having people creating our own communities. We did this fun thing where we actually kind of found club captains who would run individual games.
00;03;23;20 - 00;03;45;19
Oscar Clark
So volunteer. So this is the sort of very early days of volunteering, but we did some interesting stuff with it. So we had a hierarchy where I had a tournament manager who was in charge of a staff member who was in charge of the process. He had community liaisons who were volunteers who were kind of semi paid. They were volunteers, but they had lots of benefits.
00;03;45;19 - 00;04;04;24
Oscar Clark
And, you know, we had to bear in mind things like holiday pay and things like that because of a ruling that happened with Yahoo! Previously. Then there were club captains, so these were pure volunteers and they may have their own deputies. So we had this like pyramid structure. Sounds very dodgy. It's not dodgy honestly, but it's very structured in order to scale.
00;04;04;24 - 00;04;27;25
Oscar Clark
Even that's relatively small number of players. And we actually introduced a whole bunch of things like game pros who were basically there to make sure that you didn't have people stuck in silos. Like if you were playing Duke Nukem 3D which is actually the game that a lot of club captains played when we were discussing what was happening in the, in the experience. It was either that or poker.
00;04;27;26 - 00;04;50;12
Oscar Clark
We often played the not for money that wasn't for money, but we used to play sort of fun poker at the same time. And these are like guy talking to the community directly and engaging. But that was the sort of primary direct route in and it was all community focus, which all kinds of games. I mean we launched the chess service, and had Garry Kasparov come along.
00;04;51;15 - 00;04;51;27
Crystin Cox
Oh, nice!
00;04;51;27 - 00;05;08;05
Oscar Clark
He approached us. The funniest thing of that was when you get Garry Kasparov at the, you know, going into this event for trying to make chess a sport. And I had this Russian voice saying, Oscar, is that you? And it turns out to be Garry Kasparov. Weird, weird things that happen to you when you get up early involved in these things.
00;05;08;13 - 00;05;09;07
Crystin Cox
Right? Yeah.
00;05;09;07 - 00;05;16;18
Crystin Cox
But the more fun one though is actually thinking about kind of we were also doing bridge. We had a really strong community approach.
00;05;17;05 - 00;05;17;28
Crystin Cox
I can see that.
00;05;17;29 - 00;05;50;03
Oscar Clark
Which you wouldn't think of now. Yeah. So yeah, early days, it was all this stuff. But the really interesting stuff that we innovated on for things like there was no rack system then. It was all done on UNIX space platforms. There were minis, right? If anyone remembers what Mini was, we actually built, not me, but the team in Colchester, that was a pair of twins called the Ashby twins who did amazing, amazing work. They between the team built these rack systems.
00;05;51;02 - 00;06;05;05
Oscar Clark
So basically they literally tore apart laptops, soldered together to make quake servers. And that was basically some of the very earliest rack systems on the planet to run quake servers.
00;06;05;13 - 00;06;22;11
Crystin Cox
Yeah. I mean, that is the way it goes, right? Like games really often do lead the way with this because we are pushing the boundaries. We're trying to get people the ability to do things that no one else is really thinking of yet. There's nothing more in a lot of ways, there's nothing more motivating than fun.
00;06;23;01 - 00;06;45;00
Oscar Clark
Yeah, exactly. And I think it's also interesting one because I spent a little bit time at Nvidia watching the kind of the way the knock on of technology in general has in all sorts of areas of life. And, you know, I think the way we work with servers, the way we work with chips, the way we work with usability has an amazing, profound impact on society, which we often underestimate.
00;06;45;05 - 00;07;17;03
Crystin Cox
Yeah. It's like thinking about, you know, those early days. I think about this quite a bit because I also started my career in the virtual world era where there is lots of virtual worlds. We were, lots of people were making virtual worlds. There was a little bit, you know, pre especially like pre World of Warcraft. And EverQuest was around but very, you know, early there was a little bit more of a broad idea of what might be an online game or might be a persistent online game.
00;07;17;11 - 00;07;41;27
Crystin Cox
And I think about this a lot. As you said, now there's so much attention on this idea of metaverse, this idea of these of persistent worlds. And it is is funny and maybe it's just, you know, the aging in me you go like, I've seen the cycles over and over again, but it is kind of exciting to see some of those ideas come back because I think there were really wonderful things that were happening even like back in those early days.
00;07;42;05 - 00;07;44;29
Crystin Cox
And you're hitting on pieces of it with community there
00;07;44;29 - 00;08;02;13
Oscar Clark
No completely. And actually, what you remind me of is when I got to go into SOE. So I had had my EverQuest account for years and I think I actually had only just canceled it like two or three years after I'd stopped playing. And so I got to go into a SOE team and see how they were doing live monitoring.
00;08;03;02 - 00;08;26;26
Oscar Clark
So it was really fascinating being part of the Sony family and seeing that working and seeing that the attitude that they had. And trying to bring some of those kind of core principles of operational management which general public never see. We would generally only doing like data in terms of like, actually I'll get into data with PlayStation Home later because there's a quite a weird story about that.
00;08;27;21 - 00;08;41;28
Oscar Clark
But just watching operational data I found fascinating, you know, they had this talk of so much we could actually see how many people were in a particular scene at a particular time so you can monitor its performance. So if it dropped, they knew something was wrong.
00;08;41;28 - 00;08;43;01
Crystin Cox
Yes. Yeah.
00;08;43;03 - 00;08;43;17
Oscar Clark
Love all that kind of stuff.
00;08;43;26 - 00;09;19;05
Crystin Cox
Yeah. Alerting and that in that kind of thing. And it's become interesting now because so much of that back end infrastructure has moved to the cloud, which is has many, many benefits. But it definitely I think I have seen less teams take a sort of hands on approach to monitoring and alerting because a lot of it really does tend to get outsourced to to your back end, which can be great if you're working with the back end that kind of like understands games but can also be rough if you're working for with a back end that is more sort of tuned to application uptime which games are just so much more sensitive.
00;09;19;13 - 00;09;38;00
Oscar Clark
Absolutely and also you know because a lot of the stuff that we do in terms of configuration we've got to think about a whole stack. It's not just plain and straightforward. I mean when we're at Sony, even though cloud was there when we launched PlayStation Home, we had to have redundant hardware in situ.
00;09;38;11 - 00;09;38;20
Crystin Cox
Right?
00;09;38;20 - 00;10;04;04
Oscar Clark
Because we weren't allowed to use cloud services at that time properly because it hadn't been proven by the Japanese technology team. So we oh, I mean Hugh, who's an old friend of mine who actually I work with at WirePlay weirdly who was at Sony Europe when I joined, he and the other team did an amazing job, of just making sure that we didn't fall over, you know, that was the one thing we couldn't let happen.
00;10;04;04 - 00;10;06;11
Oscar Clark
We couldn't let the game fall over on the first day.
00;10;06;20 - 00;10;25;11
Crystin Cox
Yeah, it is wild that it kind of we are leaving the era like it just so much of I remember spending so much time with so many teams flying to these locations all over the world to do, you know, server center buildouts and like to make sure everything was going to be fine. And we really are transitioning away from that.
00;10;25;11 - 00;10;31;09
Crystin Cox
Like there isn't, there isn't that kind of planning that goes into like, well, how many racks are we going to buy and where are we going to deploy them?
00;10;31;09 - 00;11;05;16
Oscar Clark
Exactly where you what you reminded me of is that I'm that old that I remember we were plugged in to Earth Station one, which is basically BC tower. B.C Tower Oh well technically it's the actually the general bank, but there's a particular connection in BC Tower where the Wireplay server was plugged in and then the same basically the same switches had the Tesco mobile, you know, not Tesco mobile, sorry, Tesco Internet, you know, these various like virtual Internet providers were all plugged into the same boxes.
00;11;06;19 - 00;11;09;29
Oscar Clark
So when Wireplay went down, most of the UK went down.
00;11;10;14 - 00;11;37;23
Crystin Cox
Yeah, everything went down right. It was so much more centralized. I remember one place that I worked earlier here, we moved our offices in Los Angeles close to LAX because that's where the Trans Pacific L3 pipe comes in. That's where the joke with people that the internet comes from LAX. Like that's where the Internet comes from for pretty much all the Western United States.
00;11;37;23 - 00;11;43;00
Crystin Cox
And we moved to literally what we wanted to be like. One hop away. We didn't want to have any additional hops.
00;11;43;12 - 00;12;01;05
Oscar Clark
No, because now we're all talking about, you know, I think we're way past all that last mile stuff now. That we're not like with the, the early noughties we were talking about last month. Now we're so, you know, we have cable to the to the front door almost everywhere. Yeah. Try to explain to people dial up.
00;12;01;05 - 00;12;11;01
Oscar Clark
But I still get laughs when I talk about this with particularly young game dev students and I talk about dial up modems half of them look at me like I'm an alien from outer space.
00;12;11;11 - 00;12;20;14
Crystin Cox
And it is an odd concept if you think about it, right? If you are if you weren't there to experience it. It's like, yes, we just we plug it into the phone and then it made this noise.
00;12;20;26 - 00;12;51;15
Oscar Clark
Particularly when I started playing in 1984, playing things like the MUDs you were talking about earlier, I was playing a thing called Shades which was US based from the UK and I plugged my phone, literal phone handle into these rubber sockets. Yeah, you get a blistering 96 baud connection on my Apple 2E. I was the envy of all my friends because they were watching War Games, the movie with Matthew Broderick and going, Oh my goodness, you can do that.
00;12;51;27 - 00;12;54;19
Crystin Cox
Yeah. Yes. You're, you're just, you're just like the movie.
00;12;55;13 - 00;13;04;03
Oscar Clark
Yeah. But this just shows you just how far we've come I mean, let's face it, we literally have the Internet in our pocket constantly and have done for more than ten years.
00;13;04;03 - 00;13;22;24
Crystin Cox
Yes. Yeah. And it's been amazing that as we move into the cloud era as well, like, there's I think it's kind of nice to look back because it certainly makes me feel confident that the amount of time that things have changed, the innovation that we've seen, it makes me feel very optimistic about like we're going to see another round of that.
00;13;22;24 - 00;13;36;27
Crystin Cox
We're going to see people figure out all kinds of interesting ways to connect with each other using the cloud, using that like totally ubiquitous online and the ability to like literally just watching something and say, I want to be doing that right now.
00;13;37;11 - 00;13;58;03
Oscar Clark
No, it's like that. And I think it's just like there's so much potential with so much opportunity. I mean, of course, what's also interesting is we've seen the epic fails. We've seen the attempts to do really interesting stuff which have failed. But then when they come back, I mean, the classic one, of course, is where are we with VR? I mean, we've seen VR come in and come in and go.
00;13;58;03 - 00;14;14;14
Oscar Clark
Come in and go. Are we there now? I don't know. I think for some people we are. But is it going to be the same mass market thing or is it going to take an Apple to come and do the glasses? I don't know. And that's the joy of it. We don't know. We can see these hints.
00;14;15;20 - 00;14;20;26
Oscar Clark
We can see what the issues are going to be, but we don't know who's going to do the disruption.
00;14;21;05 - 00;14;41;07
Crystin Cox
I think that's the thing that excites me about working in LiveOps, too, is that I think we're really well positioned in games that we get all this technology that's out there. We can take that and then actually go turn it into something that players want to engage with. We can innovate and try things and see what sticks because you're talking about VR and it's like, is it there yet?
00;14;41;28 - 00;15;02;07
Crystin Cox
I have no idea. Right? And it's actually even possible for VR that we'll look back and say, Well, those were really the proto-VRs. Then someone made this amazing leap and that became the thing that we think of as augmented reality or VR or whatever. And games just have such a strong hand in helping us figure that out.
00;15;02;12 - 00;15;23;07
Oscar Clark
They do you again, going back to the networking activity side of things, it's the scale of the thing. So for example, we're already seeing streaming an integral part of many games, but in such a way that it's invisible. You know, it just happens. I mean I'm, obviously, you know, massive fan of playing my Xbox Gamepass on my phone with the controller.
00;15;23;07 - 00;15;32;13
Oscar Clark
I mean, I playing Fallout 76, my phone makes me smile. I don't do it very often because I don't want to run my battery down. Battery still remains the biggest problem.
00;15;32;15 - 00;15;32;27
Crystin Cox
Yes. Absolutely.
00;15;33;03 - 00;15;35;25
Oscar Clark
But, the fact that I can. That changes everything.
00;15;36;07 - 00;15;57;11
Crystin Cox
Yes, and it's starting to become somewhat seamless. And I think the place where this stuff always strikes me and I think you'll have some interesting, and you hinted you might have an interesting thing to talk about with PlayStation Home specifically here, is data is just the story of data, especially coming from sort of the earlier days of LiveOps.
00;15;57;11 - 00;16;05;02
Crystin Cox
When we had data, we were doing things with it and maybe a lot of the rest of the industry was much less so at the time.
00;16;05;19 - 00;16;28;16
Oscar Clark
Yeah. So I mean, I came from Telco, so I was the games guy in British Telecom, I was the games guy in Three. Telecoms love data so I learned how to do data properly. I mean, I'll say properly, anyone who's a data scientist who looks at my work will laugh at how pathetic I am. But I know the principles and I've got some kind of, you know, self taught ways of doing it.
00;16;29;05 - 00;16;35;17
Oscar Clark
But data is in my blood. I mean, data-filiac. I love data. So I turn up at Sony and there isn't any.
00;16;36;01 - 00;16;36;19
Crystin Cox
Yeah.
00;16;36;19 - 00;16;54;04
Oscar Clark
This is 2008. They're not... we don't do data. We make, we make craft. And I love that too. Don't get me wrong, I totally appreciate that. But it was really interesting. It was a real uphill struggle. In fact, there was a friend of mine, Martin Shenton, who was head of the sort of what they call it now.
00;16;54;05 - 00;17;16;00
Oscar Clark
It's like Global Grains Group or something. It was a group inside London studio. And he and I kind of hatched this plan. It's a bit of a political plan. So he would make a data capture process for Home. I got Adrian who is the lead client engine coder to plug things in, but I had such resistance. No one wanted it.
00;17;16;00 - 00;17;38;08
Oscar Clark
They would no, no, no what they want. And then we go to launch it and then the Japanese team, network team decide to kind of take us through months and months of testing because they don't want it. Then GDC happens 2011 and half the team are sitting in the Eve Online talk about the use of data. By the time I get back everyone is a data convert.
00;17;38;20 - 00;17;44;28
Oscar Clark
We basically and then everything I've done, which has been a compromise just to get something isn't good enough.
00;17;44;28 - 00;17;45;11
Crystin Cox
Yes.
00;17;45;25 - 00;18;10;10
Oscar Clark
But I mean eventually we get it and it's great. But it's like, it's interesting for me because I mean watching Martin doing work with like Killzone 3 to get a heat mapping on a first person shooter. That's magic. That's black magic stuff. And if you can do the same kind of principle, you authentically care about the player experience, then that's great.
00;18;10;19 - 00;18;35;21
Oscar Clark
But then you contrast that with now. So I mean, we just did some work for the UK, ICO, the Information Commission Office, because with GDPR changes, but also their new regulation for the UK, which is called the age appropriate design code, the rules are so much harder now. So for example, you cannot assume if you're selling a game to somebody in the UK, you cannot assume that they're over 18.
00;18;35;27 - 00;18;36;16
Crystin Cox
Oh, interesting.
00;18;36;16 - 00;18;37;12
Oscar Clark
You have to prove it.
00;18;38;01 - 00;18;47;08
Crystin Cox
OK, so they're going to move to a not like. Yeah, and that's sort of not assume unless told otherwise. Now it's go get active consent or.
00;18;47;21 - 00;19;14;11
Oscar Clark
Go get the active consent active proof. So we had to provide them with some frameworks to say this is how you structure data if you're a game developer to be safe. So that's all been incorporated in all their recommendations now. But what is really fascinating when you've got all this stuff like the ID faith for mobile, you know, with Apple no longer having these ID for advertisers where you've seen these restrictions and understandable privacy controls, which is actually a good thing.
00;19;14;21 - 00;19;26;29
Oscar Clark
I think some of the, some of the positives of that are we have to move back to making sure we are taking the player with us. We have to think about their interests. We have to build brands that they care about.
00;19;27;06 - 00;19;28;17
Crystin Cox
Yeah, build trust.
00;19;28;23 - 00;19;31;22
Oscar Clark
But it is harder. It's hard, but it's so much harder. [ad break music]
00;19;36;24 - 00;20;13;11
Crystin Cox
[ad] The Art of LiveOps is presented by Azure PlayFab. PlayFab is just one of the many Microsoft tools that game developers use to build games on any platform. From the industry leading development platform, Visual Studio, to the award winning physics of Havok, Microsoft makes the tools that power game creators. To learn more about all the ways Microsoft can help you on your game development journey, head over to MicrosoftGameDev.com to find documentation, pricing and other developer resources. That's MicrosoftGameDev.com [end ad]
00;20;13;11 - 00;20;36;23
Crystin Cox
Welcome back. You're listening to The Art of LiveOps podcast and we are talking to Oscar Clark today. We're having a great conversation, diving into the history of LiveOps and some of the early days. We're talking about data, we're talking about pipelining, we're talking about community. We're really covering a wide range of topics and there's more to get to. So let's get back to it.
00;20;41;28 - 00;20;56;21
Crystin Cox
It is harder. I'm actually really interested to see how it goes because I feel like what I've seen is started from a place where we couldn't afford to collect mountains of data, and so we were very picky about what data we would go get, and we were very targeted.
00;20;56;22 - 00;20;58;28
Oscar Clark
Oh, I remember those days.
00;20;58;28 - 00;21;31;14
Crystin Cox
Yes, right? Where we were literally always having, you know, always having my storage teams. Like, I don't know if we can add this event log. And we went from there to this sort of explosion of data. And some of what I've seen with the teams I work with is I think that they will welcome some of these restraints because it will help curb this beast of sort of unlimited data where a lot of times they'll now say the problem is not that the data isn't there.
00;21;31;15 - 00;21;50;20
Crystin Cox
That's never the problem anymore. There's the yes, we're instrumented all over. There's so much telemetry coming in. Then the problem becomes they're like, OK, well, now I have this data. What do I do? Like, what do I do with all of this information? And it's still the same process, actually, from back in the early days.
00;21;51;02 - 00;22;03;24
Oscar Clark
Yeah, but I don't know if you feel the same way about it because we started out where we had to think about what data was important I've always maintained that attitude. So we had this debate in Sony where do we want to capture every footfall?
00;22;04;08 - 00;22;04;19
Crystin Cox
Right.
00;22;04;25 - 00;22;09;27
Oscar Clark
No, no, actually, we don't. What we need to know is when you do something, where were you?
00;22;10;13 - 00;22;10;23
Crystin Cox
Right. There's a very different...
00;22;11;11 - 00;22;26;23
Oscar Clark
So I want the X, Y, Z location of where you were when you did that thing? That way I can try and map this meaningful about what happened in the game. So for me, it was always about there's no point in collecting everything if I can't have insight from it.
00;22;27;08 - 00;22;49;22
Crystin Cox
Right. For me, a lot of times where I push really hard is on when we actually go to create reporting and dashboards and things. I'm always very strict about saying, I don't want to see it unless you're going to do something about it. Yeah, yeah. If it's just going to look pretty, I don't I don't even want to see it.
00;22;50;10 - 00;22;53;24
Oscar Clark
So you don't like total all time downloads?
00;22;53;27 - 00;23;15;10
Crystin Cox
Yes, I'm not a big fan of total.. Yeah. Like these are things that to me, I say to them, like, if you want to create a status update on a quarterly basis or something, you want to pull. Interesting stats, great. But if we're going to look at this and we have we all have limited bandwidth to think about numbers, if you're not going to do anything with it, then I don't want to see it
00;23;15;10 - 00;23;33;19
Oscar Clark
For me it's always that rate of change. I remember I was, when I was at Three, the mobile operator thing, I was pressed for my sales numbers, you know it's product earning. You own the business. It's a business within the business. And I had some of my numbers on a regular basis. And they used to come to me three times a day.
00;23;33;25 - 00;23;40;00
Oscar Clark
and I used to say, no, I'm not telling you for a week. Because it makes no sense. You know, giving you a number is meaningless.
00;23;40;00 - 00;23;41;02
Crystin Cox
Yeah too much noise.
00;23;41;02 - 00;23;52;21
Oscar Clark
Too much noise. I'm going to tell you what they what actually matters, ie, what was the rate of change over the period of activity? This is so... this is why we were so success..., I think why we're successful at the time.
00;23;52;21 - 00;24;19;15
Oscar Clark
It's because we thought about things like what is the frequency people came back to the service? Well, the more you updated the service, the more frequently they came back because there were things for them to come back for provided it was predictable. If it wasn't predictable, they didn't come back. Yeah, it's obvious. Oh, you know this, we know this, but I still to this day have to tell developers to think about predictability.
00;24;20;03 - 00;24;39;14
Crystin Cox
I agree. This is actually a really tough thing for people, for developers, to internalize, like very challenging. And I get it. I am actually quite sympathetic to the fact that games are innovative. It's difficult to be predictable. We have, you know, there's many memes about the video game industry to how bad we are predicting ship dates and things like that.
00;24;40;02 - 00;24;55;04
Crystin Cox
But you really do have to internalize it, because if you're going to run a live game and you've got to be able to it's I told teams all the time, it's more important that you be predictable than the quantity of what you deliver on that schedule. You'll get there.
00;24;55;04 - 00;25;17;18
Oscar Clark
Absolutely. Absolutely. I always refer to Colombo, the TV show. I know it sounds a bit weird, but the reason is that, you know, he's going to say "just one more thing" five minutes before the end. And I want to make my LiveOps so it delivers space for the players to expect. I don't have to have volume then I just have to satisfy that itch.
00;25;17;26 - 00;25;18;04
Crystin Cox
Yeah.
00;25;18;05 - 00;25;31;17
Oscar Clark
In the right timeframe, as long I've satisfied the itech, it doesn't have to be huge volumes of content it has to be something that they're predicting or expecting. And that that sense of emotional journey, you know, it's really, well you know Columbo is the best TV show ever made.
00;25;31;26 - 00;25;40;01
Crystin Cox
Yes. Yes. I'm a big fan of sort of a classic detective and mystery shows. I'm a big Columbo fan as well.
00;25;40;21 - 00;25;46;02
Oscar Clark
But the reason I like the Columbo specifically is that it is a whodunit where you know who did it.
00;25;46;14 - 00;25;46;21
Crystin Cox
Right.
00;25;46;29 - 00;25;50;02
Oscar Clark
And it's still fun. And that's why I think it's such a good metaphor.
00;25;50;02 - 00;26;12;15
Crystin Cox
Right, because it's the interesting thing is watching him put the pieces together. And I think that, as you know, you're talking about this like this adages like predictability is so important. It's more important than quantity. I think that's just going to become more and more true because there's more and more things for people to do and less and less friction to do them right.
00;26;12;15 - 00;26;30;07
Crystin Cox
And as you said, you're like, well, if I want to, I can play my Xbox on my phone. I don't even have to go home and sit on the couch. Right. So the more and more there is, the more I think it is important for people who want to have long term relationships with their players to start thinking like, how am I delivering what they want
00;26;30;26 - 00;26;42;04
Crystin Cox
on a predictable schedule and fitting into the rest of their lives so that they don't have to make this choice? Well, I like this game, but it's so demanding. Like I can't stick with it and do these other things I want to do.
00;26;42;06 - 00;27;05;15
Oscar Clark
But I think it's also, you know, the other side of the fence is that as a developer, it's very tempting to think that your LiveOps is building new content and that's all it is. And then you end up on this content treadmill. You end up with teams that aren't paying attention to things like sick leave. They're not paying attention to the fact that sometimes staff will leave and you'll have to hire new staff and they'll take time to be on board.
00;27;05;19 - 00;27;34;24
Oscar Clark
They're not thinking about tech debt. They're not thinking about UX. You know, because if you're not constantly thinking about UX improvements, you're not going to get new players engaging with the game. You know, so I'm constantly trying to get people to think to do less and actually do less smartly. So, for example, the one thing I love about the backends that we're using now, so when I'm you know, I've got a game, I got a .json file in my PlayFab and I can configure, you know, the gravity.
00;27;34;24 - 00;27;59;19
Oscar Clark
I can the number of spawns. I can redesign an entire level if I designed it right and I've got a decent CMS, I could basically make the whole game dynamic enough and I can release content on a schedule that's pretty planned. That means dev team can focus on fixing stuff and the content team...In fact, the community team can focus on making the game engaging.
00;28;00;07 - 00;28;24;18
Crystin Cox
Right? Yeah. Get really focused on the iteration that I agree. I think it is challenging for a lot of teams to accept that the thing you're going to spend the most time on is iteration, not really new content development depending on the type of game that you're producing. But, but as you said, the danger you always run with, you get really focused on content is what the game is going to get bigger and bigger and bigger and more and more complex and you're building up a lot of debt.
00;28;25;00 - 00;28;48;10
Crystin Cox
But the thing I really was sticking on there because I've been having conversations with a lot of teams that I work with recently about this question of, wow, how do we keep the team healthy as we continually develop? How do we create this space? Because we're so used to working right up to the metal, that it's difficult to back off.
00;28;48;22 - 00;29;06;23
Oscar Clark
But you can't do that because, I mean, the constant one for me is the pipeline. So for example, I'm going to design a new feature for the next quarter, OK? So I've got to have the design ready for when the dev team are ready to do that. And also they've got to have the art lits spec, you know, the asset requirements for the art team to do that.
00;29;07;01 - 00;29;24;14
Oscar Clark
and I don't, I can't afford for the art team or the dev team to have massive downtime where they're waiting for the design to happen. But then the design is not going to get the data of how the last thing you designed worked until after that's released. So I can't wait to do that. He's got to plan the next one.
00;29;24;28 - 00;29;49;25
Oscar Clark
So for me, actually embracing the pipeline, you know, thinking about I mean, personally, I try to separate content from feature. So content, meaning doesn't need a coder. Feature means it does. And I try to break down the content so I know what my quarterly, in a quarter, I know what my three monthly themes are. I know what each week is going to deliver.
00;29;49;25 - 00;30;17;24
Oscar Clark
I try to make a narrative of those elements for each of those things that's nice and self-contained. I can handle that. I don't have to rely on the data to get something useful, but that means features I have to be a bit more smart about because if I'm spending a third of my time on tech debt, which I recommend people to at least allocate right, and a third of my time on UX improvements, I've only got a third of, you know, one month in a quarter to think about features.
00;30;18;06 - 00;30;33;25
Oscar Clark
And if I don't do a feature a month, I'm not going to be getting that level of confidence in the game. So, it's got to be small features, it's got to be small things that you can understand and deliver and configure. Otherwise you're just going to drown.
00;30;34;03 - 00;30;52;01
Crystin Cox
You know it really is an art, as you said, it really is an art of balancing out, like setting yourself up with enough planning and direction so that you can actually comfortably operate, but with enough flexibility that you can react when you need to and you can be, you know, monitoring the data coming in. It really is an art.
00;30;52;01 - 00;30;54;10
Oscar Clark
Someone should do a podcast called The Art of LiveOps. Oh wait...
00;30;54;10 - 00;30;59;03
Crystin Cox
I agree. I think it is some kind of art. It's amazing.
00;30;59;21 - 00;31;20;17
Oscar Clark
But it's also, you know, the trouble is that all of the stuff we know and we're talking about sounds dreadfully scary if you're a new developer, the number of times I get people saying that LiveOps isn't for me. And it's because they are, they hear all the horror stories of crunch with development. And then they hear all the horror stories to try to maintain the content treadmill.
00;31;20;24 - 00;31;21;05
Crystin Cox
Right.
00;31;21;10 - 00;31;46;12
Oscar Clark
But it's about designing a pipeline you can deliver consistently. And scaling what you deliver to that pipeline. It's not the other way round. Planning and systems and processes save you a ton of effort. So it's literally it is be smart, not be busy.
00;31;46;26 - 00;32;08;22
Crystin Cox
Yeah. And I definitely understand the intimidation. I always really try to focus when I talk to people also on the beautiful thing about LiveOps is it's all about iteration. And your goals are always is it getting better? You don't have to set yourself an impossible, lofty goal.
00;32;08;22 - 00;32;09;19
Oscar Clark
And you can experiment.
00;32;09;26 - 00;32;11;10
Crystin Cox
Yes. And you can be wrong.
00;32;11;16 - 00;32;11;25
Oscar Clark
Yeah.
00;32;12;00 - 00;32;34;27
Crystin Cox
And then you're going to have an opportunity to learn from that and do something different. And if you just stay focused on that and like breaking it down and really saying like, you know what, all that matters is we're learning and we're getting better it's OK if we're not perfect. We don't operate, and you need to try and set those expectations because they'll look at teams like, you know, Supercell and say, well, I have to do everything that they do.
00;32;34;27 - 00;32;38;23
Crystin Cox
And that's not very realistic for the beginning of your operation.
00;32;39;01 - 00;33;00;28
Oscar Clark
Is actually one of the things as the designer part of me of me loves about LiveOps is that I can basically not never have to kill my darlings, because if I've got a feature I want, I just put it to a later release. They get killed naturally because you never get around to prioritizing them if they're not right, they just sit there in the backlog, never turning up
00;33;00;28 - 00;33;01;14
Oscar Clark
It's so true.
00;33;01;14 - 00;33;05;25
Oscar Clark
and that's OK. That's ok. It's a lot less emotionally damaging.
00;33;05;26 - 00;33;21;23
Crystin Cox
Well, you can take a little pressure off of yourself, right, to be correct all the time. To be, you know, that's this is the curse of design, right? It's really fun, except for all that pressure to be brilliant all the time and have, you know, clever ideas.
00;33;21;28 - 00;33;51;06
Oscar Clark
Exactly. Or actually, one thing I do always, constantly regret it with PlayStation Home is I never manage to get anyone to buy into a back button, essentially a way of going back from where you came from easily and smoothly. And it's the ridiculous idea. I know, but there's something about virtual worlds where the architectural design of things is fine, but having a nice, smooth experience really matters.
00;33;51;13 - 00;33;57;06
Oscar Clark
You want to do it in person. Anyway, so that's one of my regrets. Never managed to fix that one.
00;33;57;21 - 00;34;09;24
Crystin Cox
I think we've all got ones like that too. We've all got our little pet projects that we want to... my team jokes with me that I'm always attempting to get chairs into every...sittable chairs into every game I've ever...
00;34;10;15 - 00;34;14;23
Oscar Clark
You've got to have sittable chairs. I mean, what's the point in having chairs that aren't sittable? I mean, come on.
00;34;14;26 - 00;34;15;25
Crystin Cox
Got to have. You've got to be able to sit in the chairs.
00;34;16;00 - 00;34;16;07
Oscar Clark
I like toilets that flush.
00;34;16;23 - 00;34;19;21
Crystin Cox
Yes, there you go!
00;34;19;21 - 00;34;24;19
Oscar Clark
It's Silly. But I know, but it's Duke Nukem that did that to me.
00;34;24;19 - 00;34;27;26
Crystin Cox
Right. It was so impressive how the toilets and the mirrors.
00;34;27;26 - 00;34;31;26
Oscar Clark
And of course that's before they killed...that's before they killed Duke Nukem forever but that.
00;34;31;26 - 00;34;36;18
Crystin Cox
Sure. Oh my gosh. Yeah. Don't open that can of worms.
00;34;36;25 - 00;34;37;05
Oscar Clark
Yeah.
00;34;38;06 - 00;35;05;03
Crystin Cox
No, but yeah. I mean I think one of just the things I try to do, like sell people on, because sometimes it really can feel when we talk about LiveOps and teams that it's like, oh, this is a whole nother set of things I have to do for my game now, and you sometimes get people be like I used to just be able to make a game and then I can ship it and it was done and it's like, no, this is exciting because there's all of these wonderful things that are opened up to you when you can iterate and you can get to build with your community.
00;35;05;08 - 00;35;29;01
Oscar Clark
But this is kind of where you are trying to pull off because I think developers like developing and we felt that, you know, given our background in LiveOps and understanding that what would be really interesting is if we could take on the LiveOps part of the community and the configuration and the scheduling and the marketing and that kind of stuff, the stuff the publishers do too. Then let the developers do what they're good at.
00;35;29;07 - 00;35;50;14
Oscar Clark
Write the game. But to do that in a way that's structured and pipelined so that it feeds in this virtuous cycle of what living games look like. One of the biggest things to do that though, is being trying to help people work out how can we change the milestone process. Because a lot of the kind of classical masses that really work.
00;35;50;14 - 00;36;04;10
Oscar Clark
So we're actually trying to learn from hyper casual games because if you can test early and make sure the layers of the game work, then you're building on solid foundations. At least that's our plan, you know?
00;36;04;19 - 00;36;26;29
Crystin Cox
Yeah. And not get stuck in the very... it's very waterfall. A lot of a lot of production process and then obviously still very waterfall, which, you know, I will say like listen I went through the Agile Revolution, too, and there's a lot of. Yes, wonderful things about Scrum and all that. But when you really look and think objectively at game development, you understand why it's so waterfall, right?
00;36;26;29 - 00;36;42;26
Crystin Cox
Because you look at it and go Well, yes, there's a ton of dependencies on a large team. It's very difficult to get away from waterfall unless you can, as you said, commit to this idea of getting functional working versions very fast.
00;36;42;29 - 00;37;07;05
Oscar Clark
You've got to box the sta... the steps and stages, and that's why you have a KPI so we do everything based on KPIs. So what does our day one retention look like? What's our day 30 look like? What's our return on advertisement? What's CPI look like? These things we can test early by having it that way we can say, here's the high level waterfall, here's the blocks that we're going to and then do agile within each block
00;37;07;05 - 00;37;08;14
Crystin Cox
right yeah.
00;37;08;19 - 00;37;18;08
Oscar Clark
Because that, I still think that the backlog approach of here are what each one do and you just build what you can in the time frame. That's OK.
00;37;18;09 - 00;37;19;08
Crystin Cox
It can still be great.
00;37;19;08 - 00;37;27;07
Oscar Clark
And it actually works with LiveOps because the things we didn't get around to doing, as long as we've got our KPI we can do later
00;37;27;07 - 00;37;48;26
Crystin Cox
Yeah. And you'll be focused on the right things. As you said, that's all about pipelining, right? It becomes at some point, I think there's also that harsh reality where every team I talk to, especially when we're just starting like what should I focus on? I was like, here, I guarantee you though, you don't have enough focus on tools. I just... whatever you think you're going to do on tools, more.
00;37;48;26 - 00;37;57;24
Oscar Clark
It's kind of difficult because I'm trying to not do things on tools. I try to use tools other people have made as much as possible.
00;37;57;24 - 00;38;03;28
Crystin Cox
Sure that's a great solution. That's, that is a great solution too, right? Just use it off shelf but the...
00;38;04;10 - 00;38;05;08
Oscar Clark
But they're never perfect.
00;38;05;11 - 00;38;20;01
Crystin Cox
Well, the danger comes that I think especially with triple-A teams with when I'm telling them focus on tools, it's because the other option for them is we're going to hand build everything. Right? And we're going to just miss...a programmer we'll just do it. And it's like, Oooh, rough.
00;38;20;12 - 00;38;39;16
Oscar Clark
I, I... Having been a Unity evangelist for so long, I mean, initally I was focusing on ads, but I still spend a lot of my time saying to people, why are you building an engine? I mean, have you got 400 people building that engine and you're an indie game studio and you're spending two thirds of your time building an engine you're only going to use once?
00;38;39;26 - 00;38;43;14
Crystin Cox
Yeah, it's a rough investment. It doesn't make sense for many people.
00;38;43;14 - 00;38;57;10
Oscar Clark
It does nothing new. Yeah. I mean, there are exceptions. I saw this fantastic particle based engine I once saw that was very perfect for that particular game they were making. Fine, I buy that. That makes sense. But they were never reselling it.
00;38;57;10 - 00;39;10;27
Crystin Cox
Yeah, if you've got a situation where you're like, we're building something really specific and we know we can build great pipelines that are going to work for us, better than anyone else can. Fantastic. But so few indie teams I think are really staffed to even get that done.
00;39;10;27 - 00;39;16;01
Oscar Clark
But this is the thing I think it's really important. When you're looking at these things. It's workflow. We're back to pipelines.
00;39;16;02 - 00;39;17;10
Crystin Cox
Yeah, it's all pipelines.
00;39;17;11 - 00;39;34;04
Oscar Clark
The engines that work best for the team are the ones that got the workflow that works best for the team. That means the art dev team, that means the code team, that means the scripting for any levels or design or moments or whatever it is. It means the backend team needs to be a part of it. Workflow with everything.
00;39;34;05 - 00;39;45;05
Crystin Cox
Yeah, I know we're getting close to time here, so I want to make sure that I get to ask you our favorite question, which is do you have any LiveOps disasters to share?
00;39;45;25 - 00;39;47;28
Oscar Clark
I've forgotten most of them, but.
00;39;48;18 - 00;39;49;17
Crystin Cox
That's probably for the best.
00;39;49;29 - 00;39;59;16
Oscar Clark
And actually the one I want to talk about most normally is the biggest one, which was the Sony PSN fall through. But I actually left the day it happened.
00;39;59;17 - 00;40;00;17
Crystin Cox
Oh, wow.
00;40;00;17 - 00;40;25;13
Oscar Clark
So I got to observe from a distance as the entire Sony PSN was hacked and everything went down across the board. Now, I'm glad to say Wireplay was the first thing that went back up. So I must have done OK, or at least the team did OK. But the specs we did obviously were appropriate. But the one I think's the most fun is actually one of the very earliest ones.
00;40;26;11 - 00;40;52;27
Oscar Clark
So we didn't have general email staff at the time. Wireplay had its own email server and someone hacked it, of course. And it turned out it was a 14 year old kid. Forgotten where exactly they were. But being British Telecom, they had access to a particular part of the police. So I sent police to the door of this kid.
00;40;53;20 - 00;41;20;11
Oscar Clark
The parents were completely horrified. They had no idea. I can't even remember his name, this kid was basically he was absolutely mortified. He was just mucking about. He'd worked something out. He thought he's being clever and knock, knock, knock. Hello, hello, hello...log arm of the law. Now we don't prosecute him, obviously. Which obviously we could have done absolutely could have done.
00;41;20;23 - 00;41;26;18
Oscar Clark
However, we decided not to. He became the best community manager we ever had. He was amazing.
00;41;26;18 - 00;41;46;17
Crystin Cox
I mean, I think that that's there's so many stories like that. I think early. And I do think you touched on it. I'm going to pull it right back to the very beginning, this conversation that early in the day there was so much community involvement. There was a lot of volunteering, there was a lot of game masters and things like that that came on.
00;41;47;01 - 00;42;04;03
Crystin Cox
I actually do wonder how that will be integrated into the future of people building things like meta-verses because I do think there's still an incredibly strong value in having that community effort be a tangible part of everything that you're doing.
00;42;04;21 - 00;42;25;18
Oscar Clark
It exists in every walk of our lives. You've got these super fans who bring with them a momentum. We're seeing it with influencers. You know I shouldn't call them influencers. I've just got in the habit of saying it. Content creators though. They are creating communities of their own we're watching what they do and how they play games, and we're letting them. We're watching their experiences and
00;42;25;23 - 00;42;55;10
Oscar Clark
it's informing our choices, but this is no different from having somebody who runs the, you know, local games club in your physical board game shop. You know. I mean, they were important part of how we think the people who run political parties in local town halls and things like that. There's so many areas of the walks of life that we have where, you know, we need somebody who's going to say, oh, let's go play Jackbox tomorrow even in that...
00;42;55;10 - 00;42;56;07
Crystin Cox
Exactly.
00;42;56;19 - 00;43;20;19
Oscar Clark
But human connection is what this is all about. And to me, games are a natural extension of human connection. We've been playing games for as long as we've been human, probably longer. And that is all about social I mean, part of our social brain comes from that we can play an experiment. And actually play, I think, is such an important part of separating from our daily routine.
00;43;20;19 - 00;43;39;04
Oscar Clark
It's the escape which we're not going to get into play to earn and the issues with that. But the key to me is sticking with why play is so special. And if we're not authentic and support people not just to play, but also to create their communities around their play, how can we do this properly anyway?
00;43;39;25 - 00;43;45;20
Crystin Cox
Yeah. Well, I can't think of a better sentiment to end on. Thank you so much for coming on. This has been a wonderful conversation.
00;43;46;00 - 00;43;47;27
Oscar Clark
Always fun. Lovely chatting to you again.
00;43;51;01 - 00;43;53;15
Crystin Cox
Thanks for listening to The Art of LiveOps Podcast.
00;43;53;15 - 00;43;58;14
James Gwertzman
If you liked what you heard, remember to rate, review, and subscribe so others can find us.
00;43;58;22 - 00;44;03;22
Crystin Cox
And visit Play Fab.com for more information on solutions for all your LiveOps needs.
00;44;03;22 - 00;44;04;17
James Gwertzman
Thanks for tuning in.