E43: Bentham's Body, Hypothesis Testing & Marginal ROAS (w/Eric Seufert)

Eric Seufert joins to dissect AI hype, marginal ROAS, Jeremy Bentham's legacy, and managing a multi-million-dollar marketing budget that falls empirically short. WE discuss:

• How do you evaluate an “AI startup” in 90 seconds without being duped?
• Can LLM-driven hypothesis testing replace the Monday creative meeting and outperform it?
• If marginal ROAS is the real constraint, why do teams still optimize to averages?
• When should a Battlefield-scale launch actually spend less on day one and wait two weeks?
• Why did free-to-play economics conquer games but stall on platforms like Twitch or Spotify?
• Will AI-driven volatility make electricity markets funky?

Transcript

Speaker: 0:00

Let's start with utility. I don't understand what it even means.

Speaker 3: 0:05

Everybody has some kind of U tools in

Speaker 2: 0:06

their head that they're calibrated. There's hardly anything that hasn't been used for money. In fact, there may be a fundamental problem in modeling. What I wanna model,

Speaker 4: 0:18

my name's Eric, if people have heard of me, it's probably'cause of my blog, mobile dev memo, which I've been running for like 12 years. Started out basically just entirely focused on like mobile free to play concepts and, and really expanded over the years to cover concepts related to like. At the digital economy, consider myself, call myself just an independent analyst. I don't call myself an economist'cause economists get angry when I do that, uh, because I don't have a PhD. But I'm, you know, interested in topics related to how products grow, how digital advertising ecosystem delivers value and how, how companies can sort of succeed given, you know, various dynamics. Um, at play, at the interplay of those two things. I wrote a book called Freemium Economics,

Speaker: 0:57

one of the, an oldie but a goodie. It was one of the first ones out there. You and Will Luton had another book, this, uh, consultant from the uk.

Speaker 4: 1:04

Yeah. I mean it still sells like reasonably well, which is shocking. You know, it's funny'cause I wrote that thinking like, well, okay, you know, I'll get like two years of mileage out of this and like, to this day, I mean, people will bring it to conferences when I'm speaking and ask me to sign. So it's, it's pretty cool to see that the kind of has, has had a longer shelf life than I thought it would. I mean, there's stuff that I'll go in and read and it's like embarrassing, right? Like, but, but I think for the most part, it. And, you know, hopeful, I think I'm being objective. I think it, it holds up. Uh, but the, the, the funny thing about that was when I wrote that, it, it, there was, there was not this sense of inevitability with free to play or freemium, right? So freemium economics, I think people thought like, this is a flash in the pan. I mean, this is a trend, this is a fad. And, and I mean, I, I, I think I was like pretty steadfast in, in my. That it would be the, the way that consumers engage with content just because of u ubiquity of the growing, you know, the, the sort of the, the, the, the coming ubiquity of smartphones. I mean, that's just, that would make more sense than trying to sell stuff. Like you just give it away for free and then monetize the, the people for whom it, it delivers the most value and find ways to deliver more value to individuals based on their, their behavioral patterns. And like, what's cool now though, is like, there's resurgence in a lot of those ideas because of the, the personalization capabilities that a lot of the AI tools, um, bring to bear. So yeah. Anyway, I wrote that book. Um, I run the Mobile Dead Memo podcast and, uh, and then just Python

Speaker: 2:19

tool. Let's not forget that.

Speaker 4: 2:20

And thesis. Yeah. So I'm the, I'm the creator of thesis, which is like a open source, a Python library for a marketing cohort analysis.

Speaker: 2:26

Can I give you my interpretation of following you and like how I, how I've like composited you and see if I'm wrong. My understanding, this is just my recollection, is that you did a master's. Econ in Estonia. I think you worked for Skype after that. There's the digital chocolate phase. There's Ro and then there's kind of like the consulting turn. But then I hear like all this other stuff, like you're going back and you're doing a master's at Harvard, which is super interesting. You're continuing to write, you're doing more investing and I'm kind of, I'm curious what it, what it kind of like adds up to, I know your wife is Estonian. That's very interesting. You don't talk about that a lot. You play Call of Duty War Zone quite a bit. You have like one piece on product, which I find interesting on Ultimate Online and everything else has been really geared towards advertising. So I find that curious too, like why you don't write more about product. These have been the things that have, uh, have puzzled me over the years. I feel like Phil's fanboying out hard. I know, I know you live in Austin. You've mentioned that once and you're like about to buy or rent an apartment or like a office. This is kind of creepy. Phil,

Speaker 4: 3:26

you mostly got it right? So I I did it, I did a master's degree in London at UCL. So UCL is part of like the Russell Group. Mr. Mr. Bent. Yeah, that's right, that's right. Good, good. Uh, did you see the body? Yeah. Oh yeah. It's right in the what body? It's right in the main building. So the body of Jerry b Bentham is, is encased in glass, in, uh, in the main building of UCL, like the actual, is it like preserved with like formaldehyde? I think it's preserved. They've got a mask over his face, so it's not, you're not seeing like a skeleton and he is wearing closed, but like, that's his, that's his real body. And then, so what I, what I did was that, that, that degree was, was fully funded by the European Commission. And this was back when Oh, when the uk Yeah, so this, but the UK was part of the, the EU at that point. Um, it was before Brexit. And so the, the deal was we, we did our major coursework at UCL and then we did, we had to go to like some partner institution for like, the thesis. And so I ended up gonna Estonia to, to write my thesis and then met, met my wife. And did, and then got the job at Skype and stayed in the, stayed in the region ish. Went to Finland and, and worked at gaming companies other than that. And, and now I live in Boston'cause I'm a student at Harvard.

Speaker: 4:30

Are you a Patriots fan yet? No, I don't. I don't, I'm not a football Okay. Guy does the Red Sox, the Celtics. Is anything grabbing you Bruins? I went to, I've been to

Speaker 4: 4:39

a couple, couple, couple Red to red sauce games. It's fun. It's fun to go to Fenway. Do you have an opinion about Dunking Donuts? I like, I like Dunking Donuts. Yeah. I like it a lot actually. I still prefer Starbucks, but if, if, you know, if there's a dunking on my, on my route, I'll, I'll, I'll grab a dunking.

Speaker 3: 4:53

The Dunkings on the East Coast are way better. The, the Dunking Donuts on the East Coast are way better than they are everywhere else. I used to think that my friend, uh, who lived in New Hampshire was an idiot for saying the Dunking Donuts was good until I had dunking in Boston. I was like, oh, okay. This is, this is not terrible. Why did,

Speaker 4: 5:09

why did you decide

Speaker 3: 5:09

to go

Speaker 4: 5:09

back to school? Why was that important to you? So I'm, I'm doing a pro, so I'm doing a program called Applied Computation. It's in, uh, the engineering school. And so it's, it's like a mix of like CS, math and, uh, stats. And, and the reason it was important to me was that I saw that my interpretation of like. The rise, I guess, or, or just the, the sort of AI's kind of application to everything is that you can't understand how AI will provide value or how it'll even be applied in like a systemic or, or substantive way unless you really understand what's, what the underlying math is. And it, it really started to worry me, especially with the fund, because I was getting pitched on AI products that I had no ability to evaluate. And I do think there is going to be like a reckoning with a lot of these AI tools that are getting funded now, that have like no substance to them that go to zero. And I think that will be entirely result of investors having no idea how to determine whether these are legitimate products, whether what they're doing is even possible, whether what they're doing even really invokes ai. And I, I can give a good example, right? So I got pitched on this product and this was part of the reason that that was, that, that. Catalyzed the, the desire to go and, and learn these, these topics. I got pitched on this product and it was, I won't give too much detail away, but like, it was, it was essentially like, it was an AI play, right? It was, it was an AI thesis. And my, I, I have a, I have a friend who's, who's an investor in, in my fund and, but who's also just like a person I trust, uh, you know, very much, uh, with, with respect to like his, his investment acumen. He's, he's a fantastically successful investor. And I was like, you look at this with me'cause I don't know if I understand how or why this could work. And we just kind of dug in and like, we didn't do that much diligence, right? Maybe like an hour or two. And we just kind of dug around on like LinkedIn just seeing who's employed at, who's, who's, who's employed at this company. How are they, you know, where did the founders go to school? Like, I mean, they were promoting their, their AI bonafide days. Like where did they go to school? What did they study? And it was like that level of superficial announcement and, and it, it was immediately obvious that they were lying. That, that, that they just, they were, they were, they, there was no substance to what they were pitching. It was, it was not ai. They couldn't be doing this with ai. They just, they didn't have the technical skillset. They themselves, they, they weren't employing anybody that to do it. And they were employing a lot of people that seemed very obvious to be freelancers who were just basically like doing mechanical Turk type stuff. And so we passed and a month later they raised a massive seed check from like one of the most famous investors in, in the world. And, and I, and my immediate thought was, it'll be harder to detect these, these, these fraudulent pitches in the future. Yeah. And I, and I, I need to understand what's going on here. And, and like in the process of doing, you know, just a, just a master's degree and learning a lot of stuff that I didn't know, I mean, I, I studied cs. Undergrad, but I didn't study machine learning. And, and certainly, you know, like, you know, the, the, the, the transformer architecture self attention. I mean, this is 10 years old. This is not, this goes back further than, you know, or my, my undergrad studies go back further than that. Uh, and, and it's changing so fast that you, you, you just, just, just, just like, not only do I do, I wanna understand these topics to, to make, you know, just, just hopefully like, you know, profitable investments, but I wanna build in this space.'cause it's really, really fascinating.

Speaker 5: 8:17

Is, is there a big market for lemons in the, uh. AI startup world where like people, it's like hard to tell what's legit, what's not, and you know, kind of everything gets kind of polluted because of that.

Speaker 4: 8:28

I, yeah, I would say it's not, it's not, I wouldn't, I dunno if I call it a market for lemons, I would say that it's, it's almost the entire market is, is is nonviable. Like, almost everything that's being pitched is nonviable. Like al almost every investment you see that is, is sort of like, is claims to be enabled by AI is, is misrepresenting what it's capable of doing. And so, but there are like the, the two, I mean this, this is like the, the kind of just general distribution of venture, but like in any sort of, in any sort of like sea change like this, you know, in free to play games, this is like this, but not, not not, and not, not in, not to the extent that people were misrepresenting what they were doing with free to play games. It's just that there were gonna be a couple that won, a couple companies that won and everything else was gonna be, uh, was gonna go to zero. Well, it, it, with ai, the differences, I mean people are. Actively and knowingly misrepresenting what their companies do and, and their ability to build what they're claiming to build.

Speaker 3: 9:18

Would you say there's like two issues? One is over promise of a technology, which is like not unique to like startup world. I mean, you had Web3, you had mobile gaming before that, like a ton of insane promises that didn't end up coming true. But you know, you still end up with a market, you still end up with a very valuable industry. It's just not worth the, you know, trillions of dollars that was promised. There's that, which is kind of a tale as old as time, but there's also, with ai, like I have found, it's harder to figure out whether somebody actually knows what they're talking about when they're communicating to me, especially virtually. It's like, okay, are you, are you looking this up? Like, did you do a little bit of research before this with Gemini? Like what's the, what's the actual level of expertise? And I imagine that team that you talked to was doing a ton of work with ai. Vibe coding or just vibe chatting with the AI about what a technology would look like. What would a founder know about what are the different, you know, features of a good product. It's almost impossible to tell whether somebody you're talking to actually knows what they're talking about. You almost kind of need that degree or whatever it is. Well,

Speaker: 10:25

why, why does this matter though, right? Like is what has been, I'm curious what the confusion has been like if I think about a company like Cosmic Lounge, like they're the ones making the AI match three puzzle, right? And they're pretty open about what they're trying to do, which is like build an engine that you can generate assets into. I think they're trying to build other components that match three developers would find useful. I mean, you know, are those things covered by LLMs or is it just like, hey, we're building like the Monte Carlo simulation. But if I think about a company like that, what do you think is is wrong about, I guess, what are you looking for in like bullshit detection? Like hasn't the argument always been like there's going to be this model layer, or at least that thesis becoming increasingly clear that there's the model layer and there's gonna be the application layer. And those things are different values in the, in the value stack,

Speaker 3: 11:06

right? There's a big difference between ai, vibe coding, a demo that runs locally on your machine and vibe coding a product that can be delivered to a million people.

Speaker 4: 11:15

People put so much effort into the facade that they could have probably actually built a real product in that time, right? But, but not an AI product, because they can't, I mean, I would say that, you know, if you think about like, and look, I mean, I think PE-people have, you know, been shocked, right? By like, just, just, just the eye popping size of the salaries that Meta has been offering to machine learning researchers, quote unquote AI researchers. But don't forget, there was a team of people that quit Google because they got so rich off their bonuses that they never needed to work again in their lives. And this was like seven years ago, right? This isn't really that new. I mean, what's new is the access to the compute. That's what's new. And the ability and the access to the data to, in order to, to train these, the, these models of the scale, right? But that, but, but the, the, the value. Of these individuals. And there are maybe like, I don't know, the, the, the kind of person, like, so a person who could, who could, who could devise like a frontier model. There's maybe a hundred of those. I think I would guess the people that could train a model, you know, and build the architecture to, to advance an existing sort of model. There's like a thousand, right? And, and that's it. And so like by definition, when you're getting pitched on ai, an AI product where the person claims that, like we're building a foundational model or something, and it'll be, the odds are just against that being possible. But like, it doesn't take long. I mean, it's, and this may sound like condescending, but I could just look at someone's LinkedIn and no, almost instantly if they're lying or not, because there's just, there's just no way they could have the capacity to do what they're claiming with a undergrad degree in CS from 15 years ago. And it's not, it's not to, it's not to sort of denigrate like, you know, their, their education. It's just to say that like they are not part of that group of people that can do what they're claiming to do.

Speaker 3: 12:58

Yeah. They haven't written 50 publications on. Machine learning applications.

Speaker 5: 13:03

Now you're getting a master's in machine learning. I stalked on LinkedIn and like 10 years ago you got a master's, you know, you're studying electricity markets. I'm curious, how do you think the growing demand for AI compute will impact electricity markets?

Speaker 4: 13:14

Oh man, that's, that's, that's funny that you asked that. So I bought all these electricity trading books this summer because I do think it's gonna introduce a lot of volatility that could be profited from, and, and I was telling a, a buddy of mine who's a, a a A professor at MIT, we, we were having lunch and I was like, yeah, I got all these books. I'm excited because I think there's gonna be this, this opportunity to trade it. And he was just like, you like the new Enron? Or something like, how would you wanna do that? It, it can't be very fulfilling. Um, but yeah, no, I think it's gonna introduce a lot of, uh, of volatility and a lot of, a lot of demand too. I mean, you saw Microsoft did that, you know, when you talk about like, the CapEx that these companies are spending, I mean, it's, it's just like almost, uh, unfathomable, these, these sums of money, incomprehensible, these sums of money. I mean, you know, uh, oh, oh, Oracle just did that$300 billion deal. Mm-hmm. Um, with open ai. You know, Facebook's saying they're gonna invest 600 billion in infrastructure. And it's not just data centers, it's not just GPUs. A lot of this is electricity, getting access to electricity. Microsoft did a deal with the, the nuclear provider in, in Washington state to get access to the electricity. I mean, it just takes tremendous amount of electricity to power these things. But I wanna go back to the, the other point because I think there is a fundamental distinction here between, like, if you look at these, you call like innovation, innovation wave. You talk about free to play as an innovation wave. I think it, I think it was, and, and that was something I was very excited about. That's, that's just a, a new business model to my mind. And, and I think the, the, the distinction here is AI is far more systemic. I mean, this is gonna have deep consequential implications for society going forward. It's not, it's not a, it's not an innovation wave per se. This is a structural realignment of society. And that's, and so like, you can make money off of that. Sure. But it's gonna change the way that we engage with technology fundamentally. It's gonna change the way that we engage with each other. It's gonna, it's gonna change the way that we, we, we sort of like understand our perception of self. And so it's like, if you wanna deeply understand this stuff, it's not just reading machine learning papers. It's, it's, I think it's, it's digging deeper than that. It's, it's, it's thinking about like the construction of the human psyche, the construction of the sense of self. And that has, it's not just a matter, and I'm not like an AI dor by any, I'm the most, I'm the biggest AI Stan. I'm, I'm just, I'm the, I'm the absolute biggest AI cheerleader that you've ever met. So I think like we, we should be moving faster than we are. But, but the thing is, like, we just, I think if you wanna think about the opportunities, it's not like. There's, there's a new business model that's gonna enable people to play games on their phone. Um, and, and, and that that will sort of piggyback on the mass availability of smartphones. And so there's an opportunity there. It's not like, it's not that it's not that discreet opportunity because there's some new form factor that. Makes it possible to engage with the content this way, it's going to change the fabric of society. And so there's ways to, to make money by investing in companies that, but it's not, it's not an a market, it's not a product. It, it's, it goes much, much sort of deeper and more, more conceptually powerful and like profoundly,

Speaker: 15:53

I think back to your, your AI stages of marketing piece, which I thought was really great. And I like always go back to that as like a realistic, sobering view about where we are in terms of the AI evolution. And it feels like marketing, marketing is like sometimes always the first best customer for some of these things. Sometimes we say that for gaming, but sometimes like marketing's willing to like get in, roll around in the dirt, crank, crank out some like AI icons. I've been seeing it on Instagram. The amount of AI ads are incredible. I mean mm-hmm. The whole creative, and it's going deeper and further up into the pipeline. But right now, like we haven't seen, for instance, like personalized OUIs. Like I remember when Stein was called like, what was it? Five traits or six traits, like, I'm gonna see this ad, it'll have an emotional resonance, and then it'll be in like a personalized oui. And like none of those things really came true. When I think about those sobering benchmarks, the fact that even those visions haven't been realized, like it, I have hard time squaring that with this is a fabric of reality change because where, where are the things now that would suggest that, that this is the case? Is growth supposed to go to the moon? At some point? When I, when I look at the evidence in front of me, I struggle with it.

Speaker 4: 16:53

Right? But that's because it's, these are existing product paradigms that are trying to apply ai, and you can't do that. You have to build a product from first principles with ai. And so if you think about that's, and no, trust me, it's happening. There's, there are tui being personalized with ai, right? And, and, but you have to think about like, how would that, how would that impact the entire user journey, the entire economy and like put aside games because I think games is a very specific case. So here's where I'll tell you, I'm, I'm not at all excited. I'm not all excited about content production for games using AI tools. Diffusion models. It's not exciting. That doesn't create value. That's just saving costs. I don't, I'm not excited about the use of diffusion models for creating ad creator, right? I wrote a big piece piece about it the other day. I mean,'cause it's, it's obviously relevant, but I don't think that creates a lot of value to reduce cost. Right? What I'm excited about and what I, my research is focused on is hypothesis testing using ai. So how can I actually figure out what a resonant concept is without actually having to test it? That saves me money, but it also drives more value because I'm gonna come up with a better concept, hopefully as a result of that. Now I'm gonna reach a different audience with that. Now, think about if I am. Hypothesis testing on the creative at the very top of the funnel. Where else am I applying those learnings? I'm, I'm applying those through the tu, I'm applying them to the onboarding, whatever you wanna call it. I'm applying them to the economy right now. There's, there's diffusion, you know, generative tools that probably allow me to personalize the, the things that they see, but that's not really the value creation. It's the things that they, um, are exposed to in the first place is determining those things. Right now I've worked with companies that are, you know, working on how to, how to implement the, the personalization in the tui, and there's like a lot of opportunity there. But like, you know, when, when you start thinking about a product that genuinely realizes value, real value, not just saving cost, but actually deriving value from these tools and these. These concepts, it's actually built from the ground up to do that. And like those will come to market at some point soon, but like they're not there yet.

Speaker 5: 18:37

When you say hypothesis testing, do you mean generating the prototype more quickly or using AI to evaluate the prototype or something else? Yeah, using,

Speaker 4: 18:45

mostly using an LLM to try to distinguish between concepts that have been using in the past to to, to determine like which will be successful and then using those to create derivatives or to create, so like parsing apart the actual qualities of the concept that maybe are not even perceptible to to humans. Right. So like there was a really good job market paper that I read. I'm trying to get the person that wrote it on the podcast, he did this, but not with ads, but with headlines. Um, uh, I wanna say it was Upworthy, I think it was the, the publication that they, they published all of their headlines and they always used to ab test every headline. And so what he did was, and there's, you could read the paper, I I, I'm blanking on the name, but you know, there was a job market papers. It is pretty long. But, but he, he just fed this into an LLM and, and, and he didn't, he didn't tell it, which one won the AB test. It was like, try to parse apart the concepts that you think would've won. Um, and, and it produced headline and then create headlines based on, here's some news stories, create headlines on the, on that basis. Right Now, you think about how you apply that to ad creative. It's really exciting. Um, and so that's kind of what, what my research is focusing on, um, in this program. But like, I think it's, it's, it's, that's gonna be like basically transformative. I, I think if, if you, because if you think about the, and, and we could, we kind of root this in games because it's gaming podcast, but like, if you think about the process for doing. Creative testing, um, it is very easily biased by the limitations of the humans conjuring up ideas, right? Like you, you walk into any team and usually what, like any, any UA team, right? And usually if there's a creative meeting and you know, it's a Monday meeting call, Hey, we gotta come up with some new creatives. Go around the room. What, what are the ideas? And it'll be, you know, the head of creative or the, the, you know, the sort of art creative director, but you'll also have like a managers, you also have the analytics person, and it's just like, uh, and people just conjuring up stuff. Now obviously that's gonna be, there's, there's gonna be like, just, just serial correlation there. I mean, it's just gonna be like, oh, what, what do we have done in the past? And maybe I can change that a little bit. And then the other piece is, well, let's bring in some competitor ads. Let's just look at what they're doing. Let's go through Facebook, uh, ads library. Oh, we should just do a, a ver a variant of that right now that works. And it, you know, you could move the performance in sort of like marginal ways, but that's, you're not gonna conjure up wholly new concepts from. Like X nilo from that or, or parse apart pieces of the concept, aspects of the concept that actually drove performance. That that not necessarily easily perceptible, just, just by looking at them. And I think that's where a lot of the, and, and that's not, that's not generative. That's not, I'm not, I'm not, I'm not running a diffusion model to create the thing. I might do that later, but the real value is coming up with the concept. And I'm not tethering that in this like, serial correlated way. Serially correlated way of like, well, we did this before. And so that's kind of what I'm tethering my id, my ideation to.

Speaker: 21:26

So there's this idea that dsh asks a bit, quite a bit, which is that like we have all these models right now and we've, we've fed them the whole body of like human corpus of knowledge. They haven't been able to invent anything new yet. Think the suggestion of that question is it becomes bearish that we've, we've already gotten this far, and why haven't they been able to invent a new vaccine or to advance even methodologically in some sort of field. It feels like those results are really small. Small right now. What do you mean not

Speaker 5: 21:51

invented anything new? Like produce plenty of stuff.

Speaker: 21:54

I, when you think about like purely original thought, like at the forefront of a field, it's they've, they've

Speaker 5: 21:59

advanced like the cutting edge, some matrix multiplication algorithms and like, you know, some abstract math problems like sphere packing and stuff.

Speaker: 22:06

Antidotal evidence that, hey, you know, a math professor, the AI model helped me. Prove out. Help me clarify my proof for particular theorem. I have seen that, but relative to the hype and what should, what I would argue should be happening if we believe in ai, to the degree that we seem to have faith in it, why hasn't it been able to do more of this? It feels like it's impact on GDP is like pretty modest right now, and I guess I would expect more with all this knowledge.

Speaker 4: 22:30

So I don't, I, so I don't think the impact to GDP has been modest. I think it's been substantial. I think it's masked a lot of weakness in the US economy. I mean, a lot of this GDP growth is driven by AI investments. Yeah. Not, not, that's not AI output. That's not AI performance or whatever. That's just investments in ai. But you can't, I mean, that, that matters. But, but I think that's a good question. I think it's, it's, it, it's a function of the prompt, right? I mean, if you think about like, so, so large language model, I think people like, you can't conflate AI with large language. First of all, AI is, is a meaningless term. It doesn't, it doesn't mean anything. People use it as a catchall. I mean, most of the time when people talk about ai, they're talking about either pretty boring machine learning, but that's just being, that's just being pursued with a massive amount of compute. So the, the architecture of the model can get much more, can get larger essentially. Um, and, and, and, and therefore find, you know, sort of like more nuanced patterns and, and process larger amounts of data. Um, or you're talking about, like a lot of times people are talking about lms, right? Which are very specific model. And the thing about an LLM, it's, you know, it's, you build this on the transformer architecture, it's using self attention, it's taking some tokens and it's, it's, it's, it's simultaneously sort of like, you know, observing all of them. And then, and then what it's doing is it's just probabilistic of coming up with the next token based on this set of tokens. And then it's appending that token and it's running self attention again. And then it's picking, you know, it's, it's just running like a, a, uh, classification process to determine what the next word is, come up with the best one, and then append that, and then run the whole thing again with the entire token context. And so like, well, that's just prediction, that's just step ahead prediction, right? And so, well, why would that if you, if you just said, Hey, find me a vaccine, like, well, what are existing vaccines? And like, why you just change something about this vaccine, you know? And essentially that's what it is, right? Like,'cause I mean, LLMs don't need to run with text. Their token could be anything. Tokens just a vector. I mean, it's a embedding, right? So like, you know, it could be anything. And so, but, but, but that's a function of prompting. But I wouldn't say I wouldn't, I would say LLMs are, are one, you know, sort of application of this. But like, I wouldn't say that like large scale deployments of machine learning haven't done anything new and novel or have been applied in new and novel ways. I mean, they certainly have

Speaker 3: 24:29

LLMs in terms of the predictive capabilities to me, are just mostly consumer grade products. I, it was listening to music in the car the other day and Spotify DJ came up and I didn't know that that was a feature. It's a new feature. It's basically their version of, you know, it's, it's a, it's a really fancy kind of personalized search model that I can prompt it and say, oh, I, I'm in the mood for this type of music. Can you play a couple of these things? Basically just repurposing a bunch of Spotify's algorithms, but they're using an LLM to make it more interactable and more kind of social. I look at a product like that, and I agree with Phil. I don't see that pushing. I see it accelerating Google search. I see it accelerating, you know, development of apps. I see it, which is, it is on the output side of things, right? The, the production function, like a times capital, times labor. It is accelerating like how quickly each unit of capital and each unit of labor can produce output. Eric, you mentioned some interesting like mathematical concepts, and I'm curious about like which areas of research. Can these types of models improve? It's just sped up the, the process that, that I typically would take, I'm starting to dabble in like software. I'm starting to dabble in helping engineers write code and, and interpreting code that I get from engineers. Um, you know, it's making our process much faster, but are we inventing anything or is the AI inventing anything, should I say? I, I haven't really found that personally, both in my consumer life and in my, my life as an employee of a, uh, of a B two, B2C business. I'm kind of on Phil's side in terms of the, like applicability.

Speaker 4: 26:06

I think there's like two, two, maybe they're competing thoughts here. It's like, is is this So, okay. So like I can't go to chat GPT and have it invent a new vaccine for me. Like solve, solve some, like, some like human scale medical problem. It hasn't solved cancer yet. Okay. It hasn't done that. You're right. Um. Can accelerate the efforts of 10 cancer researchers to discover that cancer a solution to cancer faster than it otherwise would've happened. Well, okay, so is that valuable? Yes. Is it, is it, is it on its own curing cancer? Probably. No. Right, and that's kinda why I talk about, like, prompting is really the, the question here. But then the other thing is like, so, so yeah, it's, it's got, it's got value. But so, so like, okay. Is that, is that transformative though accelerating the timeline or making people much more efficient? Is that, is that transformative? And I would argue yes, it is. Because now imagine that, you know, these 10 cancer researchers who have PhDs in biology or whatever, molecular biology. I don't know what a cancer researcher looks like in terms of profile, but these 10, these 10 cancer researchers, if there, if there had been a hundred of them, they would've discovered this, this cure for cancer on, on, on the same timeline that the 10 do assisted by ai. Okay. There's value, there's obvious value there. But the thing is like, now can I become a cancer researcher without going and getting a PhD and, and doing all that stuff? Like maybe that's, that's an, that's maybe a more interesting idea here. Or is there like an agent that is itself a cancer researcher, right? Who, who and, and, and that, that it's, it's imbued with all the knowledge that a, a human would've had to acquire to become a cancer researcher. And I can just spin it up with like a, a like a, an AWS instance or something. And that's, that's working on my behalf. Right. And so now we don't have a hundred cancer researchers in, in the same, in the same, in the terms of output, because we have the 10 with ai, but we have a million or a billion cancer researchers. Right? Like that to me is, is, is really interesting. But, but the, the other, the other question here is now, okay, take it away from cancer research and say that I can, I can allow like this workflow, this, this. Workflow that is imbued with all the knowledge that any person would need to acquire to become like the absolute world's leading expert in that, in that domain that is available across all specializations and in, in ways that, uh, that were previously sort of like excluded those people from the economy. Right? And that to me is like really interesting. So, so the, the example that I've used, I, I did this podcast like a while ago called eCommerce at the Limit. Now if you think about like. The typical sort of like setup. And this has gotten like sort of easier and more and more seamless over the years as a result of companies like Facebook and, and Google investing in these automation tools like Performance Max and Advantage Plus, they just made it easier to spin up marketing, right? To make to, to, to sort of like bring all the component pieces into one place and like press a couple buttons. Now you still need creative, but even they're, they're moving in the direction where you might not, you know, you still need some analytics, but they're moving in the direction where you might not now that's great, that made an existing team faster and exist an existing team more efficient. But one, once it gets so seamless that you actually make that, that uh, that practice available to people that were otherwise completely excluded from that is where it becomes transformative. And so this commerce that the limit idea I had was like, think about now every solopreneur in the real, in the sort of the physical economy. The person selling haircuts, like the person running a halal food truck, uh, or whatever, halal guy's, food truck, those people now can advertise. Those people are brought into the advertising economy that is truly transformative. And so like, it's not just the, the sort of access to this, this corpus of knowledge because that caps out at some point, and maybe it already has.

Speaker: 29:42

Why, why does that level, the playing field, why, why isn't the, why is that the equilibrium? Because if everyone has an agent, now I'm just investing into the better agents right now. Our agents are duking it out and that's reflected in relative prices and click through and like all the normal things we know with the marketing apparatus, you, you've made the point that personalization is what unlocked, like a lot of these mid-size PCG companies and a, a lot of went, went away with, you know, a TT was their, their ability to target and to go after particular subgroups. But I'm not, I'm not sure what the. Spinning up an ad creative does in equilibrium. If I'm, if I'm scopely, I'm gonna, now I'm gonna buy 10 agents to spin up ad creative, I'm still gonna insert marginal dollar equals marginal benefit. That feels like it's, it, it's virtually a perfectly competitive market in advertising. Even if there's a new ip, you know, people will rush to the Supercell Meta or the Supercell Art style, or the Fisher Price art style, the Royal Match Art style. Like everyone really rushes to that. The, the voter median theory. What

Speaker 3: 30:35

I think he's saying is that right now there are n equal a thousand and equilibrium is E. What he's saying is like, n equals 10, 10 equals

Speaker: 30:42

100, but you're still bidding on a scarce resource though. You're bidding on time. You're always, you're, you're bidding in time and relevancy. Right? And, and feel free to take me to town on that, but like that, that is scarce. You're trying to still fit all of that, that production function you need to graph it onto that scarce resource. I'm skeptical that this new equilibrium is gonna be more enhancing for the Halal guys versus, you know, the Scope Lees or the Last War guys. Whoever it may be a big advertiser to, to employ the same model and still scale it with capital.

Speaker 4: 31:12

So there's couple, couple things because the advertising, the advertising economy use cases is, is very specific, right? So, so there's a couple, there's two things, right? So in that podcast I talked about, yeah, you're, you're expanding the size of the, the, of the economy of advertisers, pe more people are participating, whereas before there was just excluded, right? And so that's, that's growing the advertiser base. But the other thing you're doing as, and, and what they, what these companies are doing as a result of these tools is they're actually making targeting better. So yes, the resource, the, the, the, the resource is the constraint, right? The resource is like the diagonal line. It's just time and attention and just availability of these ad impressions. But actually what's better is the, the pricing power is that I, I, I'm gonna better reach these individuals now. Now take this to the, the physical economy. These people, not the digital economy, the physical economy, the hair, the barbers, the local restaurants or whatever. They're able a lot of that now they, they can bid on this inventory that otherwise probably went unsold for people that like are maybe not attractive from, uh, banana Republic doesn't wanna bid on me because I'm, I don't look like an e-com shopper, but I live in that area. There's every, so there's not, there's more advertisers'cause they're empowered with this ability to advertise. There's also more advertisee or people receiving advertising, people that were otherwise not targeted'cause they weren't seen as economically valuable for e-comm.

Speaker: 32:22

I think, I think about that as price discrimination then. Is that how we think about and model it, that we're reaching people who wouldn't otherwise have reached like, this is the matinee discount for seniors?

Speaker 4: 32:30

No, no, no, no, no. Because it's, it's, it's, it's, it goes the other way. The the, uh, the intentionality goes the other way. It's, I'm not trying to reach, I'm not trying to like change consumer surplus or reduce consumer surplus. I'm reaching people that were not, there was, there was no economic value in reaching'em before because they're not an e-commerce shopper, or they're not, they're gonna play a game, but they do need to get their hair.

Speaker 3: 32:48

It's probably difficult to like distill the impact of AI into one economic concept like price discrimination or, you know, shifting the production function to the right. Like, I, I did wanna get that, that I think you can totally map it to our normal models. Why, why can't we? I don't understand what well can, I'm just saying like you're trying to, uh, it sounds like you're trying to distill it down to like, oh, is this price discrimination? It's like, yeah, well it's

Speaker: 33:08

price discrimination. I think we have a tool set to understand it, and I think this has been, the disappointment I have had is I don't think economists have taken enough advantage of this and trying to model this out and think about like where, where this all ends up. I'd say the same thing for Web3 by the way.

Speaker 3: 33:22

It depends on, it depends on the economist that you're talking to. If it's a macro economist, they don't care about this because they, like I said, they just start taking a, and they're increase. They're like, this just increases a, it just increases production. If you think about like a, a labor economist, what are they thinking about? They're thinking about the marginal productivity of labor. Okay. NPL goes up, probably going to also increase marginal production, marginal revenue increases.

Speaker 4: 33:45

There's the economic A Im impact, but, but I think just, just going back to like why it's not price discrimination, because it's not that those pe, those people are not paying it. It's not that they, the ad is an ad being shown to that person is a function of their perceived value to any given advertiser. And like, so if they're not getting an ad, usually it means that they're not clearing some CPM floor for, you know, when the platform would deem like. Interrupting the native content to be, to be worth it. Right. And so like there are people that are not being advertised to. It's like, so not, not because the product doesn't become any cheaper. It's just because the, the, the universe of products avail, like where they, they were in the targeting group, it's determined by the platform that they just weren't, wouldn't convert at any rate that made it, that, that, that, that'cause the, the sort of, the, the price that's paid in advertising is based on an auction. Right. And so, but the au the, the bid is, is, is adjusted by the conversion prediction. And so it it whether this person will convert or not. And so like, they were just perceived to not convert on any of that stuff because they're not an econ shop, but they're not a gamer. But they do live in some place and they need to go get their haircut. And so they're become, they're adding inventory to the system because previously just not, no ads are being shown to them. They're gonna free ride. Right. There's a relevancy

Speaker: 34:59

that was never being reached

Speaker 4: 35:01

Right. But even if you, even if you take that piece out of it and just the targeting gets better and there's more advertisers being brought in, and let's say that like, well, you're gonna have to displace the ad that's currently being shown to this person, right? Um, so like, there's no ccpm floor, so something's being shown to them. Well, the targeting getting better just probably allows more irrelevant things to be shown to them. And so you're creating more conversions. Now, the issue, the, the thing here is like, well, and, and I've, I've written about this, like, let's, let's take this to some, you know, sort of like theoretical extreme or some unrealistic extreme, or these platforms get so good that there's always an ad available to any given person that would convert with a hundred percent prob predicted probability. So every ad. Then, okay, you can't show that many ads because what happens? The person's gonna run outta money. So if every ad could be so relevant to them that they would definitely click on it, they would, and then very, very likely convert and buy. They actually have to show fewer ads. Right. So you're actually winning some, some of that, you're winning some of that time back as a consumer. Is that, well, okay, now let's say that, you know, I was being shown a thousand ads a month. Well now I'm being shown 20. And my, my, you know, the, my expenditure is the same because they're such good ads. Well, now I get more time with my social media instead of seeing the ads.

Speaker 3: 36:11

So do you, do you see that as like the biggest impact that AI will have on gaming? You don't seem in interested in, you know, product. You, you mentioned you, you don't think that it's gonna have a big impact on like, productivity That's not interesting. Like, oh, cool, we can create more art assets times as fast as we could before. What is kind of the, the marginal productivity of workers? Isn't that interesting? It does seem like search and finding, you know, higher. Return on ad spend seems like the highest potential for AI and

Speaker 4: 36:37

video gaming. Oh, that's, that's gonna be transformational for gaming. It's just like, and, and then the ability to build more varied types of games. Right. So, so like, if you think about that, that targeting becoming very good, right? Based on like preferences. Well then, okay, I can, I can sort of explore the whole preference space with development because it's become a lot easier to just develop stuff. Right? And then I'll map those, those games, these new genres of games or whatever, stuff that people wouldn't make before.'cause there's, it's not enough of a market to it, but if I can meet the five people that I'll pay$10,000 a month, well maybe there's a business there. Whereas before there's just, there's there couldn't have been because you're not gonna find those five people, not five, but call it 500. Imagine. Imagine a game with DAUA thousand that made a million a month. Like, is that, is that unrealistic or is that just not possible now because we can't find those thousand and they never churn. They never churn. They will play that game for 30 years.

Speaker: 37:31

So to, to talk about the marketing piece for a little bit.'cause I, I do, I do wanna talk about like, the core econ piece, which I, I've been reading you for a very long time. And my favorite, I dunno, seism, I don't know, I think, I think you're ready for like a branding on some of these catchphrases that all, all marketing is, performance marketing has been something that I go back to all the time, right? You can always look at any marketing dollar regardless of whether or not it's direct response with a cost benefit analysis lens, you're always trying to drive growth with spend. Mm-hmm. What I'm curious though is I still don't know because you're, you're also very charitable to what we might consider traditional brand advertising, non-direct viewing. Like you're, you're, you, you, you take a sarcastic tone with it, but, but you know, it could work. Like that's something that, that kind of some come, comes across in your writing. And I don't have anything specific in front of me, but I guess like, if I think about you managing a marketing budget. How do you even think about something non-direct response advertising when you don't have the opportunity to run incrementality tests when you don't have the opportunity to apply that performance based framework onto it? How would you approach something like that? Like, is the Super Bowl useless? You know, is that, is that like no man's zone? Is that never profitable? Like under what conditions can we think about these type of things as profitable for firms? Y

Speaker 4: 38:43

No, I think the Super Bowl probably drives a lot of value. Right? I think, uh, you, you, so here's what I, here my, my, my take here is that you shouldn't do something if you have no, if, if you have no hypothesis around how it can be measured. And I'm not saying you need, you know, apriori, you need a perfect solution to that. I mean, you can't have one, but I mean, what's your approach? What's your mental model for how this drives value and how we can test it, how we can measure it? Mm-hmm. And like that, that, that just should be a, a precondition for spending a single dollar on marketing. Now for direct response, the problem is a lot of people just take that as, um, first of all, as precise, which it never was. Even pre att t it never was. And also you miss a lot of the interaction effects, right? So like, I, I don't, I don't, so I, I definitely see brand advertising is, is really important, even for digital products. I mean, I, I, I've, I've written about how it, a lot of times the difficulty of measuring that gets seen as an excuse to not measure it, you know? And so, well we, we have a vague sense, we have a gut feeling that this is gonna drive value and it's gonna make this other stuff more valuable and, and more performant. But, you know, since we can't really measure it, which just give us a blank check. I mean, just let us do it. And it happens a lot. I mean, people, people accuse me of using a straw man when I say that, it's not, I've seen it. Yeah. Right. And, and it's, it's common. Uh, and so that, that's just, that's corrosive.'cause that just makes everyone feel like they have less, that there's, there's, there's, there there's less of a requirement for them to be, you know, to, to, to justify their. Existence. But, but yeah, I think so. How do you do that? Well, first of all, you formulate a way to measure it, and you can, you can't, I mean, first you said like if you can't do incrementality testing, but most times you can. I mean, you can, you just have to figure out how to do it. And it's hard and it's, it's, it's, it's this persistent thing you have to do all the time. It's not a one off, but you know, if you can, and since you can, you must. And then if you do that well, then you can measure the impact, or, or at least, or at least come to me with a lot of assumptions. Like, and, and then, and then just let me, let me, uh, you know, let, let me, let me, I can't necessarily test the assumptions, but I can use my accumulated experience to, to tell you if those assumptions are, are reasonable or not. Right? So like an assumption would be, well, I mean, we can't do perfect incrementality testing, but what we can do is we can segment parts of the US and we, we can, here's some assumptions. The assumption is that the behavioral patterns are similar within these regions. And there's, you know, similar size, similar, you know, g per capita, similar cultural sensibilities and. Like, that's, that's an assumption that I'm making. And then that's how we're gonna do the test. And it's like, okay, well that probably seems reasonable. But now if you're picking Missouri versus California, I'm gonna say no, that's not reasonable. But if you're, you know, if you're picking things that like you can justify, you can make the case, that's probably okay

Speaker: 41:13

if I follow this, like down the funnel, I mean, isn't, shouldn't, shouldn't the phrase be like, all spend is performance spend, because why wouldn't I follow this down the funnel and think about this for ab testing, like a product feature. Like there's nothing about this that's specific to marketing, isn't it? Shouldn't, shouldn't we take this approach with, with everything? Well, what do you have in mind? I'm not sure I follow. So like, consider that I'm a product manager operating on a game and you know, I have X amount of capital that I can spend towards acquiring users or let's say spinning up another team. All spend is performance spend. Like I'm always trying to get high ROI there, it's just a question of having a falsify belief at which one will gimme the highest return. Isn't that like strategic finance? I, I, I think it's just capital allocation. Right? But there are games typically PC console where. This is really hard to apply. This is something I struggle with quite a bit. HD companies almost never have AB testing. If they do have AB testing, it's maybe on a small seg segment of the store. How, how would you approach then for, for something like that, like for Battlefield Battlefield's coming out? If you're a marketer on Battlefield, what do, what are you trying to prove? What would you, what would you try to, to say? Like, you know, you know, they put a banner in Med Warrior plots in here in Stockholm. We can do that mental math of how many people walk through. That square with the conversion rate. There's this thesis that, that I've seen the Chinese push where they, where they think there's, at some point your CPIs dip when you cross a certain threshold of spend. You know, if I'm miho yo and I push, you know,$150 million down, down all these channels, that there's just this general awareness that goes up that decreases your threshold to click, which means you actually face lower CPIs even if we can't draw a direct connection to it. Do you think like those type of arguments hold water, even if we don't have incrementality evidence behind them, we can still try to evaluate them?

Speaker 4: 42:47

There's competing effects, right? I mean, you know, if you're, it depends on, you know, if you're, if you're pursuing like truly a saturation campaign, like you're gonna run into capacity issues and the ccpm, the CPMs are gonna go way up and you're competing against yourself essentially. So, I mean, there's, there's some, and, and no one's really explored that whole, like, it's, it's, it's difficult because you, no, no one is. That's a one shot, right? You can't, yep. You, you can't test different levels of spend at that scale. Right. And so, you know, you'd have to make the case that like, you're not actually just saturating all channels and competing with yourself and driving your CPMs up. So I, I don't know that you could take that as like a general rule. Now it could be true in some cases, certain levels of spend, but like, I think with a battlefield, you know, it's, it's, it's interesting because like, you know, when we launched Inverse two, I was a VP at ua at Romeo and launched in Inverse two. And it's like, you know, we spent a lot of money, but we didn't, I mean, I don't, if we hadn't spent that money, I don't know that it would've made a difference in terms of the original install numbers for that launch because it was just a high profile launch. And if I was the VP of marketing at Battlefield, I'd say, let's just sit this out for two weeks and then. We come in and we, we do the marketing for the people that we didn't reach in the launch. Just like that long tail of people that weren't just already waiting for this game to be launched. And, you know, obviously, so, you know, the platforms have a incentive to promote that game'cause it's gonna drive a lot of revenue for them. So let them do the outreach and, you know, there's gonna be a lot of earned media. So like, do we need to even spend money now? What, what would be the, what would be the actual incremental value of that? And would we just be poaching people that would've discovered it Anyway, that's, that's, I mean, that's the part of incrementality. So I think those are just like case by case. I, I don't, I don't know that like, battlefield is a ton of, to spend a ton of money on awareness or on call it, like direct response advertising.'cause I think, you know, they just need to make, make people aware of the fact that this game's coming out and they're gonna wanna buy it anyway. And you probably have a lot of emails, you know, the, the platforms are gonna assist you because it, it's in their interest, right. And so, uh, but yeah, I mean that's why we did. The, the celebrity TV ads, it's like you drove, you, you, you have this like sort of ambient, you, you increase like sort of ambient level of awareness and that makes people more receptive to the ad when they see it. I mean, it worked until it didn't, right? I mean, like, that was, I mean, was that, was that a successful outcome? I mean, I don't know. I don't think this is like a controversial thing to say. It wasn't,

Speaker 3: 45:03

I wanted to return to this idea of incrementality, you know, you mentioned needing to at least understand the metrics and how much you can spend. Um, I mean, basic understanding of return on ad spend in the context of like Web3 and that whole situation four or five years ago, billions and billions of dollars being funneled into this industry. I still work in Web3. My whole new thesis is that, uh, there was just a complete misunderstanding about, first of all, measuring the return on ad spend. What is the actual value of a customer? And like, how much can we spend to acquire a new customer? Did you see a misunderstanding of how UA works in gaming? Uh, when you were being pitched these Web3 companies and did, like, when you were getting pitched, what was, what were the things that, that like didn't pass the sniff test and were you surprised by the outcome?

Speaker 4: 45:52

Yeah, I mean, I, so, I mean, I did two Web3 investments, right? I did one into a studio that, you know, I was probably not doing well. I haven't gotten any updates in a while. It's not a good sign. Did you get an NFT at least? No, I didn't. Uh, I made one investment in studio and, you know, I knew the founder. I was kind of just, you know, I wasn't, that wasn't a Web3 investment per se. That was a this person investment. Yeah. Um, and then I made an investment into a company called Spindle, which got acquired. It was, it was a positive outcome. They got acquired by Coinbase, and, and the thing is. And so I, you know, just, I didn't say yes to any other Web3 investments because I, to, to answer your question, no, I, I, or yes, I do think that they had a fundamental misapprehension around growing a game and, and a lot of these. Marketing campaigns were just like, well, we'll spin up a discord, we'll do some, uh, token giveaways, and there's gonna be a lot of interest. And, and then that'll turn into like, you know, retained high value, high engagement players. And I just thought that that was like a absurd approach. Right? But spindle, they were building essentially like the ads infrastructure for, for Web3 gaming, really. I mean, that was the, the kind of principle product category. And like their pitch is very compelling. It's like, look, I mean, look at all of the, what, what, what, what better technology could you conjure up for attribution? They did, like they were doing attribution, like ads attribution. What, what better technology could you conjure up for ads attribution than the blockchain? Because look at all the waste in mobile app attribution, because there's, like you, when you do mobile app app attribution, you have multiple attribution calls. Like it's just waste with the, if you put it on the blockchain, like every interaction is, is an entry on the blockchain. So you can trace that path. Yep. Like perfectly right with, with, you know, deterministically. And so that made a lot of sense. And then the other part of that. My investment thesis was like, look, if you go back to the history of free to play gaming, none of this took off. This category didn't take off until you had the advertising and measurement and and attribution infrastructure in place. So if you think, if you think about the sequence of events. You had, um, and I used to know these dates, I'm probably a little fuzzy on them now. Uh, but you had Amplifier, which became Unity Ads, I wanna say launched in 2010. You had Apps Flyer launched in 2010. You had, I think, uh, iron Source launched I think in 2011. You had, um, uh, a just launch in 2010. 2011. You had App Loven launch in 2010, and then. The biggest free to play games launched because they could scale, they could grow, right? And so and so the thesis was like, look, Web3 gaming won't take off until this is in place, until this ads infrastructure's in place. Because it didn't take off for free to play gaming until that infrastructure was in place. Now, you know, I think the issue with Web3 gaming was it was still too hard. Like there was no platform that supported discovery very easily. It was still too hard to onboard users and, uh. You know, I think it, it just wasn't just, just fundamentally wasn't like a, a mass market, uh, product category be because of the complexity of, you know, getting a wallet and all the fraud that ended up, you know, sort of taking, taking root. Right. So I think those are, those are issues. But nonetheless, I mean, I'm very pleased with my. Investment in spindle and it, it was a positive investment. I made money and it was like a one year or 18 month holding period. So that's great.

Speaker 3: 48:54

We, I know exactly how wealthy my player base is. Like I can, I can look into each and every single wallet. I know what they own, where they got it from, who they interact with. I have a graph of every single wallet and who it's trading with and, and you know, the programs it's interacting with, it's, it's an incredible amount of insight and, and data that you have at your disposal. But if nobody is building these products, and I, I think it was kind of the case that, you know, it's in free to play. You had the free to play games were perfected, well not perfected, but you know, you had these fun experiences that were playable and then you had the network come in after the fact with Web3. It was like you had the network and you had all these people who were like, you know, picks and shovels, picks and shovels, but there were no actual products. And by the time a product came out, I was like, this is pretty clunky to actually interact with. So perhaps it was mistiming. I'm curious if that. That understanding of the network being super valuable will lead to a research and some Web3. I don't know.

Speaker 4: 49:49

I mean, to be honest, I I, I'm not very familiar with, with the current state of the Web3 market. I would be, be happy to hear what you all think on that.

Speaker 3: 49:56

As somebody who works in the industry, it's not great.

Speaker: 49:59

It's, it's just impressive how little you can learn after repeated failures. How, how hard can you fall on your face over and over again? One of the things you wrote a really long time ago was the ultimate online piece where you explored kind of old Ultima Online and how it could be applied to Web3. And I I thought that piece was fascinating because it's one of the few times I've ever seen you write about core economic or core product problem rather than some of the other things you tend to write about. And I'm curious about why, like, why don't you write about Call of Duty War Zone? Why don't you write about what they're getting wrong in their game loop? Is that, is that not interesting? Like you've been around games for so long? I'm sure you would have amazing things to say. I

Speaker 4: 50:32

mean, maybe I like. I don't know. I, so first of all, those articles are, take a long time to write. I mean, those are, there's, there's a lot of research that goes into that. So just a

Speaker: 50:41

function of time, but you don't go in and shit on like, call of Duty, like Call of Duty, war Zone store. Right. Like, I don't know if you've, you've had a chance to visit it, but like there's certain things that I know that you would look at and you'd be like, holy shit, they're getting this right, this Right. And they're getting this wrong. I've spent thousands of dollars

Speaker 4: 50:57

on Call of Duty War zone. I, I know what they're getting. Right? I, I think, I mean, so here's what I don't think is, so, so I think they, they overcomplicate the, uh, tech trees per season. I think you can make that a lot more simplistic and some just like straightforward and you'd probably make more money. I also think that the, a lot of, so they try to pair, you know, it's very obvious that they view the kind of constant content treadmill. As a function of, of, of getting people to spend money purely for optical reasons, purely for cosmetic reasons. And then, you know, utility and like, that's actually like really challenging to do. And I think I would probably just get rid of all the, a lot of the cosmetic stuff. And I think if you could imagine having fewer, and this is maybe it sounds like it's contradict what I just said, but it's not because they're tech trees per season are related to like achievements, right? You have to do these things to unlock the ability to have this, this other set of things, right? If you got rid of a lot of like the silly cosmetic stuff, you could have guns that were like much more co uh, um, configurable. Like you could have built like, like, uh, build outs, like, uh, that, that were much more configurable. And I think that would actually create like much more of an incentive to really invest. Um, and that would also create, um, a lot more breadth in like, uh, strategy and, and which, which I think, you know, you, a lot of people like are, are frustrated with Call of Duty War Zone because a lot of times you see that people that are. The best at that game are just kids who are really quick at jumping and like very fast at reacting with their controller. Now what if you brought in like a much more strategic component to that and you had guns that were like just, you didn't need to be the fastest to press the button or the best at jumping, like you could actually be, just have the most strategic use of the combination of the gun, I mean the secondaries. And so that to me feels like maybe an opportunity. And I feel like they're just, they're just building this massive base of content that, that actually doesn't make the game more fun from a strategic standpoint, which I think is leaving a lot of money on the table. It's just, it's a, it's cosmetics and it's b it's like being the first to like exploit some, um, you know, like a lot of the games are like the, the guns that come out. They're way too powerful and they get nerfed and like you, you know, I follow a lot of the streamers and like they're, oh, this gun's really awesome. You gotta have it, you have to have it in your, um, your loadout right now. They said, well, no, a gun is obviously, it was just mismanaged. They, they, they, they, they, they brought this, this, uh, new blueprint and it's just, it's the dumbest gun ever. It is just like way too powerful and, you know, they're gonna nerf it. And so like, I'd rather you have like a smaller universe of guns, but you could actually make them, they're much more configurable and I think people would spend a lot more money on that. And it would also, it would sort of like mitigate that, that. Benefit of just being like really fast at, at pressing the buttons. I mean, that's why like the thing about like eSports is like you, you're aged out of eSports by like age 19. Like you have to be a kid because you just have to be super fast. It's just a reflex. It's a reflex game. And so like, I think you could just bring a lot more people in the tent who were competitive if there was a much more strategic component to that. Like imagine if it was a kind of RTS obvious, there's, there's. It's a first person shooter, so you're running around and you have to be like, reactive. But like, I think Call of Duty gets portrayed as like a strategy game when it's really not. It's just a, it's, it's just, it's a game about like, uh, response times.

Speaker: 54:05

I asked you this like three years ago and we, we discussed this a little bit. When I think about games like Call of Duty and, and Battlefield, why aren't they using performance based advertising like this? This just blows my mind. Like they have the ability to do this and nearly all of it is non-direct response, what we might call brand advertising with budgets that are eye-popping. Yeah. You know, hundreds of millions of dollars. It, it just baffles me. That is, is it centripetal force? Like what's holding them?'cause I've seen this on Amazon Game studios, like we installed the Adjust SDK at one point. I'm just. Why doesn't this happen more? Does no one care? Well, I think it's,

Speaker 4: 54:37

it's, there's a couple pieces, right? And you know, it's, it's funny'cause like whenever I say there's just like, you know, I'll, I'll talk about just like inefficiencies, right? I just talked about like, you know, the brand team that says, oh, well we can't measure it. So like, don't even, like, let's just like, we can't measure it. And so therefore we can just spend without any accountability, you know? And people are, oh, that's a straw man. That doesn't happen. It does, it does happen. I mean, you, you can't, I think you, you put your economist hat on, you say, well no, that, that would've been eliminated.'cause it's obviously wasteful. No, it doesn't. And so there, there is just like institutional momentum for that practice. And like you'd say, oh, that can't be the reason because otherwise these companies that are, you know, very maniacal about efficiency, they would've eliminated that or they would've made it more. No, that is just the case. You have the person running it for 20 years and this is always the way they've done it. The other thing is just like the suitability for that with the platform, it's just really hard to do that for console games. Let's, let's on steam, right? Steam in particular is really difficult to do user tion for because it's just, there's, it's just a. You've got the event horizon and there's no information that that can come back, right? Like you get the person onto the game, you can't retrieve anything. There's no ability for a feedback loop to take shape, right? And so that's, that's an issue with, with PC console. Now, that said, there are really smart teams that do, do user requi, like, you know, direct response or performance user requisition for PC console, right? There's companies that specialize in attribution now that, that's probabilistic attribution. There's no deterministic component to that. But still there, there are smart companies that do that, right? I worked with WB for a long time, and we did that. They, they, I, I wasn't part of that team, but they did that for the PC console business, right? And so there are smart teams that do it, but it's, it's just, there's a lot of institutional momentum and there's just, you know, the, the platform doesn't lend itself to that.

Speaker 5: 56:13

Yeah. I'm seeing some games use like creator codes to do marketing on Steam, where like, you know, you have to enter the creator's code, get a discount. Do attribution.

Speaker 4: 56:21

I'll do a little bit of plug. So I'm an advisor for Discord and you know, we're building, um, the ADS product. And, and you know, this is, I think it's a, it's a, it's baked in attribution, right? People are familiar, but it's the Quest system. And so you get rewards. It's, it's almost like the rewarded UA in, in, in mobile. Um, and so that's a really good way to do attribution. Who collected the, who saw the ad and who collected the reward? And well, I can map those two things.

Speaker: 56:42

I've been all on tight to that one. I've, I've got hope, I've got faith. I just need more ad inventory to like get, get more games on it. It's like people are ready to spend an hd. Like I, I think there's, there's been a lot of businesses, especially like the failed VC cohorts. There's been a lot of businesses that are still able to persist and are having stable DAUs. Like they're not a hundred million dollar products yet, right? But they're hitting like 50 million a RR and like they want to grow through marketing and they just don't know how to do it. They just set a loss, which is a real shame. Well, there's a lot of mobile companies that wanna spend, they just can't.

Speaker 4: 57:11

I mean, they're just constrained, right? I mean, that's why I think the rewarded space took off to the degree that it did. I mean, there's just so much pent up demand for new inventory, and if you added measurement in like, then that's great. People will spend, and they, you know, they'll spend a lot. So, I mean, I think, you know, kind of counterintuitively, I think now would probably be

Speaker 5: 57:28

a good time to launch a mobile ad network. A question about kind of marginal analysis here. It seems to me most performance marketers look at average bros. Whereas like, you know, you expect, you should, you know, economics says you should look at marginal impacts. Is, does the ad networks, does the market. Have standard market dynamics, you know, mar increasing marginal cost, decreasing marginal LTV or like what, why, why aren't they looking at the margins?

Speaker 4: 57:50

Some companies do, I, I wrote a piece about this a month ago, optimization models for digital marketing, and I talked about that. Like, what are you optimizing for? Are you optimizing for, so what's your, what's your constraint? Right? I talked, I just presented this as like two different optimization problems. So my constraint is marginal ROAS should be for most people. I don't wanna, the average is irrelevant to me because there's gonna be a lot, you know, if I just looked at that on a monthly basis, I'd probably wasted a lot of money. Um, so, so that should be the constraint. It is just that marginal roas, that cap. And then, you know, so if, so for a lot of companies, if you go to. A lot of mobile companies, and this is probably more true back in like the, the, the, the golden era of, of free to play gaming, which is, you know, free to play mobile gaming, which is probably like 2014 to 20 17, 20 18 maybe, where you'd go to a studio and say, we have no budget. Like budget is we could, we could, we could, we could go to our VCs and get$50 million tomorrow if we could show a model that says we'd be spending that profitably. Right. And so when, when you are in that position, and no one's truly in that position because you can't spend 10 billion this month, right? Like, you know, you're, there's, there's other constraints there, but like, you know, if you'd say you'd put a soft cap on that like. I could probably, probably spend like a up to 20, a hundred million dollars in a month without like absolutely destroying my unit economics, right? For some, some game, right? Maybe. Okay, well then, then, then you're not concerned about, there's no budget constraint, right? Then the constraint is the marginal row as that's what you really, that's, that's, that's what you really care about, right? And then there is, now it's, it's more common to see people say, wow, well we are budget constrained. We're gonna spend this much, and, you know, we understand that we're leaving money on the table. But that's fine. Like, so, so then, then you, you're just looking at it, it's a different optimization problem. The constraint is the budget, right? And then you're just trying to max out roas. You might have a ROAS target, which is one 20, but the budget constraint results in, and so then what are you trying to do? Well, you're just trying to max roas across all the channels, and so you're trying to distribute money such that that happens, right? So like I, it might turn out that I have like 150% blended ROAS across the board this month, but because I was budget constrained, I couldn't spend more than that. And usually you'd want that to be one 20 at the margin for the next dollar that was spent. It would be one 19.9 or something so that you that, that you hit the marginal constraint. In any case, like the budget constraint or the, the, the hard ROAS constraint that's driven by marginal ROAS on average, average is almost irrelevant.

Speaker 5: 1:00:00

But are people actually looking at and measuring marginal roas or is'cause when I've looked at UA reports and I'm not super deep in it, it it's always spitting back averages.

Speaker 4: 1:00:08

No, I mean, so again, there's institutional momentum and UA is weird because I started this like website a while back called Qumar and I had this idea that like if you look at the sums of money being spent by some UA teams, I mean it just rivals capital under management. Assets under management to some, like some small hedge funds. Why aren't uua people being paid? Like, why are they not being compensated like that? Like hedge funds, you know, like, like a PM at a hedge fund. Like, or a, a pm at a, like, you know, a, a performance manager within a pod, at a hedge fund. They might spend, they might have less capital under management assets under management than what, like a Uua Uua team for a really scaled game spent in a month or certainly in a year. Right? And so, like, why aren't they compensated like that? Well, they don't have the same skillset. That's why they're not competing for those jobs. You're not competing with the hedge fund for those people. It's a different group of people that's coming into it. And so I think, you know, I, I'll tell this to people, like, yeah, people don't really look at marginal, like, how is that possible? That sounds totally inefficient. You can't, you can't be correct that, that te you know, that would've been solved for already. No, it hasn't. Most people look at average and it's like, yeah, you understand that. If that average came in right at the ROAS target, you probably wasted a lot of money that month. And people will say, what do you mean? I don't understand. And, and then you start talking about average. I mean, it's just it, but it is, it is, it can be shocking in some cases, but some, some teams do.

Speaker: 1:01:27

We talk about the free to play model. It's been relevant towards games, not towards other apps. By and large. Why does this model emerge for games and not for other ad products or other app products?

Speaker 4: 1:01:39

Yeah, that's a great question. I think there's a couple reasons. One is like, I mean, freemium certainly is, is utilized by a lot of other product, uh, product categories.

Speaker: 1:01:45

Shareware though, right? It, you know, one time purchase stuff like not true LTV building,

Speaker 4: 1:01:51

right? Right. And, and so in the, in the, in the book, I mean the three examples I give at the outset are Candy Crush. Mm-hmm. That a game. Um, and then Spotify and Skype, right. Which Skype's implementation of free to play. So I was working at Skype. Um, like right at a, at a grad school and, you know, that's when I sort of caught the free to play bug or the freemium bug'cause they were implementing it. But I thought in just a very superficial way, really, it was just free to use and you could buy a phone number. So it, it, and, and I think it was a subscription, so I think you paid every month, but like, you'd have a phone number, so if people called the phone number, it would ring your Skype. So if you're on your phone and you're on wifi or whatever, like this was before kind of pre smartphones. More typically you'd be on your desktop. Um, and, and so then I looked across, you know, the Baltic Sea and I saw these Finn gaming companies doing really cool stuff with Freeplay, and that's how I got into gaming. But why does it work better for games? I think, well, gaming is a total sandbox. You can make a game be whatever you want. I mean, you're not bound by the laws of physics. You're not bound by, you know, any sort of like, you know, constraints related to like realism or whatever. You could have a game about zombies, they don't exist. You could have a game about time travel that doesn't exist, right? And so you can make a game whatever you want, and you can monetize it through whatever means you want and selling whatever you want. You can make whatever consumable you can make, whatever cosmetic item you want. You could just conjure it up and sell it and put it right there. And so you can make that catalog of things to purchase much more diverse and broad and therefore relevant to different groups, right? And you take that to its logical extreme. You could make products specifically for a person now using generative tools based on what you know about them. And I think that's what's really so exciting about the shift to web stores. But that's like maybe an aside. So I think that's just the reason. I mean, with Spotify, what are you gonna do? I mean, you can think of ways like to bundle access.

Speaker: 1:03:33

If think, think about the transition from selling individual songs, right? Would be more consistent with an MTX model or like an IEP model versus the subscription all you can eat Spotify model. Like even if we just take that comparison, that felt very value destructive to me, moving from the single song purchase to the subscription. But I think about the peril story in games like Game Pass. Game pass would've had been the same narrative if it had taken off, but it failed. So there, there seems to be a difference, something fundamental between those two things when I think about those two models.

Speaker 4: 1:04:00

But I think, um, so I mean, that's iTunes, right? I mean, iTunes, you can buy songs. It's like, why did, why did Spotify succeed where iTunes failed? I think it's just that ease of use, right? Like I, so, and, and first of all, you need a free, you need, you need the, the freemium, the free to play. I mean, there's gotta be something that's free. So it can't be, you just get into the software and everything costs money to, that's what iTunes is. So like, then what's the, what's the, what's free and what's not? Now, now you could think about like, with Spotify, maybe everything's free, but there's ads. Well, that's freemium in a sense, right? And, and that exists. They have that tier. Or you could say, well, everything's free, but if you wanna build a playlist to, to pay money to buy the playlist, ability to do that. And I think, like, I'm sure they, and maybe they modeled some of this out, actually, they probably didn't, they probably just said subscriptions are really nice because it's like reliable recurring revenue. That was the business model and, and I think if someone tried to come in and say, Hey, well let's actually play around with some free to play concepts, they just wouldn't, they, they wouldn't be allowed to do that because you can't really. Disrupt the business model in that way.

Speaker: 1:04:57

But even when I think about Twitch, we get, we can think about other, other platforms, but when I think about people that have tried to make more MTX based models leveling up servers or buying character cosmetics for Twitch, something, something along those lines, it still feels like the game MTX IP model has actually failed these other paradigms. Like shouldn't, if it was successful, why, why isn't it, you know, like a classic game theory model where it's reproducing and it's taking over? Where's the imperialism from the model? If it's as successful as, I mean, I think, I think it's, I think it should. I'm just, I've been stunned. It hasn't taken over more things.

Speaker 4: 1:05:28

I, so I think, so let me go back to the Spotify example though, because there's another, there's another, another fundamental impediment there, which is that licensing costs. So you can't give away too much music for free because you, you've gotta recover the licensing costs, right? So that's an issue. It's not, it's not. So one of the, one of the, one of the sort of fundamental properties of, of free to play is that there's no, there's zero marginal cost of distribution. So you have to be able to just sell, replicate the software and sell it. And then with Spotify, that's not true because if you, every time you replicate every, someone downloads the Spotify app, they're, they're sort of accessing this pool of content that they, you have to pay for as a result of them consuming it. Right? Which is not true with gaming. I mean, maybe it very tiny amounts based on like server costs or whatever. But so I think that's another issue. But I, I, I still do think it's, with games you can conjure up whatever you want. I mean, it, there's no constraints, there's no, there's no limitations there. What, what you could present to be purchased by the user in terms of like, you know, and, and, you know, you go back to like the typical implementation and there's the soft currency, hard currency paradigm. But even then, I mean, whatever you're buying after that, you could just be whatever you want. It could be skulls, it could be crystal dust, it could be whatever you want. Whereas when you're selling some thing, you're, you're, you're, you're limited by the appeal of the thing. But like, I can make anything in a free to play game and I can give you any amount I want. I can price that however I want. It is just a total blank canvas. And I think that's probably why it worked in free. And, and then the other thing is the scale, right? So now you say, well, okay, but then why didn't that work on console? Because the console scale of the player base is on console. And even PC is much smaller than what could be on, on mobile. And mobile is everyone. But on PC console it's actually pretty limited. And there's, there's the hardware hurdle, right? And whereas you've got a hardware hurdle with phone, but not. Not for the gaming use case.

Speaker: 1:07:04

Everyone has a phone. So if I, if I just think about social platforms, so we think about like Discord, which just try to mo monetize platform economics, uh, and Twitch like cosmetics. But then I think about like TikTok, Facebook, Twitter, the blue check mark, the fabled blue check mark. It feels strange to me. There's no monetization tentacles into what is essentially a status game. If you look at that as a game economist, it just feels baffling to me that there isn't deeper tentacles in those things. Or even when I look at like the Twitch and Discord examples, like that hasn't been enough to deliver them to the promised land. It has to be advertising that gets them there. Yeah. Like they're barely break even in a lot of these, these paradigms. And I just, I'm surprised by that and that I get, you know, is there, is there something we're missing?'cause I feel like you can do those things on social platforms, right? We can play with status, we can do server upgrades or cosmetics.

Speaker 4: 1:07:48

Yeah. So it's not just status though. The blue check also impacts your distribution now. Right. So like, I mean, I wrote about this, it was like, I, I can't remember the name I gave it, but it was like, there was a, I invoked some concept from like John Locke. But, but, but anyways, the idea that we, we always agreed with social media. My distribution is a function of the ideas that I disseminate. It's, it's not limited by what I'm paying or not. Like, and, and that was broken. I think it was, I called it, I don't think I called it a social contract, but anyway, but that was, and that was, that was broken. And I made the point that was broken'cause of, at t because you couldn't do that, you couldn't advertise to people in the same way anymore. So you, a lot of these people lost rpu. And so you just kinda had to break that con contract and say, well, you want mass distribution now, you gotta pay for it. Uh, and so there's more than just status with the blue checks a lot of the time. But yeah, I don't know. I mean, I, I, what could Twitch offer?'cause you're, you're constrained then by just the product. I mean, what could they offer? I mean, they, because it's cosmetics, eh, you don't really make that much money. It's gotta be something that's fundamental to the experience. Like, is it minutes consumed? Because then you're talking about metering that's not really the same, you know? And so like, where, where, how do you, how do you augment the experience? Because the whole, and another fundamental principle of free to play and freemium is that what you wanna do is you wanna match the value that's that a user perceives from something with the amount of money that you're charging them for it, right? Such that, like, I could just scale that up endlessly. If I could figure out a way to deliver more value. That's way that's whale behaved, right? But how do you do that with Twitch? I mean, it's, it's just like, there's only, how, how do you make that a more valuable experience? Because a lot of it's just like cosmetics or tipping or stuff like that. It's not, it's not making it fundamentally more valuable to me as a, as a consumer, as a user.

Speaker 5: 1:09:25

Uh, this idea here about like, kind of like game genre UA fatalism, where like a lot of people will say stuff like, okay, so ROAS is what dictates what games gets to market and who can scale. And games that just cannot compete on ROAS are doomed to fail. And so one example, for example, is like, if Tetris launch today, they'd be competing with Candy Crush. Candy Crush has much higher LTV, they're marketing towards a similar audience.'cause they're both kind of colorful abstract puzzle games. And Tetris would just never succeed because they would get outbid by Candy Crush every time and fail that basically ROAS dictates what games are successful or not. Uh, I dunno, how much do you buy into this thesis? Like I, I, I see it a lot when people try to justify why their game failed, but um,

Speaker 4: 1:10:08

yeah. Yeah, that just sounds like a post hoc justification to me. I mean, there's not, it's not, there's no ROAS determinism or Yeah, it's ROAS determinism. It's not, it's not LTV determinism because you could have 150% ROAS on a 7 cent LTV. If, if you're buying installs at 5 cents. And so like, so then, then the question is like, well, why are you chasing this, uh, competitive market and, and scale too? It's like if you can appeal. Better to some, even if it's a very competitive space, even it's very competitive segment based on the audience that's being pursued. If you can appeal better, your clickthroughs are gonna be, lower costs are gonna be lower. You could have lower LTV, you could have higher roas. So it's like, I think it's, it's, yeah. I think when people talk about like, oh, well, we couldn't compete with the CPIs, that candy crushes pink. That's nonsense. That doesn't make any sense. The CPIs are not what you're competing against. You're competing against the relevance, you're competing against the, uh, willingness to click. You're competing against the audience size. You're competing against your own ability to monetize. So you're, you're saying the CPIs are higher. It sounds like you suck at monetizing. It's not an, it's not a, it's not an advertising problem. It's a you problem. It's a, it's a monetization problem and,

Speaker: 1:11:13

and, or it's an audience problem. Do you buy into predatory pricing? I mean, I, I know this is always a, a critique that comes up in certain markets, but I, I hear it rarely brought up in mobile advertising that people are just basically buying break even or neutral traffic to suck up the market. Right. There's a, there's almost like a crowding out effect. Do you think there's anything to this or it's all just hogwash?

Speaker 4: 1:11:31

I don't think anybody is, I don't know. I don't, I don't think anybody is deliberately crowding people out or, you know, I mean, so like, in order so to, in order, in order to do that, you'd have to believe that you'd actually kill off the competition, but you wouldn't. Mm-hmm. I mean, there's. You just get more people lining up. I mean, if once you, you might kill off some existing set of companies by crowding them out, but then, you know, they stop competing and the ad prices drop and then more people enter.'cause it's an attractive space. So like, it's not, I don't, I don't know that you could permanently kill off the competition because it remember like the, there's, there's very, there's almost no hurdle to make a game, especially now, right? I mean, you could have a three people get carved out of a studio to go make some game because you perceive there to be an opportunity and from Yeah. It's not

Speaker 5: 1:12:15

like railroads or oil refineries. Right, right.

Speaker 4: 1:12:18

Exactly. Yeah.

Speaker 3: 1:12:19

You're talking about KPIs a lot and obviously you're an investor. I'm curious, two questions. The first one, if you had to pick a KPI that you were judging a, a company based off of which one would it be, and then is that something that's even a part of your investment thesis? How do you make decisions in, in terms of what's, uh, you know, investment in 2025?

Speaker 4: 1:12:40

Well, so I invest company formation stage, right? So there's never any data. Except, except what exceptional things has the founding team done? Ideally. Ideally, together. And so I think, you know, building a successful startup, I mean, that's, that's incredibly challenging to do. It's, it's, it's an exceptional feat. You're, you're telling me that you endeavor to do something absolutely exceptional. Yep. And so, well, what's the track record of doing exceptional things? And like, so this, you know, and this maybe is, um, controversial, but there's any number of exceptional things. Like exceptional things are not, they're not, you know, it, it, it need not only pertain to like how much money you've made at a company or whatever have you, did you get into a really good school? That's an exceptional thing to do that I would count that he said with the Harvard flag behind him. I mean, okay, maybe that's elitist, but that's a signal. And now if you, if you went to an accept, but, but, but it's also like the difference of, like, did you, did you go to some, you know, impressive Ivy League school because you went to Phillips Exeter, or did you get into some impressive Ivy League school going to the high school that I went to, that, you know. So and so that would be, except that's an exceptional thing to do. I'm, and I'm just making the point that it's a, it, there's a, there's a sort of like broad set of things that you could do that I would consider to be exceptional. But the thing is like, did you work as employee number, you know, 17,000 at King, is that truly exceptional? I mean, if you, if you, yeah, we launched Candy Crushed. Great. That's exceptional. Now I joined Candy Crush, you know, 12 years after it was launched and I worked on something. I was, well, okay, so, so, so the thing is like, well, no, and, and you can say, well, hey, like, what a, you know. And, and people, this is like a, just, just flawed logic, but people, then you're gonna miss the next Mark Zuckerberg. No, he wouldn't. He, I mean, well, he, he did exit it or Harvard, so maybe, I don't think that's exceptional. Okay, fine. But like, you mean the once in a lifetime world historical figure, maybe I would miss that. Yeah. Based on this rubric. But the thing is like, well, and then you're saying, well, hey, I never had the opportunity to exceptional things. Okay, well then now, now you do right now, right. The second you can go and join some company and like launch the, I just think like there's just things I need to see that you've accomplished because the big, the hardest thing to do is to launch a game and, and, and just, just go be this, you know, how hard launching a game is to getting from, getting from nothing to the point where you have an MVP and pushing that out. You know, how many game companies have seen die? Where they had something that was kind of playable. They were just, they were just too embarrassed or, or they were just, they were just, they were just too paralyzed by anxiety or, or just analysis paralysis to actually launch it and start iterating on it. I need to assume that you've done something, that you've completed something from start to finish. I'm not making the point that you have to have a degree. That was not the point I was trying to make. I, I hope I'm, I hope that comes across, but that's just a wide variety of examples that you've start. If you yourself have done this thing that was improbable, and if you've done that, then like, I think like where people make mistake, you know, they, they, and they might get funded. I mean, certainly this was the case, like 2021, like companies were spinning out of these big companies and they were raising a lot of money, and you know, it's because that, that, that company was impressive. But with the things that they didn't, that they did, there were maybe not that impressive. And like, that's not really possible to do that anymore. But like, yeah, that's what I wanna see. That's the KPI, it's like, what, and ideally as a team, this, this group as a team accomplished something exceptional. And therefore I believe that they have the capacity to do that. But if it's, you know, if it's just a kind of a string of jobs and you were just sort of like you brought, you came in and it was already a successful product and you scaled a little bit, that to me, that, that, that, that doesn't communicate the ability to do zero to one. Like what you're gonna have to do, especially if you're launching a game, but, but really with any startup.

Speaker 3: 1:16:10

So failures are for you, a bad signal?

Speaker 4: 1:16:13

I don't, I don't wanna say on their own, they're a bad signal. I would say if, if a person doesn't have any successes, it's probably not. And, and that's not to say that that person's like a loser or something, or, you know, there's something wrong with them. It's, yeah. My, my, my, what I would say to them is like, go start a product in a bigger company, you have the chance, like you're, you can do that right this second, like, and then talk about starting your own company. But like, you know, like it to me that it's, I'm just not seeing the, the, the, the, the, the track record of having completed something. But it's not to say you're, it's, it's, it's not to say that you can't do that. I'm not saying it's, I'm not saying you're incapable of doing it. I'm saying I don't, I don't see any proof that you can do it. And those are two very different things, right? Yeah. And if I don't see any proof that you can do it, how do I take the risk? Like, show me proof that you can do it and then I'll feel comfortable with that risk.

Speaker 3: 1:16:58

No, just slightly, uh, against this narrative that I see. It's like failure, you know, is, is amazing. Like fail, fail, fail, uh, which like, yes, you have to fail in order to succeed, but I feel like it trivializes almost, it doesn't encourage failure, but it's like, oh, failure is whatever. It's a part of the process. It is, but it's not what we should strive for.

Speaker 4: 1:17:17

I would say like some history. I mean, I failed a lot of stuff. Failure is not disqualifying, but it's also not a proof point that you're capable of doing exceptional things and so therefore I just kind of ignore it. Is Web3 the sub-context

Speaker: 1:17:27

here, Chris? No, no sub-context, no sub-context. I keep getting money. Chris. Um, our guest today has been Eric Suer. Um, you can find him my mobile dev memo. Uh, you're on Twitter, uh, you're on Ben Thompson's podcast. I love, I love listening to you there, by the way. Uh, I hope, I hope you get a regular seat. Sharp Ad Tech would sound nice. Is there any other way, uh, people get in touch with you for investments? You can just, just

Speaker 4: 1:17:47

reach out. I'm on, I'm on LinkedIn a lot. LinkedIn and, and X and Threads are like my primary channels. Just reach out. I mean, always happy to chat with people. Uh,

Speaker: 1:17:55

thank you so much for being here. We appreciate it. Yeah. Take care guys. It was fun. We should teach this to our children. Economics is major, major, major. Everyone has to major in economics, number one for personal survival. Economics is major.