An N of 1 with Peter Judodihardjo

Peter Judodihardjo is a Behavioural Scientist and YouTuber from London. Pete has received millions of views on his videos exposing academic fraudsters and addressing flaws in the research process. During the day, Pete works as a Senior Auditor of Behavioral Risk at one of the UK's largest banks.

🔹 Follow us on LinkedIn: besci-io

🔸 Join our BeSci community on Slack: besciio-public.slack.com

♦️ Enhance your career and knowledge of applied behavioral science with our courses: learnappliedbesci.com

Transcript

 I'm so excited to be joined by, Pete today.

Um,  like, you know, you have been, uh, really evangelizing behavioral science, which I love. Um, and this is going to be kind of a unique perspective today because you and I are separated by a few years and a few strands of hair. And so, you know, uh, many of the people that we've interviewed in the past  Behavioral science really wasn't a mature, applied behavioral science wasn't a mature discipline when they started.

And so, we've heard a lot of stories of how people sort of find their way to this. And you, I think, really came about right at the, kind of the perfect time to just be the sort of starting piece. So first, I know,  People call you Pete Judo, like, do you want to introduce yourself and say that last name that I, uh,  Yeah, so my real, my real name is Peter Judo Deharjo, and I shortened it to Pete Judo on YouTube for SEO reasons more than anything else.

Um,  but yeah, my real name is Peter Judo Deharjo, you can call me Pete or Peter, I don't, I don't mind.  And, uh, yeah, I've been, yeah, I've been, I've been  into behavioral science, uh, since about the age of 15 or 16, and I'm 25 now. So it's almost a decade of my life and a big, big chunk of it, so very, very happy to talk about it.

Yeah. And so talk to me about this, because I know, let's start maybe even at the high school level. So wait, so you're 15 and someone gives you a book, you're 15 and you see it, like, what's the sort of Yeah. Germ that, and we should set this off, right? So this is being recorded such that, you know, you were 15 in what year? 

So I'm 25 now.  Um, so 15, I was nine years ago. Nine, 10 years ago. So this is like 2014. 2015, yeah. Um, and  yeah, so what kicked it off was.  My brother went to university, my older brother, and I was home alone with nothing to do. And so I decided to just watch, uh, a ton of TED Talks. Because I was a nerd, and that's what nerds do.

Um, so I was watching loads of TED Talks on everything from architecture to beauty to all kinds of stuff. And then one of the TED Talks I stumbled across is Dan Ariely's TED Talk.  Um, Are we really in charge of the decisions we make? I think it's called. And it just blew my mind, right? And like he was talking about the research that he was doing.

He talked about the decoy effect. Um, and I just thought this is like the coolest thing in the world. And I, I instantly became obsessed, uh, with his research and his work initially. And then I found, um, Rory Sutherland's TED talk. And Rory, uh, if you don't know Rory, he's, he's the, the co founder of, uh, Ogilvy Change or Ogilvy's Behavioural Science Practice.

And he's a real evangelist of applying behavioural science in business. And  I became fascinated with him as well and his way of thinking and how he used behavioural science as a tool for innovation and product development. And so I thought, That's really what I want to do with my life. Like it seems fine.

It kind of blended my love of creativity and science together Um and business as well And I thought like this is just the perfect field for me to get into so I really became obsessed with it  From that point. This is me age 16  and  And I had a friend of mine who was incredibly intelligent. And I said, do you know anything about this behavioral science stuff?

And he was like, Oh, I heard there's this book called thinking fast and slow that everybody talks about. Maybe you should give that a read.  That's what I did. So that summer I read all of thinking fast and slow. Um, and that was my initiation point. Uh, to the field and it was really off from there. Um, from that point, I know you did your A levels in, in math and econ.

Is that because of a love of behavioral science or that was sort of already the plan and then, and it's kind of bit with that.  So yeah, that was because of, I knew I wanted to do behavioral economics or behavioral science. And so I told my dad, Hey dad, I'm going to study psychology.  And he was like, no, you don't make any money in psychology.

So he said, do economics  and, and math. So that's what I did. And he was like, you know, then if the psychology thing doesn't work out, you can go and work in the bank,  which is funny. Cause we'll get onto why later. Um,  so, so that's what I did. I ended up loving economics and I loved, uh, and I was pretty good at maths, but then when I got to university,  um, I didn't get into the economics course that I wanted.

And then I was like, well, I was, I called up the university and I was like,  have you got anything else? Like, I'm desperate here. You got anything else for me? And they said,  well, we have space on the psychology course. And I was like, perfect.  Absolutely! Hey dad! Sign me up! I could fit into the econ class, but they have The universe I tried.

I tried. And, and, but wait, before we get there, you went to Warwick.  Was that like, because, you know, they have a fantastic applied behavioral science program, right? Yeah. Known for it. That  why you chose them? Is it, you know, sort of, were there other considerations? Like how did you sort of, you know, bridge over into work from your, from your A levels?

It was so, so just like at any, any kid going to university, I, I went to open days and lots of different unis and at first I thought I was going to Oxford. So I went to Oxford and I talked to the econ professor there and I was like, do you do any behavioral economics?  At Oxford. And he goes,  no, we study real economics here. 

And I was like, wow. Okay. Oxford thing to say in the world. Right now. Now I've heard, I've heard since then that there are, there are a bit more receptive these days, but at least back then, you know, it was, it was a very clear signal that like, this is not the place for me.  And then when I went to the open day at Warwick university, it was a completely different energy.

And they were really forward thinking the professors that were talking about Kahneman. They were talking about, you know, system one, system two thinking. And I was like, Perfect. Like, this is The place for me to be where people are really embracing these ideas and not regarding them. It's not real Economics, right?

And so I apply I I did I did something very stupid which I don't recommend which is I only applied for one university  Um work or bust  Yeah, I was all in yeah, I I kind of have that attitude to life, um, and uh  And yeah, and then, and then yeah, through this roundabout way, I ended up on the psychology course instead, and it turned out to be the best, um, accident, because I reckon if I did straight economics, I would have dropped out.

Like I would have been so bored by just all the econometrics and all the mathematics of it. And with psychology, I really  found myself like, I really, I loved psychology. It's so interesting. You know, everything from, from the neuroscience to the mental health stuff. And then of course, all of the social psychology stuff, which is so closely related to behavioral science.

And I, yeah, I fell in love with it. And so I'm really glad actually that I ended up on the psychology course and not the economics course. And, um, and it was while at Warwick.  That I actually, um, weaseled my way into the behavioral science team, uh, that was at that university. Should we, should we talk about that?

Yeah, of course. Like, anytime someone uses the word weasel. So we've already heard one thing that we hear a lot, which is applied behavioral scientists are stubborn bastards.  That that's sort of like, we're only going to apply for one school. We're only going to, you know, there is a sort of.  single mindedness, and I, it's hard to tell if that's associated with applied behavioral science or if it's just the nascentness of the field and the people that survive in nascent fields are people who are sort of  gritty, right?

Yeah. So this definitely is a theme that emerges. The word weasel is also a theme that emerges, meaning, you know, people find ways to, when you're in a scarce environment, people find ways to navigate these paths to sort of get themselves into unlikely places. So you're like, you know, a freshie, uh, here at, uh, here at Warwick, and like,  Sort of, did you, before you came, did you know there was a, was a Behavioural Insights team there? 

I didn't, uh, and it, I only found out because they run, they run this event every year called the Behavioural Science Summit. It's like a, it's like a conference, basically, where they invite a bunch of speakers to come. And then I just saw it advertised on campus and I was like, this sounds amazing, let me go.

And so I went at the end, like you say, I was a freshie, end of my first year. And, Uh, and then at the end I like, I ran up to the organizer and I was like, how do I join your team that organizes this thing? He was like, well, we're all like post grads here. Like we're all like masters and PhD students and you're like some kid.

But, uh,  you know, but like, you know, if you're, if you're eager, like come on down. But you're handsome and eager and we will be receptive to that.  Yeah, I mean, I think they were, I think they were looking for, for younger people, actually, and they were finding it harder to reach sort of the undergraduate audience.

And so they, they were glad to have me on. And so I worked my way on. And like you say, like,  you know, behavioral science was a little bit more established when I was younger, but it still wasn't like that. established. And so I was trying to figure things out, just trying to find anyone in the industry. And it turned out to be a great decision to join that team because I stayed in that team until, until the end of my time at Warwick.

And, uh, by, by my third year in the team, I was leading it.  Um, and I was organizing the summit that I went to in that first year. Um,  yeah. And I met, I met a lot of good friends. I met Merla who I think you've had on this by a side chat before. She's a good friend of mine now. And we met in that team at Warwick.

Yeah.  Yeah, I mean at work is a,  uh,  as an American as a yank, like not a school I knew anything about, but like has really become, you know, look, the LSE program is obviously a wonderful program. I, you know, I'm not saying work is the only program in the UK, but like, it really has emerged. As like a very, very, very strong applied behavioral science.

You know, if we look at the behavioral science slack, you know, there's tons of people there that are either at work or came through the work program. Yeah. Um,  and it's an interesting sort of case study in like,  you know, distinguishing yourself in a thing that really nobody else is paying it. You know, they really sort of snuck a march on there and I'd be interested.

I wonder how they sort of got there. Right. Was it like one professor who sort of recognized this thing or?  I think, I think it's actually the newer nature of Warwick because, you know, there's a lot of very old universities. in the UK, right, Oxford, Cambridge, obviously really old. Um, but then, you know, Warwick is a newer university and also a very high performing university, like consistently in the top 10 universities in the UK.

And so it was uniquely placed as like, you know, we're kind of, we're, we're very high performing, but we think we're a lot more forward thinking than these old kind of stuff, universities that have all these sort of legacy, um, you know, ways of working. And so. I think that kind of culture really allowed for behavioral science, which was this kind of like wacky, weird, tangential field at the time, to thrive in a place like that, where people were willing to embrace new ideas and new ways of thinking and challenging the old system.

And that's kind of, that's really how behavioral economics was in the beginning, right? It was all about, this is what classic economics says, this is what behavioral economics says, and like, not many universities are willing to embrace that way of thinking. Warwick was a really good place to do that. Yeah, I mean it really has produced some fantastic folks like yourselves and others Um, so you're on the bit team you sort of weasel way you're on there This is also, you know, you start working for wendy wood as like a social media consultant around this time, too Was social media a long standing interest for you?

It came up as part of  Powerful behavior change tool. Like where did that, that, cause you know, you are unique, uh, your, uh, dedication to science communication, right? I mean, I do some, I do these talks, right? We do some science communication, but you know, I spend a huge amount of my time, you know, focused on political science.

You, you really have.  Embrace the community. Where did that sort of,  is that echoing here at Warwick? That comes in later? Yeah, so that really started out in that BIT team, because what I found was that I loved teaching other people about behavioral science. And so,  as part of my role in that team, I was, basically acting as a behavioral science evangelist on campus, just like hosting all of these talks, introducing students, staff members to this field of behavioral science.

Um, I would bring it into like every project I had, uh, during my degree, uh, as much as even when it was like, they were like, okay, that seems a bit random, but like, I guess, I guess it's relevant. Um, so, so yeah, so, you know, that's, I found that I just loved. teaching behavioral science.  And then what happened was that in, you know, in my, in my final year of my undergrad degree, I was organizing this big conference and we had everything set up, all the speakers lined up, including the headline speaker of Rory Sutherland,  right?

The guy who got me into the field in the beginning was lined up to be the headline speaker that year and COVID happens,  right? It's 2020  and everything goes into lockdown and it's a bit hard to hold a conference in a lockdown. Um,  And so we were gutted. And basically what happened was that, uh, we had to tell all the speakers, we're, we're, we're canceling the in person conference.

If you can send us a talk, we'll upload it to our YouTube channel and kind of have sort of a conference on YouTube. Um, but yeah, it was definitely not, not a great feeling. And then I reached out to Rory  to do the same, fully expecting him to say no. Right. Like as our headline speaker, busy guy. Like, there's no way that he says yes to recording some silly YouTube video for us.

Um,  and he said yes,  which was unbelievable. He's a really top guy, he's so generous.  So he recorded that talk for us, and then while I was sort of coordinating with him to get that off him,  he wanted to have a call with me.  Which was crazy for me, right? You imagine as this guy who I watched the TED talk of so many years ago suddenly wants to have a call with me.

Um,  so I have a call with him. I'm a nervous wreck. Uh, and he says, you know, he says, it's really great that you're uploading all these talks onto YouTube. He said, there's actually not a lot of behavioral science YouTube content out there. Um, you know, here at Ogilvy, we host this thing called nudge stock and that's.

Like a good amount of content that we upload every year, but it's really not that much out there. Well, and it's one big, you know, Nudgestock is a ton of talks, but it's, you know, 24 hour. You know, it's sort of, it's not a series and you, and Warrior's right. I mean, I think there's a couple of things that come up in there.

One, you know, I love behavioral sciences open community. That's a theme that we hear time and time again, you know, experiences just like yours where people are like, there's no way this person is ever gonna, you know, this thing is ever going to happen. But you know, in the behavioral science community, it kind of does right to your point, you know, Rory, who I know very well, right?

He is going to hop on a call with a young person, he is going to upload that photo, he is going to do that thing, you know, I have open office hours, so that anybody in the world, you know, can take 30 minutes with me. And, you know, it is, it is amazing the people who sort of show up and are like, I can't believe you're willing to talk to me.

And, you know, I think a lot of behavioral scientists are very generous with their time. And I think it's a generous thing to create content. I mean, I think, you know, the work that you've done, and we're going to talk about the pivot and the work that you've done somewhere in here.  Because that is important.

But, uh, uh,  I, I really do think that that is that willingness to take a chance both on, on your end, right? Hey, I'm willing to talk to, you know, I'm willing to reach out. I'm going to make that ask. I'm going to make what feels to me like a big ask. And then the willingness of the receiving person to sort of do it.

I always remember how I ended up at my first startup. Um,  uh, you know, in the tradition of back at this is 2000, like seven, eight. So this is like really early days, you know, academics just have their phone number on their, like, you know, academic website, you know, you can call their office. And so this startup called Dick Thaler and was like, Asking him for advice about their startup.

And he talked to him for like an hour. I think this was probably pre Nobel prize. They then asked if he would join the startup and they were like, he was like, absolutely not. But you can, Matt Waller guy. Right. So. You know, they found me through Dick Taylor, but even the fact that Dick Taylor, like future Nobel prize winner was willing to like, get on the phone with them and like have a conversation is a very behavioral science community.

Sort of thing that we do here.  Yeah. Yeah, certainly. And I, and yeah, like you say, it's a, it's a sentiment that's been echoed across lots of my colleagues in this field where, you know, It's just such a, people are so generous in this field and it's something that I've tried to, to carry on myself now, now that I have somewhat of a profile online, like, people reach out to me, I still try and make time to, to meet people whenever I can.

Yeah, and come on shows like this one, which we appreciate.  So, so you're starting to do some social media for, for Wendy Wood, you're starting to create your own content, right? So Rory kind of brings up YouTube and you go, hey, there's a niche here, right? This, this is starting to be a thing for you.  Exactly.

Yeah. So, so it's locked down. I can't get a job. So I'm like, fine. I'll, I'll, uh, I'll, I'll start a YouTube channel and I'll do my master's at the same time remotely. Let's talk about that. I mean, cause you went right undergrad to master's, right? No break. You were right. Break. Yeah. The plan was to take a break initially, but then the pandemic, nobody's hiring.

Might as well upscale, right? That was like the good advice that everybody had. So. So that was, so you didn't come into undergrad being like, Hey, I'm going to go get my master's immediately. It just sort of worked out because of the pandemic that way. Exactly. Yeah. And I, I did, I mean, I had applied previous to the pandemic happening.

Um, and then Warwick had a conditional offer for me and then the pandemic hit and they were like, it's not a conditional, just come,  just come if you want to do it now. Yeah, exactly. So, yeah, I mean,  so yeah, it was a great experience to do that. Actually. It was a great use of that time being locked inside to just make YouTube content and really, um, yeah, I knew a little bit of video editing from, from like stuff I've messed around with in the past and obviously I've been teaching behavioral science at Warwick in person for, for a couple of years already.

So I was used to communicating behavioral science. I knew what methods of communicating behavioral science really resonated with people, at least with a live audience. And it was just about trying to translate those skills that I learned From the live audience and the editing and filming skills that I learned as a hobby and turning that into this YouTube channel.

And so that's really where it started. And um, how I came into contact with Wendy Wood was that  my second ever video on the channel was about Wendy Wood's book, Good Habits, Bad Habits. Uh, it's a book which I, which I love and I still recommend to everybody today. And um, yeah, I was, I was talking about one little part of the book where she's talking about the marshmallow test. 

Um, you guys know the marshmallow test, right? The kids have to resist the marshmallow. If they don't,  then they're You know what I mean? Yeah, okay. So I'm talking about this, uh, this famous experiment. And then, I just tweet it. I tweet it at Wendy.  And she sees the tweet, and she reads it. Uh, and then she reaches out. 

Which I never expected. And then she was like, hey, you know, I'm looking for someone to  help me with my social media stuff. Do you know anyone?  And I was like, yeah, you're like, yes, I do.  Yeah, I'm, I'm more than happy to help you.  And it turned out to be  the best, uh, the best possible opportunity to come my way to work for Wendy, because essentially what she wanted me to do was to write three social media posts for her every week.

And she's very, because she's such a great academic, she wanted to make sure that everything that I was tweeting had a citation.  And so Um, even for a tweet.  And so, basically every week, I was reading three papers about habit science.  Um, and writing tweets about them. Right, and it was kind of, it's kind of, the way I think about it is like, have you heard that, that quote from Einstein where he says like, if you can't explain something simply, you don't understand it? 

I think the modern version of that is if you can't condense it into a tweet, you don't understand it,  right? And so I was reading three, reading three papers a week. I had to really understand it to turn them into tweets. Uh, and then by doing that for over a year, I suddenly accumulated this huge amount of, you know, of knowledge of the academic literature around habit science.

Um, and so by the, by the time I graduate from my master's degree, um, and I'm applying for jobs, I'm really positioning myself as like, yes, I'm a young, enthusiastic behavioral scientist, but really I'm a habits expert. So even within this niche field, I had an even. more specific niche within that field, which is habit science.

And that's really something which I only gleaned through working for Wendy Wood, because, because of this exercise. And also because like, every time I would tweet, she would, um, she would read my, my drafts and she would correct my thinking. So I had like the world's leading expert on habits, correcting my thinking on habits on a weekly basis.

And it was just this incredible learning experience, um, which, yeah, which I treasure to this day. And there's a lovely part in there about sort of learning by doing, right? There's a difference between I read three papers a week, every week for a year, and I read three papers every week for a week, and then was forced to summarize them, and then was corrected by the author of the paper about how my summarizations were flawed, right?

Right. Um, you know, I, I, Alana on my team, and I talk about this a lot, you know, she learns a lot of behavioral science just by writing something, and then me being like, Well, let's talk about how this is actually slightly different, you know, because it is those subtleties that make science communication sort of work the way it does, because it, you know, in summarization, it's easy to  generalize to an extent where something could be unintentionally  false,  which is different than intentional falsehood, which we will talk about, right?

But, but it is easy to make something sort of unintentionally false, and it's hard because  You know, all summarization requires at least some generalization, uh, which,  you know, uh, creates  ambiguity is the right word, but, uh, some disclarity in the field for lay people at times.  You're in the pandemic, you're like getting this master's, you're working for Wendy, like the world starts to open up again and you sort of end up at, at Ogilvy, right?

Right. Back, back with Rory and Cogue. Back with Rory.  Yeah. So, you know, Rory and Wendy are two, I regard them as like my two mentors throughout my career. They've really been like monumentally influential over my life. And, um, yeah, so I get to the end of my master's degree, I graduate, Um, at the same time I have this fledgling YouTube channel, but a couple thousand subscribers, uh, and I'm applying for jobs and I apply in a bunch of different places.

And then I reach out to Rory because We never stopped talking since that initial call that I had at the end of my undergraduate degree for over the year. We never stopped talking. So I reach out to Rory and I tell him like, Hey, look, I'm looking for a job soon. Like, is there anything coming up?  And he says, there is.

And so, um, so he says, you know, like there's this job opening, but like, you got to apply like everybody else and it's all anonymized. So I can't promise you anything, but like, you know, go for it.  So I apply, um, and I get it obviously. So, uh, it's, it's yeah. When I get that, it's probably like one of the happiest days of my life, like getting that call because it was really a full circle moment when, you know, I was like that, that 16 year old kid watching that Ted talk, seeing Rory talking about the work that he did at Ogilvy.

And I was like, Whoa, that's like the coolest place in the world to be. Um, and then I get a call on the phone saying like,  you got the job. Uh, it is, is unbelievable, right? It's, it's, it's such a good feeling. And I, uh, I was so grateful to have that opportunity. What other kind of things were you applying, if it wasn't Ogilvy, what, like what other kind of things were you applying for? 

So other consultancies, right. I really wanted to work in as much as possible. I wanted to work in a dedicated behavioral science team, which I'm very lucky to be in the UK where there's a few of those around, not, I know this. I've, I have fans from all over the world and they tell me like, Hey, I'm from Botswana, there's no behavioral science team where I am.

And I'm like, yeah, I don't know what to tell you. Like that's, you know, I'm very lucky. Um, so yeah. So I'm very lucky that I got to work Apply to a bunch of teams. I got through to interview on a few of them Um, but yeah in the end it was Ogilvy as soon as Ogilvy gave me the call. I was not going anywhere  Goodbye the rest of you.

Yeah, um, so  So yeah, so yeah, I got into Ogilvy and and it's it's just it's fantastic Like it's the people that are are amazing like it's a genuinely Such a creative, impressive team that they have over there and, and I learned the pace of learning, like I, I went in kind of cocky thinking like, oh yeah, like, I've got this behavioral science YouTube channel, like, blah, blah, blah.

I know things. I know stuff. I've read books. Uh, and then I go in and I'm like, like, I don't know how to the whole applied side of behavioral science is a whole thing to learn, right? Like, how do you take these concepts that are very academic  and translate them into creative solutions for your clients, especially the consultancy where you have to do that at pace,  right?

You don't have time to write a, uh, a three month, uh, research proposal, right? Like, you know, it's, you got to go straight into it. So, um, it was a fantastic experience. I learned.  You know, a lot about applied behavioral science, but it's all about business in general, um, from working there and, and, uh, and, and really, and really enjoy being part of that team and working directly under Rory, having like, you know, regular contact with Rory,  um, was just an amazing, amazing opportunity. 

Yeah, I think that that, you know, one of the things that comes up a lot is that transition from academia into applied, right? And how different, you know, it is so easy to feel like, well, because I know this science, I must be able to do this thing, right? But, you know, it's like being a physicist trying to be an engineer, right?

Like, just because you understand the physics of bridges doesn't mean you can build a bridge, right? And get, you know, 100 laborers together to like build a bridge. And so much of Building anything in the world, and that's particularly in the business world, is an understanding of the building process, right, which is, to your point, occurs at a very different pace than academia does.

Yeah. You know, um, it's interesting to me how academics sometimes perceive themselves as busy. And then I sort of listen to their schedule and I'm like, and you think that's, like,  you produced three papers this year and you think that's busy? Like,  it's, you know, it's not basic.  But, you know, it, it, it. It sounds like you got a bit of a crash course.

Is there a memorable, I know we're limited always about the things we can talk about with our clients, but is there a memorable thing that you learned in there that you would pass on to people who are making that transition?  Yeah, I think  that sort of the way, the way that Ogilvy thinks is very useful to almost any, any business that you're, you're working in.

And the way that they think is they,  they use behavioral science as a way of joining the dots between solutions in different industries.  So you'll look at something  in one industry and go, that works. And then by understanding the behavioral science behind what they've done over there, you can then look at a completely different industry and go,  I can apply the same thinking  to this field over here.

Right. And, and the way to get good at that practice is to just get good at spotting how behavioral science has been used in the real world, right. In, in, in science. As you're walking around in life, you'll, you'll notice things that people do and you go, Oh, that makes me feel good. That makes me, that makes me remember that experience.

That makes me, um, want to go back to that place. You know, why did that happen? It's because of something in behavioral science and being able to identify that is such an incredibly useful skill and something which I, which I continue to use, you know, today. It is, and it's, uh,  you know, academics by nature sometimes are a little bit separated from their environment and there are benefits to that separation, but certainly in applied, like.

You know, so often when I'm on a call with somebody, I'm like,  think about something you liked. Why did you like it? Like, what happened there? Right? Like, being fascinated by our own behavior and sort of the immediately local behavior to us, I think, is a distinguishing feature of Applied Behavioral Sciences is that sort of like, Desire to unpack why something worked or didn't work as every every time, you know, I have like a  close to 600 day streak on uh on on uh, uh, duolingo as I like I had revived my childhood spanish, uh, And every time they get me to do something i'm like, you know I sort of am fascinated by  how they're getting and failures too, right?

Like where does something hit?  hit off, right? You know, where something doesn't change your behavior, it doesn't feel like it sort of inspires you to do or have the result that people want. What's going on there? How could you redesign it differently?  Yeah. And, and you, you develop as you, as you practice this skill, you kind of develop an  instinctual sense for like what feels like good behavioral design to you and what feels like bad behavioral design.

And you within now I'm, now I'm at a point where like within a first glance at something, I could be like, I can immediately identify like, you know, where are the pain points from a behavioral science point of view in this customer journey, in this customer experience that I'm, that I'm looking at and That's something which I cultivated at Ogilvy. 

Yeah. In that really active period. And then, you know, you moved from Ogilvy to NatWest.  What made that decision? What was that transition like? You know, going from consulting to in house, like there's a lot of changes that happened there. Like, tell me about this, this,  you know, sort of new, new NatWest role.

Because you've been there, Yeah, about a year.  Um, so yeah, so I moved to NatWest. Um,  I was really looking for something that felt a little bit more pro social.  Uh, you know, Ogilvy is a wonderful place to work, uh,  for learning how to apply behavioral science to business. Um, but sometimes, I would like, well, I feel like I'd be using these, these same techniques and skills that I'm learning here and applying them to, to really help people.

And you might not think that working for a bank is particularly pro social. Um, but actually the, the team that I work for, uh, the behavioral risk team headed up by Alex Chesterfield is,  Is fantastically pro social in its focus and we really have a strong focus on improving customer outcomes and using behavioral science in the financial services sector, um, to help people with their money, right?

Behavioral science and finance obviously are so closely related and, um, you know, there are some things that just like defaults that you need to set up for people and some things are more habitual behaviors that, uh, people need to engage in. And, you know, as a function, we look at all the ways that Uh, maybe customers aren't optimizing their finances and use behavioral science to try and help them and I, I found that focus as a team to be very appealing to me.

And so, while it did, it did feel quite painful to leave Ogilvy and Rory and this team that, You know, had given so much to me, um, it did feel like the right step for my career and, um, yeah, I have no regrets about, about moving. I hear you totally on, you know, finance and health are the two places where behavioral science is most often used and, and when people ask me why, I always sort of feel like It is the place where user behavior and monetization method are most closely aligned, right?

Where, you know, if you can get people, there's a whole bunch of pieces of health, where if you can get people to be more healthy, you can make more money, right? Insurance and lots of other places. Finance the same way. If you could get people to do, you know, there are parts of the financial industry that are very exploitative, right?

You can get people to do things that are quite bad for them. But for a lot of places, right? Like  we as a bank need more depositors, like that is good for you, good for me. Like, you know, that's a, you know, these sort of triple win situations. Um, that can be very valuable. Talk to me about,  did you, you know, go do a, apply a bunch of places and do a full survey?

You sort of fell into NatWest.  Um, I, I literally just saw the job posting on LinkedIn.  Uh, somebody who shared it. But you weren't sort of actively going out and seeing what was out there in the field. Not particularly, not particularly. No. Uh, I mean, I, you know, I, I scan occasionally as everybody does, but you know, um, No, not particularly.

And it just, it seemed appealing. I was familiar with Alex Chesterfield because she spoke at Nedgestock, which Ogilvy organizes, uh, just like a year or so ago. And so I was, I was familiar with her. I'd heard about her work. I thought she was really cool. And so, yeah, I knew that at least from a leadership level, they had somebody really good at the top there.

And so, uh, yeah, when this opportunity came along, I was like, I'll give it a go and see what happens. And I got it. So that's amazing. Um, tell me a little bit about the Interview process and sort of how you, you know, what kinds of things did they, you want to know from them? What kind of things did they want to know from you? 

Yeah. I mean, they, they, they really wanted me to, uh,  explain the way that I think about, um,  financial behavior. Right. And I think it's something which not everybody necessarily thinks about when it comes to finance, but, um, they wanted to know like, you know, okay, if we're trying to find customers who are suboptimal  account, Right.

How would you identify that?  And then you think about like, what are the, what are the metrics that you'd be measuring? Right. And that, that shift for me was really interesting coming from Ogilvy where it's a creative consultancy. And so our output was all to do with like new ideas and innovative thinking.

That was really like the primary reason for hiring that team. Um, whereas. Now West is a bank and so it's a lot more data driven and they have a lot more access to data as well. And so, uh, it was lovely to, to be able to, to take those data skills that I learned at Warwick and then now apply them in my job, um, in this new place and having a team around me that supports that way of thinking, like, okay, it's great to have behavioral hypotheses,  but how would you actually test it?

Right? What data would you look for and what patterns would you want to And, uh, that's really what the interview process is trying to do. For me, was, was, was I able to think in that way? And  I suppose they thought I could. So, so I got the job.  Experimentation. You know, it's true at an agency, the hardest thing is always, you know, so we talk about the side method, right?

Strategy, insights, design, evaluation, and the hardest thing in an agency is to convince people to pay for evaluation because, right? They're just convinced you're right, right? Like, they're like, you did good strategy, insights and design. So you must, the results of the design process must be true, right?

It's important, you know, having been the head of behavioral science at frog, right? And so having done a ton of design work, it's hard to get people to pay for evaluation, even though it's true. Right. It's the hallmark of science. The hallmark of science is experimentation. Like, are we actually, you know, we think we're right.

Can we prove we're right? And so  interviewing you for those experimentation skills and, and that shift of mindset, what did you want to know from them as you were thinking about? You know, leaving Ogilvy and this team that you loved and making this big change. Yeah. What kinds of questions do you think people should be asking of their employers to make sure that's a good shift for them? 

I mean, I, I wanted to just be sure that, uh, I wasn't joining a team that was scamming people using behavioral science, right? Like, that was like primary focus number one for me. And obviously I didn't, I didn't phrase it that way in an interview, that would be suicide. But, um, uh, you know, I was, I was essentially like probing like, you know, what, you know, what does your day look like?

You know, what kind of projects do you work on? Um,  typical stuff just to make sure that like  That the work that I'd be doing would be aligned with like the reason for me leaving Ogilvy in the first place was, was to utilize these skills in a more pro social way. And so, yeah, it was very reassuring to talk to the team about that.

And, um,  uh, yeah, I would encourage people to always check, like, does the, does the company that you work for align with your values, especially when it comes to behavioral science? Cause it's such a human focused industry where we're shifting human behavior. And so you always want to be doing that, uh, to, to help create the world that you think is the right world. 

Uh, so Pete, what do you do on a daily basis and what kind of projects are you working on?  So we work on a lot of different things as with every job. Um,  it's kind of two sides to it. So, so on the one hand, because I'm part of the audit function, we do like Regulation compliance, which is not that interesting to talk about. 

Um, but essentially it's like, you know, the regulate the financial regulator of the UK has certain things that the bank has to do to me. And interestingly, some of those things are very behavioral in nature, right? So like our customers able to understand your communications is that unnecessary levels of friction and your customer journeys, those kinds of things,  sort of the diagnostic side of behavioral science, um, that I was very familiar with from my time at Ogilvy.

I still do those types of things at NatWest. Um, but really with the focus of making the customer experience better. So rather than introducing sludge into the customer experience in order to hold them into a product that they hate or isn't working for them, um, we, we work to do the complete opposite. So we say as an audit function, we aren't profit motivated.

And so, um, we say, look, we've looked at your customer journey. This is unreasonably difficult. It's unreasonably difficult to cancel this product. You have to change it. Otherwise we're going to. Tell you off basically. And so that's, that's all we get to do as an audit function. And that's, that's really fun for me to do that.

Um, cause I, it's clear the value to the customer of doing that. And it aligns with the bank's sort of overall value, which are like, we want to be the best bank and being the best bank means having the best customer experience.  Um, do you usually find that the, so, uh, nobody likes auditors, right? Like you come in, you tell me that my thing is unreasonably frictiony. 

It's met with open arms. There's some persuasion that goes on. What do you use to persuade? Because one of the skills that all behavioral scientists, you know, have to learn is how do you, how do they do evidence based storytelling, right? Because evidence doesn't speak for itself. If it did, everybody would already be doing the thing that we needed them to do, right?

If evidence was able to tell its own story, you have to help interpret that. So talk to me a little, you know, you YouTube, right? Communications, et cetera. Sure. Yeah. Tell me a little bit about that. Okay. Yeah. I'm going to use the word persuasively, although,  although that's an interesting word, but at least, how do you use evidence to sort of tell the story internal to the bank as you're trying to get internal behavior change to happen?

Yeah,  well, you're right. And it was, it was a huge shift going from, uh, behavioral science consulting, where you have a client who's paid you, you know, thousands of pounds to, to, to give them ideas. And then when you present it, they're like, Oh, You know, they're excited to hear what you have to say. And then when it comes to auditing, they're like, Oh my God, I'm being audited.

Right. And it's like,  it's like the worst, it's like the worst feeling and everybody kind of shuts down. Um, so yeah, it is a total shift. It's also a totally different shift in power dynamic. Right. And I don't like to,  it's not going to my head or anything, but like, certainly as an auditor, you have, um, some level of authority over the business part of the to say like.

This is the right thing to do. And if you don't do it, then like, you know, we're going to be in trouble. And the nice thing about working in internal audit is that even though.  People don't necessarily like it that we're kind of giving them extra work to do to fix something. Um,  ultimately, uh, Audit and the bank are aligned in terms of their goals.

And, you know, basically, we want to stop the bank from getting fined, and the bank doesn't want to get fined. And so if you, if you frame it from that kind of more holistic perspective, point of view, then people are kind of like, okay, yeah, fair enough. Um, and I think it's always about like, uh, the way that, the way that I do it is, is, is always, uh, bringing it back to like, what is the, what is the fundamental reason why you're bringing this up in the first place, right?

We're not just being nitpicky for the sake of being nitpicky, but like there's something here. There's a reason why we decided to look at this, um, and why we think that something needs to change. So.  Yeah.  Um, and tell me a little bit, so what's your daily, you know, Pete shows up in the morning and what happens? 

Um, so, so yeah, my, my job involves a lot of, I would, I would describe it as like the diagnostic side of behavioral science. Um, so we don't do so much, you know, at Ogilvy was a lot about like.  Um, ideas and coming up with new solutions and innovation. And now Wes, it's a lot about just really looking at problems and trying to understand  why, why those exist in the first place.

Um, so we'll, we'll look at customer journeys. We'll look at customer data, um, and see like, you know, what, what are the challenges, what are the challenges that we're looking at here? You know, are people, you know, do we see a group of customers who are stuck in a  Is not as good as maybe one of our other products for them, right?

And they could switch to those for free Um, but they aren't why is that happening, right? So we'll we'll use data to find those things and we'll we'll do lots of customer research like customer interviews and and stuff like that to try and Get to the bottom of those questions. And then we present those to  senior members of NatWest  to try and persuade them that something needs to change.

And so let's talk about the, we, you keep using the word. We are you doing user research and data science all contained within the, within the, the, we, that is Pete, or you have a user researcher and a data scientist and you're working together as a team.  We do have some data people who help us, but. You know, for the most part, we're a self functioning team.

And so, uh, so yeah, I will do everything from, um, helping with the scoping of like what to look at, to crunching the numbers, doing the data analysis, doing the coding, to putting the PowerPoint together, to presenting. Like I'll do it all start to finish, which is what I'm used to anyway, because that's what I use in consulting, it's the same, um, thing.

But, um, yeah. Yeah, I'll, I'll, I'll get involved with every part of it. And I love doing that. I love being involved with every, every part of a project from start to finish. I think that's much more rewarding.  Uh, you know, we talk about T shaped behavioral scientists, so one of the things we have coming out is this, you know,  try to move people away from the notion of a monolithic  behavioral scientist to sort of some role based behavioral scientists, quant researchers, qual researchers, strategists, designers, and then PMs, uh.

If you were gonna, you know, uh, specialize. So if you start to draw out the bottom of, right now you're doing a lot of horizontal work, is there a place where you feel like, hey, I'd love to go deeper in quant or qual or design or, you know, experimentation or sort of business side of things? Yeah. Um,  I mean for, for, okay.

In terms of financial services, I think.  Uh, the biggest,  the biggest area of improvement, I think, is like communications, right? Just how do we, how do we talk to customers in a way that makes sense to normal people, right? And  banks are historically terrible at this, especially older banks, like my bank, NatWest, right?

And, you know, again, I'm not, they're improving for sure, and I'm helping them improve, but like, Just, you know, banks have a whole, have a whole language that they speak to each other, right? Like, I, it was such a huge learning curve when I first joined to understand all these financial terms and a million different financial products that we offer in this completely different world.

And then you see some of that language then feeding through into like the communications we send to a plumber in, Bristol, like he's not going to understand what we're talking about. And so, yeah, I think like communications can be so powerful and we've seen in behavioral trials  in, in banking, but also in like related industries, like energy and so on that, like simple changes to communications can have dramatic consequences.

Impact on, on people's behavior change. Right. And so  yeah, just improving, for example, just one letter that you send out to a customer about a particular product,  uh, can really, you know, have big influence on, on their lives, right? Like if you can switch them to a better mortgage rate, like that's going to have a huge influence on their financial wellbeing. 

And when you multiply that by, you know, the millions of customers that a bank like NatWest serves, right, that's a huge impact.  Yeah, I vividly remember working on a finance project where they kept talking about BIPs. I'm like, what the heck's a BIP? And they said, oh, it's a basis point. And I said, well, what the hell is a basis point?

Right? That was not a clear, like, you're like, oh, it's a basis point. There's jargon for the jargon. Yeah. Everyone knows what a basis point is, right? Like, you cannot, to your point, send a, you know, a letter to a plumber in Cardiff, About the basis points of his mortgage, right? Like that is not, and yet they think that's clarification of BIPs. 

Exactly. Better than BIPs. I guess, I guess it's better than BIPs.  What's, what's, so let's talk about, you know, you have also changed the focus of your YouTube channel. So you continued throughout these jobs to create content  largely on behavioral science and largely on what I would call academic behavioral science, right?

Explaining of  Various kinds of effects and things, right? It's not, you're not interviewing applied behavioral scientists to talk about their projects, right? This is, this is largely explaining effects, et cetera. And then recently you've made a fairly large decision. Uh, tell us how you came to that. Tell me a little bit about sort of how you're freezing that decision in your mind, because this is for you. 

Yeah. So I'm glad that we've structured this chat the way that we have, because if you were to say to me right at the beginning, like,  You know, summarize who you are.  I'm in a weird spot,  right? I'm in a weird spot right now, right? That's and I'm trying to figure it out because  as I've explained, like, you know, leading up to really just a few months ago, I was all in on behavioral science, right?

It was like,  It was, it wasn't just my job, but it was like who I was, like, as a person. I am a behavioral scientist, right? That's kind of how I define myself. I am Pete Johto, well known, you know. Well known behavioral scientist. My YouTube channel is about behavioral science. It's not about anything else.

And that was kind of all I  laid all my eggs, I laid all my eggs in that basket. Um,  and recently I've, I'll be honest, I am starting to be a little bit disillusioned with the whole field, right? And  this really, like, I've been feeling this way for a while, right? There's, there's, you know, you hear these stories, there's like, there's the Dan Ariely scandal, there's, uh, you know, which, which was very personal for me as the person who first introduced me to the field, right?

That was like a real blow. And  then there's, you know, Amy Cuddy, and then there's other people, and slowly, slowly, there's, like, all these, like, little, you know, Uh, stories that you hear of just stuff not working out.  And then, uh, and then a few months ago, there's the Francesca Gino case.  And it's, it's something that goes  viral across the whole world.

Like, everybody knows about it. And, uh, and I read it and I'm like, and I'm shocked, right? Because like, these are studies that not only,  It's totally different learning about this after I've started working in the field. Because before working in the field, it's just like, Oh, okay, it's all just academics arguing with academics, whatever.

Once I've worked in the field, I'm like, I've actually cheated my clients,  right? And that's a horrible feeling. Because it's like, I've sold you on ideas based on studies that were false. Right. And it was like multiple, I've used multiple studies from Francesca Gino in my work at Ogilvy. Um, and then all of this breaks and I, and it sucks, right. 

It sucks to feel that way. Um,  I ended up covering the story on my YouTube channel  and it blows up like crazier than I ever imagined. Right. Like I was, I was a tiny channel. I just hit 10, 000 subscribers. And then I make this video on Francesca Gino and then after two weeks, it has over a million views. 

And there's like, I never hit anything remotely close to that, uh, in the past. And it goes, it goes absolutely crazy. And it, it really resonates with a lot of people because I think, what I was noticing, because I've been running, I've been doing my YouTube channel for like two and a half years by that point. 

And what I noticed was that like, people were starting to drop off. Right? Like, people were like, not engaging with behavioral science anymore. And it was, I think it's because people were really feeling what I was feeling. Which is just like,  stuff isn't working out. Like, all these ideas sound great, and then we try them, and they don't work.

Right? So like, what's going on here? And, um,  yeah. And so, and so I cover this story, and then I look into it more. And then, and then I have this incident at work. Where  somebody shares, uh, this meta analysis on loss aversion.  Loss aversion, right? Like, come on. Cornerstone! Cornerstone,  right? That's like, it's like,  you know,  principles that everybody knows, right?

System one, system two, which is debatable in itself. And then,  Loss aversion, right? It's like, it's like, it's like huge, right? It's, it's, it's the basis of so many other principles and it doesn't replicate, like loss aversion doesn't replicate, right? And  I'm, this is like a huge blow. Right. And I'm like.  I don't think I trust any,  like, well, not any, but like the vast majority of behavioral science principles, I no longer have faith in.

If I can't trust loss aversion, I don't think I can trust anything else that's even crazier, right? As an idea. Um, and at the bank, I don't mind it so much because, because the way that the bank works is so data led and we don't do anything. We don't roll out anything until we've tested it a million times with a million customers.

But. Um, you know, in the real, in the real world, uh, especially on YouTube as a, as someone who teaches something. I don't want to teach things that are false. Uh, and so I really just, I've really recently become very disillusioned with, with behavioral science and a lot more skeptical of it.  And I started to look into it.

I was like, how did this happen? Right? I was like, how, how, how did studies like this, like Francesca Gino and others notable cases in the past, like,  how, how did these studies get published? Um, how were they not caught in peer review? And you know, there's multiple coauthors on this paper. How come nobody caught any of this stuff?

Like what's going on? Like, how, why is there so much fraud and bad science happening coming from these scientists who we're supposed to trust? As practitioners, right? As practitioners, we're supposed to be able to trust these people. I don't feel like we can. And so I started looking into it and I found all of these cases, not just from behavioral science, but across the whole spectrum of academia, from, from, from physics to medicine, you name it, right?

Every aspect of academia is suffering from the same thing.  Um, and so I started making videos on that  and, uh, it's no secret that those videos are doing super well. Uh, my, my channels exploded over the last few months. Um, but I don't, I don't do it just because.  You know, they do, the videos do well, but I do it because like, I'm feeling the same way that a lot of people are feeling that like they're finding it very difficult to trust, um, academic institutions and the output that they're putting out.

Right. And it's not, it's not just your, uh, uh, anti vaxxing crowd as well. That is, that is that way. It's, it's people in academia, it's scientists, people who have done PhDs in the past, maybe dropped out from their PhD because of, they saw this kind of thing going on and they didn't want to be a part of it.

Okay. Um,  that Right. That's, that's . I, yeah. I saw that. You, you, you started when and dropped out. Right. So it's a, it is, it is a lot of people who, who've had that experience and they feel the same way. And I think the content that I'm making now is doing so well. 'cause it's resonating, uh, with that message that a lot of people are feeling. 

Yeah. It's one of my early. formative experiences between after undergrad, pre grad school, I was working in a lab for someone who was the editor of a journal. And so I was kind of doing the logistics of like trying to get papers reviewed and things, and you're like begging people to review these things.

And you have the sense that maybe they didn't read them sort of in depth, you know, they're just sort of going on the strength of the name of the person. Like, I'm like,  you know, it always sort of feels a little bit math, uh, as you, as, as you go through it and, you know, and.  Even, which isn't to say that I don't think science is real, I think science is tremendously important.

And it isn't to say that I don't think that we can't make a replicable academic thing. But I remember a few years ago getting in a violent argument with a UK based  behavioral science consultant. Uh, who, who,  You know,  strategy, insights, design, evaluation, right? That's the way I explain the process.  And I say, look, it's not behavioral science if you only do one part of this, right?

Like, unless you are doing all of the parts, you're clearly defining a behavioral outcome, you're clearly understanding why that outcome does or doesn't occur,  and then you're clearly testing that whatever the heck you designed actually does change those, those outcomes. Features and in a way that changed that behavior.

It's not because I and she did sort of exclusively sold insights as a consultancy and she was like, God, it's still behavioral science, right? You're even though we're doing no evaluation. I'm like, how do you know you're right? And she's like, well, I know because I've read all these papers that say I'm right.

And I'm like, really? Like knowing this specific, like, you know, this, you have a really specific application with a specific group of people in a specific context that, you know, this thing, I always loved one of those, one of the formative things for me was the, was, uh, God, what's his name at Columbia, the sort of ornery stats guy.

And he wrote a thing about the piranha effect, right? And he's like, look, if you took all the smoking cessation papers and you lined them up, no one would smoke, right? Because if you total up all the interventions, it's greater than 100%, right? But because they're not competing against each other, right?

They, you know, if you tried to do all of them at the same time, some of them would eat into the effect sizes of each other. And so you wouldn't, you know, you don't get this sort of thing. That was really impactful for me. I mean, I do.  I still read the literature just because it was, it's hard for me not to, right.

But, but I don't believe anything until we've evaluated it. And in this specific instance, you know, yeah, exactly. You can tell me a paper that says I should move the thing to the beginning, but until you can prove in my specific instance, moving the signature to the beginning does the thing that I want.

And, you know, this is where I think behavioral strategy is so important, is you have to clearly articulate, you have to pre register, you have to articulate in advance what is the outcome we are trying to produce. Yeah. Right? Not, we moved to the beginning and then we made up an outcome that made it sound like moving to the beginning was a good idea, but I at the beginning said, this is how I'm going to measure  success.

And then we're exactly  and that's why that's why I'm happy to continue working in a behavioral science team at NatWest is because we are able to do that, right, like you said earlier, like in consulting, it is so hard to get that evaluation stage, right? It's so hard to get the client to approve that in  as an internal in an internal team with like, with one of the richest data sets in the UK.

It's easy to do that. In fact, it's my, it's my main job is to do the evaluation. Right. And so, um, yeah, that's why I'm still comfortable like going, yes, like  here. So I know, I know there's some relevant ideas here from behavioral science that might be helpful and here's how I'm going to test them with the data that we have.

Right. And, and, and that's, that's how I'm kind of get sorting that part of, of my journey in my brain is that way. Yeah. I think it's important when we have this conversation as a field.  About skepticism or sort of how we should be skeptical or actively skeptical about our own discipline. Yeah. Making that applied versus academic distinction.

Right?  Yeah. Because, you know, it can be easy. I worry that we'll throw out the baby with the bathwater in terms of applied behavioral science. Because people will sort of say, well, Dan Ariely studied and replicated, now we shouldn't have a team. And I'm like, if your team was any good, they should have been evaluating their own stuff. 

Right, you know, in a rigorous, transparent, open to the rest of the organization. That's sort of where I was leading you with that, like, hey, when you're talking to execs, how are you using evidence? Because I think that is underrepresented in our conversation about  essentially phenomenology. Phenomenology is so fun to talk about.

It's so fun to talk about, like, you did this thing and then it's like, but it, yeah. I also think a behavioral science team is so valuable for orgs because behavioral scientists just know how to run good experiments. Yeah.  And that's, that's not something that's a given in business, right? Like a lot of people don't know how to run a proper experiment.

They don't know that you need a control. They don't know what data to gather and how to, you know, how to carry things out in a, in a rigorous way. And so just having a behavioral science team who is confident. On how to run a good experiment, how to interpret results correctly, how to do proper data analysis.

Like that is incredibly valuable in itself, almost unrelated to whether or not the theories are replicating or not. Yeah. Elizabeth Kim, formerly of Spotify, you know, has very strong feelings about this, right. About the true benefit of what we add is that experimentation layer  where, you know, experimenting with humans. 

is very different than data science experimentation, right? In the sort of way that data scientists often view it. And that's what that rich human subjects. Yeah. How do you deal with the messiness of humans and control for things and still be able to show effects that sort of help you evaluate decision making  is a really distinct skill that you see from that sort of applied behavioral science, social psychology, kind of. 

Exactly. This was so much fun. Thanks so much for joining me. And, and, you know, I really appreciate your perspective and the work that you're doing on trying to help. Curate that skepticism in the right direction, um, both internally at NetWest, but also, you know, sort of externally in the world. Um, I appreciate you so much.

Um, you know, I hear you on the disillusionment. I, you know, I like all older behavioral scientists bear some response. You know, I tried to beat this drum about  less on academia and more on this evaluatory thing throughout my career, but I didn't do as good of a job communicating that as I should have  right throughout the 20 years of my career.

And, and. You know, uh, what I hope is that, that,  that you and others can, can.  Uh, help us be skeptical about  broad over generalized claims without throwing out the beautiful thing that is science. Science is this wonderful process for figuring out whether we're right or not, right? Or whether we should do something or not, right?

And, and that continues to be extraordinarily valuable. Uh. Right. But that's a nuanced method and nuanced messages are hard and I appreciate your your desire and ability to bring nuanced messages to the masses. Um, and so thanks so much for joining me today, Pete. I hope you have a wonderful rest of your day and I'll see you on down the road.

Thank you. Thanks for having me. Yeah, it's great for talking to you. Thanks, Pete. Take care, man. Bye bye.