Interview: Phil Graves, author of Consumerology, on the hidden flaws of market research

In this episode, we chat with Phil Graves, author of Consumerology. As a leading voice on the flaws of traditional market research, he explains why what customers say often differs from what they do. He also highlights the mechanisms behind purchase decisions and explores how behavioral science offers a more reliable lens for understanding real consumer behavior. 

Transcript

MichaelAaron Flicker: [00:00:00] Welcome back to Behavioral Science for Brands, a podcast where we bridge academics and practical marketing. Each week we sit down and go deep behind some of America's most successful brands. I'm Michael Aaron Flicker. 

Richard Shotton: And I'm Richard Shotton. 

MichaelAaron Flicker: And today we're sitting with Phil Graves, author of the acclaim book, Consumerology and Founder of Shift Consultancy, a firm specializing in integrating behavioral science with qualitative research.

Let's get into it. So Phil, welcome to Behavioral Science for Brands. Richard and I, as we were talking about before the show, are on this little mission to find the best applications of behavioral science, bring them to the world of marketing. And we're so excited for our conversation today. Our listeners love when we get to bring in folks that are in the field that can just jam and talk with us.

 But before we get started. If it's okay with you, you'll indulge me as I give our listeners a little bit of [00:01:00] background on you. You are a renowned expert in consumer behavior and psychology. Your book, consumerology. The Truth about Consumers and the Psychology of Shopping challenges traditional market research methods by highlighting their limitations and accurately predicting consumer behavior.

In 2005, you started shift consultancy, which focuses on aligning market research with insights from behavioral economics and psychology. And from all that I've read your work spans many sectors, financial services, health, FMCG, retail media technology, and for those that may not be familiar with your work.

You've been at this forefront of bringing these two very important disciplines together and we're very excited to have you be with us today. So our listeners love stories. Maybe you could start at the beginning and [00:02:00] tell us how did you get introduced to the world of research, of consumer psychology, of behavioral thinking?

Where did this all begin for you? 

Phil Graves: So by education, I was a statistician. And after a couple of three or four stats that sent me into the world of market research from the kind of quantitative end of things. And as it happens in the interview for the first significant, well the first market search job that I got I was asked if I had experience with qualitative market research.

By the head of research within this large financial services company. And I said I did and I genuinely meant it, but I subsequently came to realize that I didn't actually know what qualitative research was which didn't prove to be a major herbal and didn't stop me getting the job. I was [00:03:00] sufficiently convincing in all innocence.

And then once I got into it, fine, so, so then once I started working in the field of market research. I very quickly became curious about why it was so often wrong, and that was my platform purely from a pragmatic look. My job, apparently is to understand consumers. So what we're doing doesn't seem to work either.

It's clearly wrong when we launch whatever we launch, or people who've got lots of years of operational experience, a kind of you know. Incredulous about the results that are coming out of the focus groups or whatever it might be. So I then started to say, okay, well I need to look into this. What, how'd you do that?

And you know, and this was some considerable time ago because unfortunately I'm old. There wasn't a lot of consumer psychology around academically, if any. And, and what I looked at, [00:04:00] first of all was psychoanalysis. So I was trying to reconcile how is it if you were gonna go and see a therapist that they, you go in and say, I'm really stressed.

And they say, well, why do you think you're stressed? And you say, oh, I've been, I think I've been doing a lot. And they say, well do this. And you go out and you hand it over your 200 pounds and you go, well, I didn't seem particularly insult. And of course, that's not the process of psychoanalysis. The process of psychoanalysis is that you don't really understand.

Why are you interacting with the world in the way that you do and that you've formed these structures in your mind and these ways of going about life that ultimately are proving counterproductive? And through a process of psychoanalysis, a therapist can get you to a point of realizing how it is you see the world, and how that maybe isn't the only way to see it at a risk of analysis.

So. As I started to look more and more into it, I started to come across the world of behavioral science, [00:05:00] behavioral economics, and the recurring theme there is we do stuff in experiments, we make people behave differently, but afterwards they never ascribe their behavior to what we've changed. And they go, oh, you pumped that music in while I was making my shopping?

Or, you know, you, that smell was in the air, wasn't it? That's never what happens. What happens is that post-rationalization, well, of course I did sensible things for sensible reasons, and it just opened my eyes up to what I'd seen going on in all the research that I'd done. And as I went further and further into that and started exploring more and more, and I'd emphasize this was all from a practical perspective, this wasn't an academic exercise.

What I started to discover was that. All of the cognitive biases or many of the cognitive biases that are identified through academic work in the field, behavioral economics are kind of just built into [00:06:00] most market research. They're a fundamental part of it. And a a, aside from the fact that the fundamental premise of market research is.

When it's InterG interrogated in nature that the people you are speaking to understand themselves and why they do things and everything we see from behavioral psychology says that ain't the case very clearly. It's not the case. Then you've got books like Strangers to Ourselves by to the Wilson about the, the illusion of Conscious will by Daniel Wagner.

You know, lots of work that's gone on and, and things that go back to the early part of the nine, the early 19 hundreds. Where people would stating that, you know, hinter actually understand what they're doing. And where there was of course, a Nobel Prize famously awarded Daniel Kahneman Ams Burke's work.

 And yet we come back to this platform within the world of market researchers. I say, all right, we ask people and then we'll believe what they tell us, which we kind of don't do in any other [00:07:00] facets of life. You know, you know, particularly if you're British and somewhat cynical in nature you know, with people horrible stuff, you don't go, well, that's definitely true.

I'll gobble it up. You go, well, is there anything to support that? And so, you know, that was my journey and, and I had no intention of starting my own consultancy business, of starting my own search company. But, but I kind of ended up. You know, it, doing it by default almost because I just felt that most insight work was being done so poorly.

MichaelAaron Flicker: Thank you for sharing 

Phil Graves: that very long answer to 

MichaelAaron Flicker: Yeah, no, thank you for sharing. No, thank you for sharing that. And you know, Richard and I, as we were preparing for today's talk, we found this great quote that you went, you go even further and you say focus groups surveys, standard market research is.

Perfect recipe for getting flawed data. I mean, and, and I think what you were describing is the reason why, is that right? Yeah, 

Phil Graves: absolutely. I mean, and you can, you can approach it [00:08:00] from either angle. You can approach it empirically and say where, you know, have we got evidence that this works? All the. That'll clearly not.

And anybody out there who's using market research will realize if they look back objectively, there have been lots of times when it didn't prove to be right. You know, you can look at the fact that something like 70%, 75% of new products that are launched, most of which are consumer tested fail within a year.

You know, so we are not sure to empirical evidence that this process has some question marks against it, or you approach it from the other angle and say. Okay, where's the evidence that people understand their own decision making? And I went looking for that. And that was kind of the journey that was consumer allergy.

And there, there isn't any, and there's a huge amount of evidence that they don't. And so from that point of view, you know, if you are, you then get into more specific issues with different types of research. So you know, there is a special place [00:09:00] reserve focus groups. Which were born outta psychotherapy, where groups of people were put together because they changed behavior.

And yet we focus groups and think, oh, we're getting deeper insights as a result of these social dynamics. There's no evidence for that. In fact, you know, at a viewing facility, people being watched change the way they become more self-conscious, change the way they speak, change what they think about, the, the, the desire you're talking about normally something pretty banal to the average person. This is not about matters of life and death. It's, you know, do you like this biscuit? At which point, you know, somebody offers a point of view or seems reasonable, I'll go along with that. Cognitively easier, that would fall out with you.

'cause we're here chatting about biscuits. You add in the moderator who's using behaviors that are designed to elicit response because. The worst thing for a moderator is people not talking, but those please you. Behaviors, as they're known within psychoanalysis, [00:10:00] enlists, certain kind of favorable reactions from people.

So there is all this kind of psychology going on on the surface, which means that what comes outta this is likely to be polarized artificially by virtue of being put into this group. Strangers. And. Really what those people will do. And you've stripped away all the context That's important when people are actually making consumer decisions and replaced it with this other context, which has got so little to do with how most buying decisions take place.

But then you get other problems if you're looking at surveys to do with how you're framing people, how you are framing questions, where you are giving them pre-coded responses. All of those things have a, predictable effect on what people will say and how they will respond. And again, there's lots of evidence from where different sequencing of questions been asked, where you add in or don't you have an opinion [00:11:00] to opinion poll type questions.

 And you find out if you add in the or don't you have an opinion, you change completely, change the result. And when I looked at one particular study that was done. By big research companies independently at the same time they got answers which were 60 percentage points part to the ultimate fundamental same fundamental question.

And it was, and it was explicable when you looked at the trail of kind of bias and influence that took place through the, through the questionnaires and, and it's just, just unavoidable. 

MichaelAaron Flicker: So I think our listeners will be very interested to go a little deeper into some of the different research methodologies and some of the known issues with them.

Before we do that, why do you think this persists? Why, why, after all of this evidence, do you think that the industry continues to return to these methods that have obvious. Least [00:12:00] limitations, if not downright fallacy is built into that. 

Phil Graves: Well, that, that's a delicious question for a psychologist. So do lot of people believe all sorts of stuff that patently isn't true?

And you can pick your point. You know, you can pick all the religions you don't believe in. If you happen to believe in a religion, you can pick mediums and psychics. You can pick horoscope, horoscopes, you can pick medicine that doesn't really do anything, you know anything you like. And, and there's all sorts of reasons that, that people believe in these things.

 There are placebo effects and there are placebo effects in research, you know, we are thinking about doing something scary or we've asked a thousand people and they said, go for it. So we're going for it. We feel much better now. Good, but you shouldn't. It's generally very easy to do market. You know, you send out your questionnaires, you get an agency in, there's no operational messiness.

You don't have to worry about legal [00:13:00] getting involved or, you know, align your operational team with doing anything. You just do whatever you want, ask whatever you like, go ahead and do it. You get your results back very quickly. It's not that expensive to do. And in if you look across all human behavior, you'll see a trade off between effort and reward.

Where I'll take a little bit of reward if it's easy to do over something that's quite difficult to do but would be much more rewarding ultimately. You've got the status quo element of if you're not being fired for producing rubbish insight at the moment, why on earth would you take on the risk and loss aversion of doing something different?

 No one's asking you to come in and shake things up and challenge things. And you get the confirmation bias, which is you know, you kind of select the bits of research that you remember as being really useful. Which is all the good bits and all the times things seem to pan out in line with what [00:14:00] you researched and forget about all the times it did.

And of course, you know, that's something humans are brilliant at doing, and you don't have to have a really high hit rate to keep the confirmation bias going. You know, because we are so good at dismissing things that causes cognitive dissonance that don't fit in with our worldview. And there's sort of social proof out there that everybody else is doing it, aren't they?

So you know, you look at it from that point of view and, and coming back to that point around kind of what you get rewarded for in large organizations. You get rewarded to coming in and shaking things up and challenging what's going on day to day in large organizations that kind of gets you fired.

And I speak from some personal experience in small dynamic startups. Absolutely. You're learning as you go. You're shaking things up, you're being [00:15:00] dynamic. But large companies, you know, they don't like all being rippled and people who prosper there kind of know that. So so it's, it's challenging to bring about that change.

And there's a whole heap of reasons why, why change doesn't happen. 

Richard Shotton: I. mean that's a very Potent mix that's keeping that kind of standard market research approach going. If you were trying to persuade an organization to do things differently, do you think there are tactics to break through those biases and barriers? How, how do you think it is?

Phil Graves: How do think the best way to kind of sell in an alternative is an excellent question. The, the way that you do it, and I should say, having said all those, all those things I've just shared with you I, I do have. House and I'm able to feed my dogs and children and things that I have earned, I have earned a livelihood.

The the [00:16:00] interesting thing is you can't explain that to people. All the things I said to you both about why current market research essentially doesn't work. I mean, it'll be right one time in three, but, you know. That's because it can either be right, wrong, or not. Sure. Those are your only three outcomes.

 So you know, a third of the time you've got the evidence for what you do. So you can't say to people, look let me explain to you why people don't understand themselves, why asking them questions is really a terrible idea. And, and why everything you've been doing up until now was essentially a waste of money because.

The response that anyone will have to, that is, well, either I could believe that and it invalidates everything I've been doing and that hasn't resulted in me being fired or, you know, beaten or wit by my superiors. Or I could [00:17:00] I could believe you and then just be thrown this world of turmoil. So psychologically that's.

You know, that's such a weighted equation that it's never gonna work. And I know 'cause I've tried it. So instead what you have to do is say, look, there are insights you are not currently getting. A large part of human decision making is unconscious. And then, you know, the work that Daniel Kaman and Amos and others have done in highlighting that is quite a useful.

Platform to use. 'cause people, well, you've heard about thinking fast and slow and it sounds quite important. We don't really know what to do with it. Yeah, well there's this thing called the unconscious mind. And actually, you know, I can do some tricks with people in the room to demonstrate how priming works and so on.

 And that you are all susceptible to all of this. And again, I, I've done that in the past too. So if you want to get deeper insights, you need [00:18:00] to use techniques. That recognize that aspect of how people think and can help you identify it and then make better decisions. And you have to leave them to do the two plus two means, well, hang on a second.

What about all the stuff we've done before? But fortunately, they never do that math because it's too uncomfortable. And again, the, the nature of being in a large organization is that you don't get rewarded for looking back. So, so you, you have to focus on the, here's a little bit extra. And here's how, how it works.

Richard Shotton: I, I, I really like that as an approach. I'm particularly intrigued by the, and experiments that you used to, to persuade people in, in the room, because I think there's, there's nothing better than a, a light demonstration. Is there a particular go-to experiment, you think working up a small group that you return to again and again?

Phil Graves: I've, I've used, I came up with a kind of a questionnaire which had I think probably maybe 10 different [00:19:00] questions on that heart, like different aspects of how people's brains really work. And then all you do is have two questionnaires. People don't know you have two questionnaires. And then you compare the answers that you get from people and you can demonstrate things like, you know, heuristic framing, effect, priming how social proof works in practice and, and those kinds of things.

So, so, yeah, I mean, it's all stuff that's, it's, you know, standing on the shoulders of the giants of the of the field. And then just sort of shaping it in a way that's kind of fun and engaging to people and duping them in the process. And at the end you kind of get 'em go. Oh, right. Okay. Yeah, I suppose it's just hard for me to argue with priming when you've just demonstrated that I'm very susceptible to it.

Richard Shotton: Yeah. See, you've run the experiments with this fictitious client and they've, there's a glimmer of hope and they're interested in an additional approach to market [00:20:00] research. What does that look like? Say if we can't give too much validity to folks groups and a at a stance survey approach. What, what, what would you do instead?

What would your recommendations be? 

Phil Graves: And the irony is it's really not too difficult. So you say to people, look, imagine that asking people questions was illegal. Arguably for marketing insight purposes, it should be. And it's also irritating if you're a consumer being asked questions. What would you do instead?

And kind of anything now you come up with is likely to be better? Now what I, most of my work is done by we, I've developed some techniques that recognize how people really think and make decisions and get as much of that presence in [00:21:00] the. It kind of in the arena as it is. I'm trying to find out whatever I'm trying to find out.

And that comes down to, and I wrote about this in serology, the effect criteria for psychological validity. So is what you are looking at an analysis of behavior because when you get people to behave, they're kind of doing things, they're revealing what it is they think. And so, you know, behavioral data, really useful stuff.

But also you can, you know, put people in situations where you get them to behave. Is their frame of mind representative of how they would be and are thinking when they're doing the thing that you are interested in. Because if I want to find out how you buy orange juice or whether you like a new orange juice that I'm bringing to the mark.

I don't need you thinking like some kind of [00:22:00] scrutineer at a car show, looking at my product and going, oh, so this is a new thing, is it? Right? Well, let me have a look at it and see what I think, because that has nothing to do with the frame of mind. You'll being when you are, you are a 2-year-old hanging off your ear the dog's overheating in the car.

You're just trying to get shopping done as quickly as possible. Don't put dogs in, by the way. And get your shopping done. And you may very well have a deeply embedded habit of always buying same brand. And we'll spend probably half or one second making that purchase. Well, if that's how you behave in that category, that's how I need you thinking.

When I explore annual fictitious orange juice the environment is such a huge source of unconscious influences. If you strip that away, what are you, what are you, what are you left with and what are you replacing it with? So I, I've developed questioning techniques [00:23:00] that put people mentally into the environment where they're making a decision as a consumer for the thing that we're interested in.

 And then I make sure that the focus of what I'm interested in is covert because. I don't want people sensitized to the topic ahead of time. I don't want them knowing what I'm interested in. I don't want them focusing on the thing that matters because they're not gonna do that in real life, out in real life.

It's a fun fight with every other brand, and initially, at least the timeframe in which they are responding to something needs to come back. The example I gave needs to reflect the timeframe of which they're making in the real life. In real world, because. Without those things, you are in the wrong part of someone's brain getting the wrong information, and we're all capable of thinking and acting very differently to the same stimulus in a different context.

You know, if I said, what would you do if you really disagreed with someone, I. You might think, okay, well imagine. Imagine you're [00:24:00] really disagreeing with some really call back to mind now and you can do it, but you are lying to yourself. And if you told me about it, you'd be lying to me because if you were at a football match or a soccer match and you disagreed with the referee, you'd probably be using some quite colorful language and screaming your head off.

If you were in a board meeting and the person who pays your salary has just said something that you really disagree with. You're probably gonna find a way to talk it, talk around it in a, in a different way. You may not even say anything at all, but we don't think about context unless context is put there for us.

So the techniques I've developed use those principles and, and it's not, I'm not saying it's perfect. I'm not saying it's the, it, it's, it's infallible, but it's putting as much there that we know is important in terms of how people respond to stuff. And taking out all the things we don't want. So I'm not gonna ask people.

So my new orange juice might be super healthy. Let's not get [00:25:00] into the beef, fat, sugar, and orange juice way. It really is healthy vitamin C, and it's not that. So you know, I might be positioning my new orange juice as a super healthy orange juice. Okay? So I might want to ask you whether you think it's healthy, but I can't ask you that.

Because you are not thinking about healthiness of orange juice when you walk into the shop. Probably, I can't presume that you are, but as soon as I ask you that question, that thought is in your head and you will respond to it in your associative brain way. 'cause that's the way you design and you will feel like that was the thought you always had.

But it's only a consequence of me having asked the question. So I can't ask you any questions. So again, I have to use techniques that mean I can explore things without asking people questions and let them reveal to me how they react, how they think, how they behave. And then I can go leverage psychological consistency, which we all desire for, and play around in their minds to find out all the wise.

MichaelAaron Flicker: Can you talk a little bit about, this is [00:26:00] different than ethnographies alone? Just following people when they shop, going into their homes and watching as they make decisions. It's closer on a spectrum of focus groups being at one end and what you were talking about being the other ethnographies feels closer to me, but not the same thing.

And can you maybe talk a little bit about the differences there? 

Phil Graves: Sure. Well the biggest, you know, the biggest challenge to ethnography is you know, imagine anything you do in your life and then stick a stranger by your side no longer the same thing. It doesn't matter what it's you know, it's changed and it's changed massively 'cause we're social creatures.

 Most of the time when we're sitting in our car picking our nose, we imagine that the world doesn't know we're there. You know, and it's only when we get see caught on camera you know, invading our own [00:27:00] product, our own nasal cavities that we kind Oh, right. Yeah. People can so, you know, that's a big change, you know, and we see in various experiments, you know, you tell, tell people that a chair is haunted, they act differently.

 You know, they'll behave differently in experiments. So, you know, putting someone into the mix. Makes people self-conscious and and self-aware. And, and it's, and that in itself is a fascinating term because, you know, saying to someone, you know oh, I'm gonna make you feel self-conscious, it's like, that's a really uncom thing.

But implicitly that's why ethnography does and there's no getting away from it. Now I'm not entirely debunking ethnography. I mean, this got its place and you may learn something interesting from it. And I'm sure there'll be lots of times people feel they have, however, from a fundamental psychological point of view, you know, you are already messing with the waters.

[00:28:00] So you've gotta be aware that that may have a bearing and depending on what, what it is you are looking at you know, you've got all sorts of problems that come into it. I mean, a lot of research. Works on this premise. You've got to tell people exactly why you're doing it. So, you know, are the points at which you say, look, I, and with photography, this is hard to avoid.

Look, I wanna come and watch you feeding your children to see, you know how you cope with meal times. You know, well, I'm now thinking ahead of time, right? Okay. What am I gonna give them so I don't look like a terrible parent? You know, make sure they haven't eaten before, so they're hungry at the time.

And, you know, all these things are, are just natural responses because, you know, we care about how we are perceived by other people. So you know, it, it's a, it's a can of worms and, you know, that there are better ways in my opinion. 

Richard Shotton: Yeah, I, I found one of those fascinating bits of consumerology was as you've just expressed that these [00:29:00] elements, like being watched a.

A questioner being involved, context timing, they're so important that maybe the way to get round a lot of those issues is to run live testing. And I thought that was one of the most powerful arguments in, in, in the book. Could you explain what you mean by a, a live test? And maybe we can continue that OJ analogy if it helps.

Like if I've got my OJ brand, oranges brand and I'm deciding whether run a new creative, you know, how could I run a live test to see whether this, this new intervention works? 

Phil Graves: Sure. So let me caveat this by saying undoubtedly live tests are the gold standard and they're almost always impossible to do, but we'll come back to that.

 So, so from a theoretical point of view, you know, live trials are brilliant because what you are doing is you are putting something out into the real world, coming back to the effect [00:30:00] criteria. Let's say you've got a new pack, their orange juice, you get some made up, you stick them on the shelf in the supermarket, and you stand well back and see what happens.

Okay, well people are coming in thinking like they normally would think. You get to analyze their behavior. You watch and see whether they notice it, you see whether they pick it up. You see whether, what buy it. They're in whatever frame of mind they should be in. They don't know you're doing the tests as long as you're staring the car enough away, you've got all the environmental cues represented and they're spending however much time they normally spend buying orange juice.

And you've had nothing to do with contaminating that. So it's brilliant. You add into that, well, it's like, well, let's test one pack in this store and we'll test another pack in another store, which you match and normally sell the same sort of mix of. Products and the same amounts of our current product.

 And then another one here and we've got our control condition. And we'll run these all at the same time 'cause we know that the biggest thing that affects sales drinks is whether it's hot [00:31:00] or not. And so we'll see which one of these does best. You know, and I have run trials like this and help companies run trials like this.

Incredibly difficult. There are logistical problems, there are legal problems, there are product sign off problems. There are supply chain problems. It, it just goes on and on and on. That, it, it, even with one, with one bank even in an area, you think, well, this will be easy to, we'll try some different, lets to try and bring down the amount of kind of customer service complaints that you get when you're doing this.

Yeah, we can't even change the signature on the bottom of the letters because it's hard coded into the computer system. Our computer system is so old, and in fact we're not even sure who that person was. We're certainly sure they're either dead or no longer work at the back. So, you know, those, those are the realities of of [00:32:00] trying to do large trials.

But you know, in the digital space. Should companies be dear? Should they be gearing up much more to use them? I believe they should. If I was an FMCG brand, I would want to get my own friendly chain of stores and pay them the decent amount of money so that I could put my test products into them and see what happened.

 With digital marketing, you know, you can try different campaigns, try different creative, and see what reaction it gets. Albeit a lot of digital marketing is shouting into the void and probably need to do quite the nut bit to get any kind of metric at all, but that's the story for another day. You know that these things are out there to be tried and you know, the value of getting genuine data, real data, valid data over and above.

 Spurious data even if it appears to be statistically [00:33:00] valid because you've asked lots of people you know, there there's no comparison, but it's incredibly hard to do in most organizations. You know technology young technology companies, I think are something like 27 times more efficient than traditional companies.

Established technology companies are something like nine times more efficient. Than traditional companies. And it's all down to their ability and capacity for doing something, seeing what happens, and then adjusting. And again, there are lots of stories, even going back to the early days of Google where, you know, they asked people how many search results they wanted per page.

They gave them what they wanted and then they saw, you know, results dropped away because pages were taking longer to layer. You know, all these things that you know, you ask people and then you do something, you find out completely different. Yeah, getting the real if you can find a way to get real data it's [00:34:00] always the best option.

MichaelAaron Flicker: So that, that's a lovely challenge, especially for our smaller companies that are listening. People that have more control over their brands. Maybe they are direct to consumer. They have, they have their own direct sales challenge, so they have more control. So as a statistician. My question is how much data is enough to give you the insight to make a business decision versus how much should we really push as statistically significant confidence given the practical limitations that you went through that how do you make a decision or how do you have confidence in the decision you want to make?

Phil Graves: So there are a couple of points within that. So think it's worth coming back to the small firm's point. So please remind me about that. The statistical confidence thing you know, I've got a degree in [00:35:00] statistics. I get the math, but it's bogus in practical terms because you know, the fact that I'm saying to you that, you know, I can tell you that 75% of people think this, plus or minus 4%.

To 95% confidence, you know? Yeah, yeah, I know what that means. But what's the 75%? Where's that number come from? Is that a number that comes from a question that you genuinely understand? People can tell you honestly, and that hasn't been influenced either by how you've asked it, the way you've asked it, the fact you've asked it at all.

 You know, the, like coming back to the example I gave you before is not often we get one study that, or two studies that are looking exactly the same thing. They were 60 percentage points apart. Now, that wasn't something that statistics could explain because statistically the confidence interval [00:36:00] on one number would've been two to 10, but worse, probably three to nine, and on the other would've been 60 to 70.

 So, you know, even if you'd gone to the nine, you know, it just wasn't gonna work mathematically. There was that difference because the difference was all in psychology and whether or not people answer it. So you mustn't take false confidence from confidence intervals. Are they really, again, a flippant example, I use, if I told you the height, the average height of a tree in my garden was 60 meters.

And you like, oh, that, that's really interesting. And the question is, how the hell did you measure the height you treat? It's like, well, I asked my 3-year-old how big trees were. Oh, I'm not sure about the 60 figure now. And that's kind of where we are at with with most survey data. But coming back, well, businesses, I had this experience, met with a small company yesterday, and what I [00:37:00] advise those companies to do is to.

Sit down, think about what behavioral data you ca you've either got or you can start to harness and do it work. Working on the assumption that these will be useful things to know. So they might be, where are your clients arriving from? It might be, you know, kind of what are their connections. It might be what's their net worth?

How much income have they got? It might be what's the first product they buy. Whatever it is. Being clear in your mind that you should have a very separate pot for behavioral data because it is true, and eventually it may tell you a useful story, but you should, shouldn't confuse sometimes the ease of getting that with the value that it potentially has to your business.

 And you would do better to spend time developing the, [00:38:00] the structures, whatever, to collect that data now and have it as you're moving forward. And you might get it perfectly right, but put the time into that rather than going out and dropping 20,000 pounds, $30,000 with some marketed research agency who believe that they can look into the future and ask people questions and come back and tell you what you should be doing there.

'cause they can't. 

MichaelAaron Flicker: Very helpful, very helpful. Richard, do you have a follow up question? 

Richard Shotton: No, I, I don't, I just love that phrase of this kind of false confidence and the, that I think is super useful. Maybe we should spend more time hanging a question mark at the end of, of some of the findings, and it might be a danger simply from, it's not just that the data coming in is misleading.

It's the, it's po it's actively dangerous. And I think that's kind of that, that, that's a really interesting thing for people to consider. Not like a slight improvement on not doing something. It could actually be a retrograde step. I [00:39:00] think that, that, that really hit me. 

Phil Graves: One of the things that the insight world loves to talk about is triangulate.

It's like, well do our research here and when we'll do this, and then we'll look at these numbers and then, you know, we'll triangulate and then that will get us to the the best place. It's like. Do you understand what triangulation is? If you've got full data there, it's taking you away from the right point.

 Now you might be running over the cliff with more confidence, but you shouldn't be. You should always be asking, how much can I trust this particular data point? How much can I trust this particular? And yet we keep finding that people go, well, you know, I've got more insights. That's better, isn't it?

Well, no, not necessarily. So yeah, there's, there's a big watch out there with thinking about how much you should be trusting a particular insight. Not just thinking [00:40:00] that you know, the kind of like stamps and collecting lots of them is better. I don't collect stamps. 

MichaelAaron Flicker: As, as we're coming to a close, this is very interesting.

You've committed yourself to writing the book, to having shift as a consultancy that works with, with companies to solve this problem. For folks that come to shift or wanna follow the teachings in the book, one clear recommendation you had is start collecting behavioral data and hold it in a separate.

 In a separate area because we know it's true. Is there other things when you come to work when you partner with Shift or that, when you wanna try to implement what we're learning in the, in the book that you would tell people to go out and try to do, or that you'd recommend as next steps to take this conversation and, and help give folks clear, actionable things to do moving forward?

Phil Graves: Sure. So one thing would be the effect criteria. Is it analysis of [00:41:00] behavior? Is the frame of mind accurate? Is the environment present? Is the focus of what you're looking at? Was it covert? And what's the timeframe in which people have reacted? If you are looking at any piece of insight, ask yourself those five questions.

How many of them can you say yes to? And if you find you can't say yes to very many of them. It's a, it's a good framing example, which is now the frame isn't I should trust this because it's the voice of the consumer. Now the frame is, I realize this is not necessarily psychological, psychologically valid.

There might be an interesting nugget in there. But throughout my career, the market research industry has been kind of been moaning. The fact it's not taken more seriously and doesn't have a seat on the board. Doesn't have more weight than organizations and feels like it should without ever recognizing that it gets kind of exactly the respect it deserves for what it produces.

And the fact that people do in a sense cherry pick from insight to [00:42:00] support their initial hypotheses very often. But so the effect criteria do give you a more rigorous way of saying, yes, we should put score in this, or no, we shouldn't. Several people, senior leaders, can look at together and say, actually, we need to be careful because we're working to an establish set of criteria.

And by the same token, you know, when you are looking to make decisions getting the best quality of insight and the best validity of insight should be your primary concern. And doing that, yes, there are companies like mine that use special techniques and charging quite a lot of money to do it. But, but you can approach it in a same way, in a similar way yourself and with an experimental mindset.

 And do relatively inexpensive things. You know, if you've just made a new come back, my very poor orange juice example, you know, say you just made your new orange juice. Stick some [00:43:00] in the fridge at the office, you know, or in another office, friend's office going stick them in the fridge. The people keep going back to them, picking them up.

That might tell you whether it's nice you know hand on to someone without necessarily making focus of it and see what they say. You know, if you've convinced yourself that this is the precious tasting orange juice in the world, well, do they tell you that anyway? Prompt it or do you need to prime them?

 So, you know, being more creative about insight can can take you a long way. I mean, one of the things I'd advocate that anyone does in any size organization is subject to what you are doing. Just go on, watch consumers surreptitiously go and see what they do. It won't tell you what's going on inside their head.

Without the right techniques, nor will they, but it will tell you how long do they spend at the fixture? How long do they spend? Do they look confused [00:44:00] when the service agent is talking to? If you are one of those companies that has phone in service, has customer service by phone, listen to a random sample of calls, and just as a human being say, does this feel like it's a.

A good rewarding conversation for the customer who's called in If it doesn't, okay, what would change that? You know, a lot of these things are there in front of us and the, some of the traditional insight work just gets in the way and stops us seeing what's really going on. 

Richard Shotton: I think that that's a really interesting one of just going and watching potential customers, how, how they interact, listening to goals, because as organizations.

Get larger and more bureaucratic. Market research becomes something that you go to other people to do for you. And I think that's a lovely idea of thinking, well, if you do that, you are missing out on many of these interactions. That could be, [00:45:00] could be hugely valuable. Maybe it's something that more people to think of, how can I get these experiences rather than asking someone to pre.

Phil Graves: Yeah, absolutely. And I think one of the challenges that that can produce is that, let's say we are a company and you Richard go off and you go, well yeah, we were talking about launching this product and I've been watching people and they all just grab Tropic Corner in a heartbeat and I think we might be wasting our time.

It's like, well, no, that's just your view. You know, that's just what you think and you never wanted to do this anyway, and you get all of that. Whereas, you know, it's like, well, I've just commissioned this focus group report and they're saying they not. So, you know, that's just you. And this is, this is that we spent 15,000 pounds on this.

So we'll believe that because you're just giving me your free where in fact you are, right? So one of the ways around that is to make that process a little bit more full and say, look, we're working on this. It's important to the business. All three of us are gonna go out [00:46:00] independently and see what we can see in this category, and we'll come back.

We'll write a brief summary of what we've seen and if we've all seen the same thing, that's probably quite a useful insight. Now we've seen something different. We need to dig into it a little bit in the moment. We've not spent a huge amount of money. We've got closer to our consumers. We haven't asked consumers anything at this point.

You've got that, that's something you have to handle with great care. You can learn things and one of the things I've said several times is I have no problem with talking to consumers. I only have a problem with believing things they tell us. So the problem tends to be if I say, you know, Richard, you went to a shop and watching you using, yes, I did.

And I asked a couple of people and they said they thought this would be a really good idea. Now that feels like it has weight. And it shouldn't. I mean, what are the things in in our report, you will never see a verbatim quote because [00:47:00] fundamentally we don't believe what people say. So why on earth would we.

 Looking is how they think and what's going on in their mind. And very often that's about what they don't say or about how they say something or about how what they've said relates to what they've done. So, you know, we don't need to report any of that to people. Or people still love verbatim quotes 'cause they feel like they're getting something from straight out the horse's mouth, so to speak.

 But again I shouldn't so, so structuring that sort of exercise a really good way, no one's. Spending thousands of pounds, but often can be more valid than than the alternative that you are wasting money on. 

MichaelAaron Flicker: This is lovely, really a, a, a super interesting and challenging conversation Phil. There's lots of questions I want to continue to follow up with.

The one that I'll end on is could you share something you are excited about? In the future of, of of this field, [00:48:00] whether it's how AI might impact what's happening with, with, with understanding behavioral data or, or some other trend or insight that you're, that you're excited about. What's Phil Graves most excited about in the upcoming future?

 In this work? 

Phil Graves: I think it would be a combination of AI and quantum computing. Not that I really understand either at any kind of useful technical level. I've seen the same adverts as everyone else. But I think one of the problems that we've got people have been excited about collecting behavioral data for quite a long time, and what they quickly find is in most organizations, they get too much too quickly and they cannot deal.

 I think AI can help with that. And I think more computer processing power can help with that. And together those two things will [00:49:00] see us looking a lot more to the value of capturing and using data than, than anything else that I see up ahead. So I think, I think that for me is, is an exciting an exciting future.

If, if your goal is to ultimately. Understand and optimize what consumers are doing. I mean, frankly, I don't, there's much money in it for me, but that's fine. Yeah, yeah. I'm old enough not to worry about that too much anymore. But I think in terms of getting gen, you know, caring about, you know, this, this subject, caring about understanding consumers better, I think, I think that's a, a really interesting area that has huge potential.

MichaelAaron Flicker: We'll all watch it together with a lot of excitement. That's really great. Thank you again, Phil, for joining us today. For those listening if this conversation was interesting, please leave us a comment, [00:50:00] interact with us on this topic or share this podcast with someone else who loves marketing and behavioral science as much as we all do.

And until next time, I'm Michael Aaron Flicker. 

Richard Shotton: And I'm Richard Shotton. 

MichaelAaron Flicker: Happy listening.

Advertisement: Behavioral Science for Brands is brought to you by Method One, recognized as one of the fastest growing companies in America for the third year in a row. Featured on Ink's, 5,000 list. Method One is a proudly independent, creative, and media agency grounded in behavioral science. They exist to make brands irresistible, helping people discover products, services, and experiences that bring moments of joy to their lives As behavior change experts Method one creates emotional connections that drive true brand value for their clients.

Focusing primarily with indulgence brands in the CPG space. Find out more@methodone.com.