Customer Success Playbook S3 E39 - Matt LeMay - AI Is Not a Strategy

In the finale of our three-part series, Matt LeMay takes on the tech world's shiniest object: artificial intelligence. From summarizing meeting notes to influencing million-dollar decisions, AI is here to stay. But Matt warns against using AI as a goal unto itself. Product teams must ground their experimentation in business logic—or risk becoming distractions.

Detailed Analysis: Episode 39 brings a timely and pragmatic take on AI in product management. Matt acknowledges AI’s strengths—particularly in summarizing information and idea generation—but urges caution. When AI tools lack source context, they can mislead as easily as they can inform. The key is discernment: understanding when AI adds value and when it distracts.

The team unpacks how AI is impacting product roadmaps, budgeting, and even job security. Matt shares war stories of teams told to "do something with AI," only to later realize the result consumed $200K to close a $50K deal. Oops.

But this isn't an anti-AI rant. Matt offers frameworks to leverage AI wisely while keeping business impact front and center. If you’re going to play with shiny tools, make sure they’re solving real problems.

Now you can interact with us directly by leaving a voice message at https://www.speakpipe.com/CustomerSuccessPlaybook

Check out https://funnelstory.ai/ for more details about Funnelstory. You can also check out our full video review of the product on YouTube at https://youtu.be/4jChYZBVz2Y.

Please Like, Comment, Share and Subscribe.

You can also find the CS Playbook Podcast:
YouTube - @CustomerSuccessPlaybookPodcast
Twitter - @CS_Playbook

You can find Kevin at:
Metzgerbusiness.com - Kevin's person web site
Kevin Metzger on Linked In.

You can find Roman at:
Roman Trebon on Linked In.

Transcript

Kevin Metzger: 0:05

Customer success.

Roman Trebon: 0:11

Welcome back to the Customer Success Playbook podcast. I'm your host Roman Tree Bond. Joined by uh, my co-host Kevin Metzker. It's Friday. We've made it to the end of the week, and we're wrapping up our three part series with author Matt LeMay. Kev, we've had an amazing week with Matt. I'm excited for Fry ai, AA Friday, whatever we call it nowadays. I think we've changed the title four times. Are you excited to get into this?

Kevin Metzger: 0:36

AI Friday. Always good to you sound. You sound excited, Kevin. No, I love AI Friday. It's always interesting to talk and, and get our guest perspective about, about AI and how it's impacting. They're part of the business and how they see how they see ai because it's such an evolving and really dynamic topic at this point. Matt, like I said, ai, it's transforming everything right from road mapping and user insights to potential layoffs. If automation displaces certain roles. How can product managers leverage AI to enhance the impact without losing sight of core business goals when budgets or leadership priorities are shifting?

Matt LeMay: 1:19

I'm, I'm of several minds about ai, and I think as with many of us, my opinions keep shifting. As the landscape of AI shifts, I've found that AI can be really good for sort of summary tasks, for taking a lot of information and synthesizing it and providing a. Basic defensible synthesis, which is sometimes really valuable. Like there are times where you wanna throw a bunch of meeting notes in there and say, all right, what are some themes that are coming up? Or, you know, in the process of estimating impact, which is something I talk about quite a bit in the book, what I recommend doing generally is, is Googling and looking for sources and figuring out. Kind of, you know, if you're trying to figure out, for example, what's the impact that improving password reset would have on an e-commerce business? Well, you can Google how many people forget their passwords. What's the average value of an e-commerce cart? Now where this gets tricky is AI can give you sort of a synthesized baseline for those kinds of things. But where you have to be more deliberate in the way you use AI and the way you write your prompts is making sure that you also understand the sources where these things are coming from. A lot of the white papers and things published that can help address some of these common challenges are published by companies that have a vested interest in making these things sound important. So for example. You know, if you're looking up what's the value of password reset, there are a lot of identity solutions companies that are gonna have a really strong opinion about why password reset is super valuable and you should work with them. Now, if you understand that, that can be really valuable.'cause you can look at that and say, all right, well, it's really unlikely that they would understate the importance of this, right? So if we assume that this is a maximum, how valuable could it be? But again, that requires that. That kind of element of human discernment of the motives and context and tone of information, which can get lost easily in AI unless you are really thoughtful about how you use it and checking the sources and asking AI systems to provide citations and things like that, which it will also hallucinate sometimes. So again, I think. As with a lot of things, you know, I went through this when it was machine learning 10 years ago. The question of like, how wrong can we afford to be in this case is always a really important question to ask. And I think that with AI, it can be a great way to get a pulse check. It can be a great way to take a bunch of messy information and synthesize it into a good starting point from which you can then further refine information. You know, I think for product people. A blank page can be really terrifying. And if you go into, you know, a large language model and say, if this is what my team is thinking about, can you give me what a one page product spec would look like, it's probably gonna be easier for some folks to edit and modify that than to just start writing from scratch. So I, I think as a thought starter, as a tool for synthesizing information, AI can be really valuable. But I think that that final level of discernment, that ability to contextualize sources, to understand what really matters, to, again, tie things back to the particular business model of your company and communicate things in a way that will resonate and make sense to the particular personalities in your company, is something that will always be valuable for product management.

Roman Trebon: 6:08

Matt, I'm curious from, from, uh, you, you know, you're in the, you're, you're an expert in, in the product space. Uh, do you see or can you see that AI can maybe be distracting from impact that it's such a big concept that like we, it's over indexing and it's taking away from what real impact is?

Matt LeMay: 6:26

You know, I remember seven or eight years ago, eight years ago, I think this was the peak of Pokemon Go Mania. I was working with a bank. Somebody asked very earnestly, well, what's our augmented reality strategy at a bank? You're a bank. Yeah. Yeah. You're make your app work and then we'll talk about augmented reality. But I, I think, I think there's always been some shiny new thing. And here's where I feel for product teams. Again, you know, as, as we discussed in, in our very first conversation on Monday, the business ultimately expects something from you. The business is ultimately looking to your team to deliver success, and if the business says to you, go make something having to do with ai, that's not great because there will come a time. Where the business turns around and says, all right, well, AI has either been commoditized or it's not as exciting anymore. Or, our CEO went to a conference and learned about something else, which is what we care about now, um, they're gonna turn around to that team and say, all right, well, what's the impact of your work? Why are you doing this? And if the answer is, well, you told us to do something with ai, so we did. Again, that's not a great answer. I, I've talked to some product people who say that they're being told to. Build things in service of$50,000 contracts that will cost$200,000 of just AI usage to build and. You know, I, I, I don't, I don't think I'm a business genius, but I think I could probably build a business that turns$200,000 into$50,000 if I tried. Yeah. So again, I think part of the challenge is, you know, businesses are not always rational. It's, it's people I. People get excited about things, and people want to jump on trends and, and do things that are cool and exciting and new. And part of my challenge to product teams is to say, all right, imagine a year from now, how are you going to explain the benefit to the business of the AI product that you're working on? And if the answer is well. You said, go do something ai, and we did something ai. Again, I don't think that actually puts you in a terribly strong position. And I think that come time for the realities of a business to kick in, whether that's, you know, raising around of funding or making a report to shareholders, if these go do an AI features and products are not delivering meaningful results to the business, they are probably not going to be. Plentifully resourced. Moving forward.

Kevin Metzger: 8:59

I had a couple of thoughts I wanted to actually run by. Yeah. One. I think your point's very good, right? You can't just use AI to use ai. That's from a product standpoint, from I, I realize that a lot of people are doing that. Realistically, you should be looking at, Hey, is there a way AI can make an impact that can provide value beyond what we're doing in our product today? And really. Spending time doing evaluations on where, where that works, where it doesn't work, what value it can bring. And just like any type of productization you should be looking at the value of what you're trying to accomplish and whether you're getting a return on that value. I guess that's more of a statement. I think the other, the other piece is, is a question on have you had the opportunity to use any of like the deep research tools yet? I know it's, they're relatively new to the market, but like open AI's got a deep research there. There's, there's a few tools that have the deep research out yet, and I'm just wondering if you've had an opportunity to see results from them yet, because those are getting very interesting when doing. Some of the queries that you were talking about where you're previously getting hallucinations and don't have all the references, well now you can get reports back with detailed references and opinions and it's, it's really some interesting details that you can get back. So just wondering if you had a chance to look at them. I

Matt LeMay: 10:36

haven't had a chance to use those yet, but that does seem like a really good use case for ai. I mean, and in a funny way, I, I think. With new technologies like this, the initial impulse is, oh, let's make this seem like magic. But I think when you look at things like deep research, part of what it's doing is showing its work a little bit. And I, I think that ultimately there are contexts where understanding the why behind something is, is really valuable and, and seeing some of those intermediate steps before you just get the answer. Certainly makes that information more valuable for folks who know how to read and contextualize that information. So I'm, I'm excited to spend some time with some of those tools, but I haven't had a chance to yet.

Roman Trebon: 11:16

That'll be your, your next book, Matt, your next book. Let's not easy yet, right? No, Matt, thanks for joining us this week. We really appreciate it. Your book. It's my pleasure. Impact first product team is Matt. Like I told you, I I, I read it. I love the book. I'm a huge fan. So for our audience, even if you've, I've never been in product and I loved it. I, you know, again, it's, you can replace product, insert your team. There's tons of insights, tons of takeaways. Matt, for our audience, where can they find more about the book yourself, uh, et cetera.

Matt LeMay: 11:45

Uh, Matt lame.com is the place to be. That's, I got links to the book. I've got some talks I've given, got ways to reach me, so I am, I'm pretty easy to find.

Roman Trebon: 11:56

Awesome. Matt lamay.com. Check him out there. Uh, again, definitely pick up the book. It's, it's tremendous. So that is the end. We've come to the end of our, our three-part series with Matt. If you enjoyed these discussions to our audience, again, thanks for listening. Please subscribe and share them with your team. We'll be, we'll be back next week with more tips and strategies for your customer success playbook. Until then, Kevin, keep on playing.