Episode 176: AI Coaching Meets ICF Standards with guests, Jonathan Passmore & Rebecca Rutschmann

What happens when an AI coach is judged by the same yardstick as a human? We invited executive coach and researcher Jonathan Passmore and AI coaching innovator Rebecca Rutschmann to unpack their new study benchmarking an AI coach against ICF Core Competencies—and the results upend assumptions. The machine reliably demonstrated ACC-level performance and crossed more than half of the PCC markers, especially in crisp summarizing and steady open questioning. That said, we draw a clear line between competence at the basics and the deep, sustained presence required for identity, values, and ethically nuanced conversations.

Across the hour, we explore where AI coaching shines—24/7 availability, structured reflection, accountability loops—and where it still stumbles: longer arcs, emotional complexity, and the tendency to praise rather than challenge. Rebecca argues capabilities are leaping forward with better prompting frameworks, onboarding, and conversational design, pointing to recent builds that reach deeper reflective work. Jonathan counters that human strengths remain decisive: relational humor, embodied presence, lived experience, and ethical maturity that can hold discomfort without defaulting to platitudes. We converge on a future of hybrid models that use AI for pre-work, micro-coaching, and late-night clarity, while reserving human time for complexity and transformation.

We also face the economics. With a surging supply of coaches and falling fees for transactional work, differentiation becomes urgent. If AI can do the basics well, human coaches must elevate to PCC-level craft as a baseline, specialize with domain and identity expertise, and design client journeys that blend AI tools without diluting trust. Finally, we call for new standards: if AI is an orange to the human apple, we need AI-specific metrics for safety, continuity, bias, escalation, and outcome transparency—so clients know what they’re choosing.

Curious where to start? We share practical steps for AI literacy and fluency, plus communities and programs that help you experiment safely and ethically. Subscribe, share this conversation with a colleague who’s on the fence, and leave a review with your take: partner, threat, or both?

Watch the full interview by clicking here. 

Find the full article here.

Learn more about Jonathan here. 

Learn more about Rebecca here.

Grab your free issue of choice Magazine here - https://choice-online.com/

Transcript

Welcome And Guest Intros

Garry Schleifer 0:07

Welcome to Beyond the Page, the official podcast of choice, the magazine of professional coaching, where we bring you amazing insights and in-depth features that you just won't find anywhere else. I'm your host, Garry Schleifer, and I'm excited to expand your learning as we dive into the latest articles, have a chat with these brilliant authors behind one of them, and uncover the learnings that are transforming the coaching world. When you get a chance, join our vibrant community of coaching professionals as we explore groundbreaking ideas, share expert tips and techniques, and what's most important to a lot of us coaches, make a real difference in our clients' lives. This is your go-to resource for all things coaching, but in the meantime, let's dive in. In today's episode, I'm speaking with Executive Coach, Team Coach, and Coach Supervisor Jonathan Passmore, and AI coaching innovator Rebecca Rutschmann, who are the authors of an article in our latest issue, Coaching to Unlock Joy. Their article is entitled Benchmarking the Machine: How AI Coaches Fare Against ICF Core Competencies. Jonathan Passmore is a qualified Executive Coach, Team Coach, Coach Supervisor. He's also the Senior Vice President of ESRA Coaching and a full Professor at Henley Business School. He has published widely with more than 40 books and 250 articles and book chapters, some of them in choice Magazine, even before today. His recent books include Becoming a Team Coach and the Digital and AI Coaches Handbook, of which I have a copy. He provides coaching supervision and can be contacted via his website. Rebecca is an AI coaching innovator specializing in the intersection of coaching and artificial intelligence. She is the co-founder of Viva La Coaching Academy, of which I'm a member, where she promotes AI literacy in coaching and corporate learning and also leads the development of AI-based coaching tools with external partners. She focuses on ethical AI integration in coaching practice and contributes to global research initiatives advancing responsible and human-centered use of AI in professional development. Again, thank you so much for joining me. Welcome back, Jonathan. Rebecca, I'm so thrilled that you guys did this, and we stuck through and got the article published because I think it was a very important piece to update coaches on what how AI is doing. And it's no surprise that the two of you got together. Let me tell you, reading your backgrounds is a perfect fit. And I hope you're up to more stuff.

Jonathan Passmore 2:45

We certainly are. Lots going on, Garry.

Garry Schleifer 2:49

I'm sure.

Rebecca Rutschmann 2:49

The year has just started, Garry.

How The AI Was Assessed

Garry Schleifer 2:52

Exactly. Well, and Eva from Viva La Coaching has put out her annual email and told us a little bit about what to look forward to, so I'm excited without letting any details out, just to be excited. Well, let's get into the article. This article is about a study, and your study showed that the AI coach could reliably demonstrate ACC level competencies and even meet more than 50% of those famous PCC markers. So, what surprised you most about how the AI performed when assessed like a human coach would be assessed?

What Surprised The Researchers

Jonathan Passmore 3:35

So what we tried to do in the study, Garry, was to use a human method for assessing human coaches and then put the AI tool through that same process. So we had expert ICF assessors who themselves were also MCCs reviewing transcripts from genuine coaching conversations carried out between a human client and the AI tool. And so in my mind, there was some skepticism about how good is this thing actually going to be? Where are we in the development of AI? Because many of us have followed AI since really the generative tools started to become available in 21, explored looking at what's been happening during 22 and 23, and early on they were incredibly unreliable. But we have seen a progression of development quicker than humans have been developing. So it's just getting better and better, faster and faster. So putting the tool through an assessment, what I suppose surprised me was how good that it was. I was wondering would it be hard to demonstrate, which is the purpose of our research, was it going to be able to demonstrate that it could be a " professional coach" at any of the levels? And when the assessors looked against the competencies, it ticked off, as you said, ACC. So we've got already, or at least when we carried out the study a year ago, we've already got AI tools that are as good as competent, professionally qualified human coaches. And the things that it was really good at maybe probably doesn't surprise us if we have a think about what AI's capabilities are. So it was really good at, and I'm going to use human words for this, but of course the tool is not actually doing this. It was really good at listening, it was capturing that data that was played to it and able to then play that back in a summary. Now, my experience at Henley with hundreds and hundreds of new coaches, many of whom are highly experienced professionals, senior leaders, is that actually they weren't, as human coaches, as good as the AI tool was because the AI tool is in a way predisposed to capturing data and then just summarizing that back. And it was also really good at asking open questions, sticking to that open question approach as opposed to what many novice coaches do is ask leading questions or closed questions until they begin to build up that open question muscle. So those were some of the things that maybe should have surprised us. But when I looked at this and thinking about the capabilities that we can see that have emerged by 23 and 24, the AI capability to be able to capture data and play that back, probably shouldn't really have surprised me. Rebecca, what surprised you?

Rebecca Rutschmann 6:40

Actually, I was surprised that, like due to my experience playing with Alpina back in those times, it was like the summer of 2023 when we started, no? Like 25, no, 24. It was the summer of 24, is when we started. I think June or July, when I started building it. So it's quite a long time. It's actually more than one and a half years now. And back then it showed already amazing capabilities. I was most surprised that it did not go higher yet, by like assessed by human assessors, because I have, of course, AI assessors that assess and match against the competencies. And I think they go a little bit higher, but I think it's a variety here, a variable, anyways, because it's hard to measure that. I think like what probably surprised me the most in this regard is more you know, like when we look at it to the human capabilities, that it didn't show certain differences in where AI can. I think it actually is much better in certain things than human coaches, like you explained, Jonathan. I have a hard time putting that into words. On the other side, I was surprised that it didn't show back at that time already a bit more of the emotional side, which could be back to the prompting that we did, because we really framed it well, Jonathan. That's why it was only asking, you know, like the open questions, no closed questions. It really kept very well to the framework. And we made sure that we have a really good onboarding as well, because this is key and crucial when you are applying or using any of those coaching chatbots, because you do want to have that framing right, yeah, also meeting the human competencies. And my thinking is really like where my thinking is heading is like if we really measure the right things when we measure AI, if we can really compare that with the human capabilities, or if we might not need different standards for both of that.

Strengths And Limits Of AI Coaches

Jonathan Passmore 8:52

I think you raise a really great question there, Rebecca. And I think we we were using a human metric because what else was there to look at? But I'm beginning to think that really AI and human coaching are different things, that they are apples and oranges, and that they serve different purposes. And one of the things, maybe this is going to be your follow-up question, Garry, what's the thing that maybe disappointed you in this? And that what disappoints me, and Rebecca might disagree with me on this, but what disappoints you about AI is its capacity to stay focused over the long term. So I have consistently, or more recently, in some other studies, described AI as short, shallow, and sycophantic. And that too frequently it's always in oh Jonathan, you've done so well with this. Well, do I really want my coach to keep praising me? Praise is of course good, but I also want it to hold my feet over the fire. And maybe AI, unless you're really priming it to do this, has a natural tendency to go for the praise rather than for the challenge. I think conversations have often been short, so it's fabulous when you wake up at two o'clock in the morning. But to be able to sustain a 45 or a 60 or a 90-minute conversation with AI has so far proved difficult. And to move beyond goal-focused conversations, so it's strong, maybe even stronger, as Rebecca hints here, at the ability to help individuals to think through, develop, explore their goals, develop plans or insights connected to those. But when clients want to explore aspects about identity or values or emotional depth, or things which are ethically nuanced or culturally sensitive, my experience is that it lacks the capability that a human coach who brings that wisdom, experience, insight is able to display in a sustained way across a 45 or a 90-minute conversation. So I'm beginning to think they both have a role, but their roles at the moment are distinctive. Now, maybe they we might see over the next three months or the next three years a coming together of this. But at the moment, I like apples and oranges, but they're different. What do you take of that, Rebecca? You disagree with that? You've got a different perspective?

Rebecca Rutschmann 11:41

I do. You know me. I still remember our first podcast, Jonathan, when you said, like, who wants to have sex with a robot? You know, I wouldn't want to have coaching with an AI. I don't know if you remember that.

Jonathan Passmore 11:58

I do, I do.

Depth, Presence, And Inner Work

Will AI Replace Coaches

Rebecca Rutschmann 11:58

But that was before, you know, like generative AI hit the market. And so for me, at the moment, I think, and I'm more than happy to provide you a new version, Jonathan. I'm really impressed. Like, we built some versions the last few months that are really truly going into the very inner work, still different to what a human coach would do. And I think we will see those differences now coming and popping up more and more. And I think we need to focus more on that in the future. But I've used a version with one of the big corporates here in Germany, and one of their, I'd say the one actually, or one of them who is actually buying some of these technologies, he was really impressed by some of the versions we provided them with and how much inner work they can already do. While before, and I totally agree, Jonathan, they were quite shallow. You could do like an up to, I think we measured around 20-minute conversations, they were really good at going more into the goal attainment. But what I see specifically with the latest models coming out since the five version is really the the amount of inner work that it can do with you is getting much better. And we're not gonna get to a transformational level yet in any way, but we're far beyond transactional already. I think we're entering now a level where AI, if prompted and framed well, yeah, and not you can't go just do ChatGPT. You have to really have something created around that. You need a proper framework. Yeah, to do that, a prompting framework, you need to structure that well. You need to set these guardrails, yeah, and like conversational design for a good AI coach is really essential, and you need a lot of experience on that as well. But when you apply some of this, and I know Garry, you've been doing your experiments on that now as well, and going down the track. So you know how hard it is to get something really good out of it, but you can reach really amazing levels now. And I think the last version that we did was we actually reached 81% of if we would measure it with the PCC markers, we got to that stage. But I think it also shows different things that are not to be captured yet by any of the, let's say, normal standard coaching capabilities or competencies, depending on who you're referring to in regards to coaching bodies. And I think this is where it gets it's gonna get interesting in the future, because like you said, Jonathan, we're gonna use it for different things, yeah. And I think topics like trust or presence or accountability, responsibility, or like you said before, ethical and moral applications is something that is going to always stay with the humans, even if it's just humans building an AI coach, yeah, and being responsible for them, deploying and applying that AI coach in their practice. But we need to know what we're actually talking about. Yeah, while other things, I think in AI, I did something now on the way back from the Black Forest because I was going through a couple of reflection things as well, and it's really good. I had a very long conversation, Jonathan, and you can use voice now and all that, and it goes on and on, and it's beautiful. And there are some things that are really helpful, and then when you know the tools, you know when they start getting stuck because they still go into this more narrow thing. Once they go down a certain track, it's hard to leave and go and identify that there might be something else needed now, yeah, for that person. And that's I think where they will always lack for the near future until we have different types of models. And we see already now in the industry that LLMs are actually hitting a wall. But that's more on the technology side. But we will also see that there are just things that it won't be able to reach.

Garry Schleifer 16:01

So that brings up a question that I still carry from the first issue about AI and technology. First of all, thank you both for being the friends of coaching, the coaches that are keeping an eye on all of this and bringing us this information back. What's your advice to coaches? Because I could hear some coaches being concerned right now about the word yet that was used in today's conversation. And but then again, never. So what which is it? Is it yet or is it never being replaced by AI?

Differentiating Human Coaching Value

Jonathan Passmore 16:35

I think it's too hard to say. Predicting the future, there's one thing you can be certain about. You're gonna be wrong. So I think that human coaches need to find different ways of differentiating themselves from the capabilities that the AI tools will develop over time, and some of that is things such as humor, our ability to laugh, and not a joke. AI can tell you a joke, it's usually like a dad joke, it's not that funny, but it is a joke, it's a formula that it's able to produce. But what is not able to do because it isn't relational, is share an experience with us, often an embodied experience, often something's connected to something that's happened in the media yesterday or today, to bring that in that's a shared experience with the other individual. And so humor is relational and situational. I think AI is less capable of doing that. I think it's less capable, uh, as Rebecca and I agree, on the ethical maturity to navigate the nuances and values that we bring into our decision making. I think it's less able and will be less able to have the embodied experience, even though you might have a two-dimensional Gary coach who's able to connect with you, it's different than the human who is even on a screen, but certainly different from the human who's going to be in front of you. So I think there are a number of elements that uh that we as humans can dial up our lived experience. So if you're a woman and you have a shared experience with your female coach, if you're a person of color, you have a shared experience with them. If you work as an engineer or an accountant, that's part of your identity. You have that shared experience. So some of these may be themes that as human coaches we can dial up that will enable us to differentiate and sustain ourselves in a different way than what the AI coach, however fast and far it develops, to maintain human coaching. But I still feel that we've probably reached peak coach. We've got enough coaches now, human coaches in the world, to meet the needs of human clients who can afford to pay for it. Because if we continue to grow the coaching population, there are a hundred thousand plus members of the ICF. There are probably another hundred thousand plus people who identify as a coach, some of whom are EMCC or Association for Coaching or a Psychologist or got other forms of coach training or accreditation, and then there's a whole bunch of people who might have done training or may have not done training, but still identify as coaches. If we doubled that, let's say 200 or 250,000 to 500,000, what's the effect of an increased supply in a market that uh is has only got a certain capacity to pay? Price comes down, right? And we've seen prices in coaching continue to fall. I spoke to somebody just before Christmas who were telling me that they were invited to work for an organization as a coach for $21 US an hour. Wow. I mean, yeah. So the world has been changing over the last 25 or more years that I've been involved in coaching. Fee rates have certainly not gone up, they've only been coming down, not just in real terms, but in cash terms. And if we double the number of coaches, they're only going to continue to come down. So coaches become little more than Uber drivers. So we need to find ways of differentiating our humanity to provide really high-quality coaching for clients who can afford to pay. And what AI can do is to complement that in hybrid models. And in my view, it can also democratize coaching for people who want to access it at a more cost-effective price and maybe a different service. And one of the advantages of AI is that I can speak to my AI coach with 20 seconds notice at 3 a.m. in the morning when I can't sleep. And I've got a business presentation tomorrow and thinking, well, what should I be in it? AI can coach me through uh to find a solution, have a plan, and then at 10 past three I can go to sleep, knowing that I've got a plan for the next day. My human coach is not able to do that. They wouldn't appreciate me, would they, Garry, if I rang you at 3 a.m.

Garry Schleifer 21:25

You wouldn't get me. I'm on do not disturb until 7 a.m.

Rebecca Rutschmann 21:28

But it's you if it's your three in the morning, it would work, yeah, Jonathan, because then he's still awake.

Market Dynamics And Access

Jonathan Passmore 21:34

Well, that is true. That is true. But he wouldn't get me with the night. Yeah. And so Rebecca, I'm stimulated to think maybe with 5.2 now available, maybe we should run something else as a a follow up study and think about what are the appropriate metrics? Because I share your view that we've done it once with human metrics. What are the AI metrics that we should be seeking to assess an AI tool against? And at the moment those haven't been developed and there isn't a consensus around these. But I share your view that probably AIs are oranges, not apples, so we should have a different set of metrics to assess them against.

Raising The Bar For Coach Training

Rebecca Rutschmann 22:32

Oh yeah, that's happening already, anyways. I mean, like in conversational AI, which is a conference I'm actually going to in the beginning of March. It's a European Jackpot conference. They talk a lot about you know, like what a conversational AI agent should be doing, and also in behavioral science, there are some behavioral scientists there, so it'll be interesting to see how they measure things, how intelligence is measured as well. I think there are a few things where we can draw from, Jonathan, on that. But to get back onto your question as well, Garry, on what you know, like I would like to people to do next, or what I can recommend, and where I totally agree with you, Jonathan, that coaching will become a skill set. I think the amount of coaches that the market is completely satisfied, let's put it this way, with like coaches that are kind of like in between. I think when you are a very masterful coach, you have done your inner work and all that, is it's a different story. I mainly work only as a team coach. That's where I come from. I come from IT, work mainly with development teams in my past. And AI literacy and fluency, yeah. So literacy is the basics. Fluency is really knowing how to do and apply this also responsibly, is for me a complete non-negotiable for the future. I mean, I'm coming back down from the Black Forest. There's not a lot of people up there, but one of our family like members, he runs, he's a carpenter. He works with AI in his business and he sees all the positive things about that as well for some of the stuff. I mean, as of course, a carpenter AI won't be able to put up that house in the neighborhood, yeah, or to do some of the things. But he was really open and very curious to use it. So for me, it's all about really educating yourself, just like in the beginning of the times when the internet boom was. That's where I grew up. Yeah. So I started studying in 97. And then learning to how to really apply AI responsibly and ethically, yeah, how to become also the better human in the room, yeah, how to become better as the coach. And one thing we always agreed on, Jonathan, I think you mentioned that last year also at NYU, that we should really focus on having better coaches in the market. We need to also use, in my opinion, AI to have better coaches in the market. When coaches come out of the coach training, they should have, let's go back to the competency sets, but they should be on PCC level, nothing less. Yeah. And I think we can do that. It's not that hard. If you look at other professions and how much training they have to do and how much work they need to do to actually really, you know, like get, you know, like certified on a higher level. I think that's really interesting. And I'm working with a university in Switzerland at the moment on exactly using also AI for getting people out, you know, like on a higher level level earlier. And we can use really cool AI tools to do that as well, to really become that better human being and the better coach in the future. Because when, of course, you know, like AI can do goal attainment, you're not gonna gain, you know, anything anymore or won't earn money with goal attainment in the future. That's where probably the 21 US dollars are gonna come in per hour. That's something that AI will cover easily. Yeah, so it's for me learning, applying, and and really staying curious and go with the flow. And you don't have to apply it everywhere, but see where it fits in and what fits you, what fits your clients. It's gonna be, yeah, this negotiating. Yeah, some people feel more comfortable, some less, some will use it for some things, and we still need to figure that out. That's where the yet comes in as well. We don't know yet where this is going.

Garry Schleifer 26:18

And we've lost Jonathan, but that's okay. So we're gonna get wrapping up. So here's my question for you, and I think you're the best person to have this answer, Rebecca, and that is where does one go to get their AI literacy and fluency?

Rebecca Rutschmann 26:35

Of course. Yes, you can go different ways. Of course, I mean that's why we also founded ourselves, Viva La Coaching, about a bit more than a year ago now, because we saw that huge gap out there in a market. And we saw that if people don't learn about AI now, they're gonna be lacking behind. And we hope that we can catch as many people and support as many people uh to actually like find a community where they can safely learn about AI and apply it and change ideas and all that. And I know there are several others out there in the market now as well, but I think our new programs are going to start this spring, and there will be continuous things going on so you can watch the space. We put out a free webinar every month, anyways. Maybe some more in the upcoming months that people can join just to learn more about it and expose themselves, read some research papers as well around the topic, inform yourself, talk to people that use it, like you, Gary. They can also approach you and other people from the Viva community and others that are offering AI training, but really try to look and and find you know something where you can get more knowledge on that topic. Yeah.

Practical Next Steps For AI Literacy

Garry Schleifer 27:53

Well, and I will give strong kudos to Viva La Coaching, to you and Eva for your work. I've enjoyed it very much. And if I wasn't coaching so much, I'd be at more of the sessions. But I do listen and read the newsletters, which are also very informative. So I highly recommend that people become part of your community to find out more about AI literacy and fluency. Rebecca, I want to thank you so much for joining me for writing for the article. Say hi to Jonathan when you see him because he fell off. It happens.

Rebecca Rutschmann 28:28

He'll be back, yeah. He'll be back.

Garry Schleifer 28:30

I know, like a bad penny that one. What's the best way to reach you? I'm sure it's something VivaCoach.

Rebecca Rutschmann 28:37

Yeah, it's vivalacoaching.com or LinkedIn. Yeah, so you can always pop me a message over on LinkedIn as well. That's where I'm most responsive, probably better than email, to be honest.

Garry Schleifer 28:47

Well, I can't wait to hear about future developments. If the two of you are working, yes, please let's get something published as your results.

Rebecca Rutschmann 28:54

And we will be working on the human factor this year as well in AI. Yeah.

Garry Schleifer 29:00

Awesome.

Rebecca Rutschmann 29:01

The big topic for me in 2026. Yeah.

Garry Schleifer 29:04

Awesome. Well, again, thank you so much for joining us for this Beyond the Page episode.

Rebecca Rutschmann 29:09

Thank you for having us, Garry.

Garry Schleifer 29:11

That's it for this episode of Beyond the Page. For more episodes, subscribe via your favorite podcast app, most likely the one that got you here. If you're not a subscriber to choice Magazine and you're watching, you can sign up for your free digital issue by scanning the QR code in the top right hand corner of our screen, or by going to choice-online.com and clicking the sign up now button. I'm Garry Schleifer. Enjoy the journey of mastery.