Learning Experience Leader

44 // How To Measure Business Impact of Learning with Dr. Bonnie Beresford

September 22, 2020 Greg Williams
Learning Experience Leader
44 // How To Measure Business Impact of Learning with Dr. Bonnie Beresford
Show Notes Transcript

Bonnie is a human capital strategist and performance consultant. She is the creator of the Measurement Map which helps to illustrate the causal chain of evidence between human capital interventions and measurable business outcomes. Her goal is to help organizations create environments and build skills that truly engage and enable their employees. 

Today we talk about: 

  • Getting to a shared definition of success with stakeholders 
  • Alignment with stakeholders as the critical part of Analysis
  • Bonnie’s measurement map model 

I tried something new in this episode by role-playing a scenario around a training request and Bonnie’s response is brilliant. I hope you find this as insightful as I did!

Resources
👉 So You Want to Measure Impact, Now What? (article - description of the measurement map)
👉 20 Things I’ve Learned from 20 Years of Learning Measurement (article)
👉 GP Strategies Podcast interview with Bonnie (podcast episode)
👉 How L&D Can Link Learning to Business Outcomes (podcast episode)
👉 33 // Evaluating to Demonstrate Evidence of Training Value for Stakeholders with Dr. Jim Kirkpatrick (LX Leader podcast episode)
👉 How to Measure Anything (book)
👉 Developing Human Capital (Bonnie's book) 

Support the show
Dr. Bonnie Beresford:

The first thing we have to do is figure out what is it we're going to measure? What does good look like? How would we know? What would it look like? And you've got to define that before you worry at all about collecting data and running analysis on the data. You've got to figure out what the heck is the data you want? What do you need

Greg Williams:

from the beautiful state of Utah in the United States? Hello, and welcome. I'm Greg Williams, and you're listening to the learning experience leader podcast, a project devoted to design leadership and a psychology of learning. This podcast helps you expand your perspective of learning design, through conversations with innovative professionals, and scholars across the world. Today's guest is Dr. Bonnie Barris. Bonnie is a human capital strategies and performance consultant, and creator of the measurement map to illustrate the causal chain of evidence between human capital interventions and measurable business outcomes. Her goal is to help organizations create environments and build skills that truly engage and enable their employees. Today we talk about getting to a shared definition of success with stakeholders, alignment with stakeholders as a critical part of analysis. And then we had a deep dive into Bonnie's measurement map model. I tried something new in this episode by role playing a scenario around a training request to illustrate the elements of the measurement map model. Bonnie's response is brilliant. I hope you find this episode as insightful as I did. Bonnie, I'm so excited to have you on the show today.

Dr. Bonnie Beresford:

So Greg, I'm always happy to talk with folks about something that I'm so passionate about measurement.

Greg Williams:

No, most definitely, um, and you've been doing measurement and performance, all sorts of things around performance for over 20 years, which is fantastic. And I know you've got a lot of insight here that we can discuss. One thing I'm interested in, when I talk to anybody who has such a depth of expertise on really anything, I'm always curious about what keeps you doing it, you know, why not move on to something else, what keeps you interested in measurement.

Dr. Bonnie Beresford:

I just want other people to get excited about measurement too. I want to be kind of that disciple for measurement so that other people realize they can do it, and that they get excited about what they can learn from it.

Greg Williams:

It definitely re energizes me when I can share something with somebody else and see the light bulb turn on around something I love so I can identify with that. Yeah, just to share my passion. Truly, you're right. Yeah. Yeah. Makes a lot of sense. And now in your current role, can you can you help me and listeners kind of get a better idea of sort of what is your day to day responsibilities and how that relates to some of these themes of measurement and evaluation?

Dr. Bonnie Beresford:

Sure. I currently am working for a training consulting agency, and lead the leading lead the Learning and Performance analytics practice there. And that involves kind of approaching measurement from two parts. One is I am the practitioner, and I help organizations identify what to measure what data to collect and do the analysis. The other part of it is that I teach other people within client organizations how to do this so they can self serve. So I'm both the practitioner and the teacher.

Greg Williams:

And it's all around performance. And so I want to make sure here for context, if we have folks listening from maybe higher education or K 12, or other things, I mean, would you say your work still spans that? Or is it more focused on specific behavioral types of objectives and so forth?

Dr. Bonnie Beresford:

My real focus, Greg is finding out if what we are doing whatever human capital investment we're making, whether it's a kindergarten class, whether it's a leadership development thing, whether it's a mentoring program, whether it's inclusion, and diversity, all these things that organizations, public and private, profit, nonprofit, all these things that they do to enhance their workforce. Are those things working? We're spending millions, if not billions of dollars on all of these kinds of programs? And yet, there seems to be very little evidence that tells us are we moving any needles? And that's where I have really focused is how do we figure out how do we measure this squishy stuff that people say can't be measured?

Greg Williams:

I get really excited about that question of are we moving the needle? There are other people that share this right, clearly you have it. But a common theme I feel is this in these conversations is that's not always broadly shared. Is there? Is there any key reasons or barriers you've seen of why sometimes the common denominator or assumption is not a keen interest in whether we're moving the needle or not? I was just talking with a gal yesterday.

Dr. Bonnie Beresford:

When she mentioned all of her clients, when the project's done, it's done. Let's move on to the next thing. Why do you want to keep harping on what's already done? And I don't think that's uncommon. You're right, it's, it's not uncommon. And it's kind of sad. And what I what I've seen is, in cases like that, you've got a business partner, who has this on their annual objective that you are going to develop this safety training. And that's what your objective is, nobody has set your objective of reducing workplace injuries, your objective has been to get the course out. And so you want to check that box and move on to the next thing. almost the last thing you want is to measure it and find out what if it didn't work? You're kind of scared if it didn't. And I think that fear of bad news steers a lot of people away from from wanting to measure these things, because they don't have the confidence in the first place that it's working. And that says, What's wrong with your design, if you're not confident that it's working? And that's why I want to bring measurement in right up front. It's the beginning of any initiative, let's map out in measurable terms, what success would look like? And honestly, Greg, that really helps an instructional designer when they know what they're supposed to be delivering? Yeah. Well, yeah,

Greg Williams:

I mean, that's one of the main things that I asked, you know, when I'm approaching a design, and once you ask it a couple times, and are still unable to get a satisfactory answer, you start to, I don't know, I start to feel a little worried like, Am I asked, am I asking something wrong? Am I asking a hard question? Or, or what might it be? And I think your perspective of it could be at least one of two things. One being our main objective actually isn't real change. It's, it's to do a thing and to move on. Whether that's nefarious or not, right? that this might be a mindset. Or it could be this, well, we don't want to dig too deep. Because we, we don't, we might not want to see what's in the shadows.

Dr. Bonnie Beresford:

And neither one of those answers is really satisfactory. Because if it's nefarious, or if we don't really want to see, then we are wasting corporate resources. People are spending money to develop this training, they're spending money to send people to the training. And yet, we haven't even articulated what we want them to do and what success would look like, shame on all of us for not considering the performance and the outcomes in the development of any of our initiatives, and shame on us for not wanting to know what success looks like.

Greg Williams:

That then begs the question, at least for the people in the learning and development field to examine your skill set, this is what I've been doing the last couple of months of, alright, I know these things are important. So am I doing them?

Dr. Bonnie Beresford:

In through me, any instructional designer who wants to build scrapped training? Nobody instructional designers are instructional designers, because they're passionate about helping people do better. Yeah. And so you should want to know, because nobody wants to spend their time building scrap.

Greg Williams:

And so I feel like there's a few barriers to doing this one, you know, when you and I were talking before your recording is too sometimes you've observed learning professionals can get sort of a sense of hives when we use words like analytics, and these other scary things that remind me of my statistics classes that I bombed. So how can we get over that? How can we, you know, find some sort of anecdote for that initial reaction and move on?

Dr. Bonnie Beresford:

That's been part of my, my research. And my work is how do we make measurement more approachable so that people start to realize it's not so scary, it is something you can do. And that's where this this tool I've developed called the measurement map comes into play. It allows you it's a it's a framework that allows you to have that performance conversation about what good would look like. And because Greg, the first thing we have to do is figure out what is it we're going to measure? What does good look like? How would we know what would it look like? And you've got to define that before you worry at all about collecting data and running analysis on the data. You've got to figure out what the heck is the data you want? What do you need? And once you've identified that, then chances are you Or someone in your organization has some decent data skills. Even basic Excel is a is a good start to get into the numbers. But the first thing is identifying what to measure. How do we quantify human performance success after safety training after leadership training after Front of House restaurant training? How do we determine what good looks like in measurable terms? There's research that shows that there are two big stumbling blocks to measurement. One is, I don't know what to measure. Which is this? And the other is I don't know how, yeah.

Greg Williams:

What and then with that, there's sort of a slew of skills that come with that, to understand what I guess you've got to be able to know how to talk with business leaders learn their language. And then the how piece I think really gets into what your measurement map concept is all about.

Dr. Bonnie Beresford:

Yep. And really, I can't stress enough that business acumen. If you want to know what matters to the business, you have to know what the business is measuring. And they're not measuring your butts in seats, they're not measuring your satisfaction scores. That's fine that the learning department does that, because you still need to know that they are looking at what is on their dashboard, or their scorecard. Those are the needles they are trying to move. And my advice to any instructional designer working with a new business partner is find out what they measure for their business, what are their KPIs for their business unit. That's what you want to help them improve. And when you can understand those things, and how they're where they come from, what they're counting. Now, you know, what the behaviors are that you're supposed to be developing in your students, because they're supposed to be the ones improving those metrics,

Greg Williams:

right? In a previous conversation, I was talking with another learning leader, about these kinds of conversations, right of meeting with stakeholders understanding what they value, she mentioned, the challenge that she frequently runs up against our leaders having a mindset of training of measuring it specifically the cost per person to be trained, right? That is kind of the main metric, just how much it's gonna cost. And the challenge for her on that was, I want to shift it to, well, what's the benefit in cost form, or in financial form of a well trained employee, right? It's the same thing that you're looking at, but from a very different angle. So let's say I do talk with my stakeholder and I guy, I only care about these KPIs. And there are these things that I feel like it's just a big stretch for, for learning to impact at least good learning. I don't know if that's what I'm saying makes sense. Or if you've seen situations like this,

Dr. Bonnie Beresford:

well, that's often the case, the business. Stakeholders don't have the mindset of, of an instructional designer, they don't necessarily connect the dots either. So I've got this gap in between, we've got the business stakeholders that have their goals, the learning people who want to help the business people, but they don't under neither one of them are making the alignment, they're not making the connection. And that, again, is where that this measurement map comes into play. Because it creates that causal chain of evidence from what I'm learning and training to the behaviors I'm supposed to do on the job. And if I do those behaviors on the job, then it should affect this KPI in this KPI and ultimately, it probably affects profitability. When we can draw that connection when we can create that clear alignment, that line of sight. Now, both the learning organization and the business partner can start to see how the training can affect the business results. And that is it. Having that kind of a conversation with a business partner is uncomfortable for a lot of instructional designers because they're not conversant in the business enough. But that's almost okay. Because to go in interested and concerned and showing empathy towards the business owner. They will tell you things. When you ask them, you know what you told me, You came to me mister business owner and said, Hey, we need some we need safety training. Well, do I go run off and build safety training? Or do I ask you? What? What has triggered your request? Oh, yeah, too many accidents. Okay, well, do I ask, okay, well go do that. Or do I say what kind of accidents? Where are these accidents happening? and dig in and how do you know and what data points are you tracking that show you this? And what would happen? What would it look like? And the job? What would you observe differently on the job? If people were doing things to prevent accidents? What would you see? And then they start telling you what they'd see. And they would say that well, we'd see the manager having a, you know, they would do a daily safety walk around the premises. Okay? Are they doing that now? And I don't think so we never really told him that they had to do daily safety watch, we just assumed they would, when you start digging into what these things would look like, on the job. And it really then feeds into your curriculum. But more than that, Greg, it also shows you if you need outside performance support, if it's more than just a training problem, if we're going to train frontline workers, but we're realizing, oh, it's really the manager who's supposed to be doing that. And we never told the managers, maybe we need manager training to

Greg Williams:

you. Yeah, I mean, it's, it's really clear to me, you know, the more that you're sharing these different examples, how critical your measurement efforts are to your analysis of the problem in the first place. Unfortunately, in the traditional ADDIE model, you know, the ease at the end, when really, the E is a huge part of the A and every other part of the whole process, right. And I think Agile software development and some of the other models that are coming out, are more illustrative of that. But what you're describing with the concept map, I think, I'm going to harken back to Episode 33. I had Dr. Jim Kirkpatrick come on and talk about kind of the new model of the Kirkpatrick model. You know, it's been out for 60 years or whatever. And in, in his book with Wendy Kirkpatrick, they talk about the different levels, we have this analogy of a Scottish bridge between levels one and two, two levels three and four. I feel like and correct me if I'm wrong, I feel like your measurement map is that bridge, like is the connecting piece between reaction and learning outcomes versus real behavior and business impact outcomes. And without that bridge, you have just the ongoing problems that you hear all the time between l&d and the rest of the business? I don't know. Is that an accurate summary of how I'm thinking about this,

Dr. Bonnie Beresford:

that's a good perspective. And I do know, Jim and Wendy, the and obviously, I'm a student of their model, and that is a foundational element of all learning evaluation work. Where I position the measurement map is actually even before a level one I'm, I consider it the alignment phase, you talk about Addie, and the a the Analyze, if that analyze also included alignment. And you built your measurement map right up front while you're doing your analysis, it really is part of that phase. And then you have that going in.

Greg Williams:

I really like that. I hope that alignment starts with a we should change analysis to alignment. Right? Yeah, alignment, because you are doing a lot of analysis. As a part of that. I think that's

Dr. Bonnie Beresford:

one thing I struggle with is that we do have the four levels or the five levels if you count jack Phillips with the ROI. We know those levels, but we the only ones we really know how to measure. Well, I wanted to. And this is where I think your your analogy of this of the measurement mat being that bridge does work, because that now tells us what in the world are you doing with levels three and four? What are you measuring?

Greg Williams:

So you're you have an article here. And my friend, Nicole, she did a conversation with you as well, I think it's coming out either this week or very soon. And I'm sure include that in the show notes because she adds a lot of it. She has a lot of great questions and insight that you give to her. So in this, this same article, I'm going to reference around the measurement map, you know, you do start with alignment, it starts with the alignment, which we've already talked about. And we get into the details here. Because I'll include this in the show notes. I encourage listeners to kind of to look this up. And what I've asked Bonnie to do with me is, is to go through an example or two of this that might be slightly different than what's in the article just so I can ask my naive newbie questions and we can work through the model. So I'd love to look at that in a little bit more detail. I want to make sure listeners understand this is an abbreviated version of the mapping process. We will be taking some shortcuts. This isn't a thorough, you know, realistic view of the whole thing. That said, I think there's still going to be some great value in going through it. Maybe we could start from the vantage point of I'm I've been asked to create training for new managers. Maybe we are a growing company and we're realizing that we need to scale Better and have better guidance and support for our managers. So that's, that's the general request that I get. And what what do I do? How do I,

Dr. Bonnie Beresford:

for the sake of your listeners, I want them to know, we have not rehearsed this. So what we're going to do is going to be a live performance conversation here. But Greg wants you to, I want you to change hats. I want you to put the hat on of one of the business owners what what kind of business? Is it that that you have that make up one?

Greg Williams:

Okay. Let's say I am a restaurant owner.

Dr. Bonnie Beresford:

Okay, well, let's let's focus with it with the day to day manager running the store. Sounds good. Stepping out of role here for a second, I call this the stakeholder meeting where we are going to be discussing this. So this is we're going to walk through that quickly here. And you're the stakeholder so you, you have all these managers, you're working with your franchisees and they've got these folks. So Greg, what's triggering you to ask for leadership training? What evidence do you have that these people are, are needing improvement?

Greg Williams:

We hear from frontline employees sometimes like in their exit interviews, that, that they didn't have very much support from their managers. People are kind of complaining about the level of manager support. Yeah, um, do you have a lot of exit interviews? Yeah. I mean, the expectation is that we, we have one for everybody who's leaving. doesn't always happen as thoroughly as we'd like, but it's, it's fairly consistent. Yeah.

Dr. Bonnie Beresford:

Wow. You know, and in the retails and services, industry, turnover of 50% or more is not uncommon. Where's your turnover right now? Do you know?

Greg Williams:

Yeah, that sounds about right. I mean, especially as we've really started growing in the last year or two, I think it would be even higher than that. If someone's employed with us for only a year or two, that we expect the turnover rate to be really, really high.

Dr. Bonnie Beresford:

Is there a point where your retention is pretty good? Like if they've been there? Two plus years, they tend to stay, but it's those people in the early days that might be struggling?

Greg Williams:

Yeah, I think it really is about two and a half years. You know, if they've been here that long with us, they usually we'll stay and try to move on into other positions in the company. But below that, it's it's hard to keep people around.

Dr. Bonnie Beresford:

Uh huh. Yeah, that's, that's the difficult part of your industry. So we understand that, you know, and managers play a role in that. And so I'm thinking that turnover is something that you're probably tracking and, you know, turnover of new hires. But I also would be thinking that retention, and especially perhaps three year retention, would be something you'd want to track. And that you might be tracking. Because if we can get people to get past that two and a half year, Mark, it sounds like that's a good thing.

Greg Williams:

Yeah, that's actually that's a good point, I'll have to check with, with our, our team on that. I don't have those numbers right here on hand. But yeah, I can look at that and get that information to you.

Dr. Bonnie Beresford:

And I'm jotting down here on post. It notes, these little metrics as we're coming up with them. So I've got kind of general retention, like one year retention, but I've also got three year retention, since that might be the sweet spot. And if we could increase three year retention, it sounds like that'd be good for the business.

Greg Williams:

Yeah, I mean, we do spend a lot and hiring and, and letting go people buy a lot is lost in that process for sure. So that I think that would help.

Dr. Bonnie Beresford:

Okay, so recruiting costs, yeah. all tied. Yeah, onboarding costs. And so along with your turnovers, like the turnover within there, there is research that 5% of people of new employees leave on the first day. I don't know if you've heard that research, but a lot of organizations like to look at like 90 day turnover, because that that could be a signal of, you know, poor onboarding, or poor hiring. And if you've got poor hiring, it sounds like you'd want to know that right away. So I've got those down. But let's see what else we want our managers to do. What would what would a manager be doing? That would improve this? What behaviors would you expect these managers to do to engage their staff?

Greg Williams:

We've sent in a few people to a few of our executive team to team meetings and those aren't always the most engaging or, or helpful. If we could have our managers, you know, create a solid team environment for their reports. We feel like that could really help everybody be more successful. Yeah, more effective.

Dr. Bonnie Beresford:

Right. Okay. So it sounds like you know, maybe the managers are needing help with How to do a good team meeting. So that might be something you'd put into your training. Is that what you're thinking?

Greg Williams:

Yeah, I think we could train people to hold really good meetings, and also like train them on how to build, you know, psychologically safe teams where if people have issues, they could talk to their manager about it rather than just, you know, not showing up or something.

Dr. Bonnie Beresford:

Oh, wait, can I pause? They're not showing up? So is absenteeism an issue as well?

Greg Williams:

Yeah. Unfortunately, that's, that's pretty common, especially in this group that we're talking about is like one to two year employees.

Dr. Bonnie Beresford:

Okay, so I just, you know, I jotted that on a post it note and I've got recruiting costs on a post it note and onboarding costs on a on a post it note, I also think this concept of equality of a team meeting, it sounds like you're going in and spot checking. And, and you would know what a good team meeting looked like. Because you might have a checklist on that, that you delivered in training, and you teach them what a good meeting looks like. So you could observe a good meeting. That sounds ideal, I wouldn't say we're quite at that level, you know, with how busy everything's been, we don't really have like a quality assurance checklist or anything.

Greg Williams:

I guess it's a little bit more of just a feeling of how, how good things are going and and also, if it comes up, as we're talking with, you know, in some of our skip level meetings with some of the director, direct reports of those managers of just like, Hey, you know, how's it going? And if they mentioned the meetings, but that's pretty good idea.

Dr. Bonnie Beresford:

If you think of a new manager, how do they know what good looks like?

Greg Williams:

Yeah, I think that's one reason we need this training, because I don't know if they know.

Dr. Bonnie Beresford:

Right? Okay. So as we define what good looks like, we will come up with other behavioral measures that we could look at to see if these managers are indeed doing that.

Greg Williams:

Okay, but couldn't we just like, couldn't we just, I don't know, we want to get something rolling out sooner than later. And that sounds like a lot of work. And important work. I don't want to downplay that. But maybe we could get something pulled together in the next couple of weeks to get rolled out to managers, and then sort of focus on success as we get rolling.

Dr. Bonnie Beresford:

And part of that will be defining what does good managerial behavior look like? Yeah. And so once you have that, you've got to articulate that specifically for a new manager, because they're new to the role. And if they don't know what good looks like, from a manager standpoint, it'll be difficult for them to achieve good because they haven't, it hasn't been defined yet. But we've met a few things. And just for the sake of time, if we were to turn what we've just talked about into a bit of a measurement map. I'm going to just hypothesize with you Greg for a minute and say, Well, we know that we want them to have these regular team meetings. Right. So do they have the team meetings? Yes or no? That could be the first thing because what if they don't even have the meetings?

Greg Williams:

Yeah. Yeah. And there may be there may be managers that don't have it, especially if they don't really know what's expected of them or what good looks like,

Dr. Bonnie Beresford:

right? And then if they have the meetings, are they any good? And if we have outlined what a good team meeting looks like, what gets covered in it, and so on, you could observe that whether or not you've got data collection mechanisms in place, you know, later. But do they have the meetings, because you're setting the expectation with your measurement, you are setting the expectation for that new manager, you will have a weekly or a daily team meeting. They will be of XYZ quality. And then we would expect that we're going to see more engaged employees in some more you might have an engagement survey. I think before before we met today, you told me you had an engagement survey, right? Yeah. So we could look at that. And then if people are engaged, if they're if they're being directed well by their manager, then we would see absenteeism go down because they the employees would know what they're doing. And if absenteeism goes down, then we are saving money there because we're not having to backfill or pay somebody overtime to be there to cover. And if I'm showing up to work, and I'm getting the coaching, then I'm probably not going to leave so our turnover rate is going to go down. Our recruiting costs will go down, our onboarding costs will go down and our retention will go up. And all this will lead to improve profitability, which as a business owner, that's really what you want. You want these stores and your and your organization to be profitable and not wasting money on and turn And that kind of thing?

Greg Williams:

Yeah, oh, I think this is a really good example, then. I know, we haven't walked through all of the nitty gritties. But I want to make sure we have time to kind of break down this example, with some of the language that you use in the post about the map. Yeah. And so in the situation where I don't put on a very convincing role as a sort of business leader. Do you start in a certain place when you're having this conversation? You know, we started with alignments? And really it was, I noticed you asked why a few times, in a, in a good way where you're digging, so we have invest investment, leading indicators of business results. And strategic goals are kind of the big columns. So maybe we could walk through that in light of our little roleplay, we just did.

Dr. Bonnie Beresford:

You're exactly right. That's how we lay out a measurement map is on one end, we start with the end in mind, what is the strategic goal and imagine to your listeners, imagine that's on the far right, our strategic goal is a profitable company or market share, or workers go home safe at night, or whatever. And at the other end, we've got whatever the investment is leadership development, or mentoring. And what we're trying to do is connect those two, through what I call leading indicators, and business results. When I'm having the conversation, like Greg, you and I just did, I'm not really looking to directly fill those boxes in. Okay, I'm starting the conversation. And this is what I teach in my in my workshops, is this. I've got a rubric, if you will, for the performance conversation. And it starts with what is the business issue. And you told me that you know, you're growing, you're adding more people. And you've got a lot of exit interviews, and that's bugging you. And that is turnover, right? And we then kind of get into what what's causing the turnover? And what's the manager role in there? And what would you expect a manager to do? So we're peeling that back, is to how in the world would a first level manager be able to influence turnover. And so we're talking through that, and through the course of the conversation, you're naturally going to be telling me some other metrics that are coming up, and I jot those down and, and we have this conversation. And we get back to the behaviors of what a manager has to do. And Doggone it, one of the key things is have those team meetings, then presence or absence of team meetings is something you could observe and count. And that becomes a very early leading indicator. So after we have this conversation, and I put all of these things, a little post it notes, we can arrange them now in our logic, logical chain of evidence that if I have meetings, and if they are good meetings, then people should show up to work. And if they show up to work, then productivity should go up. And we should see that in whatever our productivity measures are for these people at the store at the restaurants. And turnover should go down. And if turnover, turnover goes down, you already told me you know what it cost to replace a person. So now we're getting into the financial metrics. And you can see how that connects to the profitability. So taking it again, all the way back. If Greg, you were to walk into any one of your restaurants and see the manager having a quality team meeting, would you believe that our training was working?

Greg Williams:

Yeah, I think okay, yeah, that's what we said we set out to do.

Dr. Bonnie Beresford:

And if you saw that that was happening, and if you saw that absenteeism was improving, would you think that the manager behaviors had something to do with that?

Greg Williams:

Yeah, yeah, the logic, I think we can continue.

Dr. Bonnie Beresford:

That's exactly what we're doing here is creating this logic model, right. And when the business partner believes it, when it's the business partners metrics, then the business partner owns this causal chain of evidence, and it's a heck of a lot easier, six months from now to come back with data and show the evidence that you have built that look at 98% of our managers are having weekly check in conversations. We did a spot observation of them and of the 30 we looked at all 29 of the 30 we're having good conversations. And we saw that absenteeism at those 29 stores. was going down. Would you my business partner believes that perhaps this training of these managers is working? Yeah. Because you were the you were there as the business partner, and you helped build this logic model,

Greg Williams:

right? I was there in the alignment. It wasn't just you doing some offhanded analysis and interviews of those managers alone. But we had this whole conversation from the get go, like you were saying, alignment as a part of analysis. Exactly. Bingo. This is great. I, if anything, this is super valuable for me, I hope listeners will enjoy my roleplay there a great illustration of how this can work. And I think I know initially, as I hear you talking through it, like, Wow, that's a little overwhelming. I feel some of those hives coming on. Like, I don't think like that, necessarily. But you've been doing this for 20 plus years, I imagine it's just takes a lot of experience and patience to identify those metrics. And to keep asking questions.

Dr. Bonnie Beresford:

It is it is kind of a critical thinking mindset, where you just you just keep digging. And really, instructional designers are great at that, because they're great at analysis. And what we do with this alignment, the stakeholder conversation is really have that kind of conversation. But it's guided in it has a very specific output, which is the measurement map, which is a very tangible output from a conversation, which sometimes is more fun than just going in and interviewing a business partner and saying thank you and walk away with your notes. You've built something together, and their fingers are all over it.

Greg Williams:

Yeah. Yeah, they're much more invested in what you're doing. And I imagine respect the work you're doing if if you're able to tie it all back to the to

Dr. Bonnie Beresford:

what they want to know, now, they want to know where they're going to help you get the data because they're curious now,

Greg Williams:

before I roll into some of the final questions, that I love your perspective on I'm, what we just did was for kind of a big initiative around manage new managers or something. Now. Many training teams, I imagine if it's been similar to my past experience, can get inundated with a variety of training requests, many of them much on a smaller scale. And having these kind of conversations, at least on its face can feel overwhelming. Like there's just so many requests, how do I stay on top of this? You know, to even start doing the alignment? Because there's, there's so much coming in? I don't know, do you advocate for doing this mapping exercise on basically any kind of training request or their guide rails of where you would say, that's probably not worth? That's not worth it?

Dr. Bonnie Beresford:

I'm so glad you asked that question. Because you're right, you could just overwhelm yourself, if you tried to do this on every 15 minute web course that you ended up building. And I would not be a proponent of that. You certainly should know what business outcome you're driving for. But this process I would do if you had a curriculum, maybe that was made up of, let's say, you've got a sales curriculum that's going to have, you know, 1020 different elements in it. Or you had a major investment of some sort? I would do it there, I would not do it on one individual, quick ask topic. You could do it on a job role, almost, that might be too big. But we tend to look at it on investments that are significant, that people are really going to want to a Did it work? Should we continue it? Should we do more of it? Should we kill it? You know, if this is a one time compliance thing, and we have to do it, you're not going to do a measurement map. But if you're building out a skills curriculum for a particular job role, and you're not sure if it needs if it's if it's going to work? And should you continue it And should you have everybody go through it? It's kind of the idea of how do a lot of organizations go through the conundrum of Should I do level four on this? And we say that if you're going to you know, if you're thinking of level four, meaning you're going to get to business impact, you absolutely need to do this. And I think the the Kirkpatrick and jack Phillips they all kind of point to, I think somewhere between five and 10% of your initiatives go to level four, I sometimes I here's another place I do these measurement maps they really, really come in handy is when you're piloting something. If you're piloting a new learner experience platform, if you're piloting a chatbot if you're piloting virtual reality, people always want to know what didn't work. But they have never defined work. How do you know if it worked? We fit the pilots complete. Well, did it work? Well, what does that mean? The measurement man Process forces you to articulate? What does success look like. The other thing that the map does, Greg, and when you start measuring these pilots this way, is you're measuring both to prove and to improve the whole purpose of a pilot. Yeah, it's a proof of concept. You want to prove it. But you also want to learn what parts of it worked and what parts of it failed. So that your next deployment is better. And you can't do that you can't say if it worked, or if it failed, unless you know going in what you're going to measure? What does success look like? And then people get excited to see results, because now they have a yardstick.

Greg Williams:

Yeah, they know what they're going for. Yep. And you can actually celebrate, instead of scratching your head and wondering, Well, I hope we keep doing well as a business and they don't cut the training department, whatever

Dr. Bonnie Beresford:

you change the conversation with your business partners to so they know you care about the business, you care about their metrics. They're starting to see you as a partner, because they never thought about things in the same way either.

Greg Williams:

I love it. This has been really insightful. Along these topics we've talked about or just measurement in general, I mean, you've been doing this for a long time. So I imagine you've turned over most of the rocks that you've confined, and faced most of the challenges a learning person might face? Is there any unanswered questions you have or things you're interested in and learning more about related to some of the things we've talked about?

Dr. Bonnie Beresford:

Well, 20 years might might make me old, but it doesn't mean I've learned everything, there's always another rock to be turned over. What's fascinating me now and what I'm really excited about is some of the informal learning that's going on, and some of the x API stuff that's starting to come around, where we might be able to track some usage of this so we can see how informal learning is working. That remains a challenge. And I think you know, you're an instructional design professional, and you know, that a lot of learning is informal. And if all we're doing is tracking the formal learning, and measuring that we might be missing some of the richness of learning that's actually going on. And I think that's the new horizon for all of us is to leverage the informal space and figure out how to measure it.

Greg Williams:

Yeah, I'm glad you mentioned that. That's a whole thing of itself. I'm curious on that, too. So you'll have to share more out as you weigh in on that. We didn't dive in much like I often do beginning of episodes into your own development and growth as a professional and you've shared a post of 20, things that you've learned are reflected. I'm interested to know if there's a specific or significant learning experience for you that you've had that's helped you develop either as a person or as a professional.

Dr. Bonnie Beresford:

I'm going to give you two, I think the first one was one of my first engagements where a client needed need, he was in approved mode. He was one of the I've got to justify our training because it was a sales department. And every time there was a new product launch that went well, the marketing and design and engineering folks took all the credit. And whenever a new product launch, faltered, it was the fault of training. And he was tired of being the Fall Guy. And he said, Bonnie, I need to figure out how to show them my contribution. So that really kick started me on this journey. And my big aha there. And I'm kind of almost embarrassed to say, but I was so naive. I was I went out and found a statistician and I was looking for somebody who had the formula for causation. Cuz I thought there was a formula I had taken this dance class, like you hit and I thought, Okay, well, you know, I learned the regressions and these sorts of things and overs and but where's that? Where's that causation formula?

Greg Williams:

I could see myself asking the same thing. So

Dr. Bonnie Beresford:

what I discovered was, oh, there's no single formula that causation is really, you know, you got to have three things. Did the cause precede the effect? Is there a correlation? And can you rule out everything else? And the way you rule out everything else is by having a good logic model, which is the measurement map and building a good research design and It doesn't matter what the statistical approach you use is you need that logic model, which is your measurement map, and then a good measurement plan and research design behind that. And so that was a big aha for me. And I this a second one is very approachable for any of your listeners. There's a great book by Douglas Hubbard, who's an economist, and it's called How to measure anything, finding the value of intangibles in business. That was an early reading that I did, that gave me the confidence that I could figure out how to measure anything, because he showed and demonstrated how that can be done, because so much of learning and all of the HR stuff, it's it's fuzzy, it's soft, it's mushy, you can't it's intangible. And he has a very good thought process that I've incorporated into my own thinking about how to figure out what those intangibles are that you can measure.

Greg Williams:

I'm glad you mentioned that, that sounds like something I definitely need to read. Are there any other resources or book recommendations that you have?

Dr. Bonnie Beresford:

Well, sure. I have my own book called developing human capital. There's a whole chapter there on measurement mapping.

Greg Williams:

Okay. Fantastic. This is great. Those are wonderful resources. I want to check those out. Thank you so much for your time, Bonnie, this has been a great conversation.

Dr. Bonnie Beresford:

Well, thanks, Greg. And thanks for playing a good business partner with me here.

Greg Williams:

Be sure to check out the other episodes of the learning experience leader podcast, and if you haven't already, subscribe and share it with a fellow learning leader. I'm aiming to mix things up in the coming months with new guests and possibly new formats. Do you have an opinion about the show? I'd love to hear it. Send me an email at Gregory Spencer williams@gmail.com until next time, keep learning