Human Resource Development Masterclass
Human Resource Development Masterclass
Evaluation in Human Resource Development
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this ninth episode of HRD Masterclass, guests Dr. Darlene Russ-Eft (Purdue University), and Dr. Jie Ke (Jackson State University) discuss evaluation and its role within Human Resource Development, including:
- Different types of HRD evaluation.
- Problems with evaluating using Kirkpatrick's levels.
- Alternatives to evaluating using Kirkpatrick.
- Speaking to organizational leaders about HRD impact.
- Aligning evaluation research and practice.
- The challenges of HRD programs being evaluated by those who built them.
- Developing yourself as an evaluator.
In the first part of the episode, Darlene and Jie meet one-to-one with host, Darren Short. In the second part, they are together to discuss their shared interest in evaluation.
For full details on the HRD Masterclass series, visit hrdmasterclass.com. For more information on the Academy of Human Resource Development, visit ahrd.org - by being a member of AHRD, you can access bonus materials not included in this episode.
This episode is sponsored by: Interpretive Simulations. Find out about their services and their HRManagement simulation at https://www.interpretive.com/
Introductions
SPEAKER_01The typical HRD program is evaluated by asking people, well, what did you think of this?
SPEAKER_03Welcome to the Human Resource Development Masterclass, the new podcast series from the Academy of Human Resource Development, the organization that leads HRD through research. I'm your host, Darren Short, and throughout this first series, I'll be joined by leading authors, researchers, and scholars to explore the fundamentals of HRD and how those are changing in the 2020s. Our focus for this episode is evaluation. I will be exploring different types of HRD evaluation, aligning evaluation research and practice, speaking with business leaders about HRD impact, and much more. To help me, I'll be joined by two leading scholars. Dr. Darlene Roseft, Professor Emeritus, Oregon State University, and currently Assistant Professor of Practice at Purdue University, and Dr. Jay K, Associate Professor of Human Resource Development at Jackson State University. In the first part of the episode, I'll chat one-to-one with each of them, and then for the second part, Darlene and Jay are together to explore their shared interest in evaluation. That discussion is brought to you with the help of the generous support of our sponsor, Interpretive Simulations. Find out more about their services at interpretive.com. All of the content you'll hear in this episode was recorded during April and May of 2021. Right, let's dive in to meet my first guest.
Conversation with Dr. Darlene Russ-Eft
SPEAKER_03My first guest today is Dr. Darlene Russeft, Professor Emeritus and former Chair of Adult and Higher Education, Oregon State University, and currently assistant professor of practice in technology leadership and innovation, Purdue University. She also teaches in the doctoral program at the National Institute for Development Administration in Bangkok, Thailand. Darlene worked as a senior research scientist for the American Institutes for Research and later as Director of Research for International Training Companies before joining OSU. She's published numerous articles and books related to HRD evaluation. She's been AHRD President, editor of the journal Human Resource Development Quarterly, and a board member of the American Evaluation Association. Darlene has been inducted into the HRD Scholar Hall of Fame and elected as a Fellow to the International Board of Standards for Training, Performance and Instruction. Hi Darlene, welcome to the HRD Masterclass Podcast. It's great to have you here in our episode focused on evaluating HRD.
SPEAKER_01Hey Darren, it's really nice to be here with you all.
SPEAKER_03How about we start by exploring what human resource development means to you and how you see evaluation fitting into your understanding of HRD?
SPEAKER_01My understanding of human resource development is that it involves the development of human beings within typically organizational settings, although it could be within communities, and it has typically involved training and development interventions of some sort, mentoring and coaching, organization development, and career development. So it's developing not only individuals, but there's a development perhaps of a team of people working together as well as development at the larger organizational level. And one might consider that even within a community setting, a community development project.
SPEAKER_03And so within that, within that picture of HRD, how do you see evaluation fitting in?
SPEAKER_01Well, I think of evaluation in a maybe broader sense than I have seen within the human resource development literature. So I see program evaluation, which is what this is. We're looking at a program or some sort of intervention of some sort, and I see evaluation as beginning at the beginning, determining the needs for an intervention of some sort. Evaluation also involves examining different aspects of that intervention and figuring out whether there are some revisions that are needed in order to achieve the outcomes. And what in HRD tends to happen is uh well, we'll focus on the outcomes. That's what we want to measure, and that's what we view as evaluation. Well, that's from my standpoint, simply one aspect of an evaluation effort. I I see it as uh basically looking at all aspects of that intervention from the beginning through to the outcomes and back again.
SPEAKER_03It sounds like there's a difference then between the evaluation that we typically see of human resource development in organizations, which, as you say, is very outcome focused, and the evaluation, for example, of social programs.
SPEAKER_01You've got a very different funding mechanism, right? So HRD, who's funding that intervention? Well, it's the organization who's funding the intervention. And so the stakeholders are the executives, the managers, could be even the employees. But with social programs, these are typically government-initiated interventions of some sort by government agencies. But who funds the government agencies? That's uh the public, you and me. Uh, and so what's typically expected in um government interventions is that there be some, uh call it, accountability. Furthermore, I I would say um what's typically viewed as a government intervention of some sort, there there's going to be political issues involved. And uh I would say that within HRD, some of those same politics occur, but people don't think about them all that much.
SPEAKER_03I think it takes us then into into the area of how are HRD programs typically evaluated and and has that changed over time?
SPEAKER_01The typical HRD program is evaluated by asking people, well, what did you think of this? It's uh how did you like this? Uh do you think you might use it at some point? What do you think of the facilitator? And so on. And the famous name in the field of HRD evaluation is Kirkpatrick, and he talks about four levels. So there's a taxonomy, and the lowest level, quote, lowest level is reactions, and that's where things tend to rest. Sometimes they get into measuring has has anybody learned anything? And then are there changes in behaviors? And finally, the the top level would be, you know, uh, what's happened at the organization level? Is there a return on investment of some sort? Those approaches have continued, they started back, I would say, in the 90s and basically have grown in popularity and uh tend to be used quite widely. But I think that within the field of evaluation, there's a recognition of a whole variety of other approaches and the use of other approaches. You know, things like Brinkerhof's success case approach, which is used to some extent in HRD evaluation, probably not as much as it should, looking at uh responsive or participatory evaluation, theory-driven evaluation, uh consumer-oriented evaluation. I mean, there's a whole variety of different approaches that are used in program evaluations uh widely that go beyond what I've seen in the HRD literature.
SPEAKER_03Yeah, that's that's been my experience as well, which is the within practice, there's definitely be a been a focus on Kirkpatrick, almost a drive to generate as much data as possible to try and almost justify the HRD function or its impact in the organization.
SPEAKER_01Yeah, and the the problem there is there's a lot of generation of data, but then not much use of the data. I know in going to different clients of the when I worked in a private firm, people had reams and reams of instructor evaluations that just sat there.
SPEAKER_03In my experience, what I've found is that at the end of the day, what really seemed to matter was what a small number of executives perceived as the impact of HRD, and that their perception was often influenced by what they heard from their directs or what they heard from the floor of the company. And if they heard good stories, great, i.e., the sort of case studies you were describing from Brinkhoff, if they heard those case studies, then they were going to have a positive feeling about HRD. And if they had a negative feeling, there was almost no way the HRD function could generate enough data to overact that negative feeling.
SPEAKER_01Exactly. Exactly. And and I think, you know, one of the so the that success case method is, I think, useful not only for highlighting the successes, but also looking at, okay, what makes this successful? Is it the intervention itself? Is it the support from upper management? Is it the people involved, the team? What are the aspects or the factors around that case that make this successful or these cases successful versus, let's say, these other cases?
SPEAKER_03It almost takes you to the comment you made earlier. Sitting down with key stakeholders and saying, what needles are you trying to move? How far are you trying to move them? And how will you judge whether or not my program has moved those needles far enough?
SPEAKER_01Exactly. And that that's why uh say undertaking a needs assessment might help to point out some of those needs, and certainly involving those stakeholders in identifying, okay, what do you want to see here is critical.
SPEAKER_03So this suggests that there's a significant gap by the sound of things between the way that evaluation is practiced in HRD and what research or theory would suggest should be the practice. Why do you think that is? Why do you think there is a gap between research and practice on evaluation?
SPEAKER_01I think there's a couple of reasons. I think that it is so easy to simply mount a survey, you know, liker type scale, something or other, and ask people to fill it out. That that is so simple. Why go beyond that? You know, I can find out that people are happy, great, that's all I need to know. I think that maybe too many times the HRD people don't think about all the stakeholders involved.
SPEAKER_03So there's like this almost ease of industrializing evaluation by putting it into surveys and having those surveys automatically pushed out by an LMS and gathering all of this data.
SPEAKER_01Um easy.
unknownYeah.
SPEAKER_03So easy. Whereas I suppose the um there's a higher resource impact of doing a more targeted evaluation. But at the end of the day, it sounds like it's worth putting. If you're running 100 programs, it's probably worth putting your evaluation effort into the five that the stakeholders really care about, and evaluating those five really well than trying to evaluate a hundred poorly.
SPEAKER_01Exactly. Or the ones that are maybe the most expensive. You know, we're spending a ton of money on this. We really need to find out is it working? Are we achieving the results we expect? Why are we not?
SPEAKER_03And I think what we've seen probably in the world of practice is almost it going the opposite way, uh, with all of the attention that's been placed over the last 10, 20 years on ROI and different ways of trying to calculate ROI and to try and turn uh somehow create an objective measure out of a bunch of subjective data.
SPEAKER_01Yeah, I would agree. I I'd say that another thing that I've kind of noticed over the past while is that um there's less and less literature in our journals. So if I think about AHRD, and I just I actually did a quick survey of the last few years in the journals, and there's I think maybe one or two articles at most relating to evaluation.
SPEAKER_03Um if they're in that position and they're starting to question their own evaluation practices, what would be your recommendation as to how to start that journey?
SPEAKER_01One of the things they might do is simply look at, so there's the American Evaluation Association, and um one can simply look at their website and some of the resources that are available from that organization. There are training programs, for example, there's something called the Evaluators Institute that takes place on a regular basis. Um there are journals that highlight different approaches to evaluation, different issues in evaluation. You know, I I printed out, for example, um uh one issue is dealing with causal mechanisms in program evaluation. Another issue is looking at issues facing communities of color today. Uh so you know, um evaluating community coalitions and collaboratives. So there's there are these different topics and also descriptions of approaches that uh folks might use in their evaluations.
SPEAKER_03So it sounds like then the the call to action here is almost look outside of the world of HRD. So if you're if you're in in practice and you are starting to question or critique your own evaluation approaches, um, don't stay insular, don't look at other HRD programs, don't necessarily even look at the HRD literature, but step out of that and start looking at the wider world of evaluation as as a means, as a source of inspiration as to how to reinvent your evaluation practices.
SPEAKER_01Yes. And and I, you know, I mentioned the American Evaluation Association, but there are evaluation associations throughout the world, and there's actually an international federation uh of evaluations.
SPEAKER_03Yeah, looking at the notes I'm keeping as we go through the conversation, and I and I look looking for the themes. It sounds like there's almost three that start coming out for me. Like one is critique your own evaluation practices by looking externally. I think the second one it would be engage stakeholders early to understand um how they would measure success, what they're looking for, and and doing that potentially as part of the needs analysis, as opposed to once the program has been built. A third piece to that then is go for more targeted evaluation. So figure out which are the programs that are either a major investment or are significantly politically important in the organization and focus your evaluation efforts on those rather than doing it for all or all of the programs.
SPEAKER_01I think you've captured it. The stakeholders, whoever they might be, the people who uh what uh fund the program in some way or other uh and or make the program function uh and are supportive of the program potentially, are the ones you want to find out. Okay, what what kind of information do you need? Also recognizing that program evaluation is political, even though you're not doing a social program, it is political, it's within an organization setting. You need to also think about who are the important voices, whether they're a stakeholder or not. Is there, I don't know, a VP who tends to be totally against uh development efforts of some sort. Uh if so, I would go to that person and say, you know, I know you don't expect much out of these development programs or out of this training or whatever, this mentoring program. Uh but uh if people worked differently here or did things better, what would that look like to you? You might then get some ideas about what what should we be measuring?
SPEAKER_03There's almost a mind, a mindset shift there. So what when I listen to your answer to that question, it makes me feel as if the the mind shift is encouraging HRD practitioners to View the organization as political and as a value, an evaluation as subjective, as opposed to what traditionally has happened, which has been to view decisions as being rational and objectively made based on data. And if you can make that shift, then at that point you start thinking it like who are the key players, what are they looking for? Who are the influencers? How do I understand what will influence them? And viewing it as a subjective system rather than uh rather than a rational decision. We would like to think organizations are rational, but they generally aren't.
SPEAKER_01No, they're not. No, there's a lot of subjectivity and politics in all organizations. You're right.
SPEAKER_03Well, Darlene, it's been wonderful talking with you about evaluation. I I feel like we've laid a really nice foundation uh for the rest of the episode. And um, and so I'm I'm looking forward to you coming back later in the episode for the group discussion.
SPEAKER_00Well, I'm looking forward to it too. It should be an interesting time.
Conversation with Dr. Jie Ke
SPEAKER_03My second guest for the episode is Dr. Jay K, Associate Professor of Human Resource Development at Jackson State University. Jay obtained her PhD in human resource development from Texas AM University and holds an MBA from EDHEC in France. Her research interests include assessment and evaluation and effectiveness of e-learning. Jay served as managing editor with the International Journal of Human Resource Management and is currently a member of the AHRD Board of Directors as well as co-chair of the China Special Interest Group. She's been serving as the chair and co-chair for the assessment and evaluation track of annual AHRD International Conferences since 2017. Jay has been an external evaluator of two National Science Foundation grant projects, the principal investigator of two grants in improving the learning of underrepresented adult undergraduates, and the co-director of a Kellogg Foundation grant project. So, hi Jay, welcome to the HRD Masterclass podcast. It's great to have you here in our episode focused on evaluation.
SPEAKER_04Thank you for having me here. I'm glad to have this conversation with you.
SPEAKER_03I was thinking of starting by referencing back to something that Darlene was talking about. Because with in my conversation with Darlene, we explored what it what was meant by the term evaluation and how program evaluation differs from evaluation in HRD. And so I thought that was a good starting point for our conversation, specifically by looking at what you see as evaluation's function and role in training and development and OD and HRD in general.
SPEAKER_04Yeah, um, absolutely. Um, evaluation plays extremely important role in HRD for sure. And however, I feel its functions and roles in different components of HRD that you just mentioned are slightly different. In training development, evaluation monitors and assesses the progress of training programs and activities and helps to achieve positive changes of participants in their behaviors, attitudes, and skills. The goal of evaluation in that setting is to compare the training outcomes with a preset learning objectives and goals. In contrast, evaluation in organization development goes beyond examining the results against learning and performance objectives, and facilitates changes for the benefits and interest of the organization and its stakeholders. It is to examine whether intervention enhances learning, whether it improves performance, whether it facilitates desired changes that align with the organization's vision, mission, and values. So as you can see, the evaluation in training development takes the snapshots of training programs at various stages of the training, while that in organization development is longitudinal and will grow and develop with organization through verifying different interventions.
SPEAKER_03So what I'm wondering when I listen to your description there of evaluation for train development and OD is what questions evaluation is seeking to answer in HRD.
SPEAKER_04The question that the evaluation is seeking to ask in HRD is whether the program is uh fulfilling its uh task, whether the intervention has reached its goals, and those questions have to be asked and answered with consideration of expectations of all stakeholders.
SPEAKER_03So there's like this question that we're trying to answer, then, which is did we achieve what we set out to achieve? Does evaluation also then look into the why piece of it?
SPEAKER_04Yes, yes, absolutely. It actually started. Um, I'm glad you mentioned that. It it actually started with the question, why? And so that's why I said evaluation is extremely important. It starts at the beginning of the change process and provides answers to the change process with regard to whether the intervention is successful.
SPEAKER_03If if we achieve what we set out to achieve, it's nice to know why that happened so that we can replicate it. And if we didn't achieve what we set out to achieve, it's nice to know what caused that so we don't repeat those mistakes.
SPEAKER_04Right. We can further uh refine, we can further develop and make some change plans along the way. Um, the the evaluation does not just happen in the beginning or at the end, or you know, in the process, it happens all through.
SPEAKER_03Excellent. Now, throughout the the episode, we've used the term evaluation. Uh, and it's my guess is it's a sort of an umbrella term for a number of different types. So, what do you see as the main types of evaluation?
SPEAKER_04If we look at the timeline, the evaluation can be categorized into uh three main types. They're formative, summative, and confirmative. Formative evaluation is conducted throughout the process of program or intervention to check whether everything is conducted as it was designed and whether anything needs to be revised, refined, and further developed to better achieve the goals set earlier at the beginning of the program design stage. And the summative evaluation is administered at the end to determine the value of the program, whether we have done things right, uh, if we did, what are some successful um takeaways? If we didn't what we did wrong, and how we're gonna change that in the future. Confirmative evaluation is another type. It identifies and explains the program or intervention's long-term effectiveness, efficiency, impact, and value versus summative and formative are to evaluate the immediate effect of the program.
SPEAKER_03So if I was uh putting it in the context, say of an HRD program, let's hypothetically create a sales training program then. Um so uh a summative evaluation is asking is the program being delivered the way we intended it to be delivered? A formative evaluation would be to ask does the training program deliver an increase, the increase in sales that we expected it to deliver.
SPEAKER_04Yes.
SPEAKER_03Um and then you would look beyond that if the program ran for like a year or two, and then ask the longer question of is this changing uh the revenue of the organization the way we intended it to, once this program has been run for a period.
SPEAKER_04Yes, that's called uh confirmative evaluation.
SPEAKER_03Confirmative.
SPEAKER_04Yes, that's uh longitudinal. And uh I want to, besides those major types of evaluation, I really want to mention another type that is called meta evaluation. It is very different from all the above-mentioned types of evaluation, um, because it the purpose of it is to check the quality of evaluation itself.
SPEAKER_03What I was thinking about as you as you talk through that is like the the different words that can be associated with evaluation. You know, when I talk to HRD practitioners, they'll talk about you know, we're we're measuring the number of people who go through the training program, or we're measuring increases in sales, or we're doing tests of individuals to make sure that they've learned what they needed to learn. So there's there's different words out there like measurement, testing, assessment, and and evaluation. And so I'm wondering from your perspective, are these the same thing, or is there a difference between those terms?
SPEAKER_04They're separate terms, but they are connected, interconnected ideas and processes. The measurement basically refers to the process by which the um the attributes and dimensions of some physical objects are determined. Normally, a standard measurement will be used to determine how big, how tall, how short, how heavy an object is. And assessment is another term that was built upon measurement. The assessment is a process by which information is obtained relative to some known objective and goal, it definitely includes a testing, which is included in the measurement too. But testing is a special form of assessment, and we can test knowledge as well as skills and ability, although testing knowledge is far easier than testing skills and ability. The evaluation is probably the most complex and least understood among all the terms. From its spelling, we know evaluation is somehow related to value. When we evaluate, what we're doing is to engage in some process that is designed to provide information that will help us to make a judgment about a given situation or a context. Generally, any evaluation process requires information about the context in question. A context is an umbrella term that takes into consideration such idea as objectives, goals, and standards, procedures, and so on. So when we evaluate, we're saying that the process will provide information regarding the worthiness, appropriateness, goodness, validity, and legality of something for which a reliable measurement or assessment has been made. Um, in other words, the context really matters in evaluation. Um with that being said, um, all the differences among measurement, assessment, evaluation are not that huge. Um, so when we talk about those, we shall not forget the interconnections among them. For all three terms, we use um some set of criteria, whether it's objective or or subjective to design them.
SPEAKER_03To give an example of subjectivity, um, from what I have seen um around organizations and evaluation over the last like 20, 30 years of being in the field, um, I I feel like I've seen a shift to where maybe 20, 30 years ago, much HRD evaluation was objective in nature. Um, and uh and over 30 years, it feels to me as if subjectivity has become more acceptable and that there's been challenges that have come with that.
SPEAKER_04Um, Darren, I'm really glad you asked this question. And the evaluation in America started off as a pure measurement mechanism in the early 20th century, and it actually started with testing to the specific in education setting. But after the World War II, the feature of describing strengths and weaknesses of a program was added to the concept of evaluation. That time, evaluators were called to describe aptitude, environment, and accomplishment of the program. Both measurement-based and description-based evaluation are objective-oriented, like what you just said, and they were found insufficient to answer many questions. Therefore, by mid-twentieth century, evaluators were called to be the judges of whether the goals are achieved, performance are rendered, so it becomes more subjective to the judgment of evaluators. The most recent evolvement in the field is influenced by the paradigm shift from positivism to constructivism. Evaluation is no longer viewed or treated as a mere technical process of inquiry. Instead, it started to include human, political, social, cultural, and contextual elements because of its value-reading nature. I remember you and Darlene talked about politics and political nature of evaluation, and that is the result of this shift over the time. And the fourth generation approach of evaluation proposed by Goba and Lincoln goes beyond the previous generations, and that would that we just mentioned now. And the emergent approach indicates that evaluation is ongoing negotiation between the evaluator and stakeholder. So we evaluators are no longer just a tester, just describer, or just a judge. We are aware of the interesting properties of evaluation, such as the evaluation outcome is not the true reality or the pure reality, as we thought before.
SPEAKER_03So they're the training manager, for example, or they're even the instructor or facilitator who built the class and now teaches the class. And so I was wondering what the challenges are for evaluation if the evaluator is also the person responsible for rolling out the program.
SPEAKER_04Darren, you've asked a really good question. As a university professor and instructor, I often run into the same challenge. I'm the designer, I'm also the evaluator of my own course. I really would like to have another pair of eyes to take a look at how I teach and provide me the feedback. So I would recommend the program designer as well as the evaluator to invite another external evaluator to avoid internal bias during the process of evaluation.
SPEAKER_03As we've talked through all of this, I'm conscious that from an HRD perspective, we often think about evaluation as being evaluating what happens in the training room. So are we teaching it correctly? Are people learning? Do they know what they need to know when they leave the room? But learning happens in multiple settings. And so, how do we make sure when we evaluate that we're evaluating the full extent of learning?
SPEAKER_04Um sure, um, like what you said, um we we do need aware of learning happens beyond just in the training room and in classroom. Um it evolves a lot of other ideas and concepts. And it's it's difficult. And it's it's it's not easy to do. And research has shown that learning is more than likely to be retained if it is still there three months after the training. So an evaluation, an effective evaluation should include assessing three months, six months, and one year, or even on a regular basis after the training, if time and finance permit. And second, I would say evaluation should examine whether the training program is conducive to learning transfer. Um, additionally, the longitudinal evaluation should look at whether transfer of learning has happened in the workplace and whether training participants are able to pick up new knowledge and skills while applying learned knowledge in the training. As a program evaluator, I would like to encourage a continuous upward spiral learning curve, which is from learning to practice to new learning and ultimately making learning part of the learner's daily practice at work or you know part of the trainee's daily practice at work. So it will be the whole system, not just pieces happening in the training room.
SPEAKER_03Perfect. Well, Jay, thank you so much indeed for our conversation. I really enjoyed our chat about evaluation today.
SPEAKER_04Thank you so much for having me.
SPEAKER_03Okay, well, stay with us because we're going to bring Darlene back in for the next segment and we'll have our group discussion next. So thank you so much, Jay.
Group conversation with Darlene and Jie
SPEAKER_03Up next, we have the group discussion, where my guests are together to discuss their shared passion for the episode's topic. This discussion is brought to you thanks to the sponsorship support of Interpretive Simulations. Since 2008, students and trainees have used Interpretive Simulations HR management simulation where participants are tasked to make challenging decisions at the HR director level in a simulated environment. Students must build a strong HR function at their simulated, medium-sized organization and wrestle with the challenges of staying on budget. The simulation makes the connection between concept and practice while students learn by doing. It comes with assignments, mini cases, and quizzes to reinforce core HR principles. If you'd like to receive faculty access to review the HR management simulation, visit them at www.interpretive.com and fill out a demo request. Welcome back to the HRD Masterclass podcast. Our focus for this episode is evaluation, and I've already met one-to-one with Darlene Roseft and with Jay K. And for the final section of the episode, we're all together for our group chat. So welcome back, Darlene and Jay.
SPEAKER_01Thank you. And it's great to be here.
SPEAKER_03For our group discussion, what I'd like to do is to dig a little into some of the topics from your one-to-ones and then also explore a couple of new ideas as well. And I thought uh a nice place to start would be to uh explore one of the topics that came out of my conversation with Jay, which is when the term uh confirmative evaluation came up. And I suspect that's a term that's probably less well known within HRD. And so I was wondering if you could say a little more about what it is and how it could be used in HRD.
SPEAKER_04Sure. The confirmative evaluation is to identify and explain the long-term effectiveness, efficiency, impact, and value of the intervention. It normally begins three to twelve months after implementation of the intervention to evaluate its sustainability. The process of conducting confirmative evaluation is really no different from that of formative and assumed evaluation. But the data collection methods and data collection sources might be different as the context changes. For instance, confirmative evaluation of a training program is to see whether learning is retained, transferred, and skills are applied on the job. The rationale behind confirmative evaluation in HRD is that learning is more than likely to be retained if it is still there three months after the training. Confirmative evaluation can also be used to continuously improve HRD interventions in change management and organizational development. So the conclusion and findings of confirmative evaluation can be used as part of the needs assessment for the next round of evaluation.
SPEAKER_01I actually did an evaluation a long time ago. We measured skills from an interpersonal skills training program immediately, so before the training intervention, immediately after the training intervention, one month after the intervention, three months after the intervention, and then I think six months and maybe even nine months. And uh what we found were first of all, the the biggest gain in skills occurred at about one month after training. Uh, and they were retained, most of them were retained for three months, and there was a bit of a decline uh at the six-month level, but not significantly. However, as I recall, there were some of the skills that just dropped off and disappeared. And then more recently, one of my doctoral students was interested in examining whether one could train multitasking skills in pilots and ran an experimental study with a control group and a trained group. And yes, indeed, she found that the skills of the trained group in terms of multitasking improved at the end of training, but they were not retained after six months.
SPEAKER_03So it sounds like confirmative evaluation clearly has a role then within HRD and how HRD programs are evaluated long term. And that thought takes me back to something else that came out of our one-to-ones, which is sort of how HRD is often evaluated in practice. And I think both of you mentioned at some point in your conversations the Kirkpatrick model, which is probably the most commonly found evaluation approach within HRD practice. And I know that Kirkpatrick's approach to evaluation is often criticized, but I'm wondering what you see as the main criticisms of it and also what the alternatives are for practitioners to consider.
SPEAKER_01First of all, it it may be called Kirkpatrick model, but as uh Ed Holton pointed out, it is really not a model, it's not a theory. What Kirkpatrick's uh concept is, is is something called a taxonomy. He's basically saying there are these phases, and there's no no clear evidence, some evidence, that one connects with the other, which is what's needed for a model. Another piece, um more from let's say the practitioner side, is that it it limits what is looked at as being evaluative. That is, what what are we going to evaluate? We're gonna measure reactions or we're gonna measure learning. And maybe those are appropriate and maybe they're not. There may be other things that need to be measured as a part of the evaluation. What might be some other um approaches to evaluation? Well, certainly the the issue of say uh of responsive evaluation or participatory evaluation. There's, I think I mentioned earlier Brinkerhof's uh success case method, which I think is really powerful for identifying what are the successful cases, what are the factors, what are the the dimensions, what are the situations, why is it that these are successful and these are not? Um and then for especially I would say for uh practitioners working within an organization, they really ought to look at some of Michael Quinn Patton's work around developmental evaluation, because what Patton talks about is the evaluator is there for the long term. And so you don't just do uh this one thing, but you look at the progression over time, and and the question might change over time.
SPEAKER_04Um other things I would like to add, some criticisms include Kirk Kirkpatrick's model only evaluates what happens after training instead of the entire training process.
SPEAKER_02Right.
SPEAKER_04And um, you know, also others claims the framework fails to specify what source of change can be expected as a result of learning and what assessment techniques could be used to measure learning at each level. And and I want to add um something that I really uh you know I agree with is from the John Greekle and colleagues, and they said the Kirkpatrick's model is not well suited to fast-paced a modern economy. See, for me, um we're looking at this pandemic and everyone moves learning and training online. If we use Kirkpatrick model to evaluate those learning, mobile learning, online learning, and it's extremely hard to evaluate the last two levels of learning and of results. So that's that's one thing I think Kirkpatrick's model cannot cover while we can look at the other alternatives. Um, last but not least, the HRD evaluation should look beyond organization. And some people said organization has a social responsibility, and the the societal impact of evaluation of interventions of organization should be also looked at. And that's the one aspect Kirkpatrick's model did not um take into consideration. Like you know, what you just covered, um, there are many other alternatives um to Kirkpatrick's model. So when practitioners try to decide which one to use, they need to look at their context and what can serve the best interests of the stakeholders.
SPEAKER_03So to ask a follow-up to that, that do you think that Kirkpatrick is widely used because it's so A, easy to understand and wrap your head around, and B, it because it's easy, um the HRD practitioner can just go through what they think is evaluation with with minimal effort.
SPEAKER_01Yes, I think I think you've captured it. It's really, I mean, it in some ways it's um what do they call it? Elegant? You know, I mean, it's so obvious. Oh yeah, well, we can um measure uh people's reaction. And and I guess what makes it attractive is measuring people's reaction. You know, do you like the training? How was the you know, asking questions about the instructor, the instruction, uh, etc. Um one can then suggest to stakeholders, well, this is a proxy for the effectiveness uh and maybe even the ROI of the training.
SPEAKER_03Yeah, when I think about the the need to influence HRD practice when it comes to evaluation, I think that one of the areas where I think there's a in my experience is a real need is in the area of ROI. And I know this got mentioned earlier in the conversation. And and I feel at the heart of that is a desire from stakeholders to better understand the value that HRD is bringing to organizations, and HRD's response to that has been to try and come up with uh numbers to describe return on investment. I'm wondering what advice you have for HRD professionals in how to navigate a conversation with an executive who is asking about HRD's value.
SPEAKER_01Sure. Um I mean I I think the first thing to do is to talk with the executives or or whoever is the primary stakeholder and say, so what is it that you want to see happen as a result of this training? Is it a change in people's behaviors? Is that what would be considered a return on investment? Um if uh you know it's training for call center employees that they resolve the problems more quickly. Maybe that's it.
SPEAKER_04Yeah, I totally agree with you, darling, on this. Um the HRD professionals cannot just always do uh what executives demand them to do, but they can be very strategic and um diplomatic in presenting the results. Um to be um, I think to be a good evaluator, an HRD professional needs to be a very good listener, be a very good communicator. Um if the executive is a target audience, um obviously we need to provide, and we need to provide the answer for them and be accountable for what we are doing. Um on the other hand, we need to listen to them really well and using their language to communicate with them. Um, like the examples that you're given, uh, very innovative and acceptable. It it can be accepted by the executive, and so it will be it will be implemented in in in a different way, but also reach the same result.
SPEAKER_03Some of what you're talking about there, Jay, I think is about the need for HRD practitioners to develop the qualities of being a good evaluator. I suppose one of the one of the challenges that comes out of moving away from automation of evaluation is that HRD practitioners are going to need to develop those qualities, so the knowledge and skills. So I'm wondering like to kind of broaden that picture a little, what you see as the qualities of a good evaluator, and and how would an HRD practitioner develop those?
SPEAKER_04A good evaluator needs to be equipped with hard skills and soft skills. So hard skills here I'm in, and you know, hard research skills, um, including understanding the learning objectives and designing, developing, and implementing evaluation plans. And a good design of evaluation is half done. So a good evaluator needs to make some decision on what key variables can reflect the impact of intervention in the context. Also, they need to make decisions on from whom the data is collected and what data collection methods to use, whether it's qualitative, it's quantitative, or mixed, and also how to implement the evaluation plan and adjust the measurements along the way, and how to analyze, report, and present the findings and the results. Um also a good evaluator needs to make an informed judgment on whether the results are aligned with the objectives set earlier, and where is the gap if they're not aligned? So um going back to the last question that you have, um a good evaluator needs the good listening skills and communication skills if they understand the executives what the executive needs, if the executive needs the numbers, they will try their best to convert those numbers, convert those information into numbers, and maybe use um number-based evaluation models such as the ROR model by Jack Phillips and the Kaplan Noten's notion of balance scorecard or even utility analysis. And if the executive want to hear um the descriptions and details of what's going on as evaluator, the HRD professional can present them the pictures, graphs, and tell the stories from the data collected.
SPEAKER_01So what I think of when uh when I think of what's needed by um uh practitioners or those who want to undertake evaluation work, uh I think of a couple things. I I think um, first of all, well, one could go and take courses, academic courses of some sort uh at a university. Um the it turns out that there's also the American Evaluation Association uh holds uh something they call the Evaluators Institute. It's done uh periodically in different parts of the US, uh, and you can go to this institute and uh uh obtain training in different aspects of evaluation. Um I think in addition, uh both the there's the American Evaluation Association has a set of competencies for evaluators. Those are really focused more around uh evaluators who are working, you know, doing evaluations of federal or state uh agency programs. There's also the International Board of Standards for Training, Performance, and Instruction. They have a book out called Evaluator Competencies, Standards for the Practice of Evaluation in Organizations. The book not only describes what are the competencies, but also provides some ways in which one can improve one's skills. There's another book out by that same organization about specifically activities that one can do to improve one's skills in evaluation. Another way to obtain some of the skills around evaluation work within organization would be to hire an evaluator and specifically ask for, we want you to build our evaluation capabilities here. And uh most evaluators will understand what that means because there's a whole movement within the field uh around uh building competencies, building capabilities of uh evaluation so that organizations can undertake those evaluations themselves. So there's there are actually a lot of different resources that are available to anyone.
SPEAKER_03That actually sounds like a wonderful call to action to have at the end of our conversation. So thank you both so much for your time today. It's been uh wonderful having uh this conversation with you both. Thank you.
SPEAKER_01Thank you. Thank you. Thank you, Darren.
Wrap-up
SPEAKER_03Thank you so much for joining me for this episode. It was wonderful spending time with Darlene and Jay. If you enjoyed the episode, check out our others to explore topics such as training and development, learning organizations, critical HRD, and much more. New episodes release weekly. To learn more about the series, check out HRDmasterclass.com. And to learn about the Academy of Human Resource Development, check out ahrd.org. By becoming a member, you can access extra bonus materials not included in the podcast. Also, don't forget to look into our group discussion sponsor, Interpretive Simulations, by visiting their website at interpretive.com. I'm looking forward to being with you in our next episode. Until then, this is Darren Short signing off from the HRD Masterclass.com.
