This episode we are excited to welcome to the podcast Danielle Duran, Director of GxP Compliance and Training at Aimmune Therapeutics! As fellow advocates for education and continuous enablement in life sciences, Dori and Danielle share their insights on the current state of learning in our industry, and where we can do better to support more effective outcomes in IT and Quality organizations.
Listen in now to learn more a bout leveraging effective learning to support quality culture, shifting towards effectiveness measures of training, and, perhaps most importantly, moving towards a people-centric learning model as part core business operations.
Enjoy the episode and please don't forget to rate, subscribe and share!
*Disclaimer: Podcast guest participated in the podcast as an individual subject matter expert and contributor. The views and opinions they share are not necessarily shared by their employer. Nor should any reference to specific products or services be interpreted as commercial endorsements by their current employer.
This is a production of ProcellaRX
This episode we are excited to welcome to the podcast Danielle Duran, Director of GxP Compliance and Training at Aimmune Therapeutics! As fellow advocates for education and continuous enablement in life sciences, Dori and Danielle share their insights on the current state of learning in our industry, and where we can do better to support more effective outcomes in IT and Quality organizations.
Listen in now to learn more a bout leveraging effective learning to support quality culture, shifting towards effectiveness measures of training, and, perhaps most importantly, moving towards a people-centric learning model as part core business operations.
Enjoy the episode and please don't forget to rate, subscribe and share!
*Disclaimer: Podcast guest participated in the podcast as an individual subject matter expert and contributor. The views and opinions they share are not necessarily shared by their employer. Nor should any reference to specific products or services be interpreted as commercial endorsements by their current employer.
This is a production of ProcellaRX
Well, welcome to another episode of software quality today. Hi, Danielle, how are you today? I'm great. Thanks, Tori, for having me. So I would I am so excited about today, because we've met over the course of this last year at a couple different events. But this last event in December, we got to know each other even more. And I'm so excited about our new collaboration and friendship and all of that stuff. Love it. I'm so excited. So before we get started, I would love for you to tell everyone about your journey to where you're at today as a director of GXP learning.
Danielle Duran:Sure, so I, I took a route through learning. That's how that's how I got where where I am right now. It started my very first role out of college was teaching math and science and medieval modern world history to seventh graders in South LA. And I was I was feeling pretty passionate, by the end of my first couple of years about teaching math and turning, you know, preteens into math nerds, that was a mission that I had. But given how things worked in large public school systems at the time, I was bumped from my position and was being pushed into a role to teach more history, which I was less passionate about. And that caused me to reevaluate what I wanted to be doing, I ended up doing some recruitment for the teaching program that I was in to make sure that my students were going to have really great math teachers. But working in recruitment taught me that I wanted longer term relationships where I could coach and support the growth and success of others. And so I ended up in a role where I was coaching and supporting new teachers. And that led me to a place of managing sets of those coaches. And then I moved into, then I moved into other roles related to teaching education for a while, including some medication technology, and then I ended up in philanthropy. So on the I've done every single side of education you can do. And in that philanthropy role, I started doing a lot more evaluation of effectiveness of programs and of education programs. And, you know, what are models of doing that and working. Another role that I had was working with the state on creating an assessment for the California Community College system. And so I got to work with psychometricians. And that's when I first started learning about validation, testing validation, and that that was very interested in that. So we use little bits of things along the way. And I did study science in college, I spent a lot of time in labs. And I think there's always a little bit of something missing in my heart for that science, heart miss in me that, you know, kind of loss for words for the right right thing there. But a role opened up at a biotech company closer to where I lived. And I was, I just jumped at the opportunity, it was in corporate training. And in the compliance area. So it was, it was great, because I got to learn everything about GXP. From the from the highest level, I got to understand everything about compliance and what that means I got to work with a lot of auditors. And that gave me a really good view into what I was doing. That department was ultimately eliminated. And I moved into technical operations where I was overseeing a one of our manufacturing sites and not site one of the one of the manufacturing groups within the larger site, there were several. So I worked in gene therapy for a little while supporting that group, and then ultimately led our site training team and was working on global initiatives, and really wanted to just continue having a larger global impact. So that's, and then I found the role that I'm in right now where I get to oversee the governance of of training The training system, writ large for the enterprise, and I am responsible for all of the regulated training.
Dori Gonzalez-Acevedo:So part of your story is very common of the some of the women that I've also interviewed this past year, right, is that we've had these kind of very divergent I don't even mazes of careers, right.
Danielle Duran:Oh, yeah. It was not a straight line.
Dori Gonzalez-Acevedo:And your mom and all this other stuff are all those.
Danielle Duran:Yeah. And I had a baby during that time. Yep.
Dori Gonzalez-Acevedo:So what about that having that diversity of experience gives you the opportunity to do what You do today.
Danielle Duran:So I, in my experience so far I, when it comes to my colleagues in, in the regulated side of training, I haven't really come across that many that have a full background in education. And, and really understanding at a deep level how learning really works within the neurobiology of that is, and what, just the vast tools that are available for effective learning. And a lot of times, I think folks in my type of role will come from it or will come from equality. And then the folks in l&d that are on the commercial side, often they're, they're really successful sales trainers, and it's a, you know, a step on the way to management. But I think the advantage that I have is really understanding what effective learning is, and how and how it connects to culture, and how important the culture part is when it comes to learning. And then neuro, the neuro stuff matters, because it's how, how available, your mind even is in order to learn. And that's not something that we think about or talk about a whole lot.
Dori Gonzalez-Acevedo:Absolutely 100% agree. And one of the things in what you're saying, strikes me as a real issue over the years that I've seen is that the effectiveness piece, right, whether we're talking about effectiveness of, of Kappa, implementation or effectiveness of learning, there has been a huge gap in within the industry. And one of the things often has been said, oh, we'll just do if we didn't learn and understand, right. And I know that that bugs you out, like really, really bad, and it bugs me out to you. But so can you say more about that why this effectiveness piece is really, really critical?
Danielle Duran:I think that Well, I also think that this is something that the health authorities are paying attention to even more. So it's even more important that we have the conversation. I think in the past, it was acceptable that a training activity being complete, meant that it was effective, but especially within what you're referring to where training, the word training is equated to an attestation of understanding, without any measure of oh, what's our evidence, and I find that so interesting for this field where everything is evidence based, right? You're not going to take a product from, you know, across the phases without really strong evidence that it's got really high efficacy. So you know, what, why, how come we're not practicing the same thing when it comes to our learning, and it's expensive, conducting training activities, it takes a lot of time, even if it's just reading SOPs, which is not training, just reading. But that's, you know, it takes time, and people who only have so much bandwidth, so much mental RAM, you can say, in order to process new information, and then if they're not going to put it to you. So if you don't have an actual training activity in there putting it to use applying it, they're really not going to learn it effectively. I mean, yeah,
Dori Gonzalez-Acevedo:yeah, it's a I think it's a big issue that we seen in quality management programs, or are not such great quality management programs. Right. I think we've also, historically have been led to think that if we do the checklist, and that we provide the checklist to the auditors or the regulators, then then that if as long as that checklist is done, and all those signatures are there as that adaptation is there, then then we're good. And we can walk away, and we've done our job. Right. And and that's not necessarily the case, right? I think there's plenty of examples out there where, you know, we've put as a cap item, right, that it's due to training, right? Human error, human error is retraining and retrain. And, and how many times do we do that right over and over again, and simplify it in that sort of way, but yet, get to an end result where something is reproducible, and effective, and learned from and added value to write all of those things. And then the other part of that is walking the walk, like saying saying, Oh, go do this training, but actually invest in the dollars associated to that. So I know you have some thoughts around how how, you know, where is learning and in the scale of organizational hierarchy?
Danielle Duran:Yeah. Well, I think I think the most the most critical thing that any One can do and a lot of a lot of what I recommend is balsley. Like, quote free in that it's, it's really just how are you leveraging the resources you already have primarily people and their time. And I remember I found a quote some time ago that I wish I knew who said it, maybe someone one of the listeners will know. But it's something like, If you don't have time to do it right. Now, when will you have time to fix it? And I know, they're stuck somewhere that I've read that, you know, talk about the cost of quality and the cost of preventing an error, as opposed to fixing an error. There's something I talked to my central daughter about literally, like two days ago, I was like, okay, but is it? Is it easier if we prevent this giant mess from happening? Or is it you know? And what is what? What's the investment? How am I going to, you know, what, how much
Dori Gonzalez-Acevedo:time I think so you're talking to your seven year old around that I think most folks know that organically at some level that that is the right thing to do.
Danielle Duran:But we don't do it. So this so this? So I'll start even with this idea of human error and kappa. So there's a human error? To what extent are we holding ourselves accountable to spending the time to get to a really good, specific root cause? And if that root cause really is why I didn't understand what I was supposed to do? That's different than I didn't follow their directions. Right? Like, those are two very different conversations. I could see retrain, sure, if you just didn't do it, then like, Okay, we need to have a little heavier hand. But if I didn't know what to do, that's very different. But if you're if your kappa is retrained the exact same way you did the first time, then what's your confidence that it's going to work the second time, so it didn't work the first time. And so it's, I think a lot of times in our field, things are so fast, and no one wants to slow down, because we're all under so much pressure. Right.
Dori Gonzalez-Acevedo:I would argue that a lot of many verticals. It's not just like sciences. Right. I mean, I think that's also kind of a the the awakening out of COVID. Right? Right. I think now we can globally and universally say, we don't have enough brain mental load to go around?
Danielle Duran:Oh, yeah. Now for all the things that were demanding of everybody. So and that's, that's where? So using that as the little kernel example. But thinking bigger of if, if someone if a manager says my people it, what does it look like to be qualified? What does it mean? And then take that time to really define? What does it mean to be qualified that I feel really confident in my people being successful later? And then if they think that someone needs some additional training, then then how is the manager going to know that that training work? How do they know that it was a good investment in time? What should they be looking for later, to see? Oh, that that training did have an impact? And I think that's where I really wish one of the things I wish I saw more of was managers being super engaged in what's in my people's curriculum? What what do they need to do, whether it's SOPs, or on the job trainings, or, you know, some other type of elearning? Is that really the thing that's gonna get me my people to where they need to go?
Dori Gonzalez-Acevedo:Well, so let's, let's pause there for a moment. Because do we think that the managers at those levels actually have the skills that they need in order even to make those assessments?
Danielle Duran:Right? Well, that's a very good question. And I'm not sure they're, I don't I don't know. I think a lot of times, not necessarily, but I don't know if they're ever taught what that looks like. Right? We oftentimes, especially the mid level managers, they're not, that's not something they've ever even seen. So how are they supposed to know how to do this? That's right. And, and then there's the, you know, very common leadership training, right that people might get, but that's not teaching you this very technical skill of how am I defining qualification for my people? How will I know what a gap is in their performance and how to match that with the right, I think I've just seen some really, like heartbreaking examples of, quote, performance management when someone isn't performing, but the manager doesn't know how to coach them into higher performance. Because it's it's not it's not straightforward. And it's not it's not simple. And sometimes it takes a lot of that
Dori Gonzalez-Acevedo:so complicated. I mean, yeah, I know, even with my teenager, right? Like, there's so many complexities to learning styles, my own myself, just like sick, and it is not something that I really understood until much much later on in my life. Now that I know that I know some things that I can do in order to help mitigate that. Right? But there's there's a whole spectrum out here for for how one can consume information in general.
Danielle Duran:Right. And the the most important thing to remember is the the, there's a there's like, blinking on the term I just I just saw LinkedIn posts about it. There's, there's in like types of intelligences like the seven. I think it's Howard Gardner like types of intelligences. And then there's, I think what what we grew up being being told was learning styles, right? visual, auditory, or tactile, whatever. But what the research has shown is that learning styles isn't a thing. It's not, it's, the preferences are real. But what matters for effectiveness, and again, coming back to effectiveness is that you're tying the right style to the content. So it's kind of like, are you trying to teach how to identify birdsong? Or, I guess, in our area? You know, visual inspection of something, you know, something in QC. But are you teaching it with visuals, like you have to see visually, you can't just describe it? Are you you know, do you need to see the actual thing with examples as opposed to pictures? Right. So just being really clear with again, the most important thing is know what you're trying to accomplish, so that you can plan to that, and it takes a lot of time, it's hard work. That's very, like heavy cognitive load of, do I really know what I want to get out of this program. And that's my argument against buying off the shelf training programs. Because it's very easy to want to outsource that mental load and say, Oh, no, someone already made this thing. Someone already made these elearning. So I just have to give them to my people and make them watch these videos, as opposed to well, what do a people specifically need in order to be successful? These the learnings could be a tool, but what questions am I going to ask about it? Or what's the conversation? And how are we going to apply this new information so that they can put it to use otherwise? You know, just watch TED Talks? Or I don't know, you know, what's, what are we? What are we doing? Yeah,
Dori Gonzalez-Acevedo:and there's so much content out there, too, is like so then what do you even choose? Right? Because at that point, you're you it's just a commodity and not something specific to your organization to the actual use case that you need to the pain points that you have to the people that you have the combination of full time employees versus contractors, right.
Danielle Duran:Oh, yeah, that's tricky with employment law in some states. Yeah.
Dori Gonzalez-Acevedo:It's a big, big deal. Right. So so and we're in an industry that does do a lot of contracting. Right. And so then, whose responsibility it is to train? Right? Am I paying those contractors my time in order to train them? Or should they be coming in trained as a some base level, you know, understanding? And so whose responsibility is,
Danielle Duran:right? And how often are we evaluating? Like, do we really know? Because they say, you know, they're, they're saying that they know these things, but do I? Do I check? Or how do I check or what's, what's effective to check?
Dori Gonzalez-Acevedo:It's really hard. I mean, I struggle even internally here. And as we were building out our team here at persona, like it was like to really understand everyone's needs, right? We're small enough that we can do that, right. But yet, how to continuously monitor and engage with them in order to, to have them grow. Right. So what are the gaps? I am not a learning specialist. Right. Right. You know, and so, so, so when I approach it from a scientific perspective, and I, you know, I try, it's trial and error, right? It's, you know, put something in place, does it work? Does it not work as an effectiveness check? Right? Yeah.
Danielle Duran:But that's great that you're checking, did it work? That's the most important thing. That's the question I got asked during an inspection with the MHRA, you know, how do you know that worked? How did you know that aseptic technique training worked? And, you know, having someone pass a quiz that requires them to pass it at 100%? That's, is that really evidence that it did work? And this is, I was on a panel or I I hosted a panel in June where I had a health authority, member share, what are you really looking for when it comes to effectiveness of training? And his his response, included a lot of things like well, there's certain things that people think that we want, but that's all the floor, like yes, you should have strong training metrics of completing training on time quote, training, right SOP training. But that's that's just a standard expectation that that should just happen. What really matters is your your QMS what how what are your deviations looking like, what your cap is look like how effective are your cap is. And the thing that he really spent most time on was collaboration, which I was not expecting. And he talked a lot about documentation of procedures? And is everybody in the room that needs to be in the room when designing that procedure? Are the people who are executing there to make sure that it's going to work? And then do you test it before you make it effective? Do you go and make sure it works? And then is there improvement over time? And I that's not something that we talk about ever when it comes to training effects?
Dori Gonzalez-Acevedo:Very interesting. So I don't know, 15 years ago, when when, when we did peak user performance qualifications, right. For CSP, when we were still kind of doing that terminology for CSP, part of the methodology that I had rolled out at that time was actually executing an SOP. Right, as evidence that the system was performing as intended. Because if the SOP that's written for that system, is not able to execute and get the result that you want, then something's amiss. Right. Right. I've seen that go away. Like, that doesn't happen anymore. And I wonder about that, because I think that back to your earlier point around how much time and effort are we spending on these activities? You know, we do a lot of stuff around validation and getting things to a state of quote, unquote, you know, validated. But the ongoing maintenance, and the operation is where all this stuff really lives. Right. Right. Right. How is it working? What what's not working, even though their systems and we programmatically design them for some things. Still, there's a huge amount of input human input that's required, regardless of where we're talking about machine learning or automation to write because
Danielle Duran:there's no humans involved, which means there's still going to be, you know, room for error.
Dori Gonzalez-Acevedo:Yeah, yeah. So what would your ideal? I'm curious, what would your ideal kind of, you know, in a 40 hour week, what would you want employees to be spending their time on? I'm curious to hear what you think,
Danielle Duran:how people should spend time I think I if I had a wish list, right? If if money wasn't an object and people can, I would love that. From the what is it? There's the common ad 2010 model or 770 2010? Model have the most of your learning comes from doing your job? And I had a manager in cheese, when was it 2007? Maybe, who? Two dozen eight, maybe who told me the best professional development you can do is your job well, and I thought, I think I didn't want to hear that. Because I think I wanted to do you know, conferences and courses and special projects, but she, I I'll never forget that of her telling me you need to do your job? Well, and I think about that a lot. Because there's a lot of parts of my job that I'm I'm still, you know, things change so much. I have my scope changes, things are new, I have to do a new type of system. And if I could learn, I can keep learning, even if I'm doing only my job. Right. So I think, but I think the piece that's missing is the reflection. So if if, you know, people typically I think they say, the effective, like use of people, it's you have 70% of your time that you're actually getting anything done. The rest is who knows what. So out of that 70%. If you're only if you're doing your job job for 70% of that time, then you still need to build in I'd say at least a half an hour, once or twice a week for reflection, like what what am I doing that's working? Who did I talk to that said something really insightful? worded? I read something that I want to you know, apply. And then the 2010. That's the, you know, coaching mentoring is 20. And then 10 is the actual learning. I don't know, I honestly don't know if I know anybody who actually spends 10% of their time on learning outside outside of work. I don't I don't think I know anybody, not even me. Yeah. And
Dori Gonzalez-Acevedo:it's something we don't prioritize, right. And part of what I always say, though, is also but a different approach to how you do work. So one of the things I just heard in what you were describing, Danielle, was that how you approach work which is actually very similar to me and maybe part of why we feel so? Akin? Yeah? Is? It is learning? Right again, doing it all in, right, like, fully consumed and in the flow, if you will, right, like really asking all those questions, you know, inside and out deconstructing reconstructing is all part of the process of which I do my day to day job.
Danielle Duran:Right, right, and asking those questions and figuring out, Well, who else do I need to talk to? Or who else might know something about this? Or who's doing this better? Or? I don't know what went wrong there. But I needed you know, it's it's a lot of asking questions to understand, right. And I think that without having really a strong growth mindset, that can be really hard, and it can feel very threatening and scary.
Dori Gonzalez-Acevedo:And you also need an organization that supports that. Yeah. Right. Because if you don't have an organization that supports curiosity, and a growth mindset, what are you left with, you're left with, you have to make sure you get these 10 widgets done before the end of the week, because that's your job. And that's your task. And we really don't care about much anything else other than the task. Right?
Danielle Duran:Right. Right. And well, there's the there's the aspect of the making the widgets, where if something goes wrong with the widget, if there's not a strong learning culture, then that deviation on the widget isn't going to be escalated as quickly or as thoroughly. Right. So having that learning mindset, having that learning culture is critical for a strong QMS. Because you, you have to make people comfortable with mistakes, and learning from those mistakes. And then the last thing you want to do is have five different people make the same widget mistake, then how much waste is that right? That's wasting time we've seen product, we've seen raw materials, you know, putting in danger getting product to patient, or whatever product to whoever uses a widget. And, and that's where that bigger picture Lessons Learned comes in. And so even if we don't have time to spend 20% of your time with a coach or mentor, I mean, that's really what some of your manager interactions should be like, are you getting enough coaching from your manager? are you presenting us information so that you can be coached? And then are we really preparing our managers to be effective coaches? I don't, probably not enough, honestly, not enough. But from I think, plenty of companies that are successful have made statements about, you know, it's your competitive advantages, how quickly you can learn, right,
Dori Gonzalez-Acevedo:and adapt and pivot and all of that, right. And in order to do that, a lot of the things that we're talking about, right, the learning mindset, the curiosity is, you know, the constant questioning, I also think that questioning in some cultures is not accepted, right. And so, so to add to the complexity of this entire conversation, right, is to put in a diverse workforce, or hoping that most folks are striving for a diverse workforce. But that does require even more question, more understanding on everyone's part.
Danielle Duran:And sometimes you can't wait for, for people to ask a question you need to be inviting, right? And invite the how do you create a culture of inviting that kind of feedback, whether it's on behavior and performance or on on a product or on a process, right, like, what, and making sure that we have lots of those diverse experiences, so that we get the best? And the end is possible?
Dori Gonzalez-Acevedo:Yeah. So in your kind of exploration, I know you've, you've done some talking, as well as at some major conferences. So you have a lot of you hear a lot of things, and what are what are some of the things that are pitfalls, or the gotchas that folks could quickly just nip in the bud and stop doing in their learning program or, or do better?
Danielle Duran:think some things I'll share one pet peeve, which I think drives people crazy, even if they don't realize it, and that's gathering data that you don't use, or, or having having data that that you could use, but you don't do anything. Like what like an example could be a lot of people might have quizzes, right or some type of knowledge assessment connected to reading an SOP or doing some reading a PowerPoint, whatever it would be, but how often are you leveraging that data to improve an SOP like, jeez, this question has a 30% pass rate on the first time? Maybe we should clarify it if if you're using your SOPs as training any materials, which they're not usually written to be a training material, then then then you have to pay attention to the performance of it as a training material. that would that would be one thing.
Dori Gonzalez-Acevedo:Well, that actually goes right back to your whole the psychometrics right of a lot of stuff, right? I mean, in a, in a, in a well defined training program or learning management program, right? All of those metrics are part of the process of doing all of that, right. Which is fascinating to me, because it's there's a lot of validating talked about validation, there's a lot of validation that goes into a lot of that stuff.
Danielle Duran:There is massive amounts, like massive, massive projects with like, millions and millions of people, right as data points. That's years and years of data over time. So I, but I think the point that you just made, is my maybe like, that's the other thing that can be a quick fix is, did you clearly define your objectives? Is your training program set up in a way that it's to accomplish tasks? Or is it set up in a way to build knowledge or skill? And how clearly have you defined those requirements for qualification? So that you, you can even track that? Because if you haven't defined it, then you can't gather data to show that it's working or not working?
Dori Gonzalez-Acevedo:Yeah, I think, again, given that we have younger kids, right, what I've been trying to instill is this, you know, constant, learner, right, and learning how to learn for themselves as they go through this process, and things that I would never have thought I would say to them, like, I really don't care about grades, I really don't care about this, I don't really care about this, like things that I held very true for me, right, I've thrown out the window, right? in service of curiosity and learning being the most important skill set that they can get. Now,
Danielle Duran:yes, and the not being afraid to make mistakes I had, I had no idea how much I would be battling this perfection pressure that my daughter puts on herself. And, you know, just the, the, that's the whole idea of perfect is the enemy of good or something perfect is the enemy of complete, I don't know, one of those tech companies have everything. But that's, it's hard. I think we all get kind of caught up in that also. And it's some of that analysis paralysis of well, I want to do this thing for my training program, but it's just really huge, big thing. And so I just can't get started. Because it's so overwhelming, where, you know, just pick one thing, pick one roll, or pick one, pick one set of SOPs on one critical, you know, set of procedures that you want to improve, pick one piece of the QMS, pick one department, you know, there's you got to just start somewhere, and it'll feel so good when you're done. It will build some of that momentum to get through some of the hardware things. Future thinking?
Dori Gonzalez-Acevedo:What do you think the impact of AI and machine learning have on learning programs?
Danielle Duran:That's such a big question. And I have not been like
Dori Gonzalez-Acevedo:to ask you. But like, I think it's an important, right, because it is.
Danielle Duran:I think so far, what I know has been a trend. I don't know about a trend. But I know people have gotten a lot more involved with AR and VR. So the augmented reality, virtual reality when it comes to training, that's huge. And I'd say I had pretty strong opinions about it when I was working to support manufacturing, because my concern was that we were treating people like robots, instead of leveraging the assets that they are people instead of robots. So it's it's fine to do mented reality where you know, you have some extra directions and it helps them to do the task. But it's very easy to get cognitively lazy, honestly, is the best word. Right? Like we just want people to do this one thing and want them to do it perfectly every single time but that's just not you know, that's it might be more than it might be a higher expectation than is reasonable. And, and if you want to, you know, I just I had a hard time really believing that the The ROI on investing in AR VR was going to come back like, is it really worth the investment? And what are you really trying to accomplish? And what are you really trying to automate out because I'm sure some of it is really good. But I think just like any tool that comes into the training and learning realm, especially on the GXP side, people get very excited about tools, but not necessarily building a program around the tool, and really having clearly defined objectives and clearly defined KPIs for effectiveness. So if, if you're implementing, and you have all that in place, then I'm super on board. But if you're implementing it, because you think it'll make it easier, it won't. That's just not how it works. But when it comes to AI, I know a lot of like school systems teachers are having to really reevaluate what they're expecting. Yeah, just because how people are leveraging it.
Dori Gonzalez-Acevedo:Yeah, I think you're spot on around the expectation piece, right? I think that there's. And I can't even fathom, like, there's just so much to consume today. Right? That's the other part, right? Like that I worry about in just how much is possible. And then to your expectation point is what's necessary, right? And how to leverage the technology in service of doing better. And I don't know, if we've, as an industry, at least in the life sciences, have, we're there yet? Because we haven't really even been able to do like, the basics yet. Right? But I know it's wanted, and it's coming. Right? Whether we like it or not, it's it right. And so I I'm, I'm pondering that sort of thing. And not just in the learning space, but also in the manufacturing space. And because it's going to hit the manufacturing floor faster than ever before. And in what are we are we that far behind? And how do we get up to speed?
Danielle Duran:Yeah, I think, like anything else, like any of these tools, the man even like project management tools, or, you know, tracking, checklists, tools, if, if it's not implemented mindfully with all these other, you know, objectives, and how we, you know, it works and how are you going to train people to use it and what's Okay, and what's not, what are the guardrails? You're you're just adding more content, you're adding more stuff for people to do. And there's only so much that people can do, just period, you know, we maxed out. So I remember reading something not too long ago about advice from someone saying this is, it's a great tool in order to free you up to do the more complicated stuff like yeah, let's let I'm 100%. Let's automate all the little things. Thanks. Yeah.
Dori Gonzalez-Acevedo:Well, and there's any automation tool, right, I mean, that's, that's, that should be the expectation and the first low hanging fruit, right to get all of those tasks as automated as possible, so that we can free up our brains to take on some higher level thinking. It's just a matter of, what do you do? What's next? And how does that evolve when you your basic programs are not really hitting the marks? In some some regards? Right? So we talked a little bit about like, you know, quality as an ROI, but like, learning as an ROI is probably even harder to quantify. I suspect, I haven't tried have you to do metrics around that. Tell me.
Danielle Duran:So the the, I think one of the trickiest parts, when, when working with your cross functional partners to define what those KPIs are going to look like, is not getting caught up in Oh, it's too complex. We can't do it. Because there's so many factors. So there's the most common, I'd say, way to measure ROI, ROI and learning is the Kirkpatrick model, where it's four levels level one is, did the people like it? And that's where you get smell sheets, or you get the survey at the end, you know, are you satisfied? And then level two is knowledge transfer. And that's where the quizzes come in. So do you have evidence to show that knowledge was transferred, did did the knowledge transfer part work? And then level three is behaviors. And that's this is where it starts getting tricky, but and you have to define it on the front end. So you define I'm going to do X training. The behavior I want to change or the behavior I want to build is x and so how will I identify it later? And that's hard is is who's going to identify it? How will it be reported? How will you record it? Right? And is there different observers, and you have to validate the different observers. And that's where in education, you'd use rubrics, and then you train people to rubrics and you norm. But that's just not something I've ever really seen done here. And I think, because it's not commonly, then people want to say, Well, you can't do it well, yet, you can. That's how you grade AP essays, you know, that's how you evaluate teachers, is you have a rubric, and you have people norm, and you have standards at each level, and then you all rate until you all get the same scores. So behaviors is level three, and then level four is business results. So if you like, let's say, you're working on your deviation training, program, deviations take a certain amount of time at the you know, that's your baseline, you take a certain measure, there's so many errors, and it's this many interactions or whatever it's going to be, and then you do your training programs you test for did people like it, you do a knowledge check? Do they retain certain amounts of new things that they have to do differently than the behaviors? Do they change their behaviors? are they submitting more on time? Are they filling these things out more correctly? Are they you know, what are the behaviors and then business result well, before, before the training, deviation took, on average, you know, 28 days, or had this many errors, or to this much time period, right. And then if you calculate that into a cost, each deviation costs$10,000. But then after the training, with the follow up, net, we're saving $3,000 per deviation because of this training. So there's the business result. But it takes a lot of time. And it takes a lot of data. And I just wish I had more lovely green belt black belt people to work with. To help with all that. And that's, that's where I really wish there was more collaboration between the operational excellence or continuous improvement teams and learning, because their work is so related. I think it's funny some time, I remember, a boss I had a while ago, we were doing worked in business operations group. And we were doing an exercise a marketing exercise for our department. And one of the questions was, who's our greatest competitor, like, we offer a service learning continuous improvement, we offer service, who's our greatest competitor, and it was our customers, our greatest competitors, because they want it, they want to do it themselves, or they've always done it themselves. They're not used to having support. And so we're not involved in these conversations. And so that's the other advice I give to people is, you have experts out there probably somewhere, you know, go to your learning person, and, and partner. And, and you don't have to, I don't know why I don't know why people don't that would be a root cause I'd like to explore is why when people don't reach out to their learning departments, why not,
Dori Gonzalez-Acevedo:but we see with quality in it too, or, you know, quality in business, you know, like, it's reaching across the aisle, it's having shared conversations, it's having the nuanced conversations, it's, you know, it's really, for having the time to do it, having the time to do it, knowing who to do it with, right, who you know. And it also, you know, really breaking down the notion that your, your, your lane is your lane, and we're all connected, like we're all connected, right, and really understanding that and one of the things that I love about some of the the good programs that I've seen is really like an understanding from a quality perspective, how we're all involved, right? Like, it's, it's, it's so incredibly important to have that that connection, be illustrated be demonstrated within a whole organization. And part of that is a whole Learning Foundation, from beginning to end this just like the quality like so, quality and learning should not be separate from its, you know, it's all part of the same thing. Yeah,
Danielle Duran:we're all it's all it's all an organism. Right, that's the system thinking. So you Peter and I'm not I don't know how much again, this is to me, I would love to have a system thinking program that you know, that people understand here's, you have to consider all these other areas. And if you touch it up here, then downstream what happens just like data
Dori Gonzalez-Acevedo:moves, right so and it moves and shifts so as you put into that system, you know, it's entropy extra Pete right, like as you move in IT contracts and moves here and then it moves and shifts again, and it has to be that continuous look all the time, right of where the needs and what is in service of what and what are the in service of the need at the moment because today is different from six months. And you know, the past six months.
Danielle Duran:I mean, like weeks
Dori Gonzalez-Acevedo:Yeah, So, all of those things. So, um, this is awesome. And I think you and I also don't think this is going to be our last podcast because we have lots to chat about. I think this is a good prime prime start here. And I thank you for your time today. I really appreciate
Danielle Duran:thank you for your thank you for the conversation and I'd really love to know what people want to talk about more you know, what what are people curious? Oh, what did they wish they knew more about?
Dori Gonzalez-Acevedo:That's great. So let will ask that question when we post this out. Have folks comments on and we maybe we'll even do maybe a live q&a next time and we'll get Oh, I love that. Awesome. Yeah. Cool. All right, Danielle. Thanks so much.
Danielle Duran:Thanks, Tori.