“You can’t improve what you don’t measure. But you cannot measure what you don’t understand.” -LGen John H. Cushman
In today’s complex sociotechnical world, LGen Cushman’s words are more important than ever. Join MINDWORKS host Daniel Serfaty as he talks with Dr. Shawn Weil, Courtney Dean, and Evan Oster of Aptima about what it takes to blend the wisdom that comes from the field with the science that comes from the lab, to optimize and improve human performance in mission-critical settings.
“You can’t improve what you don’t measure. But you cannot measure what you don’t understand.” -LGen John H. Cushman
In today’s complex sociotechnical world, LGen Cushman’s words are more important than ever. Join MINDWORKS host Daniel Serfaty as he talks with Dr. Shawn Weil, Courtney Dean, and Evan Oster of Aptima about what it takes to blend the wisdom that comes from the field with the science that comes from the lab, to optimize and improve human performance in mission-critical settings.
Daniel Serfaty: Welcome to MINDWORKS. This is your host Daniel Serfaty. Today, we will explore together how science and human performance data are revolutionizing the world of education and training. This a big task. Over my thirty plus years of working tooptimize and improve human performance in all kinds of mission-critical settings, I had the privilege to work with one of my mentors, Lieutenant General John Cushman, who passed away in 2017. I learned a lot from him and there is one phrase that he used to say that really stands out to me the most as being extremely relevant to today’s challenges of optimizing training, optimizing education, and understanding human performance. General Cushman was not just a highly decorated veteran of the Vietnam War, but later in his career, when I was working with him, he was very instrumental in really revolutionizing the way the US military thinks about human performance when it comes to training our top commanders.
This is when he and I worked on a project trying to understand the nature of expertise in command. We were working at Fort Leavenworth—here you had one retired General, General Cushman, training other Generals that were ready to go to the field and teaching them how to do their job—evaluating them, coaching them, and understanding what needed to be done for them to be Field Generals.
General Cushman used to say two things that stay to me to this day and has been the philosophy behind a lot of the work we have been doing at Aptima. It was a two-part statement. First part: “You cannot improve what you don’t measure.” Basically, the false claim of, “Oh, things went well,” is not good enough if you cannot measure the before and the after of the training intervention. But I think it’s the second part of his statement that is most important if you really want to improve human performance. If the first part was, “You can’t improve what you don’t measure,” the second part is, “But you cannot measure what you don’t understand.”
Our job is to first listen to the field commander, the surgeon, the law enforcement officer, about their experience and their expertise, and then take that and frame it within what we know about human science. So that the end we understand that out of the many, many things that you could measure during action, during performance in the field, there are only a handful that truly matter. How do you find the handful? That if you measure each one of these dimensions, these key dimensions, you know when to train your people, when to put people in a situation, when to give them a technology to improve their performance. And you know also whether or not five or six dimensions are really going to improve or not. Are you going to move the needle on those key performance that emerge from that notion of understanding before you measure and understanding through human expertise in the field and human sciences in the lab. It’s not easy to do that. It’s actually very hard to do it well because when we try to identify this handful of dimensions of performance—sometimes in the military they call that “measures of effectiveness” and in the management literature they call that sometimes “KPIs” or “key performance indices”—those dimensions that really matter, we tend to either ignore a lot of so-called common wisdom or give away way too much weight on things that are not truly relevant to the one thing that counts the most, which is learning.
But today we have some very special guests that are going to help us unpack all of this for us. They are three experts in human performance who are also my colleagues at Aptima. Dr. Shawn Weil is Principal Scientist and Executive Vice President of Business Strategy at Aptima. His expertise are in research areas that include social media, military command and control, advanced training, and communications analysis for organizational improvement. Courtney Dean is a Senior Scientist and Director of Products at Aptima. Courtney specializes in developing and deploying training and performance assessment solutions for individuals and teams in both live and simulated training environments. And thirdly, Evan Oster, is a Scientist and Learning Solutions Architect at Aptima, whose current work centers on infusing cognitive science into performance assessment, analytics, and artificial intelligence.
So our three experts are not just academically fit in terms of having been trained in cognitive science or organizational psychology but they all have deep experience into implementing those principles in the field, which is really what is the topic of our discussion today. It’s blending that wisdom that comes from the field, together with the science that comes from the lab.
Welcome, Shawn, Courtney, and Evan. And I would start first by asking my guests to introduce themselves one at a time and ask them specifically, why did they choose this particular domain of all the domain of interest they we would have chosen, the domain of human performance in different flavors, by the way? Shawn, good morning.
Shawn Weil: Good morning, Daniel. Thank you so much for having me. You know, I t's such an interesting question. I started my career thinking I was going to be a college professor, a psychology professor. I've always been really interested in human performance at an individual level, but I got a little bit tired of doing laboratory experimentation while I was in grad school. And I was adopted really by folks who were doing much more applied research in human performance and the study of human behavior in real environments. I credit David Woods at the Ohio State University for finding me roaming around campus and bringing me in. And it's been a pleasure over the past almost 20 years now to look at human performance in different domains in a very naturalistic understanding of human interaction and performance.
Daniel Serfaty: Thank you. I understand that you were at the epicenter of a whole movement, cognitive systems engineering, which Professor Woods was leading. And you're going to tell us more about that later.
Shawn Weil: Absolutely.
Daniel Serfaty: Courtney?
Courtney Dean: So I stumbled into human performance, rather circuitous route that was very much self-centered. I played golf in college and was looking for opportunities to improve my game and came across some sports psychology material. And this included a course I was able to take over one summer and a series of tapes by Dr. David Cook that told some really great stories to some PGA professionals. And I listened to those tapes, and this was audio cassettes enough times that I think that they started to unspool themselves. I wanted to enter into that field, but some advisors and professors at my university guided me away from that much like what Shawn just talked about because they were pointing me towards the option of either being a consultant or being a professor. And at the time, I wasn't particularly interested in going into teaching.
And this consulting notion coincided with me just hacking to take an industrial psychology course. And I said, "Well, there's something that's much more applied." And I got into an applied graduate program that focused an awful lot on that. That led me to a public safety selection environment, and I found that to be very interesting. And then the opportunity with Aptima came along that was essentially doing the same thing, human performance, but with the DOD. And having always thought that I was going to be a fighter pilot like my dad, I thought, "Well, heck, that's a great way to get close to that environment without actually having to put my life on the line."
Daniel Serfaty: That's great. I didn't know if you were those things you just told the audience. But that straddling between domains such as sports, aviation, military and public safety is very interesting because this is also our theme today. We're going to explore basically how the power of the scientific approach and understanding what to do with the data we collect for the betterment of human performance crosses actually domains. But then Evan, how did you get to this business?
Evan Oster: So I actually started off in K-12 as a teacher, and I found that to be a very rewarding thing. But a couple of years in, I found that things were becoming the same. You may do the same tasks and jobs, and it's something that over time I wanted to change. And so I was looking for a different challenge. I love the idea of instruction and training, and I've always had a profound respect for the military. So I thought, what's a way that I could still continue doing what I love but also pull in this aspect of the military? We have so much untapped potential as humans. And being able then to jaw our focus and attention to something, to be able to accomplish something new that we previously weren't able to do, I find to be really rewarding. So being able to take my background and instruction and training and being able to bring that into the military I think is a great blend of these two things that I love.
Daniel Serfaty: Okay. So we have a lab scientist, a golfer, and a teacher. Sounds like the beginning of a wonderful joke, but it is actually not just a joke. He's actually the secret ingredient of this very challenging but extremely rewarding domain of studying human performance. We need that multidisciplinary curiosity and that combination of methods to crack the nut of the difficult problem that capturing human performance represent. So perhaps in one or two sentences each one of you can take our audience, that is, what do you do? Now we understand your background, but what do you do at Aptima? You can just pick one project or one activity that really you think represents what you do. And I'm going to scramble a little bit the order and start with Courtney. What do you do, Courtney?
Courtney Dean: So at Aptima, my primary focus is the most directly human performance, you could just about put that as line one on my resume. I've been focused on developing measures of human performance and training contexts for a variety of domains almost since the first day that I stepped through the doors of the company. This involves sitting down with subject-matter experts in their respective domains and identifying what it is that differentiates somebody who's competent from somebody who's incompetent or somebody who's excelling in their field versus someone who's not. Breaking that down to the micro level, what are the specific behaviors that we can observe that indicate that somebody possesses the knowledge, skills, and abilities that are necessary to complete said tasks?
And we've developed a pretty effective methodology for eliciting that information. And I've just run with that methodology to apply it to many different domains and utilize that to both gain an understanding about the domain that we're focused on there, and then produce a series of metrics that those individuals can then take with them into their training environment and utilize to achieve some goodness on the part of the trainees or the learners in their environment.
Daniel Serfaty: So in a sense, you have a scalpel like a surgeon, which is a method you mentioned. And you are trying to understand, deconstruct the nature of mastery or expertise in the domain that you study, whether they are fighter pilots or police officers or surgeons actually. Is that what you're saying, you basically are an analyst that decompose that and say, "These are the ingredients of mastery"?
Courtney Dean: Yeah. I would say that it's a little bit less elegant than a scalpel. The truth is that it's a little bit more along the lines of a sledgehammer and some super glue.
Daniel Serfaty: Right. Well, we'll talk about those tools in a minute. So what is it that you do at Aptima, Evan?
Evan Oster: That's a great question because when friends asked me that, I have a different answer each time, and it really depends on the day. When I am looking at human performance at a high level, that's me reviewing and conducting research on training to improve human performance. But more specifically, what that looks like is working with colleagues who are experts in their field in small teams to be able to innovate some solution that satisfies a customer's need. So that can be improving decision-making, it can be improving the instructors, it can be helping to improve the students. And I think that's what's really unique is you can take a look at human performance challenges from multiple different perspectives and multiple different ways. And each time you improve something, it improves the whole.
Daniel Serfaty: Okay. So you're not an engineer, you're not a software engineer. Many of the solutions you dream up or Courtney dreams up end up in instantiation software. How do you establish a dialogue with folks that are actually about coding and architecting software systems?
Evan Oster: So that's also a really good question, something that we face every day. And I think what it comes back to is really in understanding and good communication with one another. Relationships are huge, right? So understanding how different people work, how they view problems, how they see things and valuing those differences. And being able to clear out space for them to work as they can work best. At the same time, having a common framework and lexicon for what it is that need to be accomplished. How many times have you heard someone say the word domain? And that means one thing to one person, something to someone else, something to someone else. And so being able to have that common language and framework to operate from really helps to inform that end goal and form the solution.
Daniel Serfaty: Well, we'll come back again to this dialogue. But Shawn, I know what you do. You're the executive vice president for strategy at Aptima, but that's not the only thing you do. It sounds pretty managerial and executive, but actually you're still scientists. So how do you bring that science into your job?
Shawn Weil: It's a really good point, Daniel. I think about this in a number of ways. I wear different hats in the company, and I wear different hats professionally. Because I have a corporate role, it allows me to think about human performance in a systemic way using systems thinking. So construed broadly, that could be looking at human performance of teams and how they communicate or how they interact with artificial intelligence. It could be looking at how we bring together measurement from different modalities, observer-based or systems-based. Or it could be understanding the link between the physiological, the behavioral, and the cognitive, and trying to make sense of that.
But the other hat that I wear, that executive hat is the one where I'm helping our engineers and scientists, both Aptima staff and our partners really understand what the end users' needs are. There's something intrinsically wonderful about human performance that satisfies the intellectual curiosity of scientists and engineers. But then you need to figure out how to frame that in a way that's going to be really beneficial societally, and that requires a different perspective. So it's a pleasure of mine to be able to take on that role, put that hat on and work with our diverse staff to help them help those customers.
Daniel Serfaty: That's a very good way to describe this constant stitching of ideas into something that not just the market, it's too abstract, but the human user, the human learner out there needs. So I'm going to ask you Shawn, think of an instance of an aha moment in your professional life when you suddenly realized something new or how to articulate insight into a scientific fact, a project, something, an aha moment?
Shawn Weil: I've had a couple of those aha moments. But there's a common thread in all of them, and that is messiness and complexity. So when I was first exposed to ideas of human cognition and human performance, human behavior, they were very neatly compartmentalized. You had these kinds of behaviors in these situations, this decision-making in these kinds of environments. This kind of communication patterns with these kinds of people, and you could study them independently. When I was in graduate school, I first got exposed to the complexity of just having two people talk to each other and two people trying to coordinate towards a common goal.
So I think an aha moment came early on in my career at Aptima when I was working on a program about command control, and we were doing some experimentation. And when you start looking at not two people, but just five people in a controlled environment, the amount of complexity, the multiplicity of ways that you can look at human performance, it's just staggering. So how do you then do something useful, collect something useful? It may not be collecting everything. So the aha moment for me in that scenario was really saying to myself, what questions do we need to answer? And from all of the chaos and all of the complexity, do we know how to zero in on that subset that's really going to give us some insight that's going to help improve the performance of that organization?
Daniel Serfaty: That's interesting because that's a dimension that all of us had to face quite early in our carrier, you're right. This notion of not everything is neatly compartmentalized the way it was in the lab in graduate school. And dealing with these, you call it chaos or complexity or messiness is really a skill that eventually we all need to acquire. Courtney and Evan, any aha moment you want to share with the audience?
Evan Oster: I had one for sure pretty early on. So we were at an army installation, and we were there to help instructors provide tailor feedback to their students. And that was going to be through an adaptive instructional decision tool. And I was conducting a focus group. And I asked the instructors what made their job challenging? And they started off by complaining about the students. They started to tell all these stories about how lazy they were, and they constantly make these common mistakes. And I looked up one of the main instructors there and asked, so why do you think they're doing this? And he paused and thought about it, and he said, "Well, it's because the job is really hard." And he gave the example of, if you stacked all of their manuals on top of one another, they'd be over six feet tall," taller than most of the trainees who were there.
And then soon some of the other instructors started talking about other things that were challenging and why they're hard. And really just from one question, the conversation and the culture shifted from complaining to more of a sense of compassion where then they're starting to put themselves in the student's shoes and they're able to resonate with why it's so difficult. And as the instructor, I mean, this is easy for them, they'd have lots of practice and they've done it for years. But really what we were able to do is before we even did any software, any development, we were able to set a culture and a framework for why are the instructors there? They're really there to help.
And from that moment forward, it helps shape what that final solution would look like. That was a big aha moment for me that this human performance, similar to what Shawn said, it's messy. You look at it from so many different perspectives, and it's not clean-cut and clear. You need to go in and feel your way around and figure out what's going on and see what the best path is moving forward.
Daniel Serfaty: Courtney, you got one of those ahas?
Courtney Dean: I have one, but I actually had to relearn. I remember watching a retired army operator taking me and bring the soldier down with him. And they had this conversation, it was very quiet and very reserved, and it was technical. He was pantomiming and gesturing with his hands to mimic the use of a rifle. And there was probably a little bit of engineering and physics associated with the drop of the bullet or the rise of the muzzle, et cetera. I don't know, I wasn't totally privy to the words that were coming out of her mouth. But I watched the body language of this settled, isolated conversation. And it was before I had been on a firing range with drill instructors and new privates or private recruits where the screaming was only overshadowed by the sound of muzzle fire. Then we started to see drill instructors taking knees next to young privates who were practically quaking in their boots if they don't hit the target just right.
And you started to see some change in behavior because they went from, "I'm trying to avoid this screaming, this punishment," to, "I'm starting get some context and some understanding and some support. And I understand that maybe my inability to hit the target as accurately as I'm supposed to right now is not a personal flaw that I'm never going to be able to get over. And then I constantly forgot that. And one day I had a friend in town and my new walking one and a half year old son, and we were going sledding. And he didn't have very good mittens on or children can't keep their mittens on and snow was creeping in between his hands and his sleeves. And I wanted to put hi, on this sled because it's going to be so exciting and he didn't want to. And he started to get scared and started to cry.
And I found myself trying to force this child onto the sled. You just get on the sled, and everything is going to be wonderful. And then I stopped for a second because I was realizing that wasn't happening. But in my head, the gears were just barely clicking. What is going on here? And my friend who hadn't had a child yet took a knee and talked to my son and calmed my son down, and he got the situation back under control. And I looked at that and I thought, "Well, I guess I'm one of those old drill instructors right now, and it's time to become one of those new drill instructors."
Daniel Serfaty: These are wonderful stories, all the three stories that you're telling me. Which really remind all of us that human learning, which is the other side of human performance and the ability to improve a skill or to acquire a skill is more extremely complex and messy. But at the same time, there is an intimacy to it between the instructor and the learner, between the tool and the person reading the data that we as engineer, as designer, as scientists have to take into account. Thank you for using those stories, they illustrate very well better than any theory our job is both so hard but also so rewarding. So Shawn, looking back, why do you think it's important to be able to measure? We go back to that notion of capturing something, of measuring, whether with methods or tools or just intuition sometimes to measure humans in order to improve their performance in doing their job. Why is measurement important?
Shawn Weil: I've thought about that a lot. Courtney was just talking about his children, and I have two children who are school age. And I think about how they're measured and where measurement is worthwhile and where measurement actually might be a distraction. The reason why you need to measure is because humans have bias. Humans as they are going about their work and trying to accomplish their goals have only so much vision, they have only so much aperture. Especially in training situations, traditionally in the military at least, what they've done is they don't have professional instructors, per se, they have operators who are expert in those domains. And they watch performance, and they cue into the most salient problems or the most salient successes. And they build their training around those narrow bands, that narrow view effects.
So when you do a more comprehensive performance measurement scheme, when you're measuring things from different perspectives and different angles, especially when you're measuring aligned to objectives of what the group is trying to accomplish, what you're doing is you're enabling instructors to overcome their biases, to think more holistically about what it is they're trying to give to their trainees, to their students. And to do it in a way that is more empirically grounded and more grounded in the action they're trying to perfect or improve.
Daniel Serfaty: So Evan, you're listening to what Shawn is saying now. Can you think of instances when you've seen that actually a particular measurement changed the way people talk or people educated? How can we link right now for our audience our ability to measure in sometime precise detail certain aspect of cognition or behavior or decision-making and eventually turn that into an opportunity to train and therefore to improve learning and eventually to improve people performance on their job?
Evan Oster: Yeah. So on measuring human performance, one of the things that I think is critical is being able to challenge the bias, as Shawn was talking through. The expectations that might be placed on students maybe by one instructor and another, you're lacking consistency. And there are nuances to the training that we're trying to get a more objective measure of. So when we're looking at how to measure human performance, being able to get concrete, specific, and objectives is really critical in getting the training to be well aligned. One thing that I've seen through some of our efforts is we've had a particular training that was being trained in groups and lacked the ability for instructors to know if a mistake was made, was that by one student over another? Was it due to the team dynamics that were there, was it due to a lack of knowledge? And when they were training in that group environment, they weren't measuring in a level where they could distinguish between those nuances and those differences.
When we in and started measuring at a more granular level, we were able to then help them disentangle what was happening, when it was happening, and with whom it was happening. And that way, then the instructors were able to tailor what they were doing to specific students at a specific point, in a specific way.
Daniel Serfaty: So you're in fact making the point that this so-called objectivity, and that's a topic probably for another podcast about why the objectivity of measurement is not here to tell the teacher or the instructor that they are wrong, but more to augment what they do naturally as teachers by giving them basically a rationale for intervention, for particular instruction, for focusing on a particular competency of the student. One of my early mentors was General Jack Cushman who passed a couple of years ago. He has an old-fashioned, crusty, three-star general in the army who after he retired was actually training other generals to conduct complex operations and giving them feedback.
And he was always exasperated when people say, "Oh, we did better this time." And he was always asking them, how do you know? How do you know you did better other than just a good feeling? And he came up with this quote that I really respect a lot, I use it a lot, which means you can't improve what you don't measure. How do you know? It's such a simple statement, you don't improve what you can't measure. But very profound in the way it's changing not only our military training but also education at all levels.
Courtney Dean: I stand on that a little bit, Daniel.
Daniel Serfaty: Of course, Courtney, yes.
Courtney Dean: So three thoughts that I don't have written down, so I'll try to channel them together coherently. So number one is I know what right looks like, or alternatively stated, I'll know it when I see it. Practice does not guarantee improvement. That was on a slide that we had for years. And then finally, feedback. So those things linked to each other inextricably. The issue that we had, the bias that Shawn was talking about that you can't improve what you can't measure is all about that. I know what right looks like or I'll know it when I see it. If we don't have something, then a subject-matter expert is resolved to that bias or we don't have consistency from one biased subject-matter expert to another.
And if you don't have any measurement, then that practice can be ... And trust me, I know this one because I have years of experience in this one. Practicing the wrong things, you don't miraculously change those things. There's that critical element that's missing, and that's that third bit, which is feedback. By delivering feedback, we have the potential for a subsequent change. And that's what training is all about.
Daniel Serfaty: We'll be back in just a moment, stick around.
Hello, MINDWORKS listeners. This is Daniel Serfaty. Do you love MINDWORKS , but don't have time to listen to an entire episode? Then we have a solution for you. MINDWORKS Minis, curated segments from the MINDWORKS , but gas condensed to under 15 minutes each and designed to work with your busy schedule. You'll find the minis along with full length episodes, under MINDWORKS on Apple, Spotify, BuzzSprout or wherever you get your podcasts.
Courtney, I'd like you and Shawn to expand basically on that idea, but focusing it more on a particular toolkit that was developed at Aptima early on we call Spotlight, which is a kit that includes basically a lot of the science by articulating, what should we measure, what can be measured, and what is measured. And how we went from basically an academy concept of scales and measurement into a practical tool. I would love for you to tell me a little bit about that. What is unique about it, and what did we learn in applying it to different setting environments? So Shawn, and then Courtney.
Shawn Weil: Yeah, I love the Spotlight story. Unfortunately, I wasn't there at its inception. I suspect if you listened to some of the previous podcasts and listen to Jean McMillan, she'll tell you the story of resilience that was the origin of the Spotlight application, which at its core seems like a pretty straightforward concept. But in practice, it's a lot more sophisticated, especially in comparison to what people tend to do. So Spotlight at its core is an electronic observer measurement tool. It's a way to provide to observer-instructors the means for comprehensive assessment of live activities. So think about it this way. The way it's been done for years is you have people in a field environment, maybe they're doing some maneuver on an actual field, maybe they're doing some pilot training in simulation environments. And you have experts who are watching them, and they're writing their comments back of the envelope.
Well, back of the envelope only gets you so far. Those biases that we've talked about, the inter-rater differences that creep in, they make it so there's very limited consistency. So enter Spotlight, essentially what we've done is put together a set of measures that are comprehensive and aligned to the activities that the trainees are actually going through, then implemented that in an electronic form factor that then affords a bunch of other things. It affords that feedback that Courtney was describing. It allows for aggregation of data. The measures themselves are designed to encourage what we call inter-rater reliability, essentially consistency from one rater, one expert to another. And we've seen that really transform the way that training is done in a number of environments that Courtney and Evan have really been in charge of and really pushed forward over the years.
Daniel Serfaty: Well, thank you for giving us a little bit of history. Indeed, Dr. McMillan was our previous chief scientist, was actually at the origin of developing that tool, I believe originally for the Air Force. But Courtney, you are actually in one of your many roles the manager of these product line called Spotlight, and you've seen dozens of instantiation for Spotlight. You also mentioned the F word here, feedback. Tell me stories about when you used or recently or previously with the way you've used Spotlight? Which is after all a tablet for our audience that prompts the trainer or the observers to grade the learner or the team of learners according to certain scale that have been established as being essential to their expertise through their mastery. So tell us some Spotlight stories, especially when it comes to how people use it to provide feedback?
Courtney Dean: I've gotten my shoes dirty on a couple of occasions. I've been in the woods of Georgia, I've sat on flight lines, I've hung out and fields next to, I guess, improvised villages or foreign operations villages. And I've been in briefings at two o'clock in the morning that extended until three o'clock in the morning after all of those outdoor activities occurred. And in all of those occasions when instructors had Spotlight with them, their ability to communicate to the learner the delta between what is observed and what is expected. And then to elaborate on that with a picture of how to close that delta is far and beyond what I've ever seen when I watched the same activities go down with an instructor with an envelope and a pencil.
Spotlight is these two core components that Shawn talked about, and I'm not going to try to re-describe it because Shawn did an excellent job. You got the measures, and you've got the application that delivers those measures. And when the measures are in the hands of a competent instructor, they're able to make total sense of the student doing the job that they're supposed to be doing. Why was his arm at the wrong angle? Why did the bullet go offline? Why did the tank not make it to the way point at the right time? Whatever the context is, they're able to thread together the story that led to the undesirable outcome. And they can pick spots within that timeline, and they can communicate with that student, "Here's where things deviated slightly, it led to these consequences. Here's where things deviated slightly again as a result of those consequences."
Suddenly the student goes from, "I failed, and I don't know why," to, "I failed, and it's because I need this fundamental error here," or, "I received this incorrect information here, and I operated on the wrong frame of reference." Those pieces of information are critical for that subsequent change in behavior that I think I've repeated two and three times now. Ultimately, the student is now empowered to become better in the long run.
Daniel Serfaty: Thank you for that vivid description, Courtney. Evan, I think in a sense we see here that what science and the judicious use of data enable us to do is not just to provide the movie of what happened, which would have been the description of the scenario, of the vignette, of the way the student went through a chapter of instruction. But also some x-ray or a hyper vision, if you wish, of that movie that enable the D word, which is in this case is a diagnostic that Courtney is talking about. And perhaps that's what science gives us, the ability to see inside and then eventually say, "Yes, you didn't do that as well as you could, but here's the reason why," and for a student being able to close that gap. Can you think of in your own development and use of Spotlight in environments that sometime are different from the environments that Courtney described? Can you give us some example of how that was used that way, provide that secret insight into human behavior?
Evan Oster: Yeah, there's a couple instances that I can think of. But one in particular is when it comes to receiving that feedback. So it depends on who your trainee or your student is. And there are times where, like Courtney outlined, you have an instructor doing this back of the envelope notes, they provide the feedback. And that leaves the trainee with an option, they can accept it or they can reject it. And oftentimes when you don't have that environment in that context and the angles, a student is more prone to reject the feedback or to make an excuse for it or whatever it might be. But when using Spotlight, I've seen a number of times where that might be the first response.
And then when the student gets the feedback and the instructor shows them where they might've gone wrong here or there, they are able then to accept it and see, oh, my arm was here, or I did do this. And it's that point in that context that's concrete and objective. And then they're able then to accept the feedback and then use the data and the additional context that the instructor can provide, to use that to make a better decision next time.
Daniel Serfaty: Evan, you've seen that, what you described right now. I would like our audience to visualize, where are you seeing that, in what context? In what domain, to use a word that you like?
Evan Oster: One context is when using that in law enforcement and has been doing building search. So there are very specific ways that you can systematically progress through a building and search, and you need to do that methodically. And in order to conduct that in the right way, there's gotta be a lot of non-verbal communication, there's gotta be a lot of decision points as a team and as an individual. And you have to be constantly aware of your surroundings. A particular example would be even doing what's called lasering your partner, where if the path of essentially what your gun is pointing at were to cross over your partner, then that's putting them at risk. It's a safety concern. An instructor might say, "Hey, I saw you laser your partner." And they could say, "No, I didn't." But when using Spotlight and having that video of the feedback, they can show a distinct point when that happened. And then at that point, they can adjust how they are holding their gun or how they move through a space.
Daniel Serfaty: Okay. That's a good prompt towards the next segment where I would like to explore how we took a lot of wisdom that we learned from the particular domain, whether it's fast jet flying or a military operation on the ground to another domain when other skills are more important. But Shawn, why don't you add a little bit texture to what we just heard from Courtney and Evan before we move to that next step?
Shawn Weil: Absolutely. One of the things that I heard Evan say that really resonated with me is this false dichotomy, false separation between subjective and objective measurement, and the tendency for people to de-value subjective measurement even if it comes from experts. I'll explain it this way. So in some of the military domains where we work, let's say you have people in an aviation trainer, you're in this multimillion dollar F-16 simulator. And you're in the computer, you're doing your work in the simulated environment. So the computer could collect all of this information about your position relative to the other planes and where you're dropping your armaments and all of these things. And people value that quantitative information, maybe they over rely on it in some sets. Because if what you're actually trying to teach in those environments have to do with teamwork or communication or some other behavior that requires interaction, then the way the computer is doing the measurement isn't the right way to collect that data.
So what's happened in the past is you have those back of the envelope guys providing some feedback, which often gets devalued because it doesn't have that quantitative shell around it. But what Evan was just describing in law enforcement and what Courtney was describing in the use of structured measurement is now we put some quantitative armor around subjective opinion. It's not subjective anymore because we've given a lot of meat to the ratings that people get. And we could correlate those ratings with video evidence of what they're doing. So now the subjective becomes really, really powerful. That's what Spotlight does, it gives you that powerful view that you couldn't get otherwise.
Courtney Dean: I would like to say for the record, it's only engineers that think that the only way that you can do measurements is objective.
Daniel Serfaty: Let's not get into the scientific versus engineer dilemma. But rather I think that, Shawn, thank you for clarifying the difference between subjectivity, objectivity as opposed to qualitative, quantitative. And in a sense you can be very subjective, which is not a bad word because sometimes subjectivity include the very expertise of the teacher, which is born out of decades of experience sometime. And so, yes, it has biases, but it has also some profound wisdom with it. Adding a layer of quantitative data to it certainly strengthen it, do not diminish it. And so we have to make the difference between subjective, objective versus qualitative, quantitative.
As we move towards those complex environments, and I get the feeling from a lot of your explanation, from a lot of Courtney's description that basically those domains are complex, they are not trivial, they are not mild. In a sense, they all have some time pressure involved in them and high stakes and different sources of uncertainty and multiple objectives and complexity that makes basically the job of acquiring expertise in those domains difficult. And certainly the job of the instructor about developing that expertise in others even more difficult.
Having talked about all these levels of complexity in the domain that we are applying our human measurement and training science and technology, I was reminded of early in my career an episode where we were tasked to take our tools, our team training tools, our decision-making training tool that we have developed for warfighting, for improving the war-fighting decision-making of our armed forces and apply them in a mission that was quite different. And I'm talking about Bosnia about 20 plus years ago where our war fighters were super expert in their domain, suddenly had to change. But not just suddenly, but on a daily basis, they had to migrate between their war fighting mode with very nefarious people wanting to do them harm and wanting to do the population harm to becoming peacekeepers, and almost at some level social workers.
And that migration between those skills, very complex skills of war fighting to the very complex skills of peacekeeping was not trivial at all. We were very challenged. We're dealing with very smart soldiers, very smart war fighters. But it was terribly difficult for them to be able to maintain those two minds at the same time. It was not on the very same day two very different missions. One dealing with war fighting, another one dealing with peacekeeping. And they had to switch the way they were making decisions, they have to switch the way they were assessing the situation. They had to switch the way they were evaluating danger. What you are doing right now is taking all these tools and these smarts and these theories of human performance and these sophisticated measurement tools and bring them into a different domain, which is the law enforcement domain. Shawn, I will ask you first, how did we decide to make that jump? Was there a particular project that led you to lead this initiative to switch from one domain to another?
Shawn Weil: Yeah. I think you point out some critical issues in the way that people have to be dual hatted when you're in harms way. The reason we pivoted in some sense to law enforcement had to do with the program that DARPA was running about 10 years ago called the Strategic Social Interaction Modules program. And I won't get into the details of the program itself. But fundamentally what it was looking at was exactly as you described, are there ways to help Marines and soldiers who have joined their military services because they want to protect the country? They've now been put into roles of civil government in some sense, and it's a very hard transition to make. So one of the things that we looked at was, well, is there a model of competency for people who are put in dangerous situations but have to manage them the way you would in a civil organization?
So that's when we started looking at law enforcement as that competency model, if you will. Because police officers in the best of circumstances go from situation to situation, to situation that might be completely different. You don't know what's on the other side of the door when you knock on the door when there's been a call for disturbance. It might be somebody who is sick or somebody with mental illness or somebody who is violent, or somebody who has mental illness, or somebody who doesn't speak English. You don't know what's behind door number one, door number two, and door number three. And we found that the very methods we've been talking about in Spotlight, in measure development could be used to measure the capability of those peace officers for both the tactical portions of their jobs and the software skills that they need to do for their jobs to manage difficult situations, dangerous situations.
And it's had this nice feedback effect because some of those same measures could then be used in a number of military contacts where there are analogous activities that need to be accomplished to achieve the goals of the mission.
Daniel Serfaty: Thank you. Courtney and Evan, I'm going to ask you to give me some examples based on your own experience. And I realize that the depth of your experience is primarily in armed forces situations or military situations. But experience in the law enforcement domain in which you had not just to use similar tool and then customize you tools, but also detected that there are some things that actually are different in terms of the skills that we need to coach these folks with. Courtney, you want to jump in and then Evan?
Courtney Dean: I want to use what Shawn just talked about and characterize it as ambiguity. We had a situation with a Massachusetts police department that was very interested in getting some support in some of their training. And they had a semi-annual field exercise that was a couple of different traffic scenarios. Each one of those traffic scenarios was designed to present an ambiguous situation to the police officer. So going back to what Shawn's talking about, you don't know what's behind door number one. You could expand on that for hours and talk about all the ambiguous situations that a police officer faces. And it's not about knowing how to put somebody in restraints and put them in the car and then fill out some paperwork. It's about dealing with citizens on an everyday basis, about being an advocate, about being a supporter, about differentiating who the bad guy from the good guy is, or understanding how to keep the bad guy from becoming a bad guy or worst guy.
So this ambiguity theme existed in all of these different scenarios. And the one that sticks out in my mind the most is the officer pulls up to two vehicles, and both of the drivers, occupants of the vehicles are standing out in the street. And they're agitated, they're screaming at each other. Unknown statements are coming out because the verbal diarrhea is just anger and frustration. So what's the police officer's first move here? Oh, I'm going to put both of these guys on the ground. That's not going to get us somewhere. I'm going to figure out what's going on, diffuse the situation and try to bring these two out of their hostility environment in back into society, and then decide from there whether something needs to be done further from the law enforcement perspective or send them on their way.
And that is a snap decision that the officer needs to make because he can't sit in his car and think about it for a little while. These two could go to blows any second now. Got to get out of that vehicle, got to move on that situation in the most effective, and let's call it gentlest way that they know how so that they don't add fuel to an already sparkling fire.
Daniel Serfaty: That's a very good example of this notion that we used to call, at least in the military version of that, on the peacekeeping side of things, teaching tactical patience, that tactical patience is that tactical passivity. It mean that patience in taking step to gain control on the situation and diffuse it as opposed to being very quick, say on the proverbial trigger and moving in and trying to stop the situation without understanding first the nature of the situation, which is really something for survival and for mission effectiveness. A lot of the war fighters, the military tradition has been training people to act very fast.
Courtney Dean: And we use Spotlight in that particular case, and the measures that we included in that were a combination of technical measures as well as measures that dealt with this ambiguity. So we might've had a measure that pertained to the physics of parking the vehicle in the correct way, and that's important. But the essence of this scenario is, is the officer understanding the situation, maintaining order without exacerbating the situation? And those are the measures that are the most important to capture because those are the ones that help the instructor evaluate or deliver that proper feedback. And those were the measures that were the most contentious afterwards for the officers, well, now hold on a second here, what did I do wrong? Or I did it this way. And as Evan had talked about before the video showed, well, here's where you demonstrated some bias towards this guy versus this guy, which agitated this guy, et cetera, et cetera. And those measures really do have a powerful impact because they can support that understanding and that feedback.
Daniel Serfaty: So obviously Evan, our audience is going to be very sensitized to the current events and the different accusation both for and against the police behavior. But sometime no matter on which side of the political spectrum you stand, you have to realize that law enforcement officers are humans too, and they have their own cognitive and behavioral challenges facing ambiguity like the rest of us. The question is, can we through science and technology train folks to actually make the best out of those situations to know that there are alternative ways to control the situation without being very fast on the trigger? Evan, tell us a little bit about your own experience. You've worked quite a bit with another region, not the Massachusets region like Courtney was describing, but central Florida.
Evan Oster: Yes. That makes me think about how the training of how to respond in these situations. We can look at training, the types of behaviors we want to see. And it isn't always a train at once and you know what to do, right? But it's more of a constant and persistent thing. So when it comes to anyone doing their annual refresher training, that's helpful, but it's likely not going to have long-term effects and impacts. And what we can do using science and technology is be able to take performance that is being captured, measure and assess that and have that be something that's on a more frequent basis, which over time can help to correct certain behaviors, adjust maybe core decision-making skills, present other options or ways of handling situations.
And like we've been talking about, the other side of the training coin is learning. And so being able to correct where each person is maybe weak in areas and be able to tailor that feedback and over time provide the opportunity for that training to take different forms. And it's something that's not just a one and done or once a year, but being able to gather data and provide that feedback in meaningful ways.
Daniel Serfaty: Shawn, you want to add to that?
Shawn Weil: Yes. When Courtney and Evan were talking about ambiguous situations, it reminded me of some work we were doing in the development of Spotlight measures at a police academy on the West Coast. And this was the capstone activity for these police cadets. They went from station to station experiencing something, and they were being measured on how they would perform. Some of the things they were being trained on were very tactical. But one of them I think really exemplified this ambiguity. This is what happened. The police officer would roll up in his cruiser to an apartment building to find that there was a man on the third story holding a child and threatening to drop the child from the third story of this building. And they had to decide how to handle this.
They're too far away to be physically there. If they tried to approach them physically, the man might drop the child. The child, by the way, was being played by a doll in this situation, so there was nobody who was actually in danger. But in spite of that, the adrenaline, the anguish, the ambiguity of the situation trying to deal with use of force, and should we shoot the sky, and what should we do was almost overwhelming for some of these cadets. I saw people burst into tears on the implications of whatever their actions might be. In this particular case, what was happening was there was one moment in the stage scenario where they put the child down and they come to the edge.
And at that point, use of force is warranted, it's legal by the rules in that state. So as we were thinking about the measures that we could develop to try to capture their performance, there was a set, as Courtney was saying of the mechanics of police work. And then there was another set that had to do very explicitly about the decision-making process. Sometimes there isn't a perfect solution, but learning how to make decisions in critical situations is part and partial of police work in those most critical times. The development of measures for that purpose will go a long way in ensuring that you're clear-headed and you have a way to break situations down and make decisions that are going to save lives.
Daniel Serfaty: Thank you Shawn for this very vivid description. Indeed, I think that the audience can imagine themselves, what would they do in those cases? And it is not always about what we hear in the popular press about systemic bias or other things like that that may exist. But there are some fundamental human behaviors here and fears and dealing with ambiguity that are at work here in which I think we as human performance experts could help quite a bit. And Courtney, this one is for you because you've seen probably more instantiation of the application of tools like Spotlight, but not just Spotlight in different domain. If I ask you to envision the success for Spotlight and associated digital tools in the particular domain that is so much right now on the mind of the United States' public, the law enforcement domain and vision success of the use of the systemic measurement, the use of science and technology in one year, in three years. Tell me a little bit how that would work?
Courtney Dean: So my associates at my former line of work would take posit what I say here just a little bit, and I'll clarify it. The job of a police officer can be team posed into what we call in my industry, job analysis. And for the most part, that job analysis can relate to different departments, different places. There was a highly litigious environment where I was at before, which is why those folks would take posit to that. But you can essentially break the job down to these tasks and these knowledge, skills, and abilities. What we could see is a library of measures that pertain to those tasks and knowledge, skills, and abilities. And in particular, it's a usable library that focuses on some of those less kinetic, less technical skills and more of those soft skills that are so essential for avoiding getting into a kinetic situation. Those decision-making skills, the ability demystify the situation, the ability to deescalate.
And there are a variety of things that can be put into place that will measure an officer's willingness and then ability to do that, and then the effectiveness of their effort. Those measures could be applied to many, many different situations and scenarios with only maybe a little bit of tweaking here and there. So in the future, we can have Spotlight with that library of measures and a couple of configurable features or functions of Spotlight. And what I mean by that is that sometimes we apply our measures in a video tagging context. So we tag the video that we're seeing with those behaviors and those ratings.
Sometimes we use a typical survey type of method, sometimes we have a combination of the two. But it ultimately comes down to what is the best method for capturing data that is going to benefit the learner? We have this library of these measures that are focused on these things that are so critical for an officer to achieve effectiveness. And we use it in environments like that like the one that Shawn just described. And we lead these officers towards the desired end state through repetition, feedback, and a deeper understanding of their role within the potentially deadly situation.
Daniel Serfaty: Thank you Courtney for, I would say optimistic vision because that's our role is to look to the degree to which those methods, especially if they are really future oriented, they lean forward, the degree to which those tools and the scientific method can help. I know that one of the more pioneering thoughts in this domain was actually promoted by the Air Force. Dr. Wink Bennett or the Air Force Research Lab has promoted the notion of mission essential competencies, tor example, that is a catalog almost exactly the way you describe Courtney. A catalog of those essential skill that if you could progress on those skills, you would progress on the quality of your mission accomplishment.
And if we could do that through a fresh look into what's considered police work in its different instantiation including the ambiguity of the situation, I think we may have a chance. Shawn, what do you think? I'm asking you the same question. Given the current situation with the police department in this country, what are your hopes and fears about your ability and that of your team to use science and technology to help? And Evan, I'd like you to chime in too about it.
Shawn Weil: I'm really excited about this, Daniel. In spite of the truly tragic circumstances that we've seen across the country, there are some positive trends that I believe are going to revolutionize performance, not just in law enforcement, but more generally. So there are several of these. Number one, there's the ubiquity of measurement. I think back to the start of Spotlight and the idea of doing performance assessment still had some novelty. Now, we all walk around with cell phones in our pockets that can measure us in a half a dozen ways, where we are, who we're talking to, the accelerometers in these devices. So if you start to extrapolate to law enforcement where you're wearing body cams and you've got dash cams and you might have some wearable physiological sensors, that you might be able to use artificial intelligence to do some of the assessment that's currently done by expert observers at a much larger scale.
And use machine learning then to start to develop a more comprehensive understanding of what right looks like even in low frequency situations. If we can, as Evan was describing, change this from a one and done situation to a continuous performance assessment and feedback environment, the sky's the limit to the ways in which these professions can improve over time. So in 2030, we might be talking about a situation where the performance of our law enforcement officers is continuously refined aligned with societal expectations for public safety.
Daniel Serfaty: Wow, that's a very ambitious vision, and I hope it is realized, Shawn. Thank you. Evan, you are back to your roots as a teacher with trying to impart some skills and knowledge to your students. Can you also describe for me vision over the next 10 years or so of how these tools, not just the scientific tool, but also the technology tool, the ability to capture artificial intelligence, the ability to look through large amount of data will enable us to transform learning and teaching and training in the law enforcement domain?
Evan Oster: Certainly. So I always like to start with the end in mind, to what end and why would we do this? And I think if we're looking at law enforcement operating as public servants and wanting to improve the safety of our communities and of our country, one of the main driving factors there is policy. So before making any changes to policy, and I think this has been a common thread throughout this whole discussion and conversation is, how do we capture data? Is it validated? What types of measures are we looking at? And ultimately, the thing that I think everybody can ultimately agree upon is that data has a major influence over how those decisions are made regarding policy. So when we look at collecting this data, then we can start making decisions on how to use it based on what we're seeing.
In the future, like Shawn said in 2030, I can see starting to blur the line between training in the field. So as video is collected and data is generated, we could start using AI, computer vision to automatically start assessing what is being viewed in the video, whether it's body cam, dash cam. And be able to either prompt or be able to collect and assess and provide that feedback as a fraction or hotwash. Then that's something that you're continuously able to monitor and do in addition to providing that back to the department to see, really where are the areas that need to be targeted as far as training in the future goes? So I think it's something that we can start getting more of this training on demand and be able to have it be custom tailored to each officer and to each department.
Daniel Serfaty: Thank you, Evan, that a very ambitious future. But you know, 10 years is an eternity in our world. And Courtney, maybe you can also imagine what a life of a police officer or a police trainee, pick one, could be a day in the life in 2030 when they have many of the tools that Shawn and Evan are dreaming about. Give us your take on that. Are you optimistic?
Courtney Dean: So I'm going to try to tiptoe the realm of politics here and avoid getting too far down one path. One thing that I like is a little bit of the dialogue that is talking about taking a little bit of the officer's responsibilities off their plate. Are officers social workers? Are officers therapists? Are officers peacekeepers? Are officers custodians of our slums, of our suburbs, of our downtowns? There may be a few too many roles that have been bestowed upon police officers inadvertently, informally, accidentally that we could relieve them of and focus those back on the professionals that want to and are best suited and trained to do that.
And so maybe in 2030, we see that we have a little bit more democratic funding of our schools, and we have a little bit more judicious allocation of support resources for our underprivileged. And as a result, we have police officers who are put in a position where they're not doing a wellness check, for instance by themselves, but they're actually accompanied by a social worker. That they're executing warrants when it's a known dangerous offender and not executing warrants when it's potentially but unknown or, for instance, a psychiatric issue.
We know that that person is a psychiatric issue, so we brought the right equipment, the right people into the equation. And the reason that we know that is because people are getting help, and we're diagnosing and recognizing when folks are a psychiatric issue. And maybe in 2030 when somebody does slip through the cracks, the officers are stepping back and they're getting the right professional into that environment before things escalate.
Daniel Serfaty: Thank you. And indeed, that vision ... And I appreciate you defending yourself of being political, you're not being political here. You're basically being a scientist looking at how the complexity of the job may require through that analysis that you promoted earlier, compartmentalization of the different jobs components of expertise that we are asking our police officers to have. And maybe by distributing that responsibility, this is a beginning of a solution. Well, I want to conclude by thanking Shawn and Evan and Courtney for their insights. They've dedicated their lives to studying and also improving human performance through learning and through compassion basically. Thank you for sharing those insights with us, not only about scientific and the technical, but also about the visionary perspective that you shared with our audience.
Thank you for listening. This is Daniel Serfaty. Please join me again next week for the MINDWORKS Podcast and tweet us @mindworkspodcast, or email us at mindworkspodcast@gmail.com. MINDWORKS is a production of Aptima Inc. My executive producer is Ms. Debra McNeely and my audio editor is Mr. Connor Simmons. To learn more or to find links mentioned during this episode, please visit aptima.com/mindworks. Thank you.