How the Justice Center Uses Data to Prevent Harm

Most people know the NYS Justice Center investigates abuse and neglect. But what many don't know is that at the heart of the agency is a powerful Data and Analytics unit—a team of data scientists working to make the entire system safer. 

In this episode, we spotlight our data unit and how the Justice Center leverages incident reports to identify systemic dangers that no single provider or case file could reveal. We use a powerful real-world example: the agency's data-driven response to food-choking incidents.

Discover how recognizing a dangerous trend led to direct, actionable changes that have protected countless people supported by New York State care systems, proving that sometimes, the most effective tool in the fight for safety is preventing harm before it happens.    

Watch on the Justice Center website: https://www.justicecenter.ny.gov/how-justice-center-uses-data-prevent-harm 

Transcript

Erin Hogan (00:10):

Hello, and welcome to the “Could This Happen in Your Program?” podcast, where we find collaborative solutions for protecting New Yorkers in care. Most people know the New York State Justice Center investigates abuse and neglect, but what many do not know is that, at the heart of the agency, is a powerful data analytics team, a group of data scientists working to make the entire service system safer.

In this episode, we sit down with Michelle McCrum, Director of the Office of Data Analytics, to show how the Justice Center leverages incident reports and case data to identify systemic dangers. Discover how recognizing a dangerous trend led to direct actionable changes that have protected countless people supported by New York State care systems, proving that sometimes the most effective tool in the fight for safety is preventing harm before it happens.

Alright, well, welcome, Michelle. Thanks so much for joining us today. Can you introduce yourself and your background to our listeners so we know a little bit about what you do in the data world?

Michelle McCrum (01:10):

Thank you for having me. So, I'm Michelle McCrum. I am the Director of the Office of Data and Analytics here at the Justice Center. So, our role in this agency, and mine specifically, is really all things data. So, from making sure that we're collecting the right information for the right reasons to education about how data can be used to promote the mission of the Justice Center. So, about me, I started at the Justice Center about eight years ago, also in a support role on kind of like the project management database update world, and have kind of found myself back to my passion of data and really growing the data capabilities of the agency.

Erin Hogan (02:01):

So many of our listeners know the Justice Center, and they think of, they automatically will go to investigations. They'll go to maybe our advocacy services, maybe quality management. Can you speak a little bit to the data side of things and the role that your team plays in that mix?

Michelle McCrum (02:19):

Yeah, no, that's a great question. So, everything that you said in some way has a foundation of data and information, but our role specifically is a support unit for both internal Justice Center operations and also supporting external partners, external agencies. So, our mission is to empower the Justice Center to make informed decisions by analyzing data, sharing insights, and also keeping our data tools running smoothly. So more of that technical side of things. Our day-to-day looks like working very closely with the Justice Center's business units to build those data skills—that’s the education component, I talked about—support operational goals, and make sure that the data that we use is accurate, consistent, and used in the right way. We help people just really understand what is data, how can we use it better, how can it support decisions, and ultimately how can we use data to better serve the populations under our protection.

Erin Hogan (03:26):

So when we think about the Justice Center, I think something that is often forgotten about or overlooked, that was a really key part in our foundational history, was the fact that we were able to sit in a unique position overseeing all of the provision of care in New York State and analyzing the data systemwide to aim to create a safer New York. So, the thought process there is maybe there is a trend or an issue that's occurring in the OPWDD space, but it may also be an issue that we see happen and occur in OASAS settings or in OMH settings. And for that reason, we are able to take that kind of cross-systems look. And I think that's the very important piece that your team is able to provide. So, all that's to say, can you tell us a little bit about what this universe of data looks like and what kind of data we house here at the Justice Center?

Michelle McCrum (04:23):

So interactions with any of our services will be documented and saved, and our primary case management database, it could be saved in an Excel spreadsheet, a Word document, any information that we--

Erin Hogan (04:36):

And when you say, I'm sorry, I don't mean to cut you off. When you say our internal system, you mean the VPCR?

Michelle McCrum (04:41):

Yes. Okay. Yes.

Erin Hogan (04:43):

I think most of our listeners know the VPCR system.

Michelle McCrum (04:45):

Thank you. So the VPCR as our primary case management database. So to operationalize that a little bit, this could be the information that becomes data could be information from a call to our 24/7/365 hotline. It could be observations made by the prevention and quality improvement (PQI) team during one of their site audits. It could be documents gathered during an investigation. So all of that is distinct data.

If you think about these pieces of data as kind of separate LEGO bricks, you have them scattered all over the place. I know I do at my house, they're all over the floor. And individually they don't mean anything, right? They're just a mess. They're just a heap of LEGO bricks. But if you think, again, going back to the different information that I mentioned, our team helps take all of those LEGO bricks that are sitting on the floor or, in this analogy, sitting in our case management database, and we help build them into something meaningful. So now the pile of records becomes a meaningful insight. It becomes something with clear actions, contextualized. So, using the LEGO analogy, that pile of nonsense on the floor is now a castle because we've taken all of that, we've structured it, and we've built something, and now you can do something with it.

So our team adds structure, it adds meaning. We take all of those individual records, bring them together to tell a story, find an insight—all of that to indicate that an action is needed.

Erin Hogan (06:25):

I love that analogy. So it goes from being a nuisance, something that you could step on the floor to being something really meaningful and helpful. And yeah, I love that analogy. That's great.

Well, bring us through, though. So, one of those LEGO bricks brings us through a single report or an instant that is then brought to your team to say, “Hey, something else is going on here. Can we dig deeper?” What does that exploration look like from your end to then lead to systemic issues?

Michelle McCrum (06:54):

Yeah, sure. Another great question. So, a while back, in, I think it was grad school, I learned about the relationship and the differences between data, information, and knowledge. So, the data again could be that LEGO brick, but it's really just a raw, unprocessed, individual, distinct records. It's a single record sitting in this vast database.

(07:16):

Data then becomes information by organizing it and structuring it; information becomes knowledge by contextualizing it and having some sort of action that you can take from the knowledge that you just gleaned.

So, going back to your questions about recognizing a pattern and what we do here? Let's use an example.

So let's say over the last six months, there have been 12 incidents or calls placed to our hotline by Provider A. So on their own, that's just 12 distinct records sitting in our database. Our team will help make that more useful. So, first, our analytics database will help structure them in a more useful way. Now, those 12 records are on a dashboard, and now they're information. So we now know, okay, here's something, but what is it? How is that information presented, and what's the context? And that context is when that information becomes knowledge, becomes actionable, and becomes a pattern that the Justice Center can do something with.

(08:22):

So let's take those 12 calls, and now we look at it in the context of our analytics, and we say, well, that was actually 12—of those 12 calls—there were seven abuse, neglect, substantiated findings. So now we're diving deeper. And what we're seeing is that of those seven abuse, neglect, substantiated cases, six of them were actually failure to maintain professional boundaries. So what does that tell us? That tells us that Provider A has, there's something going on here, and there's an action needed. Is it training? Is it a refresh of procedures or better documentation at Provider A, whatever that action is, the Justice Center and the provider can now take intervening action together to mitigate the risk of more things happening and mitigate the risk of these things happening again.

Erin Hogan (09:20):

Well, and I think that's the unique piece and the role that we serve as a great example because it's the behind-the-scenes part of our work that no one really sees, right?

We are working collaboratively with providers, with the SOAs that license or operate them, to find that collaborative solution to an issue. “Hey, we've identified this. How can we help support you to make sure it doesn't happen again?” And that leads to that culture of safety that we're really trying to create here in New York State.

Michelle McCrum (09:49):

And, also, that culture of partnership, too, through information sharing.

Erin Hogan (09:57):

How would you say, you kind of alluded to it, but that systemic approach, how is that helping more than a systemic look versus just taking an individual case? I mean, I think we all know, and we're very mindful of the Justice Center, that mistakes happen. So it doesn't necessarily mean that just because there's something that occurred, it doesn't necessarily mean there's a huge problem, or even if it's a bad program, it's just something that occurred; we need to fix it.

But when does it become more of a systemic issue, and why is that a better, more problem-solving approach than just looking at individual cases alone?

Michelle McCrum (10:30):

Yeah, so I mean, I think a systemic approach is very similar to a preventative approach. It's about stopping the problems before they occur, which is I think very much what you're saying. So the patterns that are identified because we're seeing the same thing happen over and over again. And to your point, it doesn't always mean that it's a bad actor. It could mean that something just needs to be strengthened, that a control needs to be put in place. Looking at this information kind of more at a 30,000-foot view, which is kind of the more systemic approach, can really help us identify some factors of this pattern. So we look at, well, what's below the surface a little bit, what's causing this? And causation is a tricky word, I think, in analytics, but it at least gives us some contributing factors, contributing factors that we can say, Hey, this is more likely to occur when X, Y, or Z happens. And then the Justice Center can help expose that information, share that information, and educate the providers to say, okay, well, here's some tools. Here's something to reduce the risk of this happening again. So if we just look at individual cases on a case-by-case basis, we're missing those contributing factors. We're missing those trends. The seasonality of the repetition, that is how we turn insights into actions.

Erin Hogan (11:50):

And I think even beyond that, it's okay, we may not know we took it this far. We found the data and the information that says this is a problem. When we show the providers or the licensing entities, they may know more of the causation to that problem because, oh, yes, at that provider, we have had turnover more frequently than not, or we need to re-up our training in that area. So I think putting the pieces together, so to speak, marrying our data with what is received by the provider, and arming them with the autonomy to then take that and make a change.

Michelle McCrum (12:31):

And I think that's a great point. I think it goes back to that difference between information, knowledge, and how context and partnerships can really help provide that path to knowledge. So there may be instances where we have the data, we have the output of that data, but we're missing the context, we're missing the understanding of why. And that doesn't just have to be the Justice Center, and nor should it be, right? So I think you're absolutely right, leaning on those partnerships, having a collaborative approach of sharing that information to become knowledge to become action.

Erin Hogan (13:07):

So about a year ago, we launched a toolkit on choking prevention. We noticed an uptick in cases involving food-choking incidents. And I think this is a good example of your team kind of stepping in and providing some insights on what is the actual universe of this issue? Is it really an issue? Is it specific to one service setting? And what are the contributing factors there? Can you talk us through that discovery process a bit, and how your information led to the development of that toolkit and led to real change.

Michelle McCrum (13:49):

Yeah, thank you for asking about this because I think this is an excellent example of how the Justice Center is using data collaboratively to drive our mission. And I am going to keep using that partnership in context. I think it's critical, right? Data analyst, we have a lot of information, but we don't have the knowledge and the day-to-day provider world or whatever it may be. So this is a great example that highlights how valuable that those partnerships can be. So I mentioned earlier, we work very closely with internal colleagues, you guys being one of them, but also our prevention and quality improvement team, aka PQI. So PQI asked our data and analytics team to take a look at trends or patterns related to choking incidents. I think anecdotally, they see there's something going on here. There's something more that we could be doing here. So the question posed to us was very simple. Does our data support the anecdotal observations that a choking review is needed and that additional tools could be provided to help mitigate this risk? So the data analyst on our team assess years of incident and case data to answer that question, which was, yes, a review is needed to really look at the contributing factors to choking, to understand more so that we can take action to mitigate that risk. So this is another great example of turning that data into information into action.

Erin Hogan (15:25):

So we have this toolkit in your team's analysis of the data, did it reveal a single cause? Was it a confluence of issues? What can you tell us about the findings?

Michelle McCrum (15:35):

Yeah, so I kind of mentioned this a little bit before causality as kind of a wonky world in the world of analytics. So establishing causality isn't always straightforward, but we can, data can suggest connections when clear causation isn't always available. And honestly, this is again where partnerships help out. So we worked with the PQI team to contextualize this information. From there, we suggest likely drivers of choking. So in this specific situation and this specific review, we looked at information collected during site visits. We looked at the policies, procedures, and training. And what it all came down to, really, was suggesting additional training for staff on food preparation and responses to choking-related incidents that would minimize poor outcomes. So we know that there's more to this, right? We know that people with developmental and intellectual disabilities are more prone to choking. So it's about contextualizing and understanding contributing factors, kind of figuring out, well, how can we control for that? And then see, okay, well, are there contributing factors that are in our control? And by “our,” I mean the global “our,” the royal “our,” whether that be the Justice Center, the state oversight agencies, or the actual provider themselves. So how can we kind of cull those actionable insights out and how can we operationalize those? And that comes to the toolkit, and that comes to additional trainings and all different types of materials and just little things that add up in the long run.

Erin Hogan (17:23):

And then now that those interventions are in place, what steps are we taking to track the effectiveness? We have the toolkit. We have the intervention from our prevention and quality improvement team, who again, conducts site visits, interacts with the provider community very frequently on quality management issues. So what are the steps that we are taking either your team or their team in addressing the concerns found?

Michelle McCrum (17:48):

So now that the interventions are in place, we just need to keep monitoring, right? We need to understand the impact of these interventions, the impact of these tools, the impact of this additional knowledge or training. So, since the original analysis, we've actually made some refreshes to our case management database, the VPCR, to collect more granular details on an incident in a more effective way so that we can keep digging in on what we're seeing here. So this allows us to continuously monitor trends related to choking incidents against the baseline of the original analysis and the effect and impact that all of this has had. But always part of that analysis is to control for those contributing factors.

Erin Hogan (18:37):

So the Choking Prevention Toolkit is a great resource. It's available on our website. I'm going to link it up in the podcast so that everyone can download and see the fruits of your labor.

Michelle McCrum (18:47):

Yes.

Erin Hogan (18:49):

But yes, great example there. I think for our audience to see how we take one data point or a multitude of data points and extrapolate into something that's a real actionable change.

Michelle McCrum (18:59):

And then it never stops, right? It's continuous. It's just a loop that we say, okay, we delivered something. Well, what happens next? And what happens now? And did it move the needle? Are we seeing a reduction or an improvement? Depending on it's what it is.

Erin Hogan (19:13):

We solved it. Check.

Michelle McCrum (19:14):

Right?

Erin Hogan (19:18):

So if you had to summarize in one sentence, or maybe if a few sentences, one sentence is pretty tough, the most important takeaway that folks listening here can take about our approach as the Justice Center to data, what would it be?

Michelle McCrum (19:36):

Yeah. So I think it's definitely not going to be a sentence. We believe in the power of data and the value of data as a critical tool in our toolkit to fulfill our mission. So the Justice Center uses our data in a protected and intentional way to understand why things are happening and how to inform ways to stop it and protect the populations that we are charged with.

Erin Hogan (20:00):

Alright. And why do you think it's so important to bring this culture of analytics to the Justice Center's work?

Michelle McCrum (20:06):

Yeah, this is a great question. Again, how much time do we have left in this podcast? I can go on forever. I'm

Erin Hogan (20:13):

Just reeling out the good questions today.

Michelle McCrum (20:14):

So this is actually really an important question, one that I feel really strongly about. The Justice Center has invested in the value of data really by just having a dedicated Office of Data and Analytics. This is critical, and it sets us apart on its own. One group of analysts will only get us so far. Going back to my favorite LEGO analogy. It's great. We have LEGO builders, right? We have a team of LEGO builders, but if that's only in one small part of the organization, it's only going to get us so far, and we're still going to have heaps of LEGOs on the floor and not a lot of castles. So we need everyone in the agency to have the same tools in their toolkit, doing their daily work. Using data in a meaningful way doesn't have to be creating a fancy dashboard.

(21:04):

It's about knowing that you should be looking for patterns and how to look for those patterns. It's about understanding that the input of high-quality data leads to high-quality analysis, and it's about just knowing how to interpret information to turn it into knowledge. So, the culture of analytics is really about asking the right questions. It's about knowing how to objectively use the data to answer that, and it's about taking those insights and turning them into action.

So that's the culture the agency is building. It's about being curious. It's about thinking critically, and it's about investing in our workforce to do these things with innovation, curiosity, and intention.

Erin Hogan (21:45):

And I think that's so important. And Michelle's team has done an amazing job in helping our staff really use data in a way that tells a story and is shared appropriately. I know I'm often like, can you look at these charts in this slide deck and let me know if this is correct? But it really is. I mean, going back to the context, it's about responsibly sharing data in a way that doesn't expose the wrong information or paint an incomplete picture of really what's going on. So I think you've done a great job in managing that and your team.

Michelle McCrum (22:25):

I have a great team.

Erin Hogan (22:27):

Well, we've been talking to Michelle McCrum, the Director of the Office of Data and Analytics. Thank you so much for joining us today. I expect everyone to be banging down your door.

Michelle McCrum:

Thank you so much. Thank you for having me.