46. Combining data to create risk indicators for audits and risk monitoring/exploration

In this episode we discuss the use of risk/performance indicators.

These can be used by audit teams, risk teams and integrity agencies.

For audit teams, they are useful throughout the audit lifecycle - planning / risk assessment, fieldwork / conduct and reporting.

We explore why they are useful, what to do and briefly how to get started.

About this podcast
The podcast for performance auditors and internal auditors that use (or want to use) data.
Hosted by Conor McGarrity and Yusuf Moolla.
Produced by Risk Insights (riskinsights.com.au).

Transcript

Narrator: 0:07

You're listening to The Assurance Show. The podcast for performance auditors and internal auditors that focuses on data and risk. Your hosts are Conor McGarrity and Yusuf Moolla.

Yusuf: 0:20

Today, we're talking about performance and risk indicators for use within internal audit, performance audit, and then also extending to first-line and second-line, and various other individuals that want to understand how different indicators can come together to provide a view of performance, or risk, more holistically.

Conor: 0:39

When we say risk indicators, what are we actually talking about there?

Yusuf: 0:42

Well, depending on the subject matter that we're looking at, it would vary. What are the different areas that will contribute to our understanding of a particular domain?

Conor: 0:52

Use the data we have to build up profiles of an entity or subject matter that is of interest to us.

Yusuf: 0:57

Yeah, and the more data that we have from sources that are different, the better the view would be. Meaning not just bringing all data together from one system or one domain. We always talk about bringing complaints data in, and complaints risk is one of those things that we don't normally include as part of an audit or a risk evaluation. So that's an example. Open data, obviously. But then also, if you just think about across functions within the entity, across departments, within the entity, bringing some of those different data points together. Obviously, they need to actually relate to the risks or the performance area that we're looking at. But they don't necessarily have to be directly related. You can have a combination of directly related indicators that would influence strongly. And then others that have a minor influence.

Conor: 1:45

That all seems really intuitive and sensible and reasonable. There's benefits to that approach at the planning stage, where you can combine risk indicators to try and help you select the sample for your internal audit or your performance audit.

Yusuf: 1:58

In the performance audit world, it would be determining which entities are we going to audit as part of a particular topic? Within internal audit, what is the sample that we're going to be looking at? So you have within banking, within retail, within production and manufacturing, which entities are we going to look at? Which branches are we going to look at? So that sort of thing is being done. But we can apply the concept to a greater number of audits than we're currently doing.

Conor: 2:22

There's clearly benefits, in conducting your audit, where you can actually draw together indicators, whether they be from internal data or external data, to profile the activity to try and understand what's going on. And then there's also indicators that we can use as part of our reporting.

Yusuf: 2:38

As auditors, when reporting on these things, we can use it to demonstrate to whoever the stakeholder is. Whether it be the general public, or internal to the organization, or the board or the audit committee, or management, whoever. Why we did what we did. Because often people want to know why is it that you looked at a particular entity or particular sample? Also we can use it to explain, what is going on. Seeing that broader picture is enabled by bringing all of this together, for reporting.

Conor: 3:03

Taking a risk indicator approach is really helpful where you have, limited control over the work that comes into your organization. And we might see that, for example, in a regulatory field, if it's a regulator or an integrity body or, somebody that receives complaints from members of the public. Now, those types of organizations have constraints on resourcing, yet they really have no control over what comes through the front door in terms of their work. So they really are reactive in nature. And what we're seeing generally, is that a lot of those organizations in that type of arena, are now understanding that they can actually prioritize their work when it comes through the front door by really trying to build up indicators about what's going on in their operating environment and use that to drive their resourcing model.

Yusuf: 3:48

Yeah. And the same could be said for internal audit functions. The same can be said for second line risk functions,first-line functions. It's important then to target your resources at the areas that are going to have the highest impact and bringing this sort of data together enables that.

Conor: 4:05

So the use of indicators is a logical response to trying to control where you focus your limited efforts. Where do you start? What's your first point of call in trying to build up a risk indicator approach to your work?

Yusuf: 4:18

It definitely is iterative. But you do need to start off with something that at least gives you a reasonable starting point. So you want to have three or four indicators at least to begin with. There's no point in trying to bring one thing together. What's this, that's not really giving you a comparison. So you need at least three or four to start. How you get to that is bottom up top down. So the top down is, what is it that I'm interested in. Bottom up is what data do I have that I can feed into this model? The bottom up is important, but we do need to not throw data in just because we have it. So top down, what do we need? And then bottom up, what do we have to address the need? And what we don't have, how do we collect it for future use or can we collect it for current use?

Conor: 5:02

And that top-down approach of trying to determine what we need that can of course change over time. So for example, if your organizational priorities change or if your strategy changes or your focus area changes, then you should always be cognizant of that having a potential impact on the indicators that you need.

Yusuf: 5:20

Yeah. And sometimes you'll see, also impact of that through the data as well. So you see that bringing a particular indicator. It doesn't give you a better answer than you had before. or a particular indicator, regardless of the weighting that you have, it doesn't actually give you a better overall result.

Conor: 5:34

As you said, you need to be ready to iterate. And add to those indicators over time. Perhaps remove some of them that have become redundant. Or slightly tweak some of them that still remain relevant. There is possibility for a little bit of fluctuation in what that stack of indicators looks like as the business goes on.

Yusuf: 5:52

The other interesting thing that we've seen a few times now is that you can use certain indicators repeatedly across multiple subjects. And that's really useful because you just need to add to it as opposed to starting from scratch each time. However, there is a however with everything, however, you need to make sure that you're not just using an indicator because it exists. Don't just grab certain indicators and pull them across just because they may have some sort of remote relevance to the new subject matter you're looking at.

Conor: 6:17

So every time you look to bring in a new indicator, you really need to ask why is this important for our focus at the minute? And how is this going to contribute to the audit that we're trying to deliver.

Narrator: 6:27

The Assurance Show is produced by Risk Insights. We work with performance auditors and internal auditors, delivering audits, helping audit teams use data, and coaching auditors to improve their data skills. You can find out more about our work at datainaudit.com. Now, back to the conversation.

Yusuf: 6:47

I spoke about top down. How do you determine that top down? Because you do a lot more than me, of the "what indicators are we going to be using?" What's your thinking? How do you approach that when you just have a blank sheet of paper?

Conor: 7:01

I'm pretty old fashioned in this regard. And I always go back to a hierarchy of needs approach. So at the top of that might be, is there a legislative imperative for what we're trying to achieve, either for the public or in a certain domain that dictates that we need to be grabbing this sort of indicator and using it. And the next thing might be then what are our three strategic objectives for this organization over the next, usually three to five years, and try and understand what are the contributing factors to those. So for example, in the corporate world, it might be, we want to expand our geographic footprint into another continent. So, I might be interested in understanding what are some of the indicators of activity in that continent that would speak to my business that I need to potentially bring in to enable that strategy to be realized.

Yusuf: 7:51

So regardless of the audit topic, because you have a certain limited set of strategic priorities, you want to try to include those in every topic that you're looking at because there may be relevance.

Conor: 8:04

Absolutely. Of course, there's a cost benefit there that you need to make a judgment call , how much does it cost to bring these indicators in. Is the data complex, do we need to clean it? Is the benefit going to outweigh the cost?

Yusuf: 8:14

There'd be some audits that are purely compliance focused and have a very narrow objective that don't relate directly to some of the more forward-thinking strategic objectives. And in that case, may not necessarily need this sort of approach. But that's not the majority of audits. Some entities are very compliance focused, but most audit teams will have a little bit of compliance work that they just need to do housekeeping work. But the majority of the effort will be on helping ensure that we achieve or ensure that we prevent against risks to achieving those objectives.

Conor: 8:44

Briefly going back to the hierarchy. So we talked about understanding what's happening in the external environment, government priorities if you're in the public sector. What do I need to do or grab that could help the people in my jurisdiction? But there's also a tertiary. And this is to really mature how you use data within your organization. That's an internal benefit, but it's still a really important benefit. If you can grab some external data that is an indicator that would also add to your maturity of how you use data, then that's still a really helpful thing to aim for.

Yusuf: 9:12

That feeds into the cost benefit analysis. There may be a benefit that falls outside of the immediate work that's being done.

Conor: 9:17

Yep. We've spoken there about why it's important. We've talked about what you can possibly focus on. What about the how Yusuf? Once we've ticked off on those two things, we know we need to do it, we know what we need to do. What do we need to focus on for the how? What are the three top things?

Yusuf: 9:34

Okay. Yeah. glad you asked for three top things because this conversation could go for days. Three top things that we need to think about in the how. The first is, do we have ready access to the data? Are we actually going to be able to get the data within time? If we are able to get it, then the next thing is how are we going to bring it together? And that is a really important step. Ensuring that we are cleansing the data appropriately. Taking out any data that we aren't going to need. So columns that we aren't going to need or rows that we aren't going to need. There's a whole bunch of technical steps, like how are we going to join? We have five different data sets, how are we gonna actually join them up together? And then importantly, how do you correlate them? You have five indicators, you've cleansed it, you're able to join it all. How do you then actually bring that together to be able to provide that one view? So we want one metric that comes out of it. And there's quite a few different ways in which you can do this. We obviously use visualization tools. But what we found is that by throwing things directly into a visualization tool, you can do joins, you can do cleansing. But, first of all, you have performance issues because some of the data can get quite big. In terms of how the data is actually coming together, creating the scenarios that are needed. All those sorts of things are better done in a backend analytics tool and then brought into the visualization tool. You've cleansed your data, you join your data, the next thing you need to do is normalize that data. So if you are, bringing a particular indicator from a particular data set, that could look very different to a second indicator from a second dataset. You need to make sure you normalize it. So you actually bringing all of the data into the same range of values. We usually use either zero to one or zero to a hundred. And then when you joining it together and now you need to say, okay, am I adding it and multiplying it, what I'm actually going to do? Typically add it and then divide it by the number of indicators that we have. But that's a very simplistic approach. The better approach, which is slightly more complex, slightly more sophisticated, is that you use scenarios for each of the indicators so that you can weight those indicators. You know, 0/1 or 0/1/2. So that when you bring your data into a dashboard, if you like, you can actually decide which of the indicators to switch on or switch off with the 0/1 (zero one) approach. With a 0 1, 2, et cetera, approach, switch off, lower risk, higher risk, et cetera. And so that's the sort of thing that you want to do that you do, like I said, in the backend. And then when you bring it out to the front, you try to structure it such that you have that one overarching indicator. The other thing that is good to do is to make it a relative score. Remember bringing indicators together, there's nothing real about that number that's generated. But really you want to be able to see it relative to each other, that's the whole point of it. And then having those scenarios. Then in the dashboard, you have that one overall indicator, you have the different scenario options, and then a range of filters, depending on what you're looking at. So it may be year, it may be month, it may be particular entities that you want to select or groupings of entities that you want to select. And then also having the table of details below that, that you can actually drill into to understand when I'm looking at a particular number, what is it comprised of? How was that generated so that when you're exploring it or you're explaining it, you can get to that level of detail.

Conor: 12:37

That ability to visually situate those entities against each other, that's particularly helpful, I think. And like you said, that gives you the ability to further drill into what's driving the performance in that particular entity.

Yusuf: 12:51

Three things about how there. One is, can I get the data? Secondly, what am I doing in terms of normalizing the data when I get it in? And then thirdly, how do I visualize it? So getting the data, calculating all of the fields and cleansing it and getting it ready for visualization, and then making sure you have enough in the front when you visualize it.

Conor: 13:09

And try not to piece together too many indicators and be too ambitious from the outset. So don't start with 20 indicators across various data sets. Like you said. Probably start with three or four based on reliable data and just build up the profile from there.

Yusuf: 13:22

The number one thing though is to start doing it. The more we understand the data that we have available to us, the better we can understand the businesses that the data underpins. And also the more we use this data, the better we're able to drill into the details to understand some of the nuances.

Conor: 13:38

Okay. So wrapping up. First thing, if you're not already doing it you got to start doing it. Take a sensible approach. Maybe start off with three indicators, stack them together.

Yusuf: 13:46

Second thing. You can use it for planning. You can use it for fieldwork. You can use it for reporting. There's various ways in which that will work. And this is within internal audit and within performance audit. Obviously if you're in 1st line risk or 2nd line risk, this is something that can become part of your continuous monitoring or something you can use ongoing to be able to understand those risks. If you work for a regulator, it can help you identify particular entities. The application is broad, but particularly within the audit sphere it can be used throughout the lifecycle of your audit.

Conor: 14:15

And lastly, It will be a process of iteration. Don't expect to use all the right indicators, to get exactly the right outcome, from the start. You'll need to keep building on those indicators, and some will fall away, new ones will come in. But just know that, you're getting closer and closer to having accurate measures of performance.

Yusuf: 14:33

Good stuff. Thanks Conor.

Conor: 14:34

Thanks Yusuf.

Narrator: 14:35

If you enjoyed this podcast, please share with a friend and rate us in your podcast app. For immediate notification of new episodes, you can subscribe at assuranceshow.com. The link is in the show notes.