39. Four Quality Assurance Steps - Data in Audit

In using data for audits, quality assurance of our work is important.

In this episode, we discuss: 

1.  The four main QA steps

• Technical QA 
• Functional QA 
• Business QA (validation) 
• Audit QA 

2. Why we need a QA Plan

This is explored in more depth in our book, here: https://riskinsights.com.au/the-data-confident-internal-auditor

About this podcast
The podcast for performance auditors and internal auditors that use (or want to use) data.
Hosted by Conor McGarrity and Yusuf Moolla.
Produced by Risk Insights (riskinsights.com.au).

Transcript

Narrator: 0:19

You're listening to The Assurance Show. The podcast for performance auditors and internal auditors that focuses on data and risk. Your hosts are Conor McGarrity and Yusuf Moolla.

Conor: 0:20

Today Yusuf, we're going to be speaking about quality assurance of our data work or QA, as we like to refer to it in audit land. But let's start off with a fundamental. Why is QA over our work important?

Yusuf: 0:31

Most auditors will have QA steps that they undertake for all of the general audit work that is being conducted. We've spoken before about checking quality of the data that we use. It's equally important to ensure that any analysis that we do with the data that we have and outcomes that we produce that we have a level of QA over it. As we've said before and as our guests have said before, data is really just another way in which we can understand the subject matter and then produce evidence to inform our findings or audit work, but also to back up some of our findings. And so, if it is just another form of evidence or something that we use as input, then QA over that work that we do becomes as important as QA of any other audit work that we do. One thing that we are consistently in agreement on- that QA is important. We've all had to sit through or deal with lots of red pens in the old days, lots of review notes nowadays, comments and track changes in our documents from people that review our work so we've all been exposed to it. There're a few differences in the way in which data work, QA works, so we'll talk through that.

Conor: 1:41

QA is a pretty broad term and covers a lot, but they're actually different types of QA. There are four main types that we're going to discuss today. The first one is technical QA, what that means and when it's applied. The second one we'll talk about is functional QA and what that looks like. And the third one will be business QA. And lastly, audit QA. We'll step through each of those in detail. So, let's start off with technical QA. What does that mean, and when is it used?

Yusuf: 2:05

Technical QA is usually the first step. This involves self-review. So reviewing your own work, then also getting a peer to have a look at what you've done. Technical QA is ensuring that what you've done technically makes sense. So when you complete some of the work that you've done, you want to make sure that the data that you've used is and remains complete, that it is and remains accurate and that it is relevant to the period that you've covered. A few things that you want to make sure about that you've used all of the relevant records, that you've not excluded any records, that you've reconciled inputs where you are able to. You haven't altered any of the accuracy of the data by double counting any data or conducting any joins that would multiply the data out. You have not performed any calculations that alter accuracy of the data and you've correctly performed transformations, have considered all of the relevant dates, etc. y ourself before passing it onto a colleague to have a look. Ideally, you'd get somebody independent of the work that you've done, potentially somebody at the same level as yourself to have a look at that and ask questions around how you've technically combined the data, how you've joined it, et cetera. The reason you want to get somebody else involved is just like with any other QA, you often get so deep into some of these technical matters that you can't always see the mistakes that you might've made.

Conor: 3:24

It talks about the importance of self-review really in all of the work that you've done. Every auditor worth their salt would obviously be already doing that. But also the benefit to be gained from peer review. Let's talk about the second QA process, and that's functional QA. What does that involve?

Yusuf: 3:39

Again, functional QA is performed first by yourself, so that's the second step that you would conduct yourself on your data work. And then by a peer. It might not be somebody at the same level as you, but it would be somebody that understands the business process or maybe understand how the broader business operates. Because functional QA is about ensuring that you remain objective and credible in the work and that also your work makes sense, Are your results believable? Can you believe them and will others believe them? Are your results rational? With an understanding of the business process overall, does it make sense? An example of this might be something like, I've got a whole bunch of data and I imported it and technically the data makes sense and I've not excluded any data that I shouldn't have excluded, I've got the right time period, et cetera. When you do a functional QA, you're looking at things like we know overall that there might be a particular trend. It is expected. There may have been a new product that has been introduced and that new product that was introduced may have generated a certain level of revenue. Can you see that revenue reflected in the results? Can you see that particular product reflected in the results? Is there something else, either seasonally or otherwise that somebody with an understanding of the business process, or the business more generally, would know about that would then look for in that data? So in essence, does it make sense?

Conor: 4:55

Then you ask questions like, this is what we're seeing, is this important? Does this meet expectations? Is there a context here that we're missing based on what we're seeing? Those types of things. So that takes us neatly on to the third important aspect of QA and that's business QA. And I think you've led into that. So what does that involve?

Yusuf: 5:11

So business QA. This is quite similar to functional QA, but the reason why you do it, one of the main reasons that you do it is that you want to have acceptance from outside of the audit team. That's one of the main reasons. The other is that there may be things that the audit team aren't aware of because they're not that close to the business process that you would want somebody that is a potential representative of the business or a subject matter expert to help explain. So again, the questions that you want to answer here are: are the results correct? Are we missing any context where the business process may have changed recently? Or the audit team just aren't aware of that particular matter. The key question we want to answer here is will our findings be accepted by the business? So, if the work that we've done doesn't pass this step, then we need to think about things a little bit differently. Or we need to get other people to think about things as in our stakeholders need to be educated. The challenge with skipping this step is that we may have a lot of pushback when we producing our results, and because a lot of the data work that we do doesn't necessarily hit the final result immediately but forms part of the report or contributes to the report, or there's further follow up work that is done after this, we want to make sure that what we're saying from a data perspective is accepted by the business early on.

Conor: 6:32

In the business QA step we're trying to again to understand the context whether or not there're for example exceptional circumstances or sentinel events that may have impacted some of the data that we've used or analyzed. Secondly, we're applying the reasonableness test to help with acceptance of what we're seeing. Talking to the business and making sure what we're seeing here is valuable, useful and practical taking forward. That takes us on to number four and that's audit QA, so what are we talking about here?

Yusuf: 6:58

This is the final bit of QA that you do. This is"QA-ing" the QA. This is where somebody within the audit team leader in the team, maybe the head of your audit function if you're in a smaller team, or if you're in a really large team that might be a team member that focuses on quality. Some teams actually have a whole QA function that's built into the internal audit team or performance audit team. What you want to do here is make sure that there has been a QA plan established for the work that you've done. We're talking about this last, but before you would start any QA, you'd put together a QA plan. The QA plan doesn't have to be very detailed, but you need to outline some steps. The audit QA will check whether a QA plan has been established. Make sure the QA has been undertaken at the right time with the right level of rigor. And then the key thing that the audit QA step will cover off is does the analysis support the audit result? The person that's conducting the audit QA will check whether technical QA was robust, whether functional QA was robust, and whether the business validation was correct. But also then check against the audit report. Is what we're seeing in the data work that was performed, is that actually reflected in the audit result appropriately? Are we missing anything? Your normal working papers will have cross references. What have you found versus what's in the report and making sure that everything found is in the report or is explained, and everything in the report was found in the data work. So similar to that does the analysis actually support that audit result?

Conor: 8:21

Is it possible to pass the technical QA then pass the functional QA of the business validation to the business QA? And then perhaps still not be meeting the audit objective, and that's maybe where the audit QA can come in at the end. Is that a possibility?

Yusuf: 8:36

Yes. The audit objective overall, and then broken down into the hypotheses that have been identified. If you only cover let's say you have an overall objective, and as part of that you looking at five hypotheses. If you only cover four of them, you've technically done the right thing. Functionally you've done the right thing. The business won't know the difference, but overall, you haven't covered off on all the hypotheses that need to be covered off. So this is that sort of final check.

Conor: 9:01

Okay, fantastic.

Yusuf: 9:03

One final note is we spoke about it briefly is what's your QA plan? What's your quality plan? Do you want to do this initially to be able to check off that each of those steps have been undertaken. And it's particularly important when you're doing work iteratively because you can quite often forget about or miss a step when you're going through an iterative style approach. Doesn't have to be very detailed but we're just creating an overall plan that we can use to ensure that we are covering off each of those steps within each of the iterations that we have, and that the individuals that are performing those QA steps understand what is expected of them at each of those steps. You could be in functional QA stage and the person that's doing the QA may not remember or understand what exactly is expected of them. By creating a plan, it makes it a lot easier to explain what is expected of each of the individuals undertaking the different types of QA, that then just leads to a more complete result.

Conor: 9:59

So we covered the four components of QA there. Technical QA and what that was and when it's used. Functional QA and they're moving on to the business validation, which is business QA and then the overall audit QA, which is your catch all at the end to make sure that all the previous steps have been taken to the extent necessary and that you're meeting the audit objective.

Yusuf: 10:17

That's it. Thank you.

Conor: 10:18

Thanks Yusuf.