Auditing with data: for Performance Auditors and Internal Auditors that use (or want to use) data
The podcast for performance auditors and internal auditors that use (or want to use) data. Produced by Risk Insights.
Auditing with data: for Performance Auditors and Internal Auditors that use (or want to use) data
45. How ANAO uses data in performance audits - with Xiaoyan Lu and Christine Chalmers
Christine (performance audit) and Xiaoyan (data analytics) are Directors with ANAO, the Australian National Audit Office.
In this episode we discuss:
- What The Australian National Audit Office (www.anao.gov.au) does
- The five ways ANAO uses data in its performance audits
- How performance audit teams work with the central data team
- Challenges with data preparation
- The benefits of a strong focus on data quality
About this podcast
The podcast for performance auditors and internal auditors that use (or want to use) data.
Hosted by Conor McGarrity and Yusuf Moolla.
Produced by Risk Insights (riskinsights.com.au).
You're listening to the Assurance show. The podcast for performance auditors and internal auditors that focuses on data and risk. Your hosts are Conor McGarrity and Yusuf Moolla.
Conor:Today, we're gonna focus on performance auditing and the use of data in auditing. And we're going to welcome two guests on the show from the Australian National Audit Office, the ANAO, Xiaoyan Lu and Christine Chalmers. Let's start with a brief introduction. Christine, can you tell us a little bit about yourself and your role at the ANAO, please?
Christine:I'm an audit manager in performance audit at the ANAO. I've been with the ANAO for about two years. Prior to that, I worked in research and evaluation for many years in Australia, in Canada and in the UK. And my background is in political science and business administration.
Conor:Excellent. And Xiaoyan, can you tell us a little bit about your role and your background?
Xiaoyan:I have been working for the Australian National Audit Office in the last three years. Prior to that, I worked for another government agency for 11 years. And before that I was a social research consultant for four years. I have a background in applied statistics and quantitative research and mainly using data analytics to inform government decision.
Christine:Just a little aside on that, Xiaoyan and I have actually known each other for about 20 years. We worked together many, many years ago, coincidentally, probably about 2002 in social research.
Conor:Can you give us a little bit of a background on the ANAO and its functions? Christine: The ANAO exists General in delivering audits. The Auditor General is an independent officer of the Australian Parliament, whose purpose is to support accountability and transparency in the Australian government sector. And he does that through independent reporting to the Parliament and in doing that driving improvements in public sector performance. His work covers three areas, annual financial statements audits of Commonwealth entities. Secondly conducting performance audits of Commonwealth entities. And then finally auditing the annual performance statements of Commonwealth entities, on request. Our role is that once a performance audit is completed and approved by the Auditor General, we will table that report in the Australian parliament. The ANAO also shares key learnings from performance audit in quarterly audit insights, publications that are also placed on the ANAO website. Most people, if not everybody, will be really familiar with financial statements audits. But let's have a little bit of a focus on the performance audits. How does the ANAO select which performance audits it will do?
Christine:Every year, the ANAO publishes something called the annual audit work program. And that is essentially a list of potential performance audits that might be conducted over the course of the financial year. The selection of the potential audits is based on firstly, the interests and priorities of the Parliament of Australia. The audit office or the auditor general will attempt to provide a balanced program of activity across different factors. And that includes the proper use and management of public resources across the four E's of efficiency, effectiveness, economy, and ethics. So we'll try to achieve a balance across those four E's. It'll look at planning and delivery in major areas of public investment. So examples from 2020, 21 might include defense capability, large scale infrastructure such as the national broadband network, programs targeting Indigenous Australians, for example. We'll also look at the measurement of performance and impact against agreed program objectives. So including in relation to things like probity, integrity. And this year, there was also a COVID-19 audit strategy. And that's looking at specific risks that the Australian public sector faced during COVID-19, and the specific challenges to policy design and service delivery during COVID-19. So in terms of the process, generally, what happens is there's an environmental scan that's done by the performance audit teams in October to November. That's developed into potential topics in December through February. And then that's sent out for consultation to the Joint Committee of Public Accounts and Audit, and the individual entities in March through June. And the program is typically finalized in July.
Yusuf:You mentioned that towards the end of the calendar year, you would identify audits and then you have approval mid-year, are there any changes that happen to that for emergency or other reasons?
Christine:Yeah. So it's really just a list of potential audits that over the course of the year could be adapted to emerging risks. And that happens. There can also be specific requests made by parliamentarians, for example, for the Auditor General to examine certain topics and those requests will be considered.
Conor:Can you give us a little bit more information about how data is currently being used by the ANAO for for its performance audits?
Christine:The first thing I would say is that we want to define what we mean by data when you ask that question. So we generally classify data into two broad types, unstructured and structured data. So by unstructured data, what we mean there is information that's normally very text-heavy. Although it might contain some numerical data such as dates and quantities it's uncollated. It requires a lot of interpretation. Some of it's available publicly. So for example transcripts from parliamentary or public inquiries or media or corporate documents. But most of it's not public and that might include things like correspondence, meeting minutes, organizational charts, contracts, those sorts of things or ministerial briefs and cabinet documents. And it's very time consuming to analyze. We're fortunate at the ANAO to have access to a program that we use to assist us with classifying, searching and storing that information. That type of unstructured data is probably the main type of data that's used by performance audit. Structured data we would define as data that's generally collated. It adheres to some kind of predefined data model. And it's usually in a tabular format with rows and columns that are related to each other, and it's usually number heavy, although it can contain text data as well. So that kind of data is used as well in performance audit. There's five different ways in which we use data. The first one is just for descriptive purposes. But secondly, we'll look at data as a means of validating an entity's public reporting. So for example, what they might have said in their annual performance statements or in their annual reports, we'll use it to test hypothesis or explain why certain outcomes may have occurred. We will often use it to identify outliers. And then finally we use it as a sampling tool. So we might identify stratified random or targeted sample based on some risk criteria that we'll then test in a more in-depth way.
Xiaoyan:The consideration of using data for audit has already been embedded in ANAO's audit workflow for both performance audit and financial statements audit. For example for the performance audits during the audit's scoping and planning phase, the audit manager will be the first point of contact, to coordinate and discuss any potential involvement and the input from the data analytics expertise.
Christine:What we find is that it can be quite complex when data needs to be transformed into something that's useful and reliable for audit purposes. So the majority of the work we find is in relation to data extraction, preparation, transformation, and cleansing, that is really the huge effort that needs to go into the data. So there's a big cost benefit analysis that needs to be done by performance audit whenever we're looking to use structured data, that's derived from other systems just to determine whether that extraction and preparation process is going to provide enough benefit to the audit in terms of meeting the objective and answering the audit criteria.
Yusuf:We've seen the same thing in terms of data preparation. Where do you find more of a challenge? Is it in preparing open data for use or preparing proprietary data for use?
Xiaoyan:I think it's both. Accessing business data involves both technical challenge and non-technical challenges for us. Just because you can access it, it doesn't mean that it is in a usable format. Sometimes you get lots of relational data sets, but as there is no data model to help you identify the relationships and sometimes you need a specialist to transform data into ways that can be easily used by our auditors.
Christine:With respect to data quality from performance audit perspective, one of the problems that we encounter is that process of preparing the data. Yeah. Given the amount of work that's involved in doing that can take so long that by the time we're in a position to draw a conclusion about how complete and reliable and valid that data is it can be quite late in the audit lifecycle. So the lack of quality data is a real issue in performance audit particularly where the data hasn't been created for audit purposes. And that can actually be an audit finding in and of itself. And the other thing too, is that the data that we might collect from one system, for example, a case management system is really only truly useful if it can be linked to data from another system such as financial management system. It's very hard to predict at the outset of an audit, the extent of the data preparation, linking challenge and the extent of the data validation challenge. That is particularly difficult for performance auditors, who may not understand the technical data. So it becomes a real challenge, I guess, for performance auditors to work out how to integrate data analysis into their audit.
Conor:So we've got all these challenges, but obviously the benefits of getting some good data prep and analysis are manifold and can have great impact. With that in mind, can you tell us about some of the upcoming audits or projects that the ANAO is working on involving data analysis?
Xiaoyan:We announced last month that we are going to do an information report on Australian government expenditure on grants. Information report is not an audit and it is not generally developed by any performance audit approach. It is mainly based on data available publicly, and this is going to be our second information report following another information report published two years ago on Australian expenditure on procurement. Probably 90% of this information report work will be relying on data analysis work. We really look forward to share this with the public later this year.
Conor:We've spoken there about the challenges, and we've talked about some exciting upcoming projects using data. What about your organization's overall future ambitions for data analytics? Where do you see that going?
Xiaoyan:At the moment, our office is in the process of bringing Microsoft 365 into our IT environment. It will come with enterprise tools for our auditors and analysts to use for data analytics and visualization functions. This will hopefully equip our auditors in general with technology for self servicing analytics solutions and our System Assurance and Data Analytics group will be focusing on providing more advanced analytics support and standardized solutions for our auditors.
Christine:I think going forward performance audit really needs to think more creatively about when and how to involve data analytics and SADA in its work. Both in terms of improving the efficiency and the effectiveness of what we're doing. The onus, I think, is on performance auditors to integrate data analysts into their performance audit team better. Such that the data analysts truly understand the audit objective, the audited entity and the business of that entity. And a large challenge is finding a way for the performance audit team and the data analytics team to work together well, because there often, we find, can be a bit of a chasm between the two teams that's difficult to fill because the performance audit team may not have a good understanding of the principles of data extraction and preparation. And conversely, the data analytics team can lack knowledge of the audit objectives the auditee or the business area. So I think that's one of the biggest hurdles that we face is just bringing those two teams together to work effectively together. Recently the ANAO moved from assigned desks to activity-based working arrangements to try and facilitate that cross collaboration between the different teams physically locating them near us by ensuring that the data analytics team is involved in key meetings and conversations right through the audit lifecycle, not just brought in mid audit. So those are some of the things we're trying to do to increase that collaboration between data analytics and performance audit.
Xiaoyan:And our next goal is really to identify if there are any opportunities to improve the performance audit efficiency by using data by starting to request similar data early and shortening the time on data discovery and acquisition. Another opportunity is to use some of the high value, common data sets which is potentially relevant to improve our efficiency in building that understanding and running analysis.
Narrator:The Assurance Show is produced by Risk Insights. We work with performance auditors and internal auditors. Delivering audits, helping audit teams use data and coaching auditors to improve their data skills. You can find out more about our work at datainaudit.com. Now, back to the conversation.
Yusuf:Open data and improving the quality of data that is available. What role have you played as an audit office in helping to improve that level of data quality?
Xiaoyan:We did two information reports and following our first release on our reporting , We can definitely see improvement in the data quality in the Austender data. I think the maturity of government agencies is improving in producing and preparing data over the last few years. Most government entities have a data strategy in place and as they are implementing data governance on how they prepare and publish their data. Through our information reports we're promoting and advocating for better quality.
Yusuf:You mentioned that there's opportunities for performance auditors to use data more directly as opposed to asking for everything from the central team. Do you have a view as to, where the gap has arisen? So where it is that our use of data as auditors has not progressed to the extent that maybe it should have and now we almost have to play a little bit of catch up in building those skills within audit teams?
Xiaoyan:This is a very good question. The way I say this is you need to vizualize a matrix. On one axis you have the capability of the audit teams and whether auditors already have analytic skills. And the other axis is the complexity of the analysis. Some of the typical statistical analysis can be done using Excel or visualization tools. That can definitely be handled by the audit teams. And there is no question about their ability to do that. But on the other side, these days because we use more and more population data from the IT systems, sometimes we are talking about millions of records. If the data is not really usable that's where my team comes in to transform and prepare data for the audit team to use. And on the other side Sometime it's not about their capability whether they can analyse data or not. It's all about efficiency. If we can do the same thing much quicker. And in our view, that's where you need to bring an analyst in rather than using traditional Excel to handle a large volume of population data.
Christine:One of the other ways in which I think SADA and performance audit will work together in future is identifying opportunities for using data. That might be an area where performance auditors need more support. They may not necessarily understand, or be able to visualize how data can be extracted and used. So that sort of collaboration right from the outset of audit planning is really important, so that SADA can help performance auditors, identify those opportunities.
Conor:You mentioned there identifying those high value data sets that maybe can be used for multiple projects. And the audit teams we speak to a lot of them are going through that same process or have done that. What are some of the other tips and traps that other audit agencies should consider when using data?
Xiaoyan:With the publicly available data sets, the quality of them really varies. For auditors, you really need to make sure the data has a high quality, is complete and accurate. Do you have sufficient information for you to verify the quality of data before you can use it? If we talk to our colleagues across, the audit sector, there may be something we can learn from each other.
Christine:I have five tips for performance auditors. The first one is to remember that a huge proportion of the data analytics time for performance audit is going to be in data preparation and budgets and timeframes need to take that into consideration. Secondly, I believe that the quality assurance and validation of the data needs to be the first priority and it needs to take place prior to any analysis that's done. So establishing face validity of the data early on in the audit process is really essential. I'd also recommend maintaining a single source of truth. So trying to integrate all your data into a single linked data set will be the key to avoiding error and other inefficiencies in using that data. Maintaining full records of all data preparation and analysis procedures to enable quality assurance work and internal audit. And then finally just the point I was making before, thinking about how you can fully integrate a data analyst into the performance audit team as early as possible in the audit lifecycle.
Xiaoyan:Last one is if analysts are not auditors, we need to make sure whoever used the analysis, has the capability to interpret the data properly.
Conor:Thank you, Christine and Xiaoyan for your time. A few key takeaways for me. When you're accessing data for your projects, get your request into the entity early. The second thing was quality. Do not skip any steps around quality because it is so important and will pay dividends in the end. Thirdly, when you get to your analysis. Make sure you've documented your logic and that it's reviewable by a peer or by a quality assurance person. And lastly, and this was something that came through quite strong was collaboration as between the data professionals and the performance audit professionals need to get that collaboration happening as early as possible at the start of every project. Great conversation with the ANAO. Thanks again.
Xiaoyan:Thanks to both of you.
Narrator:If you enjoyed this podcast, please share with a friend and rate us in your podcast app. For immediate notification of new episodes, you can subscribe at assuranceshow.com. The link is in the show notes.
.jpg)
.jpg)
