Testing in Power BI projects

This week we discuss an often neglected part of many BI implementations: testing.  It isn't glamorous, it isn't the headline-grabbing side of a project, but it is vital.  After all, what use is a report if it doesn't show the right information.

We talk about different approaches to testing, from structured tests following a script to unstructured exploratory testing.  We discuss why both are important elements of a rounded test plan, and why it should never simply be left to the developers to test their own reports.  And to be clear - we are not suggesting development teams would cynically pass on reports they know to be broken, but rather that when you have been working on piece of code for weeks, it's hard to see where errors might be creeping in.

Finally, we talk about non-functional testing.  This is probably the most neglected part of an already neglected process.  It is vital that a report performs well, has readable fonts and suitable colours.  If any of these aren't the case, the report consumers will quickly be turned off and revert to other ways of reporting data.

You can download Power BI Desktop from here.

If you already use Power BI, or are considering it, we strongly recommend you join your local Power BI user group here.

To find out more about our services and the help we can offer, contact us at one of the websites below:
UK and Europe: https://www.clearlycloudy.co.uk/
North America: https://clearlysolutions.net/

Transcript

Andy:
Hello everyone and welcome to the Clearly Podcast, Episode 4: Testing in Power BI Projects, recorded on Thursday, January 14th, 2021.  Today, we discuss the crucial, yet often overlooked, subject of testing Power BI reports and dashboards. Why is it important? Why does it get missed? What approaches can ensure proper testing? Let's dive in. How are you doing, guys?

Tom:
I'm good, thanks. How about you, Andy?

Andy:
I'm doing well, thanks. Let's talk about testing. It's a key part of any project, but it often gets neglected. Why is that?

Tom:
Testing isn't glamorous. People don't like going through tedious checklists and test scripts.

Shailan:
Exactly. Projects often start with a test plan, but as iterations continue, testing gets pushed further out. By the time you reach it, people have lost interest.

Andy:
Yes, testing often gets cut for time. And then projects suffer because the numbers are wrong, or no one knows how to use the report.

Tom:
A lot of testing is unstructured. People need proper test plans that ensure they check if numbers are correct, filters work, etc.

Shailan:
Right, and users often see the report for the first time during testing. Without a plan, they just click around and might miss critical errors.

Andy:
Testing with real data is important. Many users can't understand or work with test data effectively.

Tom:
True. In greenfield projects, where everything is new, you must use test data. But even then, you need to ensure the test data reflects real business scenarios.

Shailan:
When testing a dashboard with lots of filters and slicers, challenge users to try and break it. This approach ensures thorough testing.

Andy:
That's a good strategy. Structured test plans are vital, but so is exploratory testing, where users try to find unexpected issues.

Tom:
Indeed. Some users are great at following plans, others at breaking things. Use both types to ensure comprehensive testing.

Shailan:
Testing should also involve multiple environments: dev, test, and pre-prod. Even when something is promoted to production, perform final tests to ensure everything is correct.

Tom:
Not every organization has multiple environments, but at the very least, have separate development and live environments. Always test in both before final deployment.

Andy:
Another dev testing your work can be very insightful. Fresh eyes can spot issues you might miss after looking at the same report for too long.

Tom:
Yes, it's fun and constructive to review someone else's work. It improves quality and catches errors early.

Andy:
Time allocation for testing is crucial. Testing must be a priority, with dedicated time blocked in calendars, enforced by management if necessary.

Shailan:
Yes, and asking testers to sign off on their work increases accountability and ensures thorough testing.

Andy:
Recording testing, bugs, and issues is important. Use tools like JIRA or custom solutions to track testing status and feedback.

Tom:
On complex projects, consider printed test scripts that testers sign and date. These can be scanned and attached to tracking tools.

Shailan:
And differentiate between bugs and change requests to manage feedback effectively.

Andy:
Non-functional testing is also important. Ensure reports are usable, fast, and readable. Check for performance issues and accessibility, like color blindness considerations.

Tom:
Absolutely. If reports are slow or hard to read, users will revert to tools like Excel. Usability is key to successful adoption.

Shailan:
Testing should start during development with peer reviews and continue through to production. It’s an ongoing process, not just a final step.

Tom:
Don’t neglect non-functional testing. Ensure reports perform well and meet user needs beyond just correctness of data.

Andy:
Use a structured system to record testing and allocate dedicated time for it. These steps are crucial for successful projects.

Andy:
Thanks for joining us today. We hope you found this discussion on testing useful. Contact us at clearlycloudy.co.uk for UK and Europe, or clearlysolutions.net for North America. We'll be back on February 8th, 2021, with an episode on self-service BI. Stay safe and have a great week!

Tom & Shailan:
Thank you!