Welcome to the Dashboard Effect Podcast. I'm Brick Thompson, and today I have with me Courtney Tewalt, who is Blue Margin's Quality Systems Manager. How's it going, Courtney?
Courtney Tewalt:It's going well, I'm very excited to be here.
Brick Thompson:Yeah, I'm excited to have you here today, too. We're going to be talking through some of the challenges around delivering quality BI. But I was thinking we might start with you just telling me what the Quality Systems Manager position does.
Courtney Tewalt:Yeah, so at Blue Margin, I get the privilege of helping create different quality systems across our departments. Right now, initially, I'm focused within our Delivery Department for our Data Visualization team and our Data Architecture team, just making sure that we have good processes in place to ensure that we're developing and providing consistent quality on all of our solutions that we're sending out the door.
Brick Thompson:Okay, so it's both managing systems and managing quality and measuring that, plus developing new systems and processes to keep making us even more efficient and better at that.
Courtney Tewalt:Yeah, definitely.
Brick Thompson:And your background... you've been with Blue Margin for a lot of years.
Courtney Tewalt:Yep.
Brick Thompson:And you were with us when sometimes our visualization engineers, were also doing data architecture. So you have some pretty deep knowledge. And at the point that you moved into Quality
Courtney Tewalt:Yep. Systems Manager, you were one of our most senior data visualization, engineers. And so you bring a ton of good experience to this. So I'm looking forward to talking through this with you. Yeah, thank you.
Brick Thompson:Well, why don't we start with sort of where.... Let's start with just sort of Blue Margin centric, where we started, where we are, where we want to end up. And then I also want to talk about just advice for people... let's say you're an a company, you're either one person or a small BI group, or even a larger one, you know considerations for them. But we'll come back to that. Let's start with sort of where we started and where we are, where we're going.
Courtney Tewalt:Yeah. So first starting with Blue Margin, we didn't actually have a lot of systems in place for quality. Our goal and aim, of course, was to get something excellent out the door for all of our customers. And that was primarily controlled through some internal reviews that you would have with your manager before you presented the final report on a call to the client. And you just kind of hoped that your manager would spot a misspelling, or help you correct some colors if they weren't looking good. But that's kind of the majority of what we had put together.
Brick Thompson:Yeah, and I can remember back, you know, eight, plus years ago, it really came down to the diligence of the engineer who was working on it, and hopefully doing a good job and catching things before we go to the client. And the client actually pointing out quality issues.
Courtney Tewalt:Yes, yes. Which is not a place you want to be.
Brick Thompson:No. And you know, we worked hard on it. And we have smart people here. So it mostly went pretty well. But we definitely learned a few things, School Hard Knocks wise.
Courtney Tewalt:Yeah definitely. Yeah. And that's gotten us to a lot of where we are now, and even helped us envision where we want to be.
Brick Thompson:Alright, so tell me about where we are now. How's it different from sort of the lone engineer being super diligent? Or even, you know, hopefully getting a good review from their manager?
Courtney Tewalt:Yeah. So I think a few things that I just want to touch on is what we have taken with us as we've continued on our iterations. So back in 2019, we read The Checklist Manifesto, by Atul Gawande.
Brick Thompson:Yeah. Great book. I loved it.
Courtney Tewalt:Yeah, it was excellent, because it helped us think about what are some of the main things that we as engineers can be consistently checking, before we get to that internal review that we have with our project team.
Brick Thompson:Right? So rather than having to try to remember 10, or 20, things you want to look at, you actually go through a checklist and get all those. Yeah, that's right. And that's made a huge difference. You made
Courtney Tewalt:Exactly. We incorporated some peer reviews, which just can really help with getting that higher 10,000 foot view when you're so used to being down in the weeds. So those are some of the things that we've taken with us. And now, you know, in early... in late 2020 or early 2021, we actually ended up adding quality as a core value here at Blue Margin. And along with that, built out our dedicated quality department. a point about peer reviews. I do think that, you know, getting a third party eye on things is super helpful, no matter what systems you have in place. Having someone who's not sort of deeply steeped in the thing you're working on makes it easier sometimes to spot things. But before you go to that person, and in order to be efficient, you want to have really covered your bases and scoured it. Yeah, definitely. So that's where we can still utilize those checklists. But now with that dedicated quality department, we have a dedicated person, myself, who gets an opportunity to look at the data warehouse solutions that we're developing, to get a look at our report builds that we're putting out there and do a quality check to make sure that the things that we've identified as standards are being followed consistently across our reports,
Brick Thompson:Right. And the quality department is, you at this point that isn't, and that, you know, you're there. We're a company of 30 plus people, and a good chunk of those are engineers. But at some point, we may need more people doing that, doing the quality position. In order to do that, there's got to be good process and good consistency about how we approach that.
Courtney Tewalt:Yep. And one thing that we've found as this, as the quality department has grown and developed, is we still have our internal review process with our project team. But our project team is able to focus in on, "Are we meeting the needs of the client?" Before there was this behind-the-scenes, back-of-the-brain thought, "I need to make sure these visuals line up across all the pages. I want to make sure that sales, which is displayed on this page and is blue, is also blue on this page." And so you were unable to really put yourself at that client level and make sure that you could focus on, "Are we meeting the needs of our client?
Brick Thompson:Yeah, the goals of the report? You're so worried about getting it technically right, getting the numbers right, getting it lined up, getting it looking consistent, that it didn't leave as much time as you might like to be focused on, "Alright, does this meet the business goal that we set out to meet?" And obviously, that's sort of the bottom line. You want to you want to achieve that. Okay. So where do you see us going? What's sort of the next evolution in terms of quality?
Courtney Tewalt:Automation, without a doubt. And we've made some great strides towards that already. Data Architecture, for example, the ETL reviews, or the solution reviews that I used to take, would sometimes take me an hour plus, just as I'm looking through some of the data, trying to make sure naming conventions are consistent, reviewing the different ETL fact views that we've written. But duet to some really smart people here who've helped me build out some queries and have tested things alongside of me, those reviews can take me now 15, maybe 30 minutes, depending on how much work has been done kind of week-over-week. And previously, for naming conventions, I used to kind of have to look down and trust my own eye, which is not necessarily the way you want things done within a system. But now I rely on conditional formatting within Power BI, to really identify, this is not in sync with our process and our standards that we've laid out.
Brick Thompson:Yeah. Okay. I love that. And so I'm assuming at some point, then you can push the use of those automated tools to the actual developer.
Courtney Tewalt:Definitely.
Brick Thompson:And then you take the final human review stuff that you're doing, you just take that to an even more valuable level, because you've covered all this stuff in an automated way.
Courtney Tewalt:Yes. And I love the point, though, about getting it back to the engineer, because that is a goal, where I would love an engineer to be able to open up this report, this automation, run it and be able to check their work whenever and not just on a scheduled quality review.
Brick Thompson:Yeah, okay, great. So let's assume you're working in a company where either you're the single BI person or you're part of a small BI group, three or five people, I guess, even potentially a larger group, what are the types of things you would tell people to do to help them improve their quality? Let's say they haven't gone through some of the school of hard knocks that we have already to develop their own systems. They will over time. But what are the one of the first things they can start thinking about?
Courtney Tewalt:Yeah, that's a great question. What initially comes to mind is spend some time thinking about what really matters in quality for you and the work that you are providing.
Brick Thompson:Okay, what do you think really matters?
Courtney Tewalt:For me, over the years, I've been able to narrow it down into, I care about consistency, and I care about attention to detail.
Brick Thompson:Okay, and so give me a couple examples of those.
Courtney Tewalt:So within inconsistency we're looking for,"Are using field names the same on all of your visuals?" So here at Blue Margin, we use Pascal casing within our data warehouse solutions. So we have client name, which is a capital C, no space capital N. That's not something that you necessarily want to surface on report for a client. You're going to want to alias that to just client or even client-space-name.
Brick Thompson:Okay, so in Power BI if you name if you name a column or name a DAX measure with that Pascal casing, that's what's going to show up on the report visual by default. But you can go back and put an alias to use a different name to make it easier and cleaner. And then in terms of consistency, you want to make sure use that same alias everywhere.
Courtney Tewalt:Yes, because at least currently, the alias is visual to visual. It's not an alias that you set permanently, and automatically gets applied to all your visuals. Okay,
Brick Thompson:So you have to really be careful that.
Courtney Tewalt:Yep.
Brick Thompson:I know, something that I've seen you really care about in consistency is that things line up tightly. Do you use a snap to grid? Do you not care? How do you how do you make sure that's good?
Courtney Tewalt:Well, you open up general properties, and you look at the X, Y axis and you make sure those are the exact same data points.
Brick Thompson:Okay, so you go to the exact pixel basically. All right, good. And then how about colors? How do you think about colors?
Courtney Tewalt:Yeah, colors are so important. And they can help ground a report user in the data that they're seeing, not just on one report page, but across an entire suite. So if you have sales as a blue variation on your top level summary page, and then three pages in sales is purple. There's some disconnect, there's some retraining that you have to do in your brain as a report user, to Okay, sales, sales is blue sales is purple. But if sales is consistently blue, across each report page, maybe even across a suite of reports, your brain just gets accustomed to. I see that blue. I know this is my sales metric.
Brick Thompson:It's way more intuitive that way. Yeah. All right. Well, speaking of colors, Caleb and I talked about this in a podcast recently, but we were saying, avoid using common KPI colors, usually red, yellow, green, avoid using those colors as sort of general colors on a report. How do you think about that?
Courtney Tewalt:Yeah, I'd say in general, I agree with that. However, there might be a really great use case, for using green and an orange or a red variation to get a point across with the data.
Brick Thompson:What's an example?
Courtney Tewalt:So I built a report once where we were showing a combination of total collections and total payments. Collections in general, are a good thing. So I painted those as a shade of green, and collect... or sorry, payments, were shaded as a shade of orange. Because that helps you without necessarily having to look at the legend and get super tied with what you're looking at. You were able to narrow in on, "Oh, yeah, my green number. This is good. This is my collections. And my orange number, these are my payments that are coming out."
Brick Thompson:Okay, so where it makes sense to help the user gain meaning, go ahead and use those KPI colors. But am I also hearing you say, you know, if it doesn't, don't use those except for KPIs, or am I putting words in your mouth?
Courtney Tewalt:No, I think that's exactly right. And even traditionally, I would maybe even try to find other colors for collections and for payments, but it worked really well on this report.
Brick Thompson:There's another area of consistency, that's sort of a pet peeve of mine, which is use of filters and slicers, and making sure as I go from page to page, or even report to report in a suite, that those are used consistently, or it can make it... just makes the user have to think too hard about how to use it.
Courtney Tewalt:Yeah, definitely. You want the order of your filters within the Filters pane to be in a consistent order, but also in one that makes sense. So if you have a hierarchy of your division goes into a region, goes into a specific location, that's the hierarchical order that you want it to be displayed on within your filters pane, or the visuals on the side of the page.
Brick Thompson:Yeah, that makes sense. And then I also, I mean this gets really into the weeds, but I think it's important that your cross-filtering is following some kind of logical structure and so that you're using that same type of structure from page to page. So it's not sort of a surprise or a hunt and peck for the user. And the same thing with thumbnails, not thumbnails, tool tips sorry, yeah. So that when you to hover over a data point, and you get additional data, that those are set up in a consistent way, so that's not surprising. And I've had the experience of going into a report and by mistake hovering over something, and getting a tooltip. I was like, "Oh, wow, I didn't know that was there." But once you know, they're there, hopefully, they're all working about the same way.
Courtney Tewalt:Definitely.
Brick Thompson:How about when it comes to the data side? So we've been talking about visualizations, which is, you know, really what the users end up engaging with. But I know part of your quality management has to do with the data, and making sure that it's correct and consistent and up to date, and so on? What are the things you consider there?
Courtney Tewalt:Yeah, so a lot of this has been put together by our Director of Architecture and our leads within that department. But the things that I help them check for and make sure are consistent across solutions, are that we have primary key identities assigned across all of our tables. We're looking for ensuring that we have ETL import and modified dates where it makes sense, because that can help with a lot of troubleshooting or report building down the road. And then we're also looking in our ETL facts and making sure that within those processes, which is where we're joining in additional data, that we're not introducing any sort of row suppression or row expansion, that is not expected. Sometimes you might intentionally blow out a table to make sure you have a row for each day, for example, but a lot of times you want your base table and all of your joins in the same row count.
Brick Thompson:Okay, and so I'm sure that we that you have standards that you follow there as well. Yep. Great. All right. When we were thinking about this podcast, you thought of a particularly painful experience that you had that really helped sort of catalyze and make us realize, "Okay, we need to devote a full position, a full department to quality." Is it too painful, or do you want to recount that a bit?
Courtney Tewalt:Yeah, I mean, it's, it's always challenging when the client is the one consistently giving you the feedback of, You have low attention to detail," because the number in this table doesn't match the number displayed in the KPI card at the top of the page. Those are things that we should be finding before a report ever gets out the door. Spelling, different details like that kind of getting out the door. It can be embarrassing and cause a nice black eye. But, you know, from that, we learned a lot and we grew a lot and it helped us focus on, "What do we need to consistently check versus what do we need to have as individualized training, maybe some better checklists, to just really make sure that we are consistently delivering quality?"
Brick Thompson:Yeah. And I think it goes beyond the black eye. I think one of the challenges, especially when you're rolling out BI, to an organization that hasn't had a lot of it before, is user adoption. And every time you have anything that's slightly off, it will cause a question mark for the user about should I trust this? Do I want to use it? And so you really want that first draft to be spot on. And I can tell you, since well, actually, for a number of years, but especially in the last couple of years, since we've had a dedicated department, I mean, every one is. I never hear of quality issues. And so I never hear about adoption issues. And I don't know that those two are exactly linked, but I'll bet you they're pretty tightly linked.
Courtney Tewalt:Yeah, I hope there's some correlation there. That would be great. Great to see that. We also have, Brick, you've alluded to this. We have an exceptional team who own the quality of their own solutions as well. And that's really valuable.
Brick Thompson:Yeah. Yeah. I mean, you have to have good people. All right, I think I asked before, and then I took you off track. But if you were if you were managing, let's say, a small BI team, and you don't have the luxury of having a quality systems manager, how would you approach that with your team? How would you approach getting them on that quality mindset? Is it just thinking about the details we've been discussing? Is it that simple? Or do you recommend that even early on you start developing checklists and peer review or almost code review type meetings, that type of thing?
Courtney Tewalt:Yeah, it's definitely about the journey and going through those processes of creating the checklists, enforcing those peer reviews, getting to the point where you have someone who can do some standard reviews, even if it's multiple people. Because again, through that, you're going to really figure out out what matters and you're able to attach to, "We have a consistent problem where we are not aliasing the same across all of our visuals or even across all of our reports. Let's do some training on this." And you can see that as a team, but yeah, just starting the conversation and really having it be a value that you own.
Brick Thompson:Yeah. Okay. So key takeaways.
Courtney Tewalt:Yeah, I think, of course, I said it just a little bit ago, but I think quality is a journey. And it's an exciting one. To kind of see where we've been, to see where we are, to envision what automation could look like in the future. That's something that is exciting for us here at Blue Margin. And it's hopefully exciting for the small BI team, or the large BI team that might be listening to this podcast.
Brick Thompson:Yeah, it's so satisfying when you deliver super high quality solutions, and your customers are loving it. And you're spending zero time talking about something that's not consistent, that type of thing. So yeah, I agree with you there.
Courtney Tewalt:Yep. And then I think on top of that is just keeping a flexible mind of what needs to be consistently checked versus what can be resolved through a couple of trainings. Those might change throughout the years. And that's good. Because as your team develops and knows the things that they're looking for, maybe that's not necessarily something you keep on the checklist, but you get to add another piece to be just continually evolving how you provide quality.
Brick Thompson:Yeah, makes sense. All right. Well, this has been a great discussion. Really appreciate you coming in. Really appreciate what you've done for us to raise our game the way you have. And I'm sure we'll be talking again soon.
Courtney Tewalt:Yeah, thanks Brick. Had a great time.
Brick Thompson:All right. Bye.