How Quality Testing Drives Business Growth

Escaped bugs can break more than your code—they can break the bank, too.

In this episode of Test Case Scenario, Jason Baum, Evelyn Coleman, and Marcus Merrell break down the connection between software quality and business impact. They talk about how to approach postmortems, why some features matter more than others, and how technical teams can align their work with business goals. 

You’ll also hear real-life stories of bugs that cost companies millions and learn practical tips for balancing technical needs with business priorities in industries like retail, finance, healthcare, and government.

Join us as we discuss: 
(00:00) Introduction
(01:26) The economic impact of testing and quality failures
(02:30) Bridging the gap between technical and business priorities
(04:18) Examples of costly bugs and lessons learned
(06:50) Building trust through proactive testing strategies
(09:39) Testing challenges for retail, finance, and healthcare
(13:17) Prioritizing workflows and key takeaways for testers
(20:38) The importance of accessibility and integration testing
(27:47) Looking ahead to 2025

We’d love to hear from you! Share your thoughts in the comments below or at community-hub@saucelabs.com.

SUBSCRIBE and visit us at https://saucelabs.com/community to dig into the power of testing in software development.

Sign up for a free account and start running tests today at https://saucelabs.com/. 

▶ Sauce YouTube channel:  / saucelabs  

💡 LinkedIn:  / sauce-labs  

🐦 X: / saucelabs

Transcript

Jason Baum [00:00:00]:

This is Test Case Scenario with me, your host, Jason Baum. This podcast is the definitive hub for knowledge and stories in the software testing and development communities. If you're new to the channel, hit the subscribe button and let's dive straight into the episode. Hey, everybody. Welcome back to another episode of Test Case Scenario. I am your host, Jason Baum, and with me are my co-hosts, Evelyn Coleman and Marcus Merrell. Good to see you both. Thanks for coming back.

Jason Baum [00:00:39]:

No, seriously, thanks for coming back and not leaving me alone. Marcus, looks like you're at home.

Marcus Merrell [00:00:45]:

I'm at home.

Jason Baum [00:00:46]:

It's great. Not on the road. Not calling from a telephone booth or airport or the car pulled over on a highway in Texas somewhere. And Evelyn, you're in a very nice living room. Love the fireplace next to the grand piano.

Evelyn Coleman [00:01:02]:

Probably shouldn't put your piano next to the fireplace or next to the window from a tuning perspective. Marcus agrees with me. You've got two pianists on the podcast.

Jason Baum [00:01:12]:

I played piano when I was five. Okay, we're in a weird mood today, so this is going to be a good episode where, Evelyn, why don't you take it away? Why don't you tell us a little bit about the topic and then we can dive right in.

Evelyn Coleman [00:01:26]:

So today we're talking about the economic impact of testing and quality. Throughout the years that we've been hosting this podcast and speaking out around quality, we have heard stories and talked about stories where escaped bugs have cost millions, billions of dollars. I think this last year, we've had seen firsthand some of the tragedies of quality that have occurred and what goes on there. So you definitely want to recap some of that. And then going into sort of the new year, there's some startups and smaller companies where you'll have a chance as engineers and folks invested in quality, to speak to counterparts in the revenue side of the business and to learn more. So we want to talk about who you should talk to and who you should bug about, about quality.

Jason Baum [00:02:24]:

Who you should bug about quality. I love that. That's awesome. All right, so, Marcus, where do we begin?

Marcus Merrell [00:02:30]:

Sometimes it's difficult to get folks who are technical to understand why it's important for them to embrace the business side of things. Technical people, as we've seen in a lot of, a lot of examples, tend to think that the tech will solve all the problems and that we don't need these people in the middle between us and the customers and that the product should just sell itself or they just don't. Don't think about It, I don't mean to paint with a broad brush, but I've been around a while, seen it, and I think that it's just important to understand that without sellers, you can't get your product to market. Without someone to connect you to the customer, it's not easy work. And they have a lot of things that they're juggling and there's a lot of stuff that they're being asked to be in control of that they are not in control of. And so it's really good to empathize and understand what they go through. In order to tell your brilliant work to your brilliant customer, you need these folks and it's really important to understand why. So I think we wanted to talk today about like sort of sell you on embracing sales.

Evelyn Coleman [00:03:27]:

And I think it's easier if we talk about it from sort of either a retail perspective or a business to business perspective. It might be a little harder to make the sort of mental jump between quality and revenue if you're working on quality for an internal tool. But it's a little bit more obvious when we talk retail and it's a little bit more obvious when we talk business to business as well. Marcus, you've had experience on the retail side of things, or at least the couponing and discounting side of things, where bugs could have potentially cost millions of dollars. Putting yourself in the shoes of someone trying to sell that type of product or trying to make the case of that kind of product after an escaped bug like that, what kind of conversations would you have?

Marcus Merrell [00:04:18]:

It's difficult when they have sort of a crisis of confidence that that happens as a result of a big bug like that. But there's a good story around, hey, that we, we figured out that that happened and we now know how to make sure it doesn't happen again that, that kind of thing. But ultimately I, I think it's. As an engineer, it's really important to understand why it is so critical to make sure that the, the fundamental nugget of the transaction can always work. If you're tracking analytics, you're tracking cart checkout, stuff like that. I think that folks in tech tend to treat all bugs and all features kind of with an equal weight. That's once again broad brush. But I've seen this happen where I'm working on this feature.

Marcus Merrell [00:04:59]:

So I'm going to work on this feature the same way I work on this other feature. And you know, if I don't have unit testing, I don't have integration testing in place. I might not Understand what this thing I'm working on right now might break in the other side. And for sure, folks in sales don't understand those kinds of deep interrelations. And so getting engineering to understand that not everything is equal and some things are far more important than others, and then getting sales to be confident that you are always focused on those things, even if you may not be working on them right now. You know, it's. It's two ways. The empathy has to go two ways.

Marcus Merrell [00:05:33]:

But what I find often is that there's not enough communication. And what Evelyn's talking about is really having an opportunity to get to understand the minds and motivation behind what. What drives people's actions.

Evelyn Coleman [00:05:45]:

Yeah, I definitely see that in terms of, like, the after effects when it comes to postmortems. Right. So if you're doing business to business or retail, and you're a salesperson and you're accountable to customers around these escaped bugs, oftentimes in order to regain that confidence, customers will ask for postmortems, which will then need the engineers or the quality folks to come back and say, like, what kind of tests are we adding? What are we doing in the future to prevent this? And depending on the feature that was affected, it can really be like pulling teeth. And I think this goes back to what you're saying marks about, like, folks not necessarily understanding that different parts of the product, different features have different weight when it comes to selling the product or helping with that sort of quality confidence. From a revenue perspective, have you ever been in a position where you're being asked lots and lots of questions to go over about something that had happened, but you were like, I'm totally ready to move on. I need to get developing. I need to go work on other tests.

Marcus Merrell [00:06:50]:

Yeah. And what I also find that I think tech people don't sometimes understand is that you'll only get that kind of drill down if they don't think you're on top of stuff. If you have had a history of delivering product that, that is, you know, the right product at the right time, then if you have a mishap, then they'll kind of tend to believe you that when you say you're on top of this. But, but if you kind of answer with the kind of like, it depends or a lot of equivocating or a lot of dancing around it, then they'll start to ask more and more questions and drill down, and they'll ask for a level of detail that, you know, they aren't actually going to understand the answers to they want to measure your confidence in how you're delivering it and make sure that you're kind of on top of things. And that'll only happen if they are not confident that you are on top of things. So if that makes sense. Like to me, that's one thing to understand about the business mindset is people tend to trust each other if history has been good.

Marcus Merrell [00:07:47]:

Not always true with engineering, but so interesting.

Evelyn Coleman [00:07:50]:

Is there anything you can do preemptively in your tests from a technical perspective to help smooth or help those conversations go faster, like comments in certain parts of tests or making sure things are labeled certain ways? Have you had any luck with being able to sort of pump out those answers more quickly as to what happened by doing some kind of preemptive work on the quality side?

Marcus Merrell [00:08:18]:

Well, what you're talking about around comments and logging, to me, I see those as tactical. To me, what has helped the most on the business side is strategic. Where I, I think we talked about this a few times before, but I think it bears repeating is that if I put myself in the, in the mind of the CEO reporting to shareholders, what does he care about? And I think, you know, for our case in the B2C area, it was that, you know, revenue is growing. Yes. But also revenue is accurate and we can predict and we can actually measure. And the most important bugs I found in the most important tests I wrote were some of the simplest where I just sort of said there are 1400 ways for the single event of having a cookie drop. There's 1400 ways to trigger that. I need to make sure that every one of those is covered and that I know the cookie is always being dropped.

Marcus Merrell [00:09:06]:

And I almost feel like you could almost skip the rest of the testing of the product as a whole if you can guarantee that event is happening because that means revenue is flowing if that event is not happening, or rather if you are not testing whether or not that event is happening and you're spending all your time in another area of the product that doesn't matter as much, then CEO will not care that you've spent time making sure that that part works. They will only want to know, does the fundamental nugget of our business actually function every time?

Jason Baum [00:09:39]:

Yeah. Well, on the making every transaction count, is it the same for all verticals? Let's move on to like banking, for example, financial institutions. Is it the same, would you say, for financial. What's different?

Marcus Merrell [00:09:53]:

I do think that there is an answer. It's going to be sort of different for every team. Every product. And I'm not going to have a pat answer. I think we could sit here and brainstorm. If we're working on a bank, what's the most important thing? And then the second and the third most. For if you are, if you are a consumer retail banking customer, what is the most important thing is that you. What, let's just, let's speculate.

Marcus Merrell [00:10:15]:

I'll say what I think and you see if you agree. As a customer, I need to understand what my balance is, what transactions have happened recently and that, you know, it's all accurate and being calculated correctly. Right. So fundamentall,y you don't even talk about it.

Evelyn Coleman [00:10:32]:

So it be calculations and it would be daily versus where I think retail you could think about load because they have holidays and times where things are being sold, times of the year where there's going to be really big spikes in usage are going to be different across regions. So things like GPS testing, language and all of that stuff.

Jason Baum [00:10:55]:

Tax.

Marcus Merrell [00:10:56]:

Right. Tax software. Yeah.

Evelyn Coleman [00:10:58]:

Banking and finance, like the numbers need to be accurate but I think it's easier to understand even if the language is a little bit off what you're looking at and just needs to be accurate every single day. Every time you open up the app and it needs to refresh a lot. Like it needs to be updated right now. You can't have a lag. Different types of testing would be the focus for me rather than like different happy paths within the app.

Jason Baum [00:11:28]:

Yeah. It's interesting you say it's like kind of like not easier but I get. Did you use that word easier or for, for financial?

Evelyn Coleman [00:11:37]:

I just said like less. There's less emphasis on maybe some things that are regional.

Jason Baum [00:11:44]:

Yeah.

Evelyn Coleman [00:11:44]:

Making sure that the numbers but the calculations for different types of money are accurate.

Jason Baum [00:11:49]:

Would you say the stakes are higher though for financial institutions? Just because if I open my bank app, I'm a consumer and my, my bank account is zero and I know it's not zero. I'll be pretty, pretty mad, you know.

Evelyn Coleman [00:12:08]:

Absolutely. I'm just saying like open my banking app and the whole thing's written in French.

Jason Baum [00:12:13]:

Yeah.

Evelyn Coleman [00:12:13]:

I'm not as worried about it. Versus like the tickets go on sale tomorrow. I open up the app and the whole things in German. I can't get around this app that I only use one time a year on this specific date. So I.

Jason Baum [00:12:29]:

Are you a Swifty, Evelyn? Is that what you're trying to tell us?

Evelyn Coleman [00:12:33]:

I was saying that like the numerical accuracy. Right. Making sure that the, the math is mapping to use like the Newspeak is important on finance apps, but like on a retail app, the regional, the branding, the language and the load I think would be really big, really high up.

Marcus Merrell [00:12:52]:

Yeah, I mean I, I would put. Still put cart checkout flow at number one. Number zero. Yeah, zero. Right. And then at my previous B2C role, we used to test to make sure that we could process incoming hits to the website. But we didn't always necessarily test the entire flow of everything. And then we figured out that we had some third party things that couldn't handle the load.

Marcus Merrell [00:13:17]:

We started to exercise some of those paths. So it's like not just us, but also, you know, one of the credit card processing companies or you know, cash transfer types of apps, but also back to banking. I'm thinking about like if you are looking at it from a consumer point of view, you always need to make sure that a user can transfer funds and that the transfer works, which means the whole workflow, external bank, internal account, everything like that. So I just, I think that everyone in the industry, everyone listening to this could write down a list of here are the five things that are absolutes that I have to make sure work. You could probably expand that to 10, but like there are going to be five. And then so to me, to answer your much earlier questions like that, to me that's how the tester can orient themselves towards the business. And I think talking to salespeople, talking to marketing folks at these kickoffs and annual meetings, it's one of the most critical things because you start to questioning a whole lot of your own assumptions about what areas of the product are most critical to most customers. Talking to these folks and saying, hey, maybe I, maybe I need to be testing this over here as opposed to the stuff I've been working on around, you know, like editing your profile or social media or something like that.

Marcus Merrell [00:14:27]:

So.

Evelyn Coleman [00:14:28]:

Oh, I completely agree. I think this is exactly the time of year to have these conversations and getting in the room, raising your hand, asking, you know, what are the top five things that drive revenue in our app, in our business, in our product, but also going a little bit deeper because people give that list and then, but if you could ask, just like we did, okay, well let's compare that to name a different app. Like how would those things be different? Is there anything we would focus on that such and such app wouldn't focus on? Dig a little deeper than you would. Don't just get the happy path if I know, okay, well customers are going to do this, this, this and this. Like think about what their Priorities are think about whether it's an app that they use every day so they have everything memorized, but the numbers need to be right, like in finance. Or is it something that they only use once a month, once a year? If you're making big changes, people aren't going to know how to navigate correctly if they don't know where the shopping cart is or the shopping cart's changed. Like, dig a little deeper than just asking, how do we make money off of our product? And how can I support that with quality?

Marcus Merrell [00:15:35]:

So let's do a little thought exercise now. How about a health insurance software company? Because I don't have a bad answer. I think it's worth the discussion. What would be their top couple items that are most critical to making sure that bugs don't escape to production?

Evelyn Coleman [00:15:50]:

Well, we've got open enrollment, right? So we've got the time thing, just like in retail, where there's spikes in usage, spikes and load. And then there's also going to be a lot of like a big push for changes right before open enrollment and maybe even a freeze before then. So something about not necessarily the types of tests that you're doing, but the rhythm of your software development life cycle and the rhythm of quality, making sure that you freeze things in time, but taking into account the time for testing in your quality strategy rather than just the time for development.

Marcus Merrell [00:16:29]:

So if you look at the suite of what's going on with a big health insurance company, there are doctors who need to interact with the system, doctors in administration, in the office, administration. Then there's the user, the person who works for a company who has a contract with this health insurance company. They're using the software, each of them is going to have slightly different needs. And so I'll throw out kind of a loaded question that you may already know the answer to. Like, how important it is, Is it for that employee whose company works with the insurance company, how important it is for the UI to be like sparkling, pretty, clean, beautiful. How important is that?

Jason Baum [00:17:09]:

I would say just for my own personal use, I want it to work like, I don't know if I care how it looks like. Claims, right? The claims center, I would say, is probably one of the top places people go to with like a health insurance, just the ability to submit a claim. I know from my own experience I've had it where the mobile app didn't work, so I had to use the browser or vice versa, and that's annoying. And people don't want to submit claims. It takes a lot of time. So shorten the process, make it easier. And I think some do it really well. And I don't think people care necessarily too much about how it looks, more about how it functions.

Evelyn Coleman [00:17:54]:

Yeah, I think on top of that, when the flows, like if you're walking through, like a clean flow, making sure the questions are relevant, like, it's the worst thing in the world when you select something and then it opens up a bunch of questions and one of them is mandatory. Like, you have to answer, but none of the answers relate to what you wrote. And you're like, am I going to lose my claim because I made of. And then at the very end, every time that happens, every time without a fail, there's questions like, did you answer truthfully to the best of your life? I did, except for that one question where you forced me to write something because the question was mandatory. So I would completely agree. Things like that. And then also support, like making sure that your Contact Us page, that the numbers and the links are all working. I think that's maybe something that doesn't get the attention that it should.

Evelyn Coleman [00:18:43]:

And sometimes I think that's intentional. Right. Like, we're going to put the quality of our Contact Us page down there because we don't actually want people to contact us. We're listening to this and you're doing quality for a Contact Us page or support part of an application. Please make it look nice. Please make it work.

Jason Baum [00:19:01]:

There's. There's also stuff behind the scenes that I was going to mention for financial. Maybe not as much retail. Well, a little bit with retail, but especially financial and especially insurance is data privacy and security, like those two things are crucial. Fraud prevention on both. Crucial.

Marcus Merrell [00:19:21]:

Yeah, that was a nice thing. At the retail situation that I was in where we initially didn't really do much with PII, personally identifiable information. We had email addresses and that was it. But as time went on, we added things like cash back, which meant that we had to have accounts and dollar amounts with each customer, which meant we had to start taking more information, which meant we had to go through a whole new set of compliance and all sorts of stuff in order to be able to interact. But it also generated revenue, so there was an ROI associated with it. But as a result of that, we started to have to put the black hat or the white hat hacker on and start to think, okay, well, now there's a whole new set of dimensions for the unhappy path testing that we need to do to make sure that we have this stuff covered. And it's funny because to your point about internal apps, otherwise, there was a whole huge set of apps that we had around content management to make sure that those offers got managed properly and ranked properly and pushed out to the right place. And that was a case where we wrote hundreds and hundreds of automated tests to test all that, make sure all that stuff worked.

Marcus Merrell [00:20:22]:

But I'll bet you only about a dozen of those tests were super critical to the actual workflow. There's an internal audience of about 10 people who use these tools, and if they're not super usable, it's not going to kill anyone. Much as we want to make them great, there's other areas to focus on.

Evelyn Coleman [00:20:38]:

So what about things like accessibility and like that testing the differences between that for like a banking app or healthcare app versus a retail app? Like, retail apps can have lots of pictures, but banking apps might have spreadsheets and really tiny PDFs that pop up that would be really hard to see just from like a visual perspective. And then nevertheless, I don't know some of the other things that we've talked about.

Marcus Merrell [00:21:01]:

Yeah, well, for healthcare and financial, you really kind of don't have a choice. It's going to be set upon you how to do accessibility testing or to what level you do accessibility testing. Retail is going to have a little bit more flexibility in it. But then again, there was a Supreme Court case around that pizza company who said, Supreme Court said if your site is not accessible and your people cannot order a pizza if they're visually impaired, it's the same as not having a wheelchair ramp on your brick and mortar store. So it is important, but it's not important for those internal tools we're talking about necessarily. Accessibility is hugely important. With government. The worst thing that can happen is if you're working on a, on a game or some other thing and your site isn't accessible, they're just going to go to a different product.

Marcus Merrell [00:21:43]:

But with government, there's no competition. You can't go to another place to get the job done that you need to get done.

Evelyn Coleman [00:21:49]:

What about integration testing for government? Because the websites tend to be a little bit older, they tend to change a little bit more slowly. Some of the UI elements tend to be a little bit more behind, but that doesn't mean that the libraries and packages and third parties that you're integrating with are that far behind. So you would need integration testing to make sure that even if you're a little behind, it's still keeping up to date with the latest and greatest in some parts of it, some aspects of it. I might just be. I don't know.

Marcus Merrell [00:22:19]:

No, I think you're right. I mean, honestly, this is where things like contract testing can help you, because you can, you can run a contract test against an external API that'll tell you whether or not, like, we used to do this, where we would run a really quick set of API tests against an external resource, a third party that we were testing against, and if their site was down, we just wouldn't run the tests and not. Not waste all that time, you know, something like that. Or if the contract had changed or was broken, we knew we needed updates and stuff, you know, that's why contract testing is huge, huge benefit to any testing strategy.

Jason Baum [00:22:53]:

It is our first episode of the new year. So first off, happy 2025. It's just so weird saying that number. I don't know. Every year of the 2000s, I'm like, how did we get to here? But something that some people like to do is a New Year's resolution. I thought we could end the show with, what are your New Year's resolutions? For me, I'll go first to give you two, because I've been thinking about this. So I don't want to have you guys be too on the hook. But I think for my New Year's resolution, I am going to try and go consistently for walks every day.

Jason Baum [00:23:43]:

I started introducing that into my daily regimen more often in 2024, and in 2025, I'm going to commit to doing it even in cold weather, because when it gets cold here, I'm like, do I really need to go for health food needs? It kind of, right? No. And then I pay for it come April when I'm like, man, I should have been walking all those months. So I'm going to commit to it. That's my 2025 New Year's resolution. And after this, I'm going for a walk. Okay, who's next?

Evelyn Coleman [00:24:23]:

Well, so for this year, I was trying to get more movement in as well. I always look to the right because I have a big 300 subscribers, a calendar next to me, and I put stars whenever I do 20 minutes of working out. And I've got so many stars. Not as many as I could have had, but so many stars. And so one of the reasons I was able to do that is this year I took off of taking night classes for those listeners who are used to me chattering about my nightly fight with math classes that I take in the evening. So I was able to do a lot more movement last year because I wasn't doing work and school simultaneously. However, come January, I'm back on my school, so I'm going to try to keep up with my stars and keep up with the movement, despite all of the schooling that I'm going to be doing in the evenings and all of the peanut M&Ms. That I'm going to be eating.

Evelyn Coleman [00:25:18]:

So that will be my next. For my next challenge, I will try to not get lazy, but also get an A in linear algebra.

Jason Baum [00:25:28]:

There you go. All the stars and all the brains. Okay, Marcus, yours.

Marcus Merrell [00:25:34]:

I think I'm going to. This is a. I don't know, a little bit more introspective maybe. I'm going to try to be more deliberately positive, by which I mean I'm allergic to toxic positivity. Like, I can't. I. I would rather somebody be angry all the time and rant constantly as I do, than be toxic positive. But I.

Marcus Merrell [00:25:56]:

I do recognize that. Been ranting a lot and it's, it's done good for some stuff. When I rant about cryptocurrency and, and generative AI and stuff like that, it's. I actually think I bring something to the table with that to fight other people's toxic positivity, but I don't need to be doing that all the time. I think that there is a lot more good stuff to embrace than I generally recognize. And I'm going to try to make a deliberate effort toward doing that. Otherwise, I'm going to build up my hobby space and get back to work with small electronics and Arduino and stuff like that.

Jason Baum [00:26:26]:

Anytime I find myself slipping into the kind of mood, you know what I mean? Like the. The world is just bleak and gray. Kids are amazing for resets. I don't know. My kid, at seven years old, she. The way she looks at the world and processes things is like so wonderful. And I'm like, you know what? There's something to be said about that freshness, that fresh mind that hasn't been corrupted and like going back to how they see the world and sometimes I just try to see it through her eyes and man, I don't know, it's just a happier place to be sometimes through the lens of a child. I don't know, they're good resets.

Marcus Merrell [00:27:11]:

My kids are basically out of the house, so I gotta. I've got to heal with my hands, as they say. So woodworking, electronics, sewing, the occasional video game. I got to heal with my hands. So. Very cool.

Evelyn Coleman [00:27:23]:

We want to make more time for video games this year.

Jason Baum [00:27:26]:

Oh, yeah, God, me too.

Marcus Merrell [00:27:27]:

If anyone's playing Elite Dangerous. The Thargoids have invaded the solar system.

Jason Baum [00:27:31]:

And over the holiday, bad for humanity. Over the holidays. I saw 8-Bit Christmas. Have you seen that movie? Okay, that is like childhood. It is a great movie. Go out as well. It's January now, but it's a very.

Marcus Merrell [00:27:47]:

Millennial, millennial childhood movie. It was a little bit. I couldn't identify with some of it.

Jason Baum [00:27:53]:

Well, thank you so much for listening. This was another episode of Test Case Scenario. Thank you for continuing with us. We are going into our third year of this show. We've done two years, two trips around the sun. Thanks for sticking with us and listening. And if you haven't stuck with us, thanks for finding us. And now you're stuck.

Jason Baum [00:28:17]:

And it's just been an absolute pleasure recording with the two of you and everyone else we've had the pleasure of recording with in the past, all the guests and former co-hosts really excited to go into a third year. We are excited to take you with us. So happy New Year. All things good in 2025. We will see you next time on Test Case Scenario. Thank you for joining us on Test Case Scenario. Share your thoughts in the comments. We'll make sure to respond to each and every single one.

Jason Baum [00:28:59]:

Don't forget to subscribe and hit that notification bell to keep in touch. If you missed our last episode, it's popping up on your screen right now, so click it. Until next time on Test Case Scenario.