NASA and the SAPA Investigation --Supplier Management

In this episode, Diane and I interview Special Agent Wade Krieger from NASA's Office of Inspector General. He shares insight and lessons learned from the investigation into supplier fraud from SAPA Materials (https://www.nasa.gov/press-release/nasa-investigation-uncovers-cause-of-two-science-mission-launch-failures).

We also discuss best practices in investigations and supplier management with Lindsey Marshall from OmegaQuant (https://omegaquant.com/), a Sioux Falls company ranked in the Inc. 5,000 list.    

Transcript

Copy Transcript Copied

Aaron Harmon: 0:00

Hi, I'm Aaron Harmon.

Diane Cox: 0:01

And I'm Diane Cox

Aaron Harmon: 0:03

Welcome to Inside Out quality both fan and I build an implement quality systems in the biotech and medical device industry. But we often get asked, Is this really necessary that we know if we are doing too much too early? Or do we even need a quality system?

Diane Cox: 0:17

Our goal is to explore questions like these through real life events and experiences shared by our guests from various regulated industries. We will show you why quality is not just about compliance and how when it's done right, it can help your product and company improve lives and make a difference.

Aaron Harmon: 0:38

On February 24 2009, a torus XL rocket launched from Vandenberg Air Force Base in California carrying NASA's Orbiting Carbon Observatory a satellite to help track the climate. The rocket got to the point of deploying its satellite when a key part of the frangible joint failed to work as designed, which caused the rockets fairing not to open, the satellite mission was lost and investigation ensued. Root causes believed to be found and after corrective actions a second mission was undertaken. That was on March 4 2001. And again, a Taurus rocket lifted off to space, this time carrying NASA's glory scientific satellite, it two ended in failure as the same joint failed to operate. In total, over $700 million was lost not to mention the value from scientific data and the blow to morale of the teams at NASA because of this failed join. When something goes awry investigations occur, root causes are determined. And in this case, the root cause took years to solve how it was solved is better told by one of the men who led the case resulting in a supplier paying $46 million in fines and restitution. That being said with us is Special Agent Wade Krieger with NASA's Office of Inspector General. Welcome to the show, Wade.

Wade Krieger: 1:48

Hi. Glad to be here. I first of all, I'd like to thank you and Diane for giving me the opportunity to speak to your audience today. It's It's truly an honor.

Aaron Harmon: 1:57

So we'd love to dive into this into the story and the work you did to help figure out what was going on in the first place with these rockets. But before we get there, the first question is, how did you get into law enforcement? What drew you in and when did you start working for NASA?

Wade Krieger: 2:12

You know, I don't remember ever really focusing on being a policeman when I was a kid, which was a long time ago, by the way back in the 1970s. But I used to love Star Trek. And I also watched like every single James Bond movie ever made with my dad. So when I got a little older, I decided I wanted to go into the Air Force, maybe because that was probably about as close to the Starfleet Academy as I was ever going to get. But most likely, it was just a good way to get away from home and have a little adventure. So, so I decided that I wanted to be a fighter pilot. But while I was in the Air Force ROTC program at Memphis State University back in the 80s, Top Gun came out and well, all of a sudden, everybody wanted to be a fighter pilot. So it became really, really competitive, and I didn't quite make the cut. Fortunately for me, the commandant of cadets at the ROTC unit where I was exposed me to something called the Air Force Office of Special Investigations. Most people just call that OSI. And I was almost instantly hooked on their mission when I heard about it. So OSI does basically the same thing as the FBI. They perform federal law enforcement duties except they do it exclusively for the Air Force. I was lucky enough out of college to be picked up by OSI as a brand new second lieutenant in the Air Force. And that's where I got my initial training as a criminal investigator. And after I spent about 10 years or so on active duty, I got married, I separated from the Air Force. And then I was hired by the NASA OIG. I stayed in the Air Force for I don't know another 11 years or so as a reserve officer, which was part time you might say that I retired from the military in 2010. So I've been working for NASA OIG at Kennedy Space Center, down here in Florida since about the year 2000.

Diane Cox: 4:08

Can you tell us about the role that you have at the OIG at NASA?

Wade Krieger: 4:12

Sure, there are over 60 Different inspector general offices that support different agencies and departments all throughout the federal government. Oh IGS were first created in the federal government after the Inspector General Act of 1978 was passed by Congress. So the purpose of this act was to promote economy and efficiency and effectiveness throughout the government. And I think they wanted IGS to serve as kind of a watchdog against fraud, waste, abuse, and mismanagement in the government. Most IGS have a division that regularly conducts audits, which are generally aimed at making sure the agency follows its own rules and makes good choices when they spend our tax dollars. An OIG is also typically have an Investigations Division that functions as a federal law enforcement agency. So kind of like the FBI and OSI. So at NASA IG, that's where I work in the Office of Investigations. So in the investigations office, our mission is to handle criminal, civil and administrative investigations that involve NASA personnel, resources, contractors or programs. The IG X is public law. And part of that law gives investigators like me access to all of the records from the agency that we serve. We also have subpoena power for records that the government does not control or possess, and we can execute federal search and arrest warrants as well. So if an IG is doing its job well and functioning properly, I like to think of it like a good antivirus software package on your computer. If it's working, right, it will root out all the viruses on your machine and neutralize them. And it will make your machine run quicker and better and more efficiently. And most of the time, you're not even going to know it's there runs in the background. So on our best days, when we're doing our jobs really well at the NASA IG, we help NASA to perform their mission more efficiently. We help them spend the taxpayer money that they get from Congress wisely. And we help them fly their missions safer. And also, occasionally, every now and then we catch a few bad guys along

Aaron Harmon: 6:19

the way. Wait, it sounds like you also could have done well in QA. Talking about investigations and audits and helping things run without problems inefficiently like that's, that's very QA sounding. I like it.

Wade Krieger: 6:36

Well, QA Are you guys are our cousins, we kind of do the same thing.

Aaron Harmon: 6:41

That's something that I've started introducing myself as an FBI Cousin. Cousin.

Wade Krieger: 6:48

I think that works. I think that works very well.

Aaron Harmon: 6:51

So the first rocket goes up and it fails to reach orbit. There's a second one that goes up and it fails. How did you hear about this? Like, is that like the news all across NASA? Was there a point where someone's like, get way on the phone now? Hey, Jim, bring them in. We did what that look like,

Wade Krieger: 7:11

actually, I remember hearing about the second one that failed. But it launched from California, not here at Cape Canaveral, where I am. And it was, you know, I was all the way across the country. So honestly, I didn't really pay that much attention to it at first. But later on, I was approached by this NASA metallurgist who had assisted my office on a few previous cases. brilliant guy. This guy, I hold him in high esteem. He's personally responsible for teaching me probably 90% of what I know about metal. Anyway, this metallurgist had performed some tests on an aluminum frangible joint extrusion in his lab at Kennedy Space Center. And, and he was actually tasked to perform one type of tests, but because he's just a really thorough guy, he also performed some other basic tests on the aluminum material properties. And some of those tests failed. So which was a little bit alarming to him so so cuz he wasn't expecting that to happen so so he contacted me because he thought that bad aluminum like this might possibly have had something to do with the with the launch feathers that had just happened. So Upon hearing this, I also learned pretty quickly that the contracts for both of the failed missions were managed out of Kennedy Space Center where I was by NASA's Launch Services Program, and that these frangible joint extrusions that were used on both flights were in fact made by the same supplier, a company called SAPA profiles that was l cated in Portland, Oregon. So i 2012, I teamed up with my p rtner on this case, a f ntastic NASA OIG agent who's o t in California, and we began i vestigating. And after we got g ing, there were several other a ents and attorneys that join t e team and they all ultimately p ayed a big part in making this s ccessful.

Diane Cox: 9:13

You're going to have to help me out Wade, what's a frangible joint?

Wade Krieger: 9:17

Okay, okay, so good question. So a frangible joint is it's a part that is essentially made to break because that's that's kind of what frangible means. It's supposed to crack apart at the right time, and release the satellite so that it can go do its mission. So if you picture a rocket, there is something at the very top called a fairing, which is where the satellite rides until it gets to space. And I like to think of a frangible joint as like a zipper that holds the fairing together with the satellite inside until an explosive charge inside the frangible joint detonates. Okay, so that's a lot so when that when this happened? that this this rail that runs down both sides of the rocket splits open cracks, open cut picture like a clamshell, two halves kind of breaking open. And then that allows the the satellites separate from the rocket and go out into space and do science. God, that makes

Diane Cox: 10:19

sense. Thank you so. So can you walk us through the investigation process, I guess both how NASA normally investigates these issues, and then also how this particular situation was investigated.

Wade Krieger: 10:33

So I guess the first place to start is with EMI B's mishap investigations boards. Anytime there's a major accident or a failure, like the ones we're talking about NASA standard procedure is to convene a mishap investigations board to try and figure out what happened. So in this case, there was an MIB for TA and another one for t nine,

Aaron Harmon: 10:56

could you explain the T eight and t nine,

Wade Krieger: 10:58

the first mission that failed in in February of 2011, was launched on the eighth chorus rocket that that the launch service provider launched so they the shorthand for that is we just call it T aid for short. And the second mission that failed was the ninth mission. So we call that one t nine. So you'll hear me say you know if I refer to the missions as T eight and t nine that's what I mean. Sorry about that. So so back to MIB s so an MIB NASA convenes a board and they they reach out and they cherry pick some of their their best and brightest engineers and technicians from a variety of different disciplines. To to look at all of in this case, they were looking at all of the flight data, the telemetry, the modal analysis, and all the readings that they got from the launch and everything they knew about the launch to try and figure out the technical reason why it failed. So after both boards spent, you know, several months trying to, you know, reviewing this data and neither board could determine the root cause of the failure. Now, the second board found that both failures happen for the same reason. They found that because a single frangible joint side rail failed to completely break on one side at the top of each rocket. So subsequently, the fairings didn't open. And the satellites, you know, never separated from the rockets. And the whole thing just kind of lawn darted over and down into the Pacific Ocean. So both boards concluded that the frangible joint was was likely the problem. And they knew the frangible joint failed, they just couldn't say for sure why it failed. So therefore, they really couldn't find the root cause

Aaron Harmon: 12:50

these bearings with the satellites are in the Pacific Ocean somewhere. How did they come to the conclusion that it was the frangible joint that didn't break apart?

Wade Krieger: 12:58

Well, they had instruments on board on on the rockets that that monitored the systems as the rocket is traveling up into the stratosphere to to, you know, as each stage breaks off, there are systems that that monitor that and so they were able to pinpoint, that was actually a and this gets a little over my head, technically speaking, but there was like a three megahertz spike in some of the the modal analysis, which is the vibrations that are measured when the rocket is traveling up. That occurred almost exactly the same time in both flights, the flight paths were identical. Both rockets launched from the same launch pad. And so the mishap investigations boards, were able to determine that the frangible joints did not function based on the flight data. But only on one side, which is actually ends up being fairly important.

Aaron Harmon: 13:56

That impresses me when people can kind of plan ahead for if something happens, how are they going to be able to sort it out later, and some of those details go into it's pretty incredible.

Wade Krieger: 14:06

It's a complicated thing. And it's actually I'm not qualified to speak with any degree of expertise on, you know, that part of the case, that's why I depended heavily on the lawn services people to do their jobs, and they did it very well. And, you know, analyzing the data that they had, and, and, you know, telling me what they could say for sure, and what they couldn't say for sure, that kind of thing.

Aaron Harmon: 14:31

Good investigations are teamwork. They actually can't do them alone.

Wade Krieger: 14:35

Absolutely. I wholeheartedly agree with that. Okay, so going back to our case, the AIG case, in 2012. After I opened it, I began talking with a couple of really brilliant Launch Services Program engineers about the aluminum that had failed in the metallurgist lab. Ultimately, this dialogue led LSP to start their own technical investigate Which kind of took on a life of its own, and picked up where the EMI bees left off. And their goal was to be able to say for sure what the root cause of these failures was, they had a business reason to know that, because they didn't want it to happen again, because of, you know, the money and man hours lost in the grief that losing two missions caused them. So, m IVs are normal in these types of situation. But this third follow on technical investigation was was a little unusual, I guess, because the MIB were not able to find the root cause and it happened twice. So their investigation went on for several years during the same timeframe. At the same time as our IG investigation, it was it was a separate and independent effort from ours. But there was a regular mutual exchange of information that I think was very healthy for, for for both investigations. So you know, looking back on it, I don't really believe that either of our organizations could have solved this case without the other, the NASA IGS office, you know, we obviously lacked the technical expertise to ever find the root cause of a rocket crash. I mean, that's, that's just not what we do. But LSP did not really have the they didn't have the ability to obtain some of the information that we uncovered through subpoenas and interviews. And as it turned out some of the technical data that we got from from subpoenas that we gave to several different companies that were involved in making the TI 89 frangible joints, some of that information turned out to be absolutely critical to LSP being able to make the conclusions that they did with regards to the failure.

Aaron Harmon: 16:52

When you were subpoenaed in getting getting that data in, I think you brought a different set of eyes to the investigation, where it's easy to rule out the frangible joint specifications, because you would see the passing data and say, yeah, it passed, it's good enough, you have a metallurgist, that kind of clues, then something's going wrong. And he's seeing something different. And so I guess, I don't know, as much of a question and more. Yeah, I just thought was, when you originally mentioned to me that you were able to see that, hey, maybe something is suspicious going on here. Let's get a subpoena to get that data. Versus someone saying, Yeah, we checked their specs in the past, it was good, we can rule it out.

Wade Krieger: 17:37

So along those lines, and I think that, that EMI bees, and I was planning on touching on this a little later, but I think EMI bees are, are excellent at doing what they are charged with doing, which is to find technical solutions to to failures, right. But but those boards as successful as they have been in the past, and is as good of work as they do, I think in these two cases, they may not have been consciously considering the possibility that someone was was committing fraud in this case, and may not have been doing what they were supposed to do. So there may have been a blind spot there. And and it's not a it's not a criticism, it's just a each of us approach a problem with, we bring to the table, the things that are in each of our backgrounds. And and, you know, like I do not think like an engineer, although after working for seven years with these guys, I've learned to think I've learned how they think and it's a it's a, it's very, it's a very good tool for someone in my job to have. So I don't know if that helps.

Aaron Harmon: 18:57

So I have a bit of a confession to make. This is your first time meeting, Diane, but Diane has known about you actually for quite a while.

Diane Cox: 19:06

Yeah. Aaron talks very highly of you. Well, thank

Wade Krieger: 19:10

you. Yep.

Aaron Harmon: 19:11

When we first talk, it was in January, and I had seen the press release. And I had reached out to NASA and ultimately ended up connecting you and I Diane and I were in the Minneapolis area doing some audits on suppliers ourselves. And you would call it so I made Diane sit with me in a car parking lot for about 20 minutes while we talk. And then cool, and then afterwards, I told her that you're my new BFF so that's been kind of like the lie. Tell Diane, I'm talking to my BFF she knows

Diane Cox: 19:44

I know. Yeah, exactly. Oh, that's

Wade Krieger: 19:47

that's that's very flattering. I can tell you that. I know a number of people in quality assurance and and I I'll go back to the My cousin's analogy because quality assurance People make my job so much easier if they're doing their job well. So the people that really care and they pay attention, and and, you know, they, they take to heart what they're there to do. They are godson to people in my line of work. So yeah, the respect is mutual my hat's off.

Aaron Harmon: 20:19

So speaking of nors, this back then, you know, kind of the coolness of the jobs that we have. You're also at NASA. And when I tell people that we're speaking to this special agent from NASA, it's pretty cool. Because you guys have some really cool people on your team. You work with a bunch of rocket scientists. What do they like to work with?

Wade Krieger: 20:41

Oh, wow, you know, they they are. It was such a great privilege for me to have work with the launch services engineers that I got to work with. They're they're truly remarkable people. And their patience, you know, because it takes a special type of brilliance to be able to, to accurately dumbed down rocket scientist so that a guy like me can follow it. So my hat is off to them. And they, they they took the time to you know, when I wasn't tracking, they took the time to lay it out for me in small pieces, and they taught me so much so it was my privilege.

Diane Cox: 21:17

Can you tell us briefly what NASA's conclusions were and why the failures happened?

Wade Krieger: 21:23

Sure, I'll give it a world. LSPs investigation agreed with the t nine MIB that a single frangible joint side rail near the top of each rocket failed to break but they went a little further, and they determined that the rail didn't break as it was supposed to, because of a combination of three factors. The first one was the fracture ligaments on the frangible joint extrusions. The second one was charged holder contraction due to temperature, and the third one was discrepant material properties of the frangible joint extrusions. Now, I probably don't have time to explain to you what those first two things mean. What they are the thick fracture ligaments and charterholder contraction. But if anyone is truly interested in that stuff, there's a there's a pretty detailed description contained in NASA's technical summary of their investigation, which NASA released publicly last year. So if you do Google searches on on these mishaps, you will come across a NASA press release with a summary hyperlinked. But the important thing to know is that these first two factors could not have caused the joint to fail all by themselves. NASA had enough data to calculate almost exactly how much of these factors eroded the safety margin, which was designed into the frangible joints, and those two things combined could not have caused it to fail. So there had to be a third factor. And NASA's technical investigation concluded that the third factor was discrepant material properties in the aluminum extrusions that were part of those frangible joints. Now, the aluminum supplier SAPA p ofiles, they dispute NASA's t chnical findings. They do not a ree that the aluminum e trusions made in their plant c used these missions to fail. A d, and they're certainly e titled to make that argument. H wever, if the rockets had not f iled, we would not have d scovered that SAPA was eng ged routinely in altering mec anical property test results on heir aluminum for almost two dec des.

Aaron Harmon: 23:41

So how long did it take to get to the point where you were like, Aha, we now know why these two rockets had failed.

Wade Krieger: 23:49

It took a really, really long time because because when when NASA OIG figured out that there were test results being changed, that still didn't, there was a whole nother the NASA technical investigation was was exhausted. And a frangible joint as I described it, the kind of the way it works is fairly simple. But there's a whole lot of complex things that go on when a frangible joint detonates. And so NASA, they looked at each component of the frangible joint in depth, and they looked at the suppliers of each of the parts. And they looked at all of the data. And and it was it was really an exhausted process. So it took it took a years before they were able to conclusively make the conclusion that they did. I can just leave it there probably.

Aaron Harmon: 24:45

Now we'll take a quick break to hear from one of our sponsors. Today's startups

Unknown: 24:49

become tomorrow's growth engines in South Dakota. We're entering a new stage of expansion for a biotech industry and you'll want to be part of it. Hi, I'm Tony Johnson, Executive Director. South Dakota biotech where the state affiliate of the International bio organization, and we're proud to be leading a state that's driving innovation to feed, fuel and heal the world. South Dakota biotech is here to inform, to connect, and to advocate for our critical industry. Whether you're directly involved in biotechnology, or looking to learn more about it, we want to hear from you find us at www that SD bio.org. Now back to the show.

Aaron Harmon: 25:27

In the world that Dan and I work, work in, you can have something like an assay that doesn't work. And you can sit down and do a root cause investigation, you can get a team together, you think you identify it, then you have someone go back down to the lab real quick on the asset again, and you see what the results are. And then they come back and say, hey, yeah, that fix it. Or they say, Oh, it didn't work again. And then we'll okay, you know, let's dig some more. Let's try these next experiments. And we'll see what happens. But that's, that's quick, and it's pretty cheap compared to let's fire off another rocket. Right? So it's pretty crazy to get your head around.

Wade Krieger: 26:03

It really is. And, and there was a lot of institutional will, I believe, to find what the root cause actually was, after so much time, and after two missions being lost, has there been a ti 10? I don't think so. I think that the launch company rebranded the line, okay, into another line of rockets. So I don't think there was ever a t 10.

Diane Cox: 26:28

I guess where I was kind of left hanging is okay. Well, how did how did you get to that actual pinpointing that it came from SAPA that the the issues basically stemmed from Safa. So I guess, maybe we just dive right in there. If I

Unknown: 26:43

understand your question, right. It was how did

Aaron Harmon: 26:44

with the EMI bees from when they formed to when we pinpoint SAPA as the supplier for you know for for these frangible joints? And and the answer is pretty straightforward because the mechanical drawing of the frangible joint extrusion they and how long is that process? Is their usual time called out SAPAs plants, which was the technical dynamics aluminum plant as the sole supplier for these for these frangible joint extrusions. So, without a doubt, that plant window? Is it pretty quick? Or is it end up being drawn out? manufactured every frangible joint extrusion that this launch company used over over probably a 20 year period. So when we found out that that the, the ti 89, frangible joints were made in the same plant, but they were also manufactured. They were assembled at the same time. Then then we became focused on on the SAPA plant in Portland, Oregon, Can you elaborate on that?

Wade Krieger: 27:55

Yeah, I can a little bit. These EMI bees, the first one lasted two to three months, I believe. And just from my memory. And usually the way EMI bees work is they convene in one location. And they are more or less I don't want to say sequestered. But they are they're supposed to remove all other duties from their from their plate and focus just on solving the MIB. So it's it's a focused concerted effort over over a fairly short period of time to try and find the answer. So the first MIB convened for I want to say two to three months. And then the second MIB convened for maybe a little longer. But I don't remember off the top of my head, how long but that's generally the way that works. I love

Aaron Harmon: 28:49

that idea of you have one job, investigate. Because most of the time, it's easy to be like, oh, yeah, I've got this going on this going on. And I'm doing this investigation and trying to like, give it the time and quality attention you need to to make that successfully. Get the answers you need. Mm hmm. Right. So kind of going going back on the story here. What can you tell us about the process of investigating SAPA?

Unknown: 29:17

Okay. Well, I can I can tell you a little bit when we first began our investigation. You know, at the IG, what we knew was that on at least a couple of occasions, SAPA had certified that their products met a certain standard for 6061 T six extruded aluminum, but that the test performed in a NASA lab, in fact, showed that this aluminum did not meet that standard in on a couple of occasions from the pieces of aluminum that were tested. So So based on this information, that was kind of our starting point, and we then use a subpoena to compel SAPA to give us their internal testing records, which, which that took some time. And then and then we also had to review all of those records. And, and during our review, we we found 1000s of instances where mechanical properties test results were changed. And our investigation during the same time as the the initial review was going on. We also interviewed quite a few people who used to work in the plant. And there were there were a lot of them that confirm that the test results were in fact being changed. So our investigation also determined that the change test results were being used on material certifications that SAPA was giving to its customers, which included the companies that built and assembled the frangible joint extrusions that were used on T-8 and T-nine in broad strokes, that's kind of that's kind of how the investigation of SAPA went. But all of that took several years, for, for a lot of a lot of reasons.

Diane Cox: 31:10

I guess it's no wonder how maybe previous missions didn't have that same failure with all this going on for a long, long time. I don't know if

Unknown: 31:19

well, that's compensated. And it's, and it's interesting, because I think ti 89 To some degree was kind of like a perfect storm. And there's a margin of error, a safety margin that's built into this engineered into most launch systems. And so there were, as I said, there were some other things that eroded the margin, to the point where, you know, this discrepant metal was probably the apple that tipped the cart. And, you know, it's possible that that discrepant metal had impacted other flights but not surpass the margin or the safety margin had not had not eroded the margin to the point where it caused a real problem.

Aaron Harmon: 32:09

And if I was going to speculate, I would imagine that that may have been happening at the same time with separate profiles, because they probably it could have started with a good process that he wrote itself over time. And as he wrote it, maybe that was what started leading to change test results show

Unknown: 32:27

they're very possible, very possible. And, and it doesn't mean it certainly doesn't mean that everything they produce was bad. That there's you know, on the contrary, there was, you know, there was a lot of material that they produced that that had no problem at all. But you know, it doesn't take, it only takes one bad, you know, or one discrepant piece of metal to cause a problem in the right situation.

Aaron Harmon: 32:59

And in the same thing, when, in our world, you don't allow things to go wrong in your process for a drug product that you have going out the door. Because even what may seem minor, if you know what's outside of that safety margin, when you release that out to the general public, there's a huge amount of risks that could go into that. So we have to draw the line as soon as it goes out of that margin. It's not conforming anymore, and it's not going to get out

Unknown: 33:26

makes total sense. And that's that is quality assurance. Is there the gate, the gatekeeper to make sure that that's why in the quality world, it's, it's, it's always interesting to me that one of the terms that you use in the quality world is an escape. Because if if a bad, you know, I like to think of it as like a burnt potato chip on the conveyor belt from the, from the losing show years ago with, you know, if one of those burnt potato chips gets down the belt, then it's seen as an escape. And quality assurance is supposed to make sure that doesn't happen

Aaron Harmon: 34:02

for potato chips.

Diane Cox: 34:05

Can you speculate on why it was hard for the team at NASA to suspect that certifications from SAPA weren't true?

Wade Krieger: 34:13

Sure, that's, you know, that's a great question. The answer, I think, will maybe provide some pretty valuable insight, I guess. I guess first of all, mishap investigations boards are comprised mostly of engineers and technical specialists. The process they use is very methodical and logical. And and I think EMI bees are probably really well suited to find technical causes for really complex failures. But But both boards, in this case investigated hundreds of different possible scenarios that might have caused these failures. So the way they would work is they would start off and they would brainstorm as many different parts possible causes for the failure as they could possibly, you know, come up with and then they would form those into different like Fishbone scenarios, and then they would assign teams to investigate each scenario and, and try to prove it or disprove it. And one of the scenarios that both boards looked at was, was the possibility that the metal and the extrusions was was not up to par. Was was not meeting standard. But when they investigated the scenario, both boards, they took some, you know, the reasonable steps that they would take. And they're, which started with reviewing the paperwork and the documentation and, and they found a material certification from the supplier saying, you know, nope, everything is good. We perform some tests, here's the test results. And and so the MIB is basically trusted that certification, they took it at face value. And then they took that scenario off the table as possible causes and they moved on. Because the way and the way an MIB works is, when you start with that many different possible scenarios, the first thing that you want to do is eliminate as many of them as you possibly can. And then you look at what's left the things that you can eliminate. So because there was a material certification from the supplier saying everything was good, and that the material certification contained test results, the MIB is basically just took that right off the table and they moved on. So I don't think that they ever really consciously considered that that fraud may have been involved. Because they're not they weren't really trained to think that way. So one of the one of the engineers later explained it to it later explained it to me this way. They said this guy said SAPA w s the largest aluminum e truder in the world. At the t me, they had 66 Different p ants in like 33 Different c untries that were a billion d llar corporation, plus they h d an ISO 9000 Quality C rtification. So the thinking w s, Surely they must be very g od at what they do, maybe even t e best, otherwise they w uldn't be so you know, so l rge and so successful. So the e gineer asked, why would a c mpany like that falsify an o der on what was really a r latively inexpensive part. I m an, I think that these f angible joint extrusions c sts, maybe a few $100 each. So I think there was definitely a f lse sense of security there. B t at the end of the day, the g vernment and and also any o her consumer has to trust the p rts and materials c rtifications that they get w en they buy things. Unless t ey're unless there's a reason n t to. Does that? Does that k nd of answer your question?

Aaron Harmon: 38:02

Does it reminds me of a saying that I've heard in the quality space, which is you trust but verify.

Diane Cox: 38:08

Yeah. So wait, what what has happened to SAPA now, y u know, the relationship with N SA especially I'm assuming t at they've they've gotten out o that relationship.

Wade Krieger: 38:17

Shopper no longer exists. They were acquired by another company. And And NASA has excluded them from government contracting for a period of time. And they have my understanding is that they have taken some aggressive steps to to fix the problems that they had, and that they've made significant progress.

Aaron Harmon: 38:41

In the biotech and medtech space, we rely on tons of suppliers. It's very normal to have a large supplier list. Are there any lessons learned from NASA that in our industry or even small startups can apply?

Wade Krieger: 38:55

There's a few. I guess the first the first one and the the one that I would foot stomp the most to and I say this to quality assurance people in regularly is, is do site visits on your suppliers do supplier audits that include site visits, doesn't have to be you know, every year, but you have to do them periodically. This is absolutely critical. In this case, the launch service provider provided no perform no visits zero between 1992 and 2011 after the second rocket had crashed. So I can't help but wonder that if there had if there had been, you know, even a few pretty thorough site visits to the SAPA a plant where thes things were being manufactured whether or not I can't help bu wonder if if this activity tha was going on might have been you know, outed years earlier and maybe these mishaps coul have been prevented. So numbe one, do your supplier audits an do site visits. And Aaron, a you just said, you can trus them, you can have a goo relationship with them. But whe you're doing the site, visit i the audit, make them verif their processes and make sur that, you know, they're, if the have quality standards in place and they have a quality manual like most, most suppliers an big companies do, make sur they're following it. So that' the best advice I can give you I have a couple more is you guy that answer that one's tha makes sense. It says, Yeah please. The second one I have i verify and document that you supplier understand you requirements. And, and this i just, it's just in the interes of being thorough if yo communicate a change to you products, you know, mak absolutely sure that supplie understands what you're changin and why you're changing it an then document it. And this i just going to make it easier t sort out any problems down th line. If you if you happen t have any I hope you don't. Bu if you run into problems later documentation is key. And it' going to make things go smoothe and getting to the bottom of it And the third one is conducted it's the one actually that Aaro just mentioned, is conducte independent testing on on percentage of the materials, an materials that are going to ge used on anything deeme critical, and certainly anythin that might pose a health o safety hazard, it would be really good idea to you know take a small percentage of th products that you get, and jus do some independent testing an verify that that materia certification is good. That'l that could potentially save yo a lot of grief down the line Finally, the last thing that have is, is questions thing that don't look right. You know and if, if the answers you ge just don't make any sense, yo might consider reaching out t your friendly neighborhoo inspector general or anothe investigative agency, especiall if you're dealing with somethin that is a critical product, o something that could affec someone's health or safety

Diane Cox: 42:19

It makes me think about so many procedures, I've read on supplier controls and how many companies will basically say if you have an ISO certification, whether it's 9001, or in my world medical device 1345. An audit is not required. And you can basically just accept their cert as having a valid QMS. And everything should be fine. And were warned by the FDA or US regulatory agency for drugs and medical device that you shouldn't rely on that alone. And that just again, validates that

Wade Krieger: 42:57

I would tend to agree with that observation. I think that ISO certifications certainly provide some value across the board. But I would be very hesitant to trust that alone as evidence. In this case. The supplier had an ISO certification for four years, during during the same time they had the supplier, the ISO certification, they were being audited by ISO and they were in the process of systemically changing mechanical properties test results, there was even a procedure in place a written procedure that kind of showed some of their lab technicians how to how to change it. So ISO certifications are not they should be used as a tool, one of one of many things that you when you look at a supplier to give you an indication on whether or not it's a good supplier. Certainly a valuable tool, but I wouldn't rely solely on that.

Diane Cox: 43:58

So based on kind of your advice here and everything else we've talked about, do you foresee changes at NASA and how suppliers are managed in the future after this investigation?

Wade Krieger: 44:09

You know, I guess first of all, NASA OIG doesn't really have the authority or the responsibility to change to actually make changes on how NASA oversees its suppliers. But our mission is often ultimately supposed to help the agency operate as safely and as efficiently as possible. So along those lines, every now and then, we get the opportunity to emphasize the importance of things like supplier quality assurance to big NASA and and the results of this investigation without a doubt, definitely emphasize the importance of quality to to senior NASA leaders and and we've gotten every indication that they heard this message. For example, NASA's acquisition integrity program, recently incorporated A ta and t nine mission failure case study into a block of, of mandatory training that every single NASA employee has to take every year. So from our standpoint, raising awareness is probably the biggest single thing we can do to prevent problems in the future. In the instance, where NASA does implements policy changes, or writes, better contract language to be inserted into contracts, you still have to depend on contracting officers and program engineers and quality assurance people down in the trenches at the field level, to implement those policies, and to make sure that they get followed. And so again, I kind of come back to you know, raising public awareness of those types of things is probably the most important thing that

Aaron Harmon: 45:54

we can do. This must have felt really good. When you were able to finish off this investigation. It's huge, as a huge undertaking, it sounds like and probably took a impact. I don't know that you would have ever foreseen where it was going to go and you started on it?

Wade Krieger: 46:12

No, I didn't. And, and it, uh, it definitely, I've been doing this job for over 30 years now. And this case was definitely one of the most, if not the most difficult, challenging cases that I've ever had. I worked on it for about seven years. And there there were times when I didn't think that we would ever, you know, figure it out. And, you know, but it's like anything, I guess you just take it a day at a time. And, and I guess we started kind of seeing the light at the end of the tunnel around 20. You know, 2015 2016 I think we, you know, as a group, and there were a lot of people in that group. We started kind of understanding what likely happened. So yeah, but it was, it was definitely an experience. Honestly, not one that I would care to care to repeat, because it was it was a lot of a lot of hard work is as interesting as it was, it was I think everybody involved put in a lot of grueling hours. So from that standpoint, we're, we're the closure is good.

Diane Cox: 47:26

How would you compare it to those James Bond movies he stole?

Wade Krieger: 47:32

I don't know. I mean, that was just more I mean, everyone when they're, when they're kids, you know, you certain characters capture your imagination, and I certainly never, you know, would have dreamed that I would actually end up working for NASA, even though I was a Star Trek fan. And, you know, the James Bond, this was a long way from James. Boring, it really is. And most of the time, it made me look glamorous from the outside. But most of the time it is going through boxes, and it's lots of coffee and late nights, so

Diane Cox: 48:11

I can imagine what a great story. Thank you so much for joining us.

Wade Krieger: 48:16

Oh, thank you for having me. It was truly my pleasure.

Aaron Harmon: 48:19

Thank you, Wade. With us today. We also have Lindsay Marshall from omega quant Lindsay, can you tell us about omega one.

Lindsey Marshall: 48:26

OmegaQuant is an independent analytical laboratory that offers evidence based nutritional tests to researchers, health practitioners and consumers with a focus on fatty acids. OmegaQuant offers the Omega three index test which was co invented by omega Quantz. president and founder. The Omega three index test is a measure of EPA and DHA in the blood, specifically in the red blood cell membranes. The Omega three index is used as a risk factor for heart disease and has been linked to improve heart, brain, eye and joint health. Our test has been used in more than 200 clinical studies, we have worked with more than 100 international research institutes, including the US Army, NIH and Harvard. OmegaQuant is located in Sioux Falls, South Dakota. And we have operations located in Australia in Scotland, which allows us to provide the Omega three index globally.

Aaron Harmon: 49:15

I feel like as I get older, I'm going to need a test. You can have one I could have brought all three

Lindsey Marshall: 49:19

of you one.

Diane Cox: 49:20

I had one. Did you Yeah. When I went to visit Yeah, one time, they were very nice enough to give me a little finger prick. I was within range, apparently. So nice. No action needed. Yeah.

Aaron Harmon: 49:31

So Lindsay, what is key to delivering high quality results for your customers.

Lindsey Marshall: 49:35

OmegaQuant is CLIA certified so CLIA stands for Clinical Laboratory Improvement Amendment. Basically, this means we adhere to established quality standards set for all clinical laboratories. This ensures accuracy, reliability and timeliness of results. Because we're also high complexity test. We have additional quality requirements for personnel qualifications, equipment, test performance and proficiency to Steam.

Diane Cox: 50:00

So Lindsay, this episode is about investigations and how to dig in and find root cause. There are many tools and tips are there any that are your favorite?

Lindsey Marshall: 50:09

Yeah, investigations in the laboratory really range and complexities. So, one tip I would have is to keep things simple. The first thing I like to do is go through a quick out of specification checklist that looks at variables. Some of these variables are procedures, people equipment, reagents and calculations. This can be similar to a fishbone analysis or the five M's in manufacturing. My second tip would be Never underestimate the impact of a change to a process. Once a root cause is identified, perform a kappa analysis to properly validate any changes that can be identified to correct prevent Euro s

Aaron Harmon: 50:47

Kappas or Diane's favorite tool. I know that yeah,

Diane Cox: 50:50

I have a sickness. I like Kappas and audits,

Lindsey Marshall: 50:53

too if people do them. Yeah.

Aaron Harmon: 50:57

That does help. So let's see. Is there anything else you'd like the listeners to know about a mega quant?

Lindsey Marshall: 51:01

Yeah, OmegaQuant has been named to the Inc 5000 list for the fastest growing companies. For the last two years. We have been growing exponentially. And we've just expanded our portfolio to include vitamin D testing, which is really exciting to be able to provide both the Omega three index and vitamin D test to our consumers at home. Because I mean, health is so important right now.

Aaron Harmon: 51:22

Yeah, it definitely is. And it's good to hear you guys are growing. Absolutely. We're big fans of the small companies around here and watching them be successful and get through those early stages is always a good thing. So yeah, gratulations.

Lindsey Marshall: 51:34

Yeah, we also have an abundance of information and research on our website, which also includes access to our podcasts, blogs, and newsletters. So if you're interested, you can check them out at Omegaquant.com.

Aaron Harmon: 51:45

Awesome. So thanks, Lindsay. So this is the part now where Diane and I kind of dive into the the main overall story. In this case, it's the FATCA investigation, and with NASA and loss of hundreds of millions of dollars over what ended up being a failure that happened with the supplier. And so like my takeaway from that story is there's two key things. One is you need to have a solid supplier management program in place. And there's regulatory requirements for that, at least in the biotech, medical device space. But then also you need a very effective investigation plan, or approach. And those tools help you get to the root cause so that if you have a failure, you can eventually get to it, solve it and prevent it from happening again. So my favorite investigation tool is to go to the site and see what happened. Whenever there's investigations and people don't go to the scene, but just stay in an office and try to listen and talk it through. I feel like you're you could miss out vital clothes that you could get being there in person. So I'm, if I was gonna pick my favorite tool. First one is go to the site, go to the site, or wherever it is that the event happened. And begin is looking at stuff.

Diane Cox: 52:58

I used to work for a company that was really big on doing cousins. And so they would, they would always stress going to gamba. And so that's what we always said, we want to go visit the site where something happened where the where the cause of the issue may or even where the issue stemmed. We would just always call it go into gamba. I love that. I agree.

Aaron Harmon: 53:18

Yeah. And in the case of the NASA story, it was looking at raw material that would have been on the rocket, some of the scraps that kind of led to understand there was a problem with the properties of that material of the aluminum. And that's kind of like going to the site in a way and then ultimately, in their investigation, they did go to the site where that material is manufactured.

Diane Cox: 53:38

One of my tips, I suppose, is to get the right team of people involved. I remember Wade, he was, you know, stressing the fact that the team members involved in the investigation are really kind of what got them to the actual cause, you know, just talking about him and his expertise, and not really necessarily being the technical person. But having his perspective. And his team's perspective on the investigation really made a big difference. So that's another key one for me.

Aaron Harmon: 54:05

Have you ever done supplier audits? Lindsay?

Lindsey Marshall: 54:07

I have not I have looked into documentation from certain suppliers that we've deemed critical to look at. And most of our suppliers are regulated under ISO standards. And so I've never gone on site.

Aaron Harmon: 54:22

You don't need to do them based on risk. So I think there's a big risk component that you have to look at, you can end up with so many suppliers, that if you try to go into each single one, if there's ways you can just focus on which ones are critical, and that begins to make it more easy, and some businesses may not have a supply that is critical in that case. So what I found to be completely fascinating with the NASA SAPA case is eventually they get to the site, they go through and look at the data, they determine that there was records being falsified. And in our world if we did an audit, you would go there you would see Right overs occurring, that there is things being changed. And our conclusion would be, you guys need data integrity training, you should go retrain yourselves. And we would probably not qualify them as an odd as a vendor, and then relief. But in this case, it went through prosecution because there was such serious downstream effects of that failure in their quality system. Do Yep. And I don't know that we often think through what the consequences are of those types of failures. And so it'd be easy for us to go to someone and say, You're not approved to be a vendor for us. But in the meanwhile, they were doing something where they were falsifying records, they could be doing that for other companies and products and jeopardizing their, their products and their safety. Yeah. So there is a very much broader impact outside of our own businesses. That's why I think every company should embrace a quality system to prevent those types of things from happening.

Diane Cox: 55:53

From my perspective, I think a lot of companies, obviously will use the same supplier, but I think the use of that supplier is for all kinds of things. It's not necessarily for one particular type of device or unit or whatever the case is. And so like Lindsay said, you know, there, they may be, are doing some reviews of the suppliers for low risk types of product, or that component that they're receiving from the supplier is relatively low risk for their use, but for another company's use of that supplier might be much higher risk. And so I think one mistake that a lot of companies make is they kind of see, you know, different companies are using the suppliers. So they must be good, you know, to kind of draw this conclusion because they, they have this big customer base. And I think still, you know, I want to emphasize that the the expectation is that you're reviewing the supplier for your use, not general use certain not for another companies use.

Aaron Harmon: 56:50

good point. And I would add on that companies are That's a large, and company like SAPA was very large as well. A component of SAPA, having an issue is mean, the whole company has an issue, right? It may mean that there's a larger failure and a Corporate Quality System. But it's very difficult to control the individual actions of people when you're a large organization. And so it's it's not uncommon to have a really good company that has maybe a site that has a problem. And that problem is what leads to that larger reputation issue, which certainly doesn't give reason to overlook it. But I think it doesn't mean that you just assume all of its bad means that there's an element that's bad. And hopefully that gets corrected and is no longer an issue. Yeah, absolutely. I think that's a wrap for this episode. Thank you for listening. We hope you tune in again.

Diane Cox: 57:44

We hope you enjoyed this episode. This was brought to you thanks to South Dakota biotech Association. If you have a story you'd like us to explore and share, let us know by visiting www.sd bio.org.

Aaron Harmon: 57:56

Other resources for quality include the University of South Dakota's biomedical engineering department where you can find courses on quality systems, regulatory affairs, and medical product development. Also, if you live in the Sioux Falls area, check out quinnbet A local Quality Assurance Professionals Network. You can find out more about wibit by clicking on the link on our website to Diane and I would like to thank several people but a few who stand out or Nate pepple for his support with audio mixing Barbara Durrell, Christian or support with graphics design and web. And lastly the support from South Dakota bio