Debugging PCB Assemblies

It's a fact of life that defects and failed boards will sometimes occur, which is why it's so important to have a debugging procedure prepared to help your CM know how to move forward.  In this episode, we talk about best practices on how to work with your CM when these problems arise.  

pickplacepodcast.com

Transcript

Copy Transcript Copied

Chris: 0:08

Welcome to the pick place podcast, the show where we talk about electronics, manufacturing and everything related to getting a circuit board into this grand old world. This is Chris Denney from Worthington

Melissa: 0:19

And this is Melissa Hough with circuit hub.

Chris: 0:23

Very nice, welcome back, Melissa.

Melissa: 0:24

Welcome back Chris,

Chris: 0:25

So as each of us are enjoying this grand old world, we are each going to be going on various vacations and business trips and not going to be able to sit down and record podcasts for awhile

Melissa: 0:38

or edit?

Chris: 0:39

or edit. Right. Who wants to edit while they have an opportunity to hang out in. So Cal and soak up some rays?

Melissa: 0:46

No, I would definitely prefer not to as much as I love this podcast. I do not think I will be spending my whole time editing while I'm away.

Chris: 0:56

Yeah. Yeah. Just to drink delicious wine and enjoy beautiful Seacoast. So, so yeah, we're going to try to record a few episodes ahead so that there's still content out there and apologies ahead of time. If we do happen to miss a release date here and there, but this again is not our full-time job and we just do our best to keep creating content for everybody. Thank you for listening now.

Melissa: 1:18

we do. You're going to be away for two weeks and then I'm going to be away. So it's like, That's why we have to record so many ahead of time. Yup.

Chris: 1:27

gotta do it. Yep. So going to Fuji for some training that'll be fun and exciting. Might, might actually be able to create some content out of that experience. And then,

Melissa: 1:37

Oh, yeah, yeah, yeah.

Chris: 1:38

Then I'll be going to Canada for my formula, one race, which I cannot wait to go for and probably no relevant content in that

Melissa: 1:47

But content that might make its way into the podcast nontheless.

Chris: 1:51

but actually, yeah, for sure. But actually I just remembered I'm also going to visit a solder manufacturer while I'm in Canada, in Montreal. actually there might be content we take out of this trip. So stay tuned

Melissa: 2:04

oh yeah, that'd be cool. Take some videos if you can.

Chris: 2:08

oh, for sure. For sure. It apparently it's a massive, massive facility, like 400,000 square feet and they like, they do like full on like, like they get crushed cars and turn them into fresh metal. Facility. I can't wait. I'm so

Melissa: 2:25

Oh, wow.

Chris: 2:26

Anybody who listens to this show and has gotten to know me even a little bit in my excitement for things we'll understand why something like that is so exciting to me.

Melissa: 2:37

Chris, Chris, doesn't just want to know how things are made. He wants to know like how, the things that you use to make the things that you use to make the things that you use are made.

Chris: 2:46

That's right. Exactly. exactly right. So how do you mine copper? Yeah, so I'm, I'm looking forward to that and yeah, by all means if you guys ever have any questions about past episodes, it's just something I meant to meant to bring up. If you ever have questions about past episodes, if you ever have comments or corrections you want to send in, please. We'd love to hear him and and speak about them. And yeah, we've enjoyed them all. But this week we're not talking about crushing cars and turning them into metal. We are talking about debugging, finished assemblies. This, this is a fun one.

Melissa: 3:23

Yeah.

Chris: 3:23

is a relevant one. Yeah. It's a totally useful one. And it's especially relevant because. We've, I've been going through this exact scenario with a customer lately, and it's like, oh man, this is so fresh on the mind. I want to get this all out there. And again, it's like these conversations take place every single day, but they take place, you know, just between two people that aren't recording this and aren't publishing this as content that can be useful to others. And it's like, ah, if, if this conversation could be recorded, it'd be so valuable. Right. That's the whole point of the show.

Melissa: 3:55

recording the unrecorded conversations.

Chris: 3:58

That's a good way of putting it. That's a good way. I remember little bit of a side tangent, but I remember Hearing or, you know, there's, this engineering newsletter is subscribed to, and very, they link to a published presentation that was supposed to be, it wasn't supposed to be secret. It was just supposed to be internal. And it didn't matter that it got published publicly, but it was, you know, kind of discussing this technical subject and the author of the newsletter said, you know, how, how much of this kind of gold is being written every day that never gets out there for the public to see. And if they did, you know, if we all did yeah. So much and so much. And, and, and, and that really stuck with me and, and got me thinking that that kind of is an opportunity for this show, you know, where we can, we can get these sort of internal conversations out there. Back on episode 41, we had an awesome interview with Duncan and Joe from fixture. And it was all about creating functional test fixtures for finished assemblies and what that sort of looks like. And some best practices. I really loved that episode. I listened back to it and I was like, man, there's some, there's some solid stuff in here. And I hope listeners enjoyed that well. So if you're just new to the show, highly recommend going back to that one, perhaps even before listening to this episode. Because there's some great relevant information that will lead into what we're going to be discussing here. Because we get into like, what is the difference between in circuit tests versus flying probe test versus functional test and all these sorts of things. So once you finished soldering a circuit board together, and all, you got all your components on there, you want to know that it works right? It's no good. Yeah. Like I remember. Somebody was saying, I forget who it was. One of our customers, they, you know, it takes like 17 hours for them to assemble their finished product. And they, they, they weren't testing our boards before they went into the finished product. And I was like

Melissa: 5:57

Oh.

Chris: 5:58

sick to my stomach when they told me that

Melissa: 6:01

Hopefully they worked.

Chris: 6:02

be kidding me, I guess, I guess they had a high rate of of success and that's what they chose to do. But Anyway. So normally you'll have some kind of a test process that you'll do to most circuit boards before you put them into your finished product, especially before you ship them to a customer, because maybe you have a high enough success rate that you decide. Yeah. I'm just going to put it right in my finished product. And then I will run my product to see that everything works and then I'll ship it that way. That might be an option. But for the most part, you're probably going to have some kind of a test procedure and a lot of contract manufacturers will have. Flying probe tester or they'll have some type of in circuit tester that you can get custom tooling made for to do all the testing all at once. And again, it's, it's, it's a little bit beyond the scope of what I want to talk about today, but suffice it to say something will fail test and what happens next, because I don't like. Things that don't work. I mean, I, we recently bought a product from a supplier. It's just a, just a mechanical fixture for somebody circuit boards. Just a kind of an ergonomic anyway. It doesn't matter what it is, but we're missing parts and that's a terrible experience. Right? You never, you never want to send somebody an incomplete product. So, I would say wholly 80%, maybe 90% of everything we build, we don't test. You know, by design, like this is the arrangement that our customers want. They want to do the testing themselves. They don't want us to test them. And, and it's, it's much cleaner when when it's done this way for the cm, because it gets really messy when the cm does the testing. I don't mean that in a sense of like, we shouldn't do the testing, like I actually really do like to be able to test the circuit boards. There's value in it for us. And you know, there's value in it for our customer. Like I totally get that. But it's a little bit cleaner and, and the reason it's not quite as clean. Or excuse me, I should say the reason we're having this episode is because it's not quite as clean and we'll get an opportunity to talk about that. what do you do when you don't have a board, a finished assembly that passes tests? What do we do next? Because most CMs, especially CMS like us. They're not going to have a team of in-house test engineers that know how to interpret your schematic and know what to look for and try to identify the root cause of a particular test. They don't know what the functional test fixtures doing sometimes that, you know, there's, there's a lot of different things that, that they don't know what to do next. When something fails test. This is where having a conversation with your cm becomes critical because one of our customers, they do actually a pretty nice job of their, their product has an LCD display on it. And so when it fails a particular test that LCD display will say, Hey, look, you know, these locations, you know, U1, R13, C43 J J4 these are all related to this particular test, all these particular components are exercised for this particular test. Take a look at those. Did you forget to populate a part? Did you populate a part backwards? Did you have a cold solder joint? Did you ha you know, like there's all these different things that you might need to look for to see if you made a mistake. That's super, super cool. Yeah. Oh, it's so useful. We'd love it. But most, I would say post, you know, boards don't have a built-in LCD. That's going to tell you

Melissa: 9:36

Yeah.

Chris: 9:36

what to look for. A lot of times these boards may hook up to a computer either via Bluetooth or USB, and then you can have the computer have some kind of a readout that says, you know, when this particular test fails, Look at these relevant components. If you can do that, if you can build that into your functional test fixture, that is amazing. Like that is just solid gold to a cm. Cause then we can make sure that we didn't do anything wrong. Cause we just, you know, we can only control what we can control. We can't control what we can't control. We can move those things around a little bit. We can start to gain control over things that we don't have control over. And one of those things is when you tell them. That you know, perhaps you have a bad relay. We can go look at that relay. We can even try replacing that relay and then it works. So all of a sudden you're moving something that's out of our control. We have no way of knowing whether or not a relay is bad. And you're moving that into our control by saying, Hey, look, try replacing K three, and then we can do it. And that's, it's a, it's a really great way of helping your cm to get through their testing and deliver a hundred percent functional boards to you. That is sort of the dream world. It's not the real world. It's awesome when that does happen, but it's, it's pretty unlikely that that will happen. So you end up having to, like I said earlier, having to have a conversation ahead of time and talk about, okay. If you guys have, you know, so you're the designer and you're having a conversation with Chris Denney, or Melissa. And you're saying, look, if you guys have a failed board tag, the board, this is the best procedure. Tell them to tag the board with where it failed, you know, write some notes on the tag and then ship those boards separately to me. And then I can more confidently troubleshoot. Right. Cause I'm the engineer. I'm, I'm the one who designed the board. I know when it failed this particular test, I P it's probably either my power controller or it's this relay or it's the Fu the sensor, you know, like you'll have a much better idea of troubleshooting it. Now don't keep all that information to yourself. So we build a thousand boards for you and five of them don't pass the test. And we note down, Hey, they failed here, here, here, and here. When they failed the test. And then you identify whatever it was that they failed at and you go, oh you know, this had a bad pressure sensor and this, you know, had a solder bridge here. Hopefully not, we should catch the solder bridge, but you know, things happen. And note that down and start to build the best way is just to use some kind of like a Google sheet or Google document or some kind of collaborative. That we can work on together with you and build up a sort of database of, Hey, when it has this failure, last time we fixed it, fixed it by replacing this pressure sensor or when it has this failure, last time we fixed it by, you know, unplugging the USB and replugging it back in or whatever it might be. You know? If, if you start to build up the sort of a database, then all of a sudden we can start to deliver it to. Completely functional boards. And we can start to do that. I wouldn't say troubleshooting for you, but we can start to do those. Well, I guess what else would you call it? I guess you'd call it troubleshooting, right?

Melissa: 13:01

It's almost like pattern matching,

Chris: 13:03

Yeah.

Melissa: 13:04

this occurs, then it it's because of this most likely.

Chris: 13:08

Yeah, exactly. I like that idea. Pattern matching we're we're we're not troubleshooting. We're pattern matching. We're seeing the pattern in the failure and we're matching it to the fix. I like it. I'm going to use that. Thank you, Melissa.

Melissa: 13:20

You're welcome.

Chris: 13:21

More solid gold. But yeah, so it's like, you know, you start to build this up and then all of a sudden. Over the course of a year. You know, if we're building a thousand of these a month or something like that over the course of a year, and now all of a sudden we've got this nice rich database that says, Hey, whenever it fails here we have, we're pretty confident it's going to be related to this part and you can swap the part. And then lo and behold, you have a you have a functional board and we can ship that out to you. And everybody's happy. That is just a great way of handling. You know, potential issues. But like I said earlier, you got to have a conversation upfront with your cm because the cm, you know, a lot of times they're not going to know what to do with your failed boards there. Do you want these, are we going to invoice you for them? We know they don't pass the test, but we did assemble them. Right? Like we, we did our job. That's the tricky thing, you know, like it is an assembled circuit board, all the components are there. They're all the right direction. There's no bridging, there's no opens, you know, everything's good, but there's some failure, like some component failed and we don't know what component it is and we don't know why it failed. And, and we don't have the technical, you know, competency to go down to your schematic and the board level and figure out what component of that is. But.

Melissa: 14:41

Yeah.

Chris: 14:41

Right. You, you, you, you paid for the engineering or you are the engineer, you have the ability to troubleshoot that, and then you can give us that information and, and we can, we can use that for you. Happy to do that, you know, happy to do that for you.

Melissa: 14:58

So you're, we've been talking about the case where the cm is doing this test process. Perhaps it might be interesting to also talk about the case where the cm isn't doing the test process. We're just shipping the boards that we do assembled and the customer gets their board, and then they say, oh no, these don't, these don't work. So I think the best thing that you should do is you should just contact your cm and say, Hey, these don't work. Can you fix them? I'm going to send them to you, right.

Chris: 15:31

Yes, absolutely. Yeah. Yeah. For sure. I, and, and like, no, I, I would say no, I would say, yeah, let's, let's put some context to it. Let's put some meat on that bone.

Melissa: 15:44

that was supposed to be sarcasm, but it failed.

Chris: 15:47

They totally failed because in my mind, I'm thinking five boards, you're thinking a thousand boards send them to the cm with instructions, right? Because you, you. You, there may be a failed you know, micro BGA on that board. I don't think a lot of design studios are going to have the necessary tools to replace a micro BGA, but we do. And you know, you can say, Hey, look, I've narrowed it down to this component. It's not working. Can you, can you replace it for me? If your board costs $20? We're going to say, thank you very much. Here's your $20.

Melissa: 16:28

yeah.

Chris: 16:30

It's just, it's going to cost us way more time and money to replace a micro BGA on a $20 board than to just refund you your money for that. So that's one scenario but maybe your board is a hundred dollars and, or it's, or it's 200 or it's a thousand. And, and if you were able to identify that particular component, we're, we're happy to help you out there and, and replace, you know, these particular components. But you know, yeah. Sending them back to us without context would be awful.

Melissa: 16:58

We have, we have gotten that

Chris: 17:00

Have some kind of, I know, I know, I forgot that we have, because we've done a much better job of vetting these sorts of things lately than we used to. Right. We have we have

Melissa: 17:10

we're learning. We're

Chris: 17:11

us out with it. Yeah. We are learning. But no, like, so, you know, if you just have just every single one of them fails, you know, then That w that would be not good. Do not send them all back because we have to get to the root cause we have to find out why every single one of them is failing. And if you say it's because you put a 10 K where you were supposed to go to one K, well, then that's on me. That should never happen from, from one of us though, because we do we do verify our resistor and capacitor values before placing them on the board, but, oh gosh, that would be awful. Oh, Terrifying to think about. Thankfully, I haven't had to deal with that yet.

Melissa: 17:51

What Chris's nightmares are made out of.

Chris: 17:53

That seriously, that would, that would keep me up at night. But yeah, if you, so, so you, you take the thousand boards on delivery, you're doing your testing. Go ahead and put them aside. You know, when they failed tests, I don't recommend like, just as a best practice, I don't recommend like troubleshooting them. Then and there, those five boards, like maybe, you know, build up kind of. Kind of a a healthy supply get to 20, 30, 40 failed boards because usually what'll happen is, you know, there might be one component on that board. That is a little flaky, right? There might be some diode or there might be some, you know, it's almost, you know, it's almost always some kind of a semiconductor, but it's not absolutely, but almost always is. And then all of a sudden you start to see this trend, you know, and you're going to have 40 of them on a bench. And like 35 of them are all going to have the exact same issue. And then you're like, you can send all 35 back and you say, Hey look, replace U4 on these. And, and they're all going to work for me. I, you know, I did it on one and it worked and and I'm pretty sure all the rest are exhibiting the exact same behavior. Just replace all those for me. We're happy to do that kind of stuff. That is not a problem, you know, and just communicate that with us, get an RMA number from us. NTCM will have the same, like they're, they're, they're totally fine with that. Pretty much every cm is going to work with you on that. Don't don't stress about failed boards. That way. We're happy to do that, but again,

Melissa: 19:20

yeah. Get an RMA. That's an important

Chris: 19:23

Getting an RMA yeah. Get an

Melissa: 19:25

Don't just send

Chris: 19:25

you just send boards back and I feel like we've probably discussed this in the past. I know we discuss it with consigned materials. But when you send boards back, send them the same way we send them to you. Okay. In other words, in bubble, within ESD bag, you want to protect these things. We've had boards sent to us. In less than great ways. Let's just put it that way and it's like, look, I I'll do, I'll replace U4, for you, but I have no way of guaranteeing that any of this is going to be

Melissa: 20:00

Yeah. Like what else happened?

Chris: 20:02

What else happened in the meantime? Yup. And that we've seen that a bunch of times. Yeah. It's, it's been bad. So sure. You know, just, just Continue to work with us. Boy, I feel like I'm having like flashbacks that we've had this discussion before, like, like total deja VU, but maybe not, maybe it's just me. I'd love to hear your experience too. If you're a listener and you've had experience with the test procedure and what do you do do with failed boards? Let us know. I, I honestly, I would love to know

Melissa: 20:28

Oh, that would be an interesting conversation to have with someone as a guest. Yeah,

Chris: 20:34

Absolutely I'd love. And if you want to share your experience, come on the show. And you're, well-spoken love to have you, because as you know, Chris, Danny has a way of dominating these sorts of conversations. So you gotta like, you know, show up,

Melissa: 20:48

you do.

Chris: 20:49

Says Chris, Denny's cohost. You do Yeah. And so I will, I will say that if you, if you have the ability to build into your test procedure, some kind of note that points to. Like check these locations. We're happy to do it. We want to deliver like every cm wants to deliver a quality product to that passes the test and is going to last a good long time. So, so don't think that we're trying to cut corners or, you know, CMEs we are by and large reputable businesses that want to do a great job. So happy to do that for you. Yeah. So a nice short one. this is I'm trying to, you know, some people have shorter commutes. I'm doing the best I can here. I should, I should mention too that most, most sort of like one and done kind of jobs where You're you're, you're just having us build maybe 500 boards or a hundred boards or 50 boards or whatever it might be. You're probably not going to get those tested. To be honest, this kind of conversation is more about the sort of ongoing orders where we're building 500 a month or a thousand a month. And it's this sort of regular repeat sort of business. That's where this sort of conversation becomes important. Yeah, you build this relationship with your cm and you build these expectations and understanding of how we're going to handle things and yeah, it all, it all it'll work out. Great. Trust me, I I've, I've done this a number of times throughout my career and You know, it, it really does work slick. All of a sudden you'll get to a point where, you know, you're heading a thousand out of a thousand, pretty much every time you might get the one-off that has some weird error you can't, you can't drill down to. And it's really confusing, you know, but for the most part you're going to, you're going to achieve, you know, just about a hundred percent yield on, on your production. If you have this nice open communication and you're working together to try to get all that.

Melissa: 22:43

Yeah, going back a little way back. To our episode with Eli Hughes, it's all about conversations and building relationships.

Chris: 22:52

Isn't it though.

Melissa: 22:53

Yeah.

Chris: 22:54

Isn't it though? That's the whole world. Isn't it really

Melissa: 22:57

So secret of life.

Chris: 22:59

secret of life? I like it. You know, what else is the secret of life? My pet peeve this week.

Melissa: 23:04

what'd you got, she got.

Chris: 23:06

It's related to the, my previous one about being able to push a bathroom door open with your shoulder. So you don't get their nasty germs all over your hands when you pull open the door. But it's, it's bigger than that. If you have a door that swings and only one direction, which I would say is probably like 99% of doors in this world, it should be obvious. Based on the hardware used for that public door. I'm talking about public doors, not in your home, whether you should push this door or pull this door. Okay. If you have a sign on your door that says push, or it says pull this is likely because people get it incorrect, right? So you, you own it. You own a coffee shop or something, and we've all been to them and right on the door, it says. Or it says pull and you're like, oh, thank you very much sign. I can read that sign. And it tells me I'm supposed to pull this door open. This is a band-aid. If you have a vertical handle or any kind of handle for that matter on the door, that is a visual indicator to the human being, approaching that door. This door should be pulled from this side.

Melissa: 24:28

Yeah, That makes sense. Okay.

Chris: 24:30

makes sense. I agree if there is no handle and there's just like one of those like horizontal bars with the, like the pushy, innie outy thing that locks in place, when you, when you lock the door, do you know what I'm talking about? That is a visual indicator to tell the human being approaching this door. This is a push door, right? You need to push this door to open it. If you put a handle. On the side of the door, that's intended to be pushed. You have failed. You failed because I look at it, I look at a pull handle and I think I have to pull this. And then I go on a yank on it and I get, get, get, get gangs. And I look like an idiot and everybody inside the store and your big glass windows is looking at me like, look at that moron. He can't read the sign that says push on it. It's like, well, maybe I don't speak French in it. You know? Cause I'm going to a French Canada soon and it's going to. say "poosh" Not push or however,

Melissa: 25:26

That's exactly what it says.

Chris: 25:30

however it's spelled in French and I'm going to look like an idiot, right? Not everybody reads English. So I say doors should they should either, like, if they're a locking door, they should have that horizontal bore us horizontal bar that, that, you know, allows you. Visually see immediately that signal that tells you that you should be pushing open this door. And, and has the VAR vertical, or sometimes they're even horizontal, but it has a pull handle, right? There's there is a hand goes here, scenario on this hardware of the door that visually tells the operator. You need to pull this door open

Melissa: 26:10

But

Chris: 26:11

it.

Melissa: 26:13

I do have a counter argument.

Chris: 26:17

Oh, I want to hear.

Melissa: 26:18

So what if it's a door? That's not a self-closing door. So if you like open it up all the way, it doesn't just close back by it. So maybe you keep the door open during the day and then you, or you need to get, and then you need to, how do you, how do you get it close? If there's no handle, how do you pull it closed?

Chris: 26:44

That's a good point. Good' Melissa: cause then grab the outside of it. And then You're going to like, oh, squishy little fingers. So then you try to close it. fully. You, you make it, you make a valid point. You make a valid point. Now I will say we have some doors like this in our building that are properly designed. They have the horizontal bar for pushing open and they have the vertical bar for pulling open. And we need to leave them open throughout the day sometimes for, for various reasons. And we have a little bungee cord that holds it open, just

Melissa: 27:11

Yeah. Oh, I know I have a solution. You get one of those little foot like hook things like you do in the bathroom doors and you just see as

Chris: 27:20

That's right.

Melissa: 27:21

Problem solved.

Chris: 27:21

That's right. But yeah, I'm, I'm a I get frustrated whenever I see a pull handle on a door and I have to push it open. I'm like, Nope, this is no, this is not right. You've you've, you've made a mistake architect. You've made a mistake.

Melissa: 27:37

I think you're a real pet peeve is that you don't look like you don't like looking like an idiot when you're in public spaces, trying to push a door when you should be pulling it and vice versa.

Chris: 27:48

Yes, I am the rare breed of human being that does not like, and enjoy looking like an idiot. I forgot how many humans out there love looking like idiots.

Melissa: 28:01

Yeah. I mean, if you look outside, It can seem like that sometimes

Chris: 28:06

It can seem like that sometimes who wears that to Walmart. But that is my pet peeve this week. Yeah, I, I just, you know, and if you own a building that has pull handles, where you're supposed to push them, you should replace those. You, you know, your architect failed you, sorry. He failed you.

Melissa: 28:25

I like it. I like this door theme we got, we got started.

Chris: 28:30

We went from packaging to architects.

Melissa: 28:32

Yeah. Yeah.

Chris: 28:35

There's more. Oh,

Melissa: 28:37

I'm sure there's so much more.

Chris: 28:40

got loads of. I've got loads of stuff. Oh, all right. Well, if you want to hear more of Chris Denny's pet peeves, you're welcome to email us a contact@picplacepodcast.com or you can tweet us at CircuitHub or at w assembly. And as I've liked to mention the past few times we recorded these episodes. If you've enjoyed this and you know, somebody who is involved with circuit boards and you think that they would enjoy it too, please let them know about the show. It's really kind of the best way we can grow our audience. And we really.

Melissa: 29:11

Sorry. I was just thinking, cause you say it said if you'd like to learn more about Chris Denney's and he's put like you email, you can, I I'd like to request another pet peeve for you to just email me.

Chris: 29:22

Oh, I've got loads of them. Don't worry.

Melissa: 29:27

like request a random pet peeve.

Chris: 29:30

You seen that Kermit the frog GIF, where he's just like hammering away at his typewriter. That'll be me. If you email me asking for pet peeves, you're just a hundred words a minute for an hour.

Melissa: 29:41

So, yeah. Even email us and you might just get a random pet peeve back.

Chris: 29:48

No problem.

Melissa: 29:50

All right. Thanks for listening to the pick place podcast. If you like, what you heard consider following us in your favorite podcast app, and please leave us a review on apple podcasts or wherever you get your podcasts from.

Chris: 30:03

Thank you very much, everybody.