Paranoids Engineering: Supply Chain Security

In this episode of the Paranoids podcast, our hosts — Shawn and Steven — explore their colleagues’ work to secure the software supply chain.

Starting with the one question you’re all asking: What does that even mean in a world of open-source software?! 

Join us in conversation to hear discussion on: 

• Defining Supply Chain Security (2:36) 
• The Prolific Nature of Open Source  (4:38)
• Improving The Developer Experience (6:36)
• Explaining Common Supply Chain Security Attacks  (7:30)  
• The Different Pieces of Software Supply Chain Security  (11:40)
• Working Within the Paranoids (18:10)
• What’s Next?! (26:28)

Hosts: Shawn Thomas (FIRE Chief) and Steven Asifo (Technical Security Sr. Manager, Governance, Risk, and Compliance)

Guests: Nate Burton (Sr. Principal Technical Security Engineer), Hemil Kadakia (Principal Software Engineer), Yonghe Zhao (Software Engineer) 

Chapters

• 2:55 Defining Supply Chain Security
• 4:57 ‘The Prolific Nature of Open Source’
• 6:55 Improving The Developer Experience
• 7:49 Explaining Common Supply Chain Security Attacks
• 11:59 The Different Pieces of Software Supply Chain Security
• 18:29 Working Within the Paranoids
• 26:47 What’s Next?!

Transcript

Speaker 1: 0:38

Uh, hello everybody and welcome to another edition of the Paranoid podcast. My name is Steven Isif, part of the G r C team here, and we are going to be joined very soon by my host, my co-host that I'll introduce. But I wanna share the topic that we're gonna be covering, which is supply chain security, specifically as it relates to software. What are some of the challenges? How are we handling this at our organization? And what are some things that we can best, uh, bestow on to you? But, uh, I'd like to introduce my co-host, uh, who we have back again, Sean Thomas. How's it going, man?

Speaker 2: 1:12

Hey, Asif. O how you doing? I'm, I'm happy to be back. Um, as, as Steven mentioned, my name is Sean Thomas. I'm the director of forensics and, uh, incident response engineering here at Yahoo. And I am super duper excited about this topic. Supply chain is a big thing in my line of work, as many people could probably imagine.

Speaker 1: 1:31

All right. And now we also have some special guest joining us from our team. We have Yahi, Zao, Nate Burton, and Hemo. Kaia Ya. You wanna introduce yourself first? My friend?

Speaker 3: 1:42

Yeah. Hi. Hi everybody. I am, uh, Yoho. I from, uh, paranoid engineering team. I am a software, uh, engineer, and, um, I mainly responsible for, uh, designing and implementing security related software. And I'm very excited to be here to share my, my knowledge.

Speaker 1: 2:04

Hey, let's go. Uh, and Nate, would you like to introduce yourself?

Speaker 4: 2:08

Hey everyone. Uh, I'm Nate Burton. I'm a, uh, senior principal engineer on the paranoid, uh, and I focus on the cloud container and compute security programs and building out, uh, those capabilities so that our platforms are security.

Speaker 1: 2:24

Okay. And last one, certainly not Liz Hamel. Uh, can you introduce yourself please?

Speaker 5: 2:29

Sure. So my name is Hamel Kadakia. I am a principal software engineer within the Paras engineering hall, and I lead the initiative for software supply chain security at Yahoo and coming up and working together with the SDLC folk as well.

Speaker 1: 2:46

All right. And maybe let, let's start right there cuz that's what the topic is about. Ammo. What, what is, you know, software supply chain and, and what does that look like from a security aspect?

Speaker 5: 2:55

Definitely. Yeah. So let, let me put it in a later term, right? Uh, let's have an analogy with what exactly. Software supply chain comparing with a food supply chain. So just like food, we, where you plant the seeds, you grow them, you harvest them, you make some food, and you finally deliver it to the dinner table. Software supply chain is quite similar. You write some code as a developer, you check in the code, you build the image, you, you build some binaries, you test it, verify it, and eventually deploy it to production. So, uh, it b the, some of the questions which come up during food is we wonder where does this food come from? What country? Is it organic? Is it vegan? Is it gluten free? All these questions that are meant to ask just for food can be asked for software as well. What about third party dependencies in your code? Who wrote the code? Where is, where was it built on? Things like those. And that's where, uh, software supply chain comes in. And as I mentioned about, uh, third party dependencies, right? So, uh, a lot of the code, which we use at Yahoo, uh, and in the industry is open source. So around 85 to 97% of our code is open source. And that means most of the application, which we use is not written by us.

Speaker 1: 4:22

And that feels like that's coming more common across the industry in general.

Speaker 2: 4:25

So there, there's actually a really fascinating, uh, comic that covers this too, from xkcd, if anybody's familiar, um, if, if you look up XKCD open source software, they have this big tower of blocks that's all modern digital infrastructure. And the one thing holding it up, it says a project, some random person in Nebraska has been thankless maintaining since 2003. It's like the perfect analogy of, of what we're talking about here and about how we use software to build these massive systems in today's time.

Speaker 4: 4:55

Certainly. Yeah, I, I think the, the point about the kind of prolific nature of open source is pretty interesting too, cuz like, you go back two decades and open source was still kind of like competing with, uh, commercial software. And at this point, like open source has essentially one, uh, it seems like it's, it's in almost everything you use, even if you don't directly consume open source, you're probably using some third party software that has a bunch of open source dependencies packaged into it.

Speaker 2: 5:26

So let's actually dig into that a little bit and, and why this is so important. Nate, I'm gonna stay with you. You know, why is supply chain security important? How does this impact businesses?

Speaker 4: 5:37

Um, UL ultimately, uh, supply chain and security is important because, uh, similar to what Hemel mentioned, uh, we, we both produce our own software as well as consume other people's software. So understanding, uh, what is in that software, how it was built and secured, what sort of vulnerabilities it might have. Uh, thinking back to the, the, the food supply analogy, like when people, uh, when when manufacturers discover that there are problems, uh, with a particular production of food, they do food recalls in the software and supply chain space, we have vulnerabilities. It's kind of a, a very similar analogy there. And so it's all about understanding like your, your posture, what you have and where it is. Um, it's kind of a foundational layer.

Speaker 2: 6:30

Excellent. Hamel, is there anything you want to add on to this?

Speaker 5: 6:33

Yeah, definitely. Uh, so, uh, for, as, as Nate mentioned, right, uh, when we need to recall vulnerabilities or if a software we know has, is susceptible to attack, we, we need to issue some sort of guidance for the developer community to make sure they fix those. And by shifting lift and catching all these vulnerabilities and issues early on in the development lifecycle, we are avoiding potential down downtime and also improving the developed experience so that they can proactively manage what they've been working on and not be busy in fixing bugs once it's deployed into production. So we making our system and software stable and also, uh, improving developer experience by ensuring they catch issues and focus more on the business value and applications they're building, and leave the security up to the professionals and make it easy for them to work on as well.

Speaker 1: 7:32

And so maybe this isn't as prevalent in like the food industry, but when we talk about like, software supply chain security, what are maybe at a high level some of the like attack chains that, you know, someone would do to try and influence, um, an organization youngi? Would you be able to like speak to

Speaker 3: 7:49

That? Yeah, yeah, definitely. So I would like to, um, say like, if you want to understand the suffer suspension attacks, you first need to understand like, what is the difference between the suffer suspension attacks and the other, uh, traditional attacks like, uh, secure injection or buffer overflow attack. So I think, uh, why, uh, suffer suspension attack is so popular today? It is, uh, it is not a attack that targeting, uh, a single machine or a single components. It is, uh, uh, targeting, uh, whole, whole system, uh, software development lifecycle as, uh,<inaudible> said earlier is there are, there are so many, so many components and steps during the, uh, from the, from the source code to the final binary. So the ator can actually attack any components inside, uh, or any links during this procedure. Um, so actually I can give you some some concrete example for, uh, for the sub suspension attacks, you can, yeah,

Speaker 1: 9:04

Yeah. Let's keep it like a high level. Like what are, like, what does that look like concretely?

Speaker 3: 9:08

Mm-hmm.<affirmative>. So do, do I, I need to share any, some attack. Okay. So, uh, it can include some very, uh, hardcore attack. Like, um, for example, uh, uh, last year some researcher, uh, researcher find the VS code in the VS code, uh, set pipeline. There's, uh, back inside it that we can, uh, get some credential and gets, uh, permission to write the, uh, s code, uh, repository. And that researcher actually created a p c, uh, push to the, uh, release branch of the VS code repository. So you can include any, any vulnerabilities and any malicious behavior. So it is, uh, dangerous, but other than that is, um, I think the server attack can be more like, uh, more like, uh, sending phish email. So kind like a social engineering, for example, there's another famous type of, uh, sub suppress attack called type of squatting. For example, if you, uh, write a Python projects, there are a lot of, uh, dependencies and there's one diff dependency called u R URL live. But, uh, once a day you make a type type, uh, you are live just missing an L and that is another, uh, another, uh, I mean package that including some, uh, crypto mining code. So it is another totally different<laugh>, yeah, totally different<laugh>, but is, uh, definitely, uh, totally legitimate. And, um, another very interesting thing is, uh, the open source, open source code maintainer can be the attack itself. So for example, last year there was some, uh, no, uh, no JS or Java screen package that the maintainer just, uh, uh, raise the poor request and merges, uh, including some, uh, his, uh, political opinions inside it and create, maybe create a, a tax file inside the user's, uh, desktop, something like this. So, um, I want, what I wanna say is, um, server suspension tech can be totally different to those hardcore, uh, traditional attack like, uh, injection. We just need to, uh, equipment the single, uh, machine or single component like database, um, to prevent the software supply chain attack. We need to care about a lot of things, lot of aspect.

Speaker 2: 11:57

So let's, let's stay on that for a second. Let's, let's kind of dive into the practicality of how this fits into supply chain security. You know, how do we take these attacks using this, this process related to open source projects? How do we make them more difficult for attackers? How do we make them impossible for attackers? Simon, do you wanna start us off there?

Speaker 5: 12:17

Yeah, sure. Uh, so as I mentioned earlier, and lot of us touched upon it, right? Like software supply chain is not just a buy one solution and you, you purchase it from a vendor and you can secure the whole system, but it's more, uh, different pieces coming together to build the software. Supply chain or the software development lifecycle includes code which is committed through a, a different tool. Callers, GitHub, we have your C I C D pipeline, which builds and test your code. You have some sort of registry where you store your artifacts. So there are multiple pieces in place, and if one has to secure their own system to, uh, guard against these open source or supply chain attacks, we need to look at each of these individually and make sure they are secured once we can, uh, individually make sure that they're secured, and we can then focus on a bigger picture to make sure that complete pipeline is secure as well. So for, for example, when I'm trying to commit a code and I, I'm trying to run it in production, I wanna make sure that the code, which I wrote is the one which is being, uh, published and being run in production. I don't want a third party code or someone else who's trying to manipulate my code, uh, is running that piece of, uh, I'm running that piece of software in my production environment. So making sure that we, we secure each service and tool, which we are using at Yahoo, uh, is quite critical and important to, uh, make it difficult, I wouldn't say impossible for the attackers to attack, but making sure if we secure each of those, we are making it more difficult for the attackers to get information or the data which they would require from an attack

Speaker 2: 14:12

For sure. And, and from the IR world, one of the things that we really call making things more difficult is imposing cost on an adversary and imposing cost on an adversary is always a good thing in our world. This, however, is a super important question. So I actually want to continue to go around, uh, youngi. Would you like to, to pick up next and dive in?

Speaker 3: 14:31

Uh, uh, yes. Uh, so let me talk about like, uh, what, what can we do to prote, uh, protect the, our some supply chain? So there are actually a lot of, uh, source standards to use, uh, for you to check. Uh, what, uh, how well did it do for, for your, uh, system? Like, uh, one of the famous one is, uh, open SF SS f uh, scorecard. It is, uh, cannot checklist. They also have a automatic check, um, c i two that you can use it to check your, um, open source repo. So actually they have about like 20 or so, uh, checks. Uh, if they find that okay for this check, you do this, you get at some point the full, full score is 10 and lowest score is zero. And also you can get a badge on your read me open source Readme fail to, uh, say, how many square did you get? It's a very, yeah, that's a very good, um, measurement to, to for also give a confidence for the users to like, how did, how well did you do? And, uh, another, uh, standard of very famous is, uh, s is, uh, s L S A, the full name is, uh, supply chain levels for, uh, software artifacts that is also, uh, uh, it is not a score, but is, uh, similar to the cannot auto autonomous, uh, driving levels in auto autonomous driving. Uh, L zero is a completely, uh, self-driving, but the L five is very smart, completely drive by vehicle. And, uh, similarly as a sales means, uh, in sales, uh, L zero is you completely do not have any protection against your software prevention. And, uh, L four is, uh, you are very secure. So each level you are incrementally get, uh, more, um, uh, security. So if you can follow those, uh, open source standards, you can get more security.

Speaker 4: 16:51

Yes. Uh, the nice thing there about, uh, salsa is it's kind of a maturity model. So you can, you can look at your end-to-end supply chain process kind of that, that hamel described and all the different parts. And then over time you can build out more capabilities, instrumenting those different steps, finding the best solutions that, that, uh, compromise, uh, balance between user experience and, and the security benefits provided. Um, and I wanted to kind of touch on a point that Hemel had mentioned there is that, uh, he had mentioned that in some cases you, it's, it's, it's kind of impossible to have a hundred percent security. Like there's gonna be software bugs, there's gonna be misconfigurations. And a big aspect of, uh, supply chain security that often gets overlooked is that you still have all the traditional defense in depth controls that you wanna have in place. So even in the event of a vulnerability, uh, in, in a particular piece of software that you might be using, you're confining, uh, any sort of attack and limiting what, uh, a malicious actor could do, uh, in your environment or in your application. Uh, so defense in depth is, is a really critical aspect of this too.

Speaker 1: 18:15

And Sean, were you gonna add something onto that, or No, I was actually just gonna pass it over to Nate. He did it for me, which, okay. All right. Um, okay, so we've kind of covered like the higher level there, and I guess I want to look a little bit at our organization and just say like, how did this kind of come about as, uh, a priority for the paranoid and how did we kick that off? Or really, if we're looking at the supply chain, like where do we say like, all right, hey, here's where this might be a pro a priority for us to start, just again, at like a high level. Nate, I don't know if you wanna start.

Speaker 4: 18:49

Um, I, I can start. I, I think one thing that's interesting to call out is, uh, kind of how we work within the paranoids and within Yahoo in general. Um, we have a, uh, a top class security organization that Yahoo, uh, the paranoids, and we have a lot of great, uh, subject matter experts in the variety of security fields. So the, the forensics and incident response teams, the vulnerability management teams, operations, paranoid engineering. Um, and so when it comes to, uh, implementing security measures, um, it's, it's a collaborative effort between like determining what the business and security requirements are, identifying where our gaps and capabilities, uh, exist, and then partnering across the organization to, to actually implement those changes. And, and so that brings, um, like having that diverse set of experiences and backgrounds helps, uh, really drive, uh, top quality solutions as an outcome. Cuz we can both partner within the, the security organization for expertise, but also reach out and partner with our platform engineering teams so that we are identifying concerns and pain points for them. Um, like maybe we're overwhelming them with too many security alerts or, or things. So like, how do we focus and refine our capabilities to cut back on noise and things like that. Um, I think, so

Speaker 1: 20:18

It's not just one team as a number of, not just one team within the paranoids

Speaker 4: 20:23

Collaborator. Yeah, we're, we're, uh, we've got a, a, a wide variety of organizations that specialize in, in areas. Um, and then I think Hemel has a bit more background maybe on the impetus around the supply chain security process that, that we built at Yahoo.

Speaker 5: 20:38

Yeah. Before, before I actually talk about what, how, how the process or how we started software supply chain security. I, I just wanna mention that within Yahoo and Paranoia in particular, uh, working with a bunch of smart security folks, uh, makes software supply chain and getting this much easier and making it, uh, company-wide a priority is quite important. And having the backing of security folks makes it a lot, a lot more easier as well. So kudos to Paranoids, uh, out here for helping, uh, making this priority at Yahoo. Sure. Coming back, we, coming back to what Nate mentioned. So, uh, a lot of people might have heard as well, and there was a, a big, so a software supply chain attack called SolarWinds, where, uh, some foreign country, uh, attacked the built pipeline system for a company called the SolarWinds. And a lot of the, uh, three letter companies and industry-wide people were using those software. And some, due to the attack being part of the build system itself, a lot of the companies were affected by it. And, uh, luckily Yahoo wasn't part of it, but that was an eye-opener for the industry as well. And people started taking software supply chain security quite seriously. Uh, this is been an issue for co this has been an issue. I

Speaker 2: 22:08

Had a years that didn't sleep for a long time with SolarWinds, actually.

Speaker 5: 22:12

Yeah. And the, uh, the thing you mentioned, right? Uh, SolarWinds was just an eye-opener, but software supply chain security has been there for quite a long time. But, uh, and when we started our journey, uh, we were quite early into the, uh, industry as well. Cause some of the tools which we use, uh, are more homegrown and proprietary to us. But now when you look at the open source world, the open source community has developed lot of products where when we started out, they weren't in place, so we had to build something from scratch. But if I would be starting all over again, I would definitely be using some of these open source, leveraging some of the, uh, expertise from open source and, uh, the security expertise from Yahoo to make it much work much, much more seamlessly as well.

Speaker 1: 23:02

And we definitely wanna get back around to like, lessons learned and, and what we Yep. Uh, would've done looking back.

Speaker 2: 23:08

Yeah, for sure. But so, so I, I, I wanna stay with you for a minute, but I actually want to go back a little bit to something that Nate had said about, you know, the, the sheer number of partners and the people who are involved in, in those partnerships. You know, how do you really measure and get feedback to, to make sure that you're moving in direction?

Speaker 5: 23:27

That's a really good question. Right. Uh, since, uh, lot of, within Yahoo, we have a lot of products, lot of teams working on different stuff. Uh, uh, it's not always easy to be aligned on the same page and the priorities for each team are completely different. But, uh, looking at the product project in general and the initiative, the, a lot of the stakeholders have been quite, uh, helpful and, uh, are working with us to make sure that we get these, uh, deployed and used by the developers as soon as possible. But with that being said, uh, there have been some of the pain, some of some pain points, which we've faced in the past, right? Uh, number one being around adoption numbers. So Yahoo has lot of services running. We have, uh, a C I C D tool, which people use and build their binaries, and there's no one standard for everyone to do the right way. So as a developer, I could just go and deploy an image using the way I feel like. So, uh, by ensure we, we need to make sure that we get those images which are deployed, scanned as well. And by relying on developers, we, we make it difficult and we increase the friction between the security folks and developers because we keep on nagging them to add these checks to their pipelines or a day, day-to-day, uh, uh, development work. So by making a lot of the stuff, uh, transparent from the users, uh, we are making sure that we increase the adoption numbers, and we also get continuous feedback by working with folks within Yahoo, like Deputy Paranoids and lot of other stuff as well. Uh, yeah.

Speaker 1: 25:24

And I guess on that, of taking the feedback and how we're measuring, maybe we can step back into that, like lessons learned and just looking backwards a bit, youngi, is there anything else that you've thought about like, Hey, if we were to do this again, or just some of the biggest takeaways from this work?

Speaker 3: 25:40

Yeah, I, uh, firstly, I t want to eco, uh, I, or just now about we have to use open source, uh, uh, open source projects more, more. And then, um, we need to get, uh, adoption number and think about how to get it earlier. And another thing I can think about is, um, we, uh, have to, uh, evangelize, uh, the software supply chain more because people, if we, uh, are enforcing, uh, any, uh, security check, they may find, okay, why, why this block might build, why is this block my deployment? So, uh, if we can have any um, uh, kind of, uh, presentation for the company, like, uh, for articles spread around the whole company so they can know, uh, the purpose, what we are doing as, uh, that will be great.

Speaker 1: 26:41

Yeah. And maybe we also like, after looking back, like look, looking forward a bit like what are, what is next, maybe at, again, at a high level for where we're thinking of going and continuing to build this workout. Nate, I don't know if you wanna share anything.

Speaker 4: 26:55

Yeah, I can, I can touch on that. Um, I think one area that like people sometimes make mistakes in is they say, security's important. We need to do security. Okay, developers, you have to be security experts. Um, and I think that's a, that's a false approach. Um, and, and we can balance that. Um, kind of in the earlier discussion there is the, the talk of the shift left approach. And that's what we've, uh, implemented mostly in our supply chain security pro, uh, process. But it's, it's all about optimizing that user experience and understanding, uh, your stakeholders and where they best operate and how they're most efficient at doing their jobs, and then trying to implement those solutions to meet them at that, that level. So, um, from next steps for us, uh, for me at the high level are, are like optimizing that user experience, reducing the noise of any sort of the tooling that generates findings, uh, making the findings more actionable so users have easy, uh, kinda playbooks for how to respond to, to a finding, um, uh, so that developers can write the code, which is their expertise, uh, to produce produce value for the company.

Speaker 1: 28:13

That's wonderful.

Speaker 5: 28:15

And just to add on, Nate, right? So since Nate and a lot of folks work with the developer community, we as more engineering, we, we wanna work more in terms of bringing more security and tools for the complete, uh, supply chain. So from Paranoids engineering, what, what we would, we are focusing and super excited to work on is something called as ASBOs. Uh, it's really hot in the market right now. If, if you just look at the executive order passed by Biden last year. Uh, and what does,

Speaker 1: 28:48

What does SBAM stand for?

Speaker 5: 28:50

Uh, it's called software bill of Materials, correct? It's, it's basically includes all the information. So, so if, uh, again, an analogy with food, uh, so when you look at a food label, it's gonna give you all the ingredients where the food, how the food is actually made. But Ebom goes one step further down saying that, which facility was the food made in? Who touched that particular food, and how exactly it got to you where you belong, right? So ebom is basically is for software where it would include information for, uh, what libraries you're using, what version of library, where exactly did you download that library from. So, and that includes a lot of information. So making sure we parse all that information and make it available to security experts to make informed decisions whether to, uh, decide whether a particular library or package is vulnerable, who within Yahoo is currently using it. So an example being locked 4k, where a lot of us were scrambling to figure out, uh, what all is currently running in production. And lot of us had sleepless nights to fix those, and some of, some people still have them. Uh, so by using SBOs and an easy way for new vulnerability team and other teams at Paranoids, uh, to find that we are gonna make life much easier for other folks as well. So that's coming up soon.

Speaker 4: 30:21

I, I, I think another, uh, area for future work is, uh, Hemel talked about how oftentimes, uh, uh, you can be ahead of the industry in a particular area, and so you have to develop kind of bespoke solutions to solve the immediate need. Um, but it's really, uh, powerful and useful to, to leverage all of the work produced in the upstream open source community, and then also contribute back. So like we can, we have the opportunity based on our history and, and the work that we've already done in the past, building our own capabilities to take those lessons learned and also work with the upstream open source communities to contribute some of that back. So I think that's an area where we'd like to do more too.

Speaker 2: 31:09

Fantastic. And, and ultimately, folks, I have to apologize for all of you who have been with us so far, but this concludes another phenomenal proceeding of the paranoids Podcast. Say that 10 times fast, everybody. Um, I'd be remiss to not shout out the fantastic team behind us, um, my wonderful co-host, Mr. Steven Asif and our wonderful guest doing some awesome work. Lastly, but definitely not least all of you who listen, hopefully this is valuable. You have learned something, whether you are a prospective employee, we would love to have you come work with us or just somebody who's interested in security stuff. So with that, goodnight. Good morning, good afternoon, everybody, depending on what time zone you're in.

Speaker 4: 31:49

All right. Talk to you guys next time.