Genealogy of Cybersecurity - Startup Podcast

Ep 10. Endor Labs on Code Vulnerabilities, Sketchy Open Source Developers, and Software Supply Chain

July 24, 2023 Paul Shomo / Endor Labs Founder Varun Badhwar Season 1 Episode 10
Genealogy of Cybersecurity - Startup Podcast
Ep 10. Endor Labs on Code Vulnerabilities, Sketchy Open Source Developers, and Software Supply Chain
Show Notes Transcript

Innovation Sandbox finalist and Endor Labs Founder Varun Badhwar discuss the magnitude of open-source vulnerabilities, highlighting the developers behind vulnerabilities like CoreJS and Log4shell, and why strategic pieces of the internet depend on libraries that sometimes rest on a single part-time open-source developer, even developers with prison records.

Varun talks about his past pioneering cloud security posture management (CSPM) with RedLock and Palo Alto Network’s PRISMA cloud, and Endor Lab’s mission to build a software composition analysis solution that truly enables developers and solves the problems of open source vulnerabilities. Including how Endor Labs is going further than simply shifting left.

You can find Varun Badhwar on Twitter @varun__badhwar or at LinkedIn.com/in/vbadhwar.

Visit EndorLabs, or find them on Twitter @EndorLabs, or at LinkedIn.com/company/endorlabs.

Send feedback to host Paul Shomo on Twitter @ShomoBits or connect on LinkedIn.com/in/paulshomo.

There's another one recently, a project called CoreJS, which is a maintainer living in Russia. This codebase runs the entire web application on the Internet, Netflix, Amazon, Google, your name and everybody's using components of this. This person is pleading on the Internet saying, I don't have financial help. Please help me. He went to jail for ten months or for an accident. During which ten months, there was no updates on the code.
Yet, it's foundational to our Internet ecosystem. Something is very wrong here. And we just need to make sure that the users and where Endor Labs is focused is helping consumers of open-source software. Do their part, which is understand what open-source your developers are using today? The genealogy of innovation is the cybersecurity startup and emerging tech podcast. I'll interview top entrepreneurs. Startup advising cisos, venture capitalists, and more. The startup world is full of innovative minds.
This is the place to explore new threats, new approaches to cybersecurity, and more importantly, attack surfaces that arise as technology regulation and business evolve. Welcome to the genealogy of innovation. I'm Paul shomo. Hi, everyone. My name is Varun Badhwar. I'm the founder and CEO of Endor Labs. Which is a startup focusing on open-source software security and governance.
Prior to this, I built two other startups, most recently a company called run lock in the cloud security posture management space that was acquired by Palo Alto networks where I went on to create Prisma cloud, had spent about three years building and scaling that up. Well, first of all, I actually want to ask you about RedLock, but first off, I want to say congratulations for making innovation sandbox finalists, big deal. Thank you so much. We're very excited. This is a third company that has got the finalist title and it's always been an amazing platform.
So we're just thrilled and humbled to be back up there and hopefully this time we can go all the way through and win the title. I do want to ask you about red light because I've been interviewing founders for 7 years and well over I don't know if I'm over a hundred. I don't think I'm up to 200 and I continually bump into founders that came out of RedLock and could you help me understand that like what about red lock was full of made it full of so many entrepreneurial people?
You know Paul RedLock had this just we had the most incredible team that we were able to put together to solve a very large problem around cloud security posture management.
We built a great product that was very well received by our customers and we just saw this tremendous trajectory of success not just creating the category of cloud security posture management but as you can now see you know there's just it's one of the fastest growing categories still 7 years later you know you see new entrants like Wiz and others coming in and challenging what was red lock and is now a Prisma cloud and I think the DNA of a lot of people there has this has had this entrepreneurial spirit has had incredible learnings of going from zero to several 100 million ARR, hypergrowth and I think people are just excited and take their learnings from there and find other ways to solve other hard problems in bigger markets.
I'm just curious I've asked one more question about this because the red light phenomena has fascinated me for some time. Was there anything special about your recruiting that you found the kind of people that would go out and do those things? You know there was a couple of things about a culture which were really aligned to and maybe one of those that was like as a CEO of that company I was always looking to work myself out for jobs so when I was recruiting for leaders in other areas it was always looking for people that could kind of get up there and be extremely successful at what they do and be very confident in the decisions they make in doing so where you know they weren't looking for me to make every decision for them.
So I think there was a part of that DNA that certainly helped. We also were not shy to find the most diverse experiences and we didn't hesitate to give people a first shot at a leadership role in the company and so I think the big thing and I look back was autonomy like leaders had autonomy to make decisions to fail fast to learn and scale up, right? And so I think that mindset and that training probably is one of the reasons why you see a lot of successful leaders in the market that's spent a few years at red lock.
But let's get back to your current area of business and I want to talk about the problem space first before we kind of get into how your product what it does and how it works. Can you kind of tell us from a mile high view if we're weighing risk across the supply chain? How big of a problem is open-source? Simply put 90% of your problem is open-source.
Here's why you know you can look at a lot of different reports somewhere between 80 and 90% and growing percentage of your modern application code is code you don't write but borrow from the Internet where you borrow it from complete strangers on the Internet off open-source ecosystems that are thriving in places like GitHub. I was fantastic because it's really fostering innovations speak to market for developers. You don't have to rebuild capabilities all the time. But also what fascinates me in these so called suppliers in your supply chain security not really suppliers.
You have no contracts with them. You have no legal support agreements with them. You don't pay them a dollar and they're just good samaritans that are providing you code as is with the warranty. Now fundamentally what that means is the onus is on you the user model that open-source code to understand those risks evaluate those risks and make risk based decisions on your comfort level of using that code on critical applications.
So you know without going too much into what Endor Labs does that was the foundational problem and why we decided to focus on this because this time I shipped it in the last 5, 7 years from where 90% of the code was first party code that you wrote and now 90% of the code being code you don't write. And open-source software, there's historically been a pretty I don't want to say fanatical but very enthusiastic supporters of open-source and it's you know it's taking a little bit of beating recently is would you say open-source software has become inherently less secure over time or is it just more widely used right now?
I think it's a bit different. Honestly a open-source is everywhere now. It wasn't necessarily like you didn't have critical infrastructure applications using open-source and things of that nature. Second, the pressure on the businesses to move faster, innovate, faster means they can't block access to the open-source ecosystem for their developers. They used to be the case. 5, 7 years ago in a regulated enterprise, you could access GitHub dot com. Guess what?
There's not a single place I could find today where you have that level of an originated blocking controls. So open-source is a catalyst for innovation. Now, the problem is we've used it in this super exciting manner with implicit trust, but what happens the bad actors figure this out? And they say, great, this is an easy way to environment, is just go get developers to use some malicious open-source or compromise their own vulnerabilities.
And great. Now I have a path where I don't have to break through firewalls and access controls and other ways to get in. Most people have underinvested in open-source security, which is why all was really interesting to me is you are now seeing the government lead way with regulation where the industry is now in front of a catch up. You don't see this very often. You've got the executive order from The White House a couple of years back that talked about open-source being a national security issue.
You now have as of two weeks ago or a week ago, a bill passed in the Senate already around securing open-source code. Any of lots of other things around S bomb requirements and other aspects may fit in in terms of requirements. So I'm glad to see the government really pulling the movable by its born and industry. We need to do more because open-source and now here to stay and I think everybody needs to make the right level of investments to catch up.
Very good point. So people back in the day you used to describe open-source as it was something that came with an army of developers to fix security issues. And for many years, there had been a perception that open-source was actually more secure than proprietary code. You're obviously, you're pointing out the flaws and open-source code, but you're not trying to get rid of it, you're trying to help it be used more responsibly.
So could you maybe help us from your standpoint of expertise and maybe compare and contrast the open-source versus proprietary development approach for what kind of security flaws or basically make that compare and contrast for us? Sure. Great question. Look, it's not that open-source software is inherently insecure. I think when the argument for why open-source should be trusted more is the code is available to everybody.
Any security researcher in the world can see it. They can find things. It's great. There's transparency. On the flip side, when people think open-source you've got to remember, most of the open-source code on the Internet that's been used today is not the one that's been created by Google, Facebook, Apple, and others. It's actually a single individual that decided to do a project as a hobby, food became something that could be reused. Now it's available to all.
Now you've got millions of people and literally millions of downloads a week happening on popular projects that are being maintained by somebody who spends ten hours a week on it. It's not sustainable. The second thing is, we got to remember nothing in this world is free. And this perspective that organizations enterprises have when they consume and their developers are using open-source, it will allow us open-source.
Okay, well, yes, it is, but the difference between Google and a traditional enterprise with how open-source adoption works is huge at a Google like place as an army of developers that are there vetting all the open-source that wants that people want to use within Google. They're maintaining it. They're providing support and contributions back to the open-source community. And when these are really critical projects, they're actually working them and then managing and maintaining it themselves.
In a traditional enterprise, which is 98, 99% of the world, that doesn't happen. They're single developer brings in this open-source hopes and praise that that person sitting in Arkansas was working on this project as a side thing is able to keep up. And it doesn't always happen. And you know the example I want to give you here is when log four J happened, obviously, we all still get nightmares with it.
Hundreds and thousands of companies send the maintainer of log four J spreadsheets a questionnaires to complete other security posture and these people just laugh them out of the door saying, I don't know it to you. You don't want to use my code, don't use it, but I'm not sitting for free and writing spreadsheets and responding to you. There's another one recently, a project called Core JS, which is a maintainer living in Russia. This codebase runs the entire web application on the Internet. Netflix, Amazon, Google, your name and everybody's using components of this.
This person is pleading on the Internet saying, I don't have financial help. Please help me. He went to jail for ten months or for an accident. During which ten months, there was no updates on the code. Yet, it's foundational to our Internet ecosystem. Something is very wrong your ball and we just need to make sure that the users and where Endor Labs is focused is helping consumers of open-source software do their part, which is understand what open-source your developers are using today.
Evaluate the risks around it, and then make risk based decisions on how, where and if you want to consume it and how you're going to manage that code base on a going forward basis. Those are actually really amazing stories. I think the perception of open-source is it's this group of people. It's this hive of all these brains together that are working to maintain it, but in a lot of cases, it's a person, and we don't really think of who that person is or their background, whether they're trustworthy or not.
Some really good stories there. Log four J was obviously the worst in longest running vulnerability that I can recall. And I think that's a pretty easy assessment to make. Obviously, there's been a rise of DevOps. Everyone's becoming a software company, codes being spread all over the place, the cloud, microservices APIs. Is that decentralization why log four J was so hard to handle? Like what's your assessment of why that was so difficult for the industry? Multiple reasons.
First and foremost, you're letting your developers bring in whatever software you need from the Internet and a lot of organizations weren't even cataloging or inventorying that software. So that's problem number one. Problem number two, the way the whole open-source ecosystem works is there's this concept of direct dependencies and transitive dependencies. So allow me to explain this. Your developer may be bringing it back in full. Okay? So in your developer's mind and in your inventory, you said, great.
I know I use food. Now, foo made itself be using as part of its development. Hundreds of open-source dependencies. And those open-source dependencies might be using hundreds of others. So you could end up with log four J 6 levels down in your code where your developers didn't even bring it in. But it's there. I call this the uninvited guest to your party. You don't have them on the RSVP list. You then track their kind of COVID vaccination status, but they're there.
And you know nothing about them. And so the first problem was just really visibility. Where do I have law enforcement? And speaking of decentralization, you'll remember a large enterprise has thousands, if not, tens of thousands of code repositories. So you can't just go in one place and look. You got to look at thousands of replaces. And so this is where most organizations spend weeks in war rooms with spreadsheets saying, please tell me if you use log forces somewhere, unsustainable.
The next problem was once you've figured out where log four J is, was am I using log four J in a manner that makes me vulnerable? In order to answer that question, you have to do code reviews to figure out if you are using the lines of code that have the vulnerability. Because then you would prioritize those things getting fixed. None of the tools today for software composition analysis actually do that.
That's one of the big things we have against the existing state of the art capabilities in the industry and why one of the reasons we think we're here at the innovation sandbox as a finalist is because we're changing that kind of technical approach to finding and discovering that. The third thing happens, okay, I have it. I'm using the vulnerable function. I need to update it. The biggest nightmare for engineers is figuring out if I update this piece of code, what else will break? Who else depends on it?
And so this just makes managing open-source hard. And this is why engineers have given it a firm. The term is called dependency how dependency hell exists because it's very complex web of code, which is hard to understand the impact of changes and it's hard to understand where you're using it. Quick question about the executive order. So that obviously came after SolarWinds and all the chaotic supply chain attacks. And that was in the United States.
Our other countries around the globe following suit with them. Yes, we have seen similar regulation now emerged in the UK and in Europe as well. So we think it's a matter of time you know. Just like food labels became ubiquitous around the world or groceries, we think the same will happen for software. And now, obviously, there's a whole host of other problems you have to solve. Once you start demanding ingredient labels and your food, then you want to actually make sure that the ingredients that go in there are good.
So you're not embarrassed when your customer reads them. And you can trust what's in there if they're certifying organic. Is it really organic? But those are the lottery shoes, but yes. We think the software transparency conversation and that's why I'm conversation is now starting worldwide. Well, let's talk about your product a little bit. So I assume you know from the broad application security perspective you know, there's testing runtime, dynamic testing, and then there's baking stuff in with a runtime interactive into the application with deployed.
I assume your ship, your shifted left into the code for analysis, is that true? Could you give us kind of the overview? Yeah, Paul. So we are. Our fundamental belief is that right when the developers are forgetting using our investigating what open-source software they should use, we need to help them in that process. We need to help them select more secure and sustainable higher quality open-source software. So that's the easiest point of intervention.
We then, once they start consuming this open-source, the design I want to use in any way, a, we allow the organization to catalog what they're using, understand and measure the risks associated with that. And the side of an organization policy level, if they want to allow or disallow that open-source component from coming in, once it's in there, now it's like having a baby. You got to nurture it. You got to maintain it. You got to keep it secure.
So you know a lot of organizations today will do this manual one time review which takes weeks and months sometimes for developers to wait and say, okay, you can use this open-source software. And then nothing happens after that. But you know we have the same year Endor, which is open-source software, ages like milk, not blind. So once it's in there, you got to make sure you're keeping it secure. You're going to monitor for updates. You got to monitor for breaking changes in the application and monitor for changes and risk posture.
We are constantly doing that for our customers. And then, ultimately, when a developer needs to make a change in that code base, and going back to what I was describing earlier, you want to help them maintain update and nurture that software, ultimately what you get as an outcome if you have these dependency life cycle management capabilities is then you can meet these compliance objectives for the executive order or the insider security frameworks. Producing as bombs that are high quality and trustworthy.
But for us, we think open-source software needs two things. One, the scanning and detection of problems needs to be very, very accurate. And so we're fundamentally changing how you do the scanning, where rather than just understanding every open-source component you use, we go many steps further to know how you use it, and what lines of code within that open-source packages you actually use in your application. So that gives us accuracy. And then we focus on completeness.
And when I say completeness, we believe open-source are a lot further than just licensed and vulnerability risks. So how do you look for leading indicators of risk that will tell you that there's a likelihood of something really becoming bad or looking for next gen supply chain attacks? Also, they focus for us. So combining accuracy and completeness is really the secret sauce of Endor Labs and doing it across the life cycle from selection to security and maintenance and updates.
So when you talk about selection and talk about whether open-source is updated, it sounds like you're really talking about you're almost like a are you like a search engine keeping track of open-source code across the web and attributing who is developing it and keeping track of it? Is that where you come from? That's a part of it, where we start is on the outside. We are basically cataloging all of the open-source software that's out there. We're analyzing the code, we're analyzing the people behind the code, we're analyzing the history of these projects and really trying to bring in what I like to call a Carfax report for open-source.
So as an engineer before you use some open-source component, you get a Carfax report, and you can decide how to use it and consume that data. And then once you've chosen which of these components, which require to purchase using the car analogy, now that cars in your garage now will continue to monitor its maintenance records, one is coming to you for service, if there's an accident, help you kind of fix the car, get it back on the road, let you know when it's time to change your tires.
That's basically what we do best is the life cycle of open-source software. That's very cool, very interesting. So you're almost a database of knowledge about open-source developers. Do you plan on releasing any kind of reports so that they could give us actually insight into the broad picture of open-source and who's developing it? Anything like that? You know, we've already started doing that, Paul, so there was a report we released in October November time frame last year, where we started bringing producing a lot of insights of where the true risks are based on this.
We then, based on a lot of our knowledge and understanding of this ecosystem produced last month, the top ten open-source risk report that is helping people understand what are the things you should really keep top of mind. And yeah, absolutely. It's an amazing dataset to mine research from and really look for leading indicators of where the bucket is headed as it relates to open-source adoption and risk management. So you've obviously been a very, very experienced and successful entrepreneur.
And you've you've been first, like obviously you mentioned red lock was very early into, well, really started the posture management. And so now you're in abstract, which has other players. Like what is your differentiator that you're coming in here and you feel strong about? Yeah, I think number one thing Paul is people are sick and tired and abstract with the amount of vulnerability reports they get from their existing tools. And ultimately getting discredited by their engineering teams because turns out 8 out of ten of those are false positives.
So the way the journey works today is you run a software composition analysis tool to scan open-source for vulnerabilities. It tells you have a half a million things wrong, use a security team really struggle to figure out how to cut that half a million to maybe a 100,000, massive list still. You send those 100,000 reported vulnerabilities to your developers, they spend the next several months going through those tickets, and then one at a time coming back to you and saying, this doesn't affect me. This doesn't affect me. This doesn't affect my code.
And then you're having this massive debate politically charged debate with an engineering team that security team security teams want to clean bill of health. From there, I see a tool, and engineering team don't want to do the work that's unnecessary. And that causes a lot of friction. So the number one thing we're solving for is really changing fundamentally the way we scan to be much more finer grained in our analysis to know picking point down to the line of code in open-source that you're using and know if that's affected by a vulnerability or not.
And thereby be able to reduce a lot of vulnerability reports by 20%. That's one. Second piece is really helping organizations think on how to get a more complete picture of risk. Like I said, there's more things than license and vulnerability compliance that are problems. And we're going to pioneering this idea of looking at risk more holistically and managing that at the very onset when developers are considering use of open-source software.
So I would put these two things kind of up there as a key differentiators against the incumbent software composition analysis products, which are the answer to open-source software. The perpetual problem that people put forward when you talk to somebody does a static code analysis is how do you get developers to fix their code. But having been a developer for many years, looking in configuration management, seeing who wrote it, seeing what feature was attached to bug reports. That's the kind of thing that developers are interested in getting a list of flaws from outside your department.
I can see this would be a very interesting dataset you could present them. No, thank you. We're really excited about it. The feedback we've gone from initial customers is phenomenal and just looking to double down. Do you want to tell us a little bit about your origin story? I was facing this problem firsthand at Palo Alto Networks right after SolarWinds are leadership team and board was asking a lot of questions about how we are managing our software supply chain risks and it was hard to produce good credible answers.
And you know when I look at this problem and then at the 68,000 alert problem I talked about at the beginning. And when I asked dozens of my peers in the industry to just figure out if this is a unique problem to me, everybody was struggling with this. And right as we were starting to think about that and I quite Palo Alto networks, two months later, log four J happen, and that just was a reinforcing factor to say yes. We really need to rethink this problem. We have far too much reliance on open-source software that we want to believe.
And we really need to think about risk management and security and then and we kind of embark on that journey. I was blessed in that process to be able to build a phenomenal team you know. We have this. We have this belief that because this was a core engineering problem that the roots, we wanted to bring a very diverse set of engineering talent in the company. So we had a rule of two. You couldn't hire more than two engineers from the same company for the first 15, 20 engineers.
So that allowed us to bring in understanding how different organizations do open-source and dependency management as part of engineering processes. We ended up hiring 7 world renowned experts and PhDs in the space of dependency management that have done some technical breakthroughs at work and academics that we then took to productize in the enterprise. And along the way, we had over 30 CISOs personally invest in the company because they've just been so excited by our approach, mission, and vision.
So all in all, we've got an amazing team, an amazing set of investors and supporters that are rooting for our success. How can you find Endor Labs on the web? And our last dot com, very straightforward. Come check it out and there's a lot of learning content and for folks that are trying to understand what are these new risks that we've been talking about for open-source, check out the top ten OSS risk report that's available under resources, or also just play around with the interactive explorer, which is just on our own page you'll see a link to risk explorer where you can interactively understand the new emerging risks and the open-source supply chain.