Security Insights - Cybersecurity for Real-World Workplaces

DevOps? DevSecOps? What Security Teams Need to Know

April 06, 2021 VP of Security Engineering Bart Westerink and Head of Endpoint Security Product Management Chris Goettl with Ivanti: Cybersecurity and Information Technology Solutions Season 1 Episode 5
Security Insights - Cybersecurity for Real-World Workplaces
DevOps? DevSecOps? What Security Teams Need to Know
Show Notes Transcript

VP of Security Engineering Bart Westerink and Head of Endpoint Security Product Management Chris Goettl discuss DevOps and DevSecOps -- and every "ops" version in between.

This conversation covers:

  • What even is DevOps? And, what's the difference between DevOps and DevSecOps?
  • Best practices on adopting and maturing your DevOps process
  • What happens when DevOps processes fall apart -- and how to fix it when the house of cards collapses





  • Next episode going live June 29, 2023!
    • New episodes publish around the second and fourth Thursdays each month.
  • For all show notes, resources and references, head to Ivanti.com/SecurityInsights
  • Join the conversation online on LinkedIn (linkedin.com/company/Ivanti)

Ivanti Insights Ep. 5


Adrian: Hi, everyone, welcome to another edition of the Ivanti Insights podcast. I'm your host, Adrian Vernon. Today I'm joined by one of our usual guests. Chris Goettl, Senior Director of Product Management, and we have a special first-time guest, it's Ivanti’s Vice President of Security Engineering, Bart Westering!

Bart, welcome to Ivanti Insights.

Bart: Pleasure to be here.

Adrian:  So Bart, before we dive into the business discussion, we have two quick questions for you. Number one, where are you located?

Bart: Sure. I'm located in the Bay area I have been working out of the mobile iron mountain view office. And I think that's going to be turned into a San Jose office soon.

Adrian: All right, so you and I, we’re neighbors, because I'm in Los Gatos, just outside of San Jose. So I'm in the South Bay, so we are neighbors here in Northern California. Chris of course, halfway across the country, out in Minnesota. All right, one other final question for you, Bart, then we'll dive into the business discussion.

Give us one main hobby that you like to do away from the office.

Bart: I like boating. I have a boat. So my spare time, you can probably find me on the water somewhere. Although these days I probably spend more time working on my boat than enjoying it. But overall I have a good time.

Adrian: All right!  That actually sounds pretty good. And the weather here in Northern California is just starting to turn for the better. So it's looking like some good boating weather right around the corner here in Northern Cal. So great to have you here in Ivanti Insights

All right, guys, let's dive in. So today we're going to talk about DevOps and more specifically how organizations can design their processes and systems to thwart cyber attacks through DevSecOps. So this is a hot topic of conversation now, a lot for companies to consider. So guys, where do we even start with this?

Chris: Yeah. Maybe a good spot is just to talk about DevOps and DevSecOps, specifically to start with. So as a product manager, I'm involved in this process, Bart is the one living in this process. So I'm going to answer with the, how do I explain DevOps to people who have very limited understanding of that first of all.  If you know anything about development, you've got the traditional waterfall approach, which is a very long process upfront and requirements gathering, and then trying to get all that all the way through to release. That was a slow and tedious process for a top heavy at the start.

By the time you release something. Usually the market has changed, the requirements have changed, something needs to already be adapted. So then there were all of these agile development models that try to help you go faster, try to fail early, figure out what needs to change, making sure that you get closest to what the need is by the time you get to the release. DevOps is trying to bring this together one step closer.  The whole agile development life cycle made it so that from development, you know, requirements, gathering development, testing, all of that was really lumped in together, but we still threw things over the wall to people like Bart on the operations side and said “here ya go, good luck with that.”

DevOps is trying to bridge that final gap to bring everything from requirements, gathering, all the way through to operational execution so we are truly executing as one.  Now, a very important distinction and for any organization that is trying to improve on, either adopt or mature, their DevOps process, I highly stress the need to talk about it in all cases with security just baked into the process. The importance of that is one of culture. If you don't think about it from the get-go as a security process, you tend to forget about the things that will make the overall process secure. And we'll go through some examples later about some ways that companies have been bit by not having that, you know, middle acronym in the whole DevSecOps equation.

Bart: Yeah, I agree with a lot of that. Any software or organization needs to implement a secure development life cycle, which is a set of activities to produce more secure code and applications. There's a number of frameworks that organizations can look at.  I'm actually a big fan of the BSMM framework, which stands for build security and maturity model and it lists about 120 or so security best practices and I'll touch on a few of them. You know, one is automation, is the ability to automate security testing through static and dynamic analysis, composition analysis, and that really goes a long way towards identifying flaws early in the life cycle and speeding up the delivery of secure code.  Security teams are typically outnumbered by developers at a ratio of a hundred to one so automation is really key to scaling coverage. 

Then you have security gates, introducing gates in a DevOps build process means that the tools can block a release.  Engineering and security essentially agree beforehand on what level of  severity bugs will break the build. And so when a critical issue comes up no one really has to blame security for holding up a release everybody knows and is already in agreement on what needs to be fixed. 

Training is another one. Making sure you have secure coding guide in place providing developers with security awareness and secure coding training on a regular basis so they know how to avoid common vulnerabilities. 

What's really important here is that you build a multi-layered security strategy, right? You start by giving developers tools to detect vulnerabilities as they write code. Then you have an internal team to run static and dynamic application security tools on a regular basis. Then you bring in the external pen-testers who perform black box/gray box testing. And finally you can set up a bug bounty program and pay security researchers to look for vulnerabilities that are more difficult to find. All of these layers increase the likelihood that you're going to find vulnerabilities in your product before the bad guys do.

Adrian: Now, let me ask you guys about third party libraries. I think we've all heard about cyber incidents that involve some kind of framework or library that was not being updated and led to the compromise. What can organizations do in that regard to avoid becoming the next big news headline?

Chris: Yeah, that's always a hot topic. Ivanti, we've had a very long tradition of helping companies better manage patch management, plugging the vulnerabilities in software. DevOps, the whole DevOps transition has streamlined this to the point where we want to get to the point where code is faster and easier to adopt and easier to update.

So you talk about third-party libraries. These could be things like dot net core, Java struts, Apache struts is one that became well known after the Equifax breach. That component got a compromise there. Each of those is basically a development framework.  They make it so that developers don't have to build everything in their product, they can take the work that's been built by others, adapt it and create more rich experiences on top of that and bring their expertise in without having to build everything from the ground up. The challenge with this is, as these frameworks have adapted, we used to have dot net framework as the kind of the framework that you would use there if you're developing on the Microsoft stack.  Now they've got dot framework and a dot net core. What's the difference between that? Dot net framework is a component you install on the system you're going to run your software from, and you can update dot net framework independent of the application itself every time Microsoft releases updates to plug security vulnerabilities.  Dot net core, you can't just push a patch out to patch dot net core on the end user's machine anymore. It's the same thing with the transition from Java 8 to Java 11, these third party binaries have changed as well. Now the developers have to update basically the development toolkit that they're running. With that they've now fixed the vulnerabilities in the components that they're deploying but now they have to basically run a new build and push that build out to their users. So the way that we need to update, so Bart and team have to make sure that the vulnerabilities in the code that they're maintaining in the operation side have been free of vulnerabilities, but the onus is now on the developers to actually update those development components, whether they're from GetHub or one of these other development toolkits.

So it's changed. We've moved where the vulnerability has to be managed a little bit in this overall equation.

Bart: Yeah. Chris mentioned Equifax, there's a great example here. Obviously these guys failed to patch the Apache struts framework, which led through there a massive breach.  Patch management isn't just about installing a scanner and a patch tool. You really have to have good policies and processes in place as well.

In the case of Equifax, they're a global threat and vulnerability management team. I knew there was an issue with struts and they emailed the alert to over four hundred people within the company basically instructing everyone to go patch up their struts incidents and to apply these patches within 48 hours.

And some of the problems that were raised in the hearings later on and in the final report is one of the top guys and ITs received the email but didn't forward it to his IT groups. He basically thought somebody else has got this and he actually received a lot of blame for that breach that impacted about 150 million customers.

Under their Equifax policy that the critical patches should have been applied within 48 hours. And in most cases the company did not patch this particular vulnerability within that timeframe and actually the server that ended up getting hacked was in patch for four months.

Here's an example of a policy in place, but it was largely ignored. And then they also didn't have a comprehensive asset inventory in place. The company relied on its employees to know the source and version of all the software running in the environment and a company like Equifax, who you're talking about hundreds of thousands of systems and applications, it's not easy to say exactly what servers and applications that are running struts and what particular version that may be on. So it's important to have a good, accurate CMDB in place to keep track of all your assets.

As defenders it's important for us to be right all the time. You can patch 99% of your servers, but the bad guys run their own automated scanners against your environment and at some point they will find that unpatched server and attack it.

Adrian: Okay.  So, I was planning to ask you guys about beyond the third-party libraries, getting into the code itself and what organizations can do to make sure what they're producing is secure and what can be done to respond to vulnerabilities when they do come up, you guys have already started to go down that route.

What else would you add to that?

Chris: I think the thing to think about here is when you're developing and releasing code within your organization, there's two parts that you really need to be concerned about. 

There's the security of the code that you're taking from others, it is the the library that I'm getting off of GetHub is that coming from a trusted repository? Is it from trusted sources? Is it being maintained? Bart talked about a bug bounties and things like that, if it's something along the lines of open source, is there actual activity around it?  Good, known, trusted contributors, and they're definitely doing good security diligence around it.

If it's small module of code that there's not a lot of activity on and it's questionable where it may have come from, that could be a problem. So there's vulnerabilities in the code that you take in and there's tools to be able to assess this and Bart could talk a little bit about, what are we using in some of these cases to analyze and make sure that those libraries that we ingest into our own code base are being sanitized. From a product perspective, we make sure that If we're taking in libraries that we've also got content-driven mechanisms to be able to update some of those binaries in production out in our customer environments. One example is if we use 7zips library for unpacking files in some of our patching technologies, we make sure that we can push an updated copy of that library out to all of our customers that utilize it in the field without having to push a new release of our product. So making sure you design for the libraries that you've got and being able to make sure that those stay secure. 

The other part of what we need to be concerned about is around our own code that we write. So then there's tools for analyzing code weaknesses so if there is a common, CWEs and CVEs are two different sides of this. CVEs are common vulnerabilities and exposures, CWEs are common weaknesses. These are areas of code that oftentimes can end up with security issues and there are code analysis tools to even analyze that and look for weaknesses within your own software and to Bart's point before, you also want to encourage external analysis of code to do things like identify bugs.  So putting out a good bug bounty program, Microsoft, Google, all of the big vendors have very mature bug bounty programs, and it really does pay off quite well for doing that. But there's two sides to that, that any DevOps organization DevSecOps organization has to look at. Vulnerabilities in your own code and vulnerabilities in the libraries you consume.

Bart: Yeah. Some of the primary tools that we use today, static and dynamic application security tools, we use software composition analysis tools to look for vulnerabilities in third-party code. And yeah, it's just a non-stop game of whack-a-mole.  There's tons of software libraries that we have in place and it's just a continuous upgrade cycle that you're looking at. So a lot of these tools can be automated. They also tend to be quite noisy, and generate a lot of false positives. So manual code reviews are going to be important as well.  In terms of responding to vulnerabilities, software has  vulnerabilities, right? So it’s important that you find them internally or by a pen-tester or someone you trust or a customer. But sometimes you don’t.  And you know, an external researcher eager to make a name for himself may come to you and say “hey, I found this nasty bug in your app and by the way in 90 days I'm going to start publishing a blog to the world about it.”

And then at that point you need to get to engineering, to drop everything and develop a patch which may be quite disruptive. Development of a patch takes time. If you're dealing with an on-prem solution, you may have to develop a patch for multiple releases. Then there's all the QA testing that comes along with it.  With cloud, typically you run a single release.  A lot of times you can silently apply a patch and not make a big deal about it. And again, with on-prem products you're going to have to really explain this to customers and obviously you're gonna have to ask them to go apply patches before the blog comes out, which can be quite a challenge.

Look, I think customers understand that software has bugs but they're really going to judge you based on how you respond to these issues more than anything else.

Adrian: Well, you know, Bart, you talked earlier about this being like a non-stop game of whack-a-mole. That was great visual that took me back to my childhood and doing that at Chuckie Cheese.  All right, we're coming down the home stretch. Bart, we're going to have you kick this off first.

Our final, big question of the day, and that is: let's look beyond the secure development cycle. What about the overall security of the environments and systems that come into contact with build process? What's the key takeaway from all this?

Bart: Yeah having a good secure development lifecycle in places is important for organizations.  Part of the problem with the Solar Winds, the recent compromise with Solar Winds probably had more to do with enterprise security or lack thereof. There are reports that the attacker spent up to a year roaming around in Solar Winds’ enterprise network.  So the question is, how did they get in? Did they get in through the VPN?  Through a vulnerable server or one that hasn't been, it wasn't locked down?  Or through an application? You don't really hear a lot about that.  But the attackers managed to install malware on their build server.  The malware with them monitored the build process, injected malicious code into the build and then it will go back and remove the code from the source code so that it wouldn't get detected. So actually fairly sophisticated attack. The problem here is that these build environments are optimized for build processes.  So oftentimes they don't have security tools enabled on them because that might slow down or impact the build process. 

So it's really important that you have good security hygiene around all of your systems, good inventory of your systems, applications, components, obviously vulnerability remediation tools and practices, configuration management and good logging infrastructure so you can review security events on a regular basis to look for anomalous activities. It's pretty hard to explain that you know, attacker went undetected that your network for over a year.

Chris: This kind of brings back some interesting memories of interactions between our own operations and security teams, and our development teams.

We had a talk about implementing additional security measures and trying to segment our dev environments and test environments from any parts of the network where we wanted to have things like email and internet browsing and things like that happening, and just trying to implement better security measures.

And you listen to a developer saying wait a second, you want me to have a completely segmented experience? I can go over here and I could send an email, but I have to go over here to do everything else for my job?  You know, how is that efficient? We have to figure out what's a good compromise in separating out some of these details but we also have to make sure that we're putting in good security measures around the entire process.

So I think it's the total path of where your code goes. One other question that was brought up in the Solar Winds incident was the fact that they had some development and testing that was happening in house. They also had some that was happening in third-party contracted environments as well.

If you're developing an application, you have to ensure security all the way out to the customer's environment. So you need to make sure that you've implemented good user controls, permissions, protect against things like phishing attempts or other types of user targeted attacks. You need to secure the devices that can come into contact with your applications, your code chain, anywhere throughout the process that could be the developer's machine. That could be a product manager that could be the build machines themselves, it could be the machines in the operations environment or the test labs.

All of these environments need to ensure that they're going beyond just the software development life cycle and looking at a broader security strategy.  You know, if you look at things like Zero Trust, Zero Trust is focused on access users and devices and making sure that anybody who's requesting access to something, that user is known and that they're, in a good, known contextual state.  Like, they're secure and that the device they're accessing from is also secure. And if we can't validate that at any point, we should deny that type of access. Well, if we implement a good Zero Trust strategy in a DevOps or DevSecOps process like this we're not only ensuring that our DevSecOps process is protected, but in general, our environments are better protected so that an attacker can't just run around for a year and finally work their way into injecting code into our DevSecOps process.

Adrian: Thanks, Chris.  And gentlemen, that's about all the time we have today. So I do want to thank Chris Goettl, who’s normally here for Ivanti Insights as usual. You all know Chris.  And Bart Westering who joined us for the very first time! But Bart, I hope we took it easy, I hope it wasn't too tough and that you'd be willing to come back and join us on Ivanti Insights again sometime in the future.

Bart: Happy to come back for another minute. Thanks.

Adrian: All right. And Bart, now that I learned that we are neighbors here in Northern in the San Francisco Bay area and that you've got a boat, if you ever have an extra seat, I'm just saying I'm around, I'm nearby.  Enjoy the spring and summer. All right, thanks for listening everyone. We'll see you again in a couple of weeks here on Ivanti Insights. Until next time, stay safe, be secure and keep smiling.