In our first episode of 2022, Chris Goettl and Daniel Spicer unpack one of last year's biggest vulnerabilities: Apache Log4j. The conversation includes:
Check out cisecurity.org and Ivanti's article on Log4j
In our first episode of 2022, Chris Goettl and Daniel Spicer unpack one of last year's biggest vulnerabilities: Apache Log4j. The conversation includes:
Check out cisecurity.org and Ivanti's article on Log4j
Chris: Hello everyone, this is Chris Goettl, and I'm here with Daniel Spicer and we are starting a whole new year of the Ivanti Insights podcast. So for those of you who were on our last December episode and are aware that Adrian has left the Ivanti team, we will miss him and we're gonna carry on and try to make this worthy of what he helped us build. So thank you, and we wish you luck Adrian. Going forward, we're gonna continue to drive interesting conversations and good quality podcast content for you guys here.
Kicking it off, start of the year, and maybe one of those things that we wish we could have forgotten about from last year that causes us all a lot of headaches, but we're gonna start by talking this year about Apache Log4j. The vulnerabilities that have cropped up there are some of the challenges that many of us experienced. We're gonna kind of tackle it in a couple of different ways. First, we're gonna talk about the vulnerability itself. So Daniel, let's kind of recap the Log4J, the vulnerability itself, the exploits that were out there and just give us the details about what this thing was.
Daniel: Yeah, absolutely. So Log4J is managed and maintained by the Apache foundation, and it is a widely used library for logging in Java applications. So Java, as a language by default, has very simplistic controls over logging, and they're just not sufficient for a lot of our more complex use cases. So Log4j kind of helps augment that baseline functionality. And real quick, we'll start out the gate, Log4j is a development library that can be included in any application that is using Java. And while it is supported by the Apache foundation, it has nothing to do with Apache's web server. It's an independent project underneath the foundation.
So most developers who are developing a Java application use open source libraries to help build their applications very quickly. And Log4j is a very common one of those because it gives them all that necessary logging functionality so they can report things like login events and error messages. Well, what happened is in early December, a vulnerability was discovered in Log4j and kind of announced in a very unusual way involving Minecraft, the video game. But essentially what we discovered is if you could present a maliciously crafted error message that the Log4j library would then take in and log, you could perform a code execution. So this sent teams scrambling, CICA classified it as critical with the highest level of score for CBE, which is a CVSS score of 10. And everyone has been scrambling ever since to try to identify all the places where Log4j is actually used and try to make sure that we're brought up to the latest version.
The difficulty of this lies in this is a development library that's included in software packages. It's not something you can go and patch directly as an individual user. You need to go back and get it patched through your vendor, through whoever provides you with the software that's included in the library. So we're over a month on, and from experience, I can tell you that Log4j is still kind of a nuisance, and hackers are definitely actively trying to use this vulnerability. From day zero, we do know that there are several nation-states that have been trying to use the vulnerability as well as some financially motivated criminals, such as ransomware actors.
Chris: Great recap of what we've experienced so far. Now, it was really apparent to me very quickly that a lot of organizations were struggling with this. So like you said, this wasn't something you could just go patch. For those of you who attend our monthly Patch Tuesday webinars, this came up very quickly and people are like “Hey, how could your solution go and patch this for me?” Well, you can't just go and scan this product and patch one component of it, that vendor had to go and do that. So in our Patch Tuesday webinars, we've actually had two cycles now where we had to talk about this as well. You know, trying to make sure people understand what the vulnerabilities are, how do they detect it if it's in their environment or not? And really trying to share the information about what was working or not, and was really kind of a key to helping people through this particular vulnerability crisis.
So Daniel, that's really what we wanna do next here, is let's talk about each of the controls that we rely on in our organizations. And let's talk about how that worked against Log4j, and we're actually gonna share a little bit about how we internally responded as well to many of these controls. And the ways that we helped to solve this and provide some guidance on other organizations. CISA, The Center for Internet Security, there are many other groups around the globe that have also tried to pull together some comprehensive guidance because this one was such a tangled mess. Let's start at the very basic, software inventory, just knowing what you've got in your environment. So software inventory is meant to go and try to detect all the software running in your environment. Now, we already talked about the critical weakness for software inventory already, and that's the fact that software inventory is looking for a piece of software, not for the component parts of that software. So Daniel, that fell short pretty quick, I think you used the word dead on arrival earlier.
Daniel: Yes, so software inventory is a fantastic tool when you're trying to understand your risk in the environment, but for the Log4j issue, yes, dead on arrival comes to mind. Because this is a component embedded in the software. So no go there, that wasn't going to help against this particular issue.
Chris: All right. So you touched on this next one already as well, but let's get a little bit deeper into it real quick. Patch management, there's over 170,000 vulnerabilities identified globally at this point, somewhere around that mark. I think it's a little bit higher than that, I'd have to go look at the exact numbers of what everybody's reporting today. Not all of those can be patched, in fact, what is directly patchable is somewhere probably closer to just around 20% or right around there. Most of these vulnerabilities have configuration changes, there are things that have to be updated through binary updates in development libraries like we're talking about here. There are protocols and ports and other things that are identified in many of those vulnerabilities that you can't just go and patch.
So when we say patch, let's put this into the category of these are the things that are easily patched. Because there's also like hey, you can go and patch SAP, they've got an update mechanism, it's very complicated. It involves multiple parts working in concert to update together. And typically an entire project team who's also there to make sure that all of their customizations and other things are also kept intact as that update is provided. So there's patchable, and then there's project-driven updates or upgrades to complex platforms, right? So patch management in this case, and Daniel, you touched on it already, but that failed pretty quickly here as well, to be able to really solve this challenge.
Daniel: Yeah, we have to remember traditional software development is not one person hacking away on a computer, and suddenly a product is born, right? We have very large teams and are building on top of a foundation of other components. And that's what makes vulnerability so complicated in this day and age. It’s not simply, is there a vulnerability in the code that a particular company has written, it’s is there a vulnerability in also any of the components that they rely on and can you even update those? And this doesn't apply for Log4j necessarily, but in other circumstances, there'll be cases where you know that you can update the component potentially manually. But will it break the system because there's some kind of dependency on the particular version? And that puts the security team and the IT teams kind of in a weird catch 22 in terms of patching.
Chris: So let's go to the slightly broader set then. So patching is how you resolve a number of those vulnerabilities. Let's step back into the broader vulnerability management space. So this is a continuous process, you have to just constantly be assessing your environment for vulnerabilities, and we all rely on a variety of tools to do that. Now there was also a bit of a sore spot with Log4j and how our vulnerability scanners that we rely on, on a regular basis, were struggling to detect this. So let's touch on that next, what made Log4j so hard to detect in this environment?
Daniel: So the biggest challenge here is Log4j is implemented into a product to perform logging. And so as that kind of implementation implies, it is up to how the particular solution wants to write logs. And so the exploit path is to find a way to influence those logs, and then you get the remote code execution. So there isn't a generic answer for how to interact with a piece of software and exploit that. A lot of the very basic scanners were looking at things like user agent, for example, something that can be logged on a regular basis, but not necessarily using Log4j for logging those types of web requests. So unless the scanner knew how to emulate all the different possibilities for all the software in the world on how it might log right to a log file, then it couldn't possibly know the exploitation path. And this is a real challenge for vulnerability scanners. They kept trying to come up with generic ways where they could force a log entry, but we're still finding things that those scanners are missing because they just can't possibly know the context of the solution in which they're scanning.
Chris: You know, one of the other questions that came up from a number of people that I talked to is “Hey, you know, Log4j has a version number, why can't the scanners just pick it up by scanning for a version as well?” After several conversations with developers and even Yako from your team, it became very clear that yes, if you go to the Apache site, you can see, and the latest versions for each of the Java editions is out there. You know, if you want Log4j for Java8, the latest version was 2.17.1, for Java7 it was 2.12.4, for Java6 it was 2.3.02. If that information's there, why couldn't the scanners pick it up? Well in this case, that JAR file, they didn't have to put that particular JAR file there with its versioning information anywhere on that system. They could take that and embed it inside of another JAR file. They could take just the libraries that they wanted out of Log4j and embed those in completely differently named JAR files as well. So there was absolutely no way for traditional file versioning to even come into play with assessing for this vulnerability.
Daniel: And just to make that a little bit more complicated Chris, you can just take out the code and paste it into a compiled file somewhere else. Or in a lot of circumstances, especially for web applications, you get these bundled up WAR files, war web application files that Tomcat will run. And so the Log4j file is actually embedded in there several layers deep, where you can't even get to it from just searching your file system.
Chris: Right, that is a great shift into the next part of this conversation, vendors. And organizations that are developing whether this was part of an e-commerce platform, whatever it was built-in into. So your DevSecOps process was very important to this and how your vendors were keeping track of what they're using, where it is and what versions those are at became very important in people being able to respond to this. So we'll start with the DevSecOps process. Those of you who've kept up with this part of the market, there was DevOps first, then DevSecOps came later because it was trying to bring the focus around security directly into the process. So organizations that have been pushing forward and really maturing that security part of their development process were better equipped to identify and respond more quickly to this. So Daniel, your team saw a pretty big variety of vendors' ability to respond to Log4j, tell us a little bit about that.
Daniel: Yeah, absolutely. I think we have learned about some new questions that will be sure to add to our vendor risk management process going forward. And honestly, we're getting into the point of the conversation where things are starting to take a good turn, right? If an organization is sophisticated enough to be performing scans against their software and libraries, and they're generating an SBOM, a software bill of materials, then you're going to get a really good response from that vendor pretty quickly on whether or not they're using that software, and that's been very beneficial. For those of you who went through the process of reaching out to your vendors, if you got a quick answer from them, you know, less than 24, less than 48 hours, that means that they had a tool in place to generate that bill of materials, and quickly check the version and see whether or not they were affected. That's great news.
There is a bit of a caveat here. There is some customization that you could have done to Log4j for the initial vulnerability which would've rendered previously reported versions that were reported as unaffected, as being actually affected by this vulnerability. And so if you just relied on your SBOM and didn't do any due diligence because you checked off on the version number, you may have still missed something. So that's an important lesson to learn. It still required hands on the keyboard, but if you knew not using Log4j, it's a pretty good indication that you can move onto the next risk.
Chris: That kind of brings up the importance of continuing to increase the security in your development processes. Whether you're a vendor like Ivanti or whether you're developing solutions internally for your customer-facing part of your business, that security part of that really does help to close gaps when a vulnerability like Log4j comes along. In fact, when we talk a little bit about the latest guidance here, that is one of the things that saved a lot of companies from having to really guess whether or not they're secure. So that was a critical part of that. One other thing Daniel, and I think you mentioned it in some of our conversations that really helped here was technologies like IPS and IDS. And if you were able to put together sophisticated enough rules, you could actually mitigate a lot of that upfront risk while you were taking the time to go and do the due diligence to figure out how exposed you were.
Daniel: So we have to remember that a lot of internet-facing applications do use Java and therefore Log4j was a very serious risk for that. You had a sophisticated enough solution in the front end, and or a sophisticated enough internal security team. There were rules you could write with these engines to help detect particular strings and block them. This magical JNDI string, which is very key to how the vulnerability actually works is something that you could look for coming from the internet into your web-facing servers and that blocks it. The trick here is writing a complex enough rule and having a solution that could actually handle a complex enough rule. A lot of IPS, IDSs out there don't give you full access to their capabilities. And in doing so really hurt teams because they can’t do the necessary evaluations against the input to make sure that they understand it. An example is if you look for JNDI, all lowercase, I can just put it all in uppercase or CamelCase, or I can do alternative casing.
There's also some really complex things you can do where you eventually evaluate the string 2B JNDI after several revisions and still get code execution. And the less sophisticated platforms really didn't give defenders the capabilities to do that work. One of the basic free tools that were available though, was mod security through Apache on your Apache web servers. It was actually very successful and being able to write something very quickly that could do an evaluation post. You know, getting through and coding and some of the different evasion techniques you can do, and then actually block those strings. But the other weakness here, you have to have a team internally sophisticated enough and familiar enough to use those tools, and you have to have them in place. If they're not already set up to do that, then this wasn't really a good option for the organization as well. So this was a great option for the top-tier organizations in the world who had the necessary toolsets and the necessary people.
Chris: Perfect. Okay, so we're gonna wrap on talking about some of the latest guidance that's available, and there are a couple more things, tidbits that we'll bring up as we go through that. So for those of you in the US market, you'll be familiar with the CISA, cisa.gov, they have a very comprehensive write-up. There's a similar one for those on the global scape for the center for internet security. So cisecurity.org also has a very comprehensive write-up. They are the two closest that I've seen out there and probably the two best. There are other groups that have good information as well, but these are two great resources. They broke it down like this, there was guidance for organizations, and we touched on a few of the technologies already. But they provided a list on GitHub or a couple of other places that were collecting these lists of all the common vendors that are out there and aggregating the information about how they've assessed and made that information available in one central place. So this quickly became a very strong tool for helping with that, that vendor risk approach to identifying if Log4j was in your environment.
The other thing, and Daniel I loved your story about how you guys took an early approach to this additional scanning to get around the limitation of network scanners. But a breed of Log4j specific scanners quickly came into being as this was developing. And I think it was only a couple of days after you guys already took your scripted approach that I saw a couple of scanners that really got down to the level of depth that you were talking about. Being able to recur through and look for this in multiple ways because it was that elusive for common scanners to get at. So let's talk about those scanners real quick. How did you guys go about utilizing that type of additional complementary scanning approach in our environments?
Daniel: When we realized how difficult it was going to be to identify this issue, we started going through very much the same exercise, Chris, that you and I are talking about. Realizing that those solutions that we typically rely on weren't going to work, we had to think a little bit outside the box, and that's when we came up with our own scanner very quickly in short order. What was key here is having an endpoint management solution that we could push a script to. And so of course, internally we're using Ivanti EPM and Ivanti neurons. But any solution will work for these kinds of custom scanners, as long as you have a place to put the results and aggregate them. And so we use the solution there, and our first iteration of this used a combination of file names and a combination of hashes.
So one of the things that we made a safe bet on was that no one's going to change the Log4j libraries too much. If at all because they're already a development library, they're fully functioning and kind of self-contained. So hashing any of the files that we found under the size threshold that those libraries would be at and comparing them against the known vulnerable hashes allowed us to get there very quickly. Our second iteration made us realize similar to what we were talking about with WAR files, that we had missed those. And so we had to add functionality to our tool to identify those files and decompress them, and then look for the JAR file in there. And a lot of the scanners are taking similar approaches, they're either using hashes, names or some combination thereof to try to find the Log4j library's local on the system. And truthfully, this is probably the most effective way of really finding that a network scan approach is simply ineffective.
Chris: Yeah, and you'll see that. So if you want to get a collection of the references that we've just made here in the last couple of minutes, my blog post for our January Patch Tuesday blog captured all of this in that write-up. It's got links to CISA, to CIS, Apache, and several others to be able to quickly get to some of this information. But they recommended a combination of the vendor assessment and these types of scanning tools to really most effectively tackle this. And those companies that are feeling confident about how well they responded, definitely took that approach in their environments.
In wrapping up Daniel, I think this vulnerability showed not all vulnerabilities are created equal. This one wasn't able to be solved by some of our more regular common tools and methods that we use. It kind of showed an extreme case to have to respond to. Not only was it an out-of-band, but it required us to use several different methods to be able to do upfront mitigation and long-term remediation of this vulnerability. I think if you look back on this and do a postmortem in your own organization about this, you'll find a number of ways that you can benefit your organization by putting a couple of things onto your security roadmap for 2022 and beyond. So it was painful, but I think a very good litness test for most organizations’ security programs. On that Daniel, any closing thoughts?
Daniel: Yeah, I think I'd like to throw out two. So first a shout out to two organizations NCSC in the Netherlands who took really early action in trying to gather information from vendors and putting it in a single spot, using a GitHub page for defenders to identify the status of products whether they were vulnerable or not. That was very helpful for defenders. Definitely please make sure you're working closely with such organizations. The other one ACSC, who actually released information about the exploitation of the Log4j vulnerability and one of the only organizations to really go out of their way to provide post-exploitation information to defenders. And then the last thing I just wanna say, obviously we took a very corporate-centric approach to this conversation. But if you would like to know more about Ivanti’s response to our products, check in the description below, and we'll leave a link there so that you can learn more about our response.
Chris: Thanks Daniel, and thank you everyone for joining us for 2022’s first Ivanti podcast episode. We're looking forward to a great year and a lot more content coming your way. Thank you!