Paranoids' Pod
Paranoids' Pod
Handling a NewVuln: Log4Shell
In our second podcast covering the Paranoids’ approach to remediating the Log4Shell vulnerability, Steven Asifo talks to Sadiah Choudhry and Lisa Hulen — who work inside Yahoo’s Vulnerability Management team responsible for handling newly disclosed security vulnerabilities.
They discuss:
- The Elements of Vulnerability Management (2:46)
- Defining a NewVuln (4:40)
- What’s an S-Bug?! (12:15)
- Responding to an Unprecedented Event (15:31)
- A Companywide Culture of Collaboration (19:03)
- Big Takeaways (26:28)
Host: Steven Asifo (Technical Security Manager, Governance, Risk, and Compliance)
Guests: Sadiah Choudhry (Technical Security Manager, Vulnerability and Control Operations Team) and Lisa Hulen (Vulnerability Management Lead)
Welcome back everybody to series two on log four shell. Today, we are joined by members of our bone management team. Discuss how they dug into this bone at a large scale. My name is Steven Ascio on the paranoid cyber security team here, but I'd like to welcome Saudi Childry and Lisa Huan to the podcast. Uh, Lisa, do you want to go ahead first?
Speaker 2:Oh no, no, no, not, not me. SA is my manager, so we're gonna let her get first.
Speaker 3:All right. Sure. Hi everybody. And, um, Steven, thank you for, uh, inviting Lisa and I to discuss how we handle log for shell and talk about our role management at the scale. So my name is SA chori, I'm the manager, uh, the vulnerability and control operations team, uh, for our security team within Yahoo. Um, my team focuses on, uh, multiple functions across the paranoids. We run operations for different functions, like bug bounty, we management, cloud security, uh, network security. And we also manage the program of the company, um, as well. Um, I have been in the space for the past 10 years or so started, right. Um, outta college, um, and have dug into different functions or domains if you call it within the cybersecurity space. Um, and currently thoroughly enjoying where I am, uh, with. Yeah,
Speaker 2:Cool. I had no idea when I interviewed her and recommended her to be hired that she become my manager.
Speaker 1:All right. So things are going well.
Speaker 3:I hope so.
Speaker 2:So far. Well, uh, yeah, so my name is Lisa Huan. I'm the program lead for our vulnerability management program. Um, and I have been in of security, uh, for 12, 13 years now. But before that I was a programmer in systems admin, uh, which really helps me empathize with, uh, the people I'm asking to do their, do the, the heavy lifting and, and patch things and, and stuff. And it gives me a really good perspective, um, uh, on what not to do and how to avoid false positives. Uh, but yeah, no, uh, log for our shell was like a once in a lifetime I hope event. And I hope I never see anything like that again, or I may just retire.
Speaker 1:Oh man. Okay. So maybe, uh, we can't predict the future, but before we go into what might bring you to retire, let's go and talk a little bit about like the vulnerability manager program and just kind of like the foundational elements of that. So maybe soya will start with you to kind of give like an overview.
Speaker 3:Absolutely. So, um, we, we have a pretty, you know, robust program that we've built, um, over the years, it's, it's taken a lot of time and, you know, I, I, I can't take credit for it at Lisa played a huge role. There were a couple of other engineers and leaders that played a huge role in establishing what we currently have today. Um, the program is broken into different, uh, components. We have a component where we, um, are tackling old wounds that exist in our environment. Um, and Ty typically we refer to them as a wound that has been in the environment for more than 90 days. Um, then we have our new wo program, which Lisa, uh, will touch on a little bit more because she's heavily, uh, involved in that one and that touches law for, um, and then we have couple of, you know, our daily exposures are ones that are critical in nature, that we look on a daily basis and we take action on that. Um, we use, uh, a vulnerability scanner, um, and that is deployed across our environment and we're using it in multiple ways to detect, uh, vulnerabilities from different, um, you know, parameters, uh, from, in, within our network, uh, outside and within. Um, so it's, you know, we're, we're capturing daily events, uh, we're doing thorough, you know, log analysis and then we're taking action based on what we see when something meets our criteria. We're taking an action. We take that action and then we run it through a, uh, our SPO program, which I know we'll cover a little bit later, but it's, you know, uh, a, a pretty robust program that we have to address anything that we see from our scanners. Yes, we will.
Speaker 2:Yeah. And on the new loan side, so some, some companies refer to what we're calling a new VA is a zero day, but to us, a new VA is, is a newly disclosed vulnerability. It may or may not have a patch. A, um, the definition of zero day for us is a disclosed vulnerability that does not currently have a patch or a mitigation strategy. Um, so we diff differentiate that, um, in our environment. So new volume, uh, looks for newly closed, uh, vulner. It's gonna depend on, you know, what it is and what it does and what product it implements on, what kind of action that we will take. Um, we can't possibly monitor every product. I think at the last last count, there was, uh, you know, 200,000 different RPMs installed across all of our platforms, just because we have such a wide variety of tools implemented and, and, and teams and products, um, that nobody is really stuck with using one particular thing. Um, and we have a lot of custom things. Yahoo is known for what they push into open source. Um, and so we also had to create some custom detection for some of our custom code and deployment strategies. Um, so anyway, we have a vulnerability, uh, intelligence vendor who, um, is awesome and, uh, sends us out all kinds of information. And we check every three hours, uh, 24 hours a day. Although we only respond during normal business hours, um, for, is
Speaker 3:That like a manual check
Speaker 2:Or is, oh, no, no, no, it's all automated. I'm trying to automate myself out of a job, do it. Uh, yeah. Uh, so we've got, um, so we've got this check and if it meets our criteria, if it's a product that we're watching, you know, we'll, uh, we'll create an internal ticket for it. And we'll look at, you know, it's severity. If it's easy to exploit, you know, what kind of, um, uh, attack vector it has if it's a remote or if it's, you know, local, um, and if it will, we bounce it against this criteria. And if it can wait for normal patch management, we're gonna let it wait. We're not gonna have people scramble when they don't need to. Uh, now vlog for J actually was not on our radar. That was not one of the products we were watching jar. Uh, Java things are really hard to find because you can have a jar within a jar within a jar they're not installed within RPM. They're just not something that's more of on a code analysis side, where we normally look for Java, the things we don't normally look on an infrastructure side. Um, and I guess I should clarify that I, I am the infrastructure vulnerability manager. I am not the vulnerability manager that handles, uh, code things or, or, or in that level. It's just, you know, for things in our infrastructure
Speaker 3:That, oh, I'm glad you brought that up because you know, when, when I do talk about the program, it is at that level. Now we do have teams within the paranoids that we do extend out or reach out to when we come across these cases and we'll, you know, have them run if they have a tool available that they can run. And let's say, for example, it's code based, you know, we have a product security team that we can run eye on. So we have these other teams. Yeah. We, we reach out to other teams to see what, you know, what their detection capabilities are and then leverage those to then, uh, you know, uh, work towards remediation of whatever the issue we're
Speaker 2:Handling. Right. Right. And that's why we don't have a vulnerability manager at, at Yahoo because we have so many different types of vulner across different kinds of things. Um, so I'm just an infrastructure pro uh, vulnerability program lead. Okay. Um, anyway, so it wasn't on a radar. Um, so it wasn't something that was ever gonna come across my, you know, my screens or anything. Uh, but it was Jeff Tolar from our, uh, data highway team, um, uh, Yahoo's data highway team that alert this the night of December 9th. So once it hit it Twitter, um, he, he saw that and he, you know, posted one of our group chat channels, which happened to be after, um, my normal business hours. Um, a, you guys might wanna take a look at this, and it's also a channel that all of the Yahoo, uh, uh, employees can, uh, see, and that kind of kicked everything off. That was, uh, where we went,
Speaker 1:The trauma begins. Yeah.
Speaker 2:Yeah. That's where we start, you know, starting to panic a little.
Speaker 1:Okay. So then it comes in from there, we have a channel, people are talking about it. So then, uh, maybe Saudi or Lisa continuing that. So then how do we even began to scope this beyond the initial estimation?
Speaker 2:Yeah, no, no, no, go ahead.
Speaker 3:I was just gonna say it's a little, it's a funny story. I think from Lisa on my side, because, uh, our team is not 24 7. We're not on call. Um, and then, you know, Lisa mentioned that it started the night of December 9th and, you know, there was, uh, my director actually was the one who tested and, uh, kicked off the incident with our incident response team, um, and worked the night. And, uh, I think it was, I wanna say 3:00 AM Eastern.
Speaker 2:It was 3:00 AM.
Speaker 3:Yeah. 3:00 AM. Uh, I'm getting page Lee says getting paged. I slept right through it in the morning. I woke up at 6:00 AM and I see like my phone, you know, with all these calls from different leaders and, uh, you know, page from our, uh, team that manages the paging, uh, process. So, um, once, once we woke up, we woke up to like a full thread of message and then, you know, I'll let Lisa go into it because then we assembled and, you know, decided to,
Speaker 2:Yeah. And I pretty much made it clear that, that if you did get me at 3:00 AM, you were getting nothing out of me that I am not a night person. And, you know, you can wait three more hours for me to wake up. Um, I'm definitely, definitely one of those people who put all my devices away, I, I I'm from, uh, originally I'm from AOL. And when Huffington post was part of AOL, Ariana Huffington would promote no, no devices in the bedroom. So I took that to heart. I am. Yeah. So no devices in the bedroom. You're not paging me at 3:00 AM. Uh, yeah, so we woke up and, uh, you know, woke up to like the world on fire, um, the internet on fire. And, uh, there were a lot of different suggestions on, you know, none of our vendors had any kind of detections at that point. They woke up and had the same, like, oh my gosh, what do we do? We need to scramble. Um, but we, we have tools and our playbooks that, you know, say, Hey, when you can't find it, you know, here's some things you can use. Um, and so there were some thoughts of, you know, let's go though, you know, send these S bugs out to everybody who's running Java. Eh, I don't like that plan. Oh, let's go, you know, send out S bugs to, to, well, what we ended, what we ended up with was we can see through our security tools, we can see running process and we found enough hits with log four J and our running processes. We knew this wasn't gonna be complete, but this was, this was a way of getting that first round out there. We did a lot of communications. We did, um, our CSO Shaza sent out a, um, a communication to our, uh, uh, tech tech channel to our, um, through our emails and stuff. Um, so we did a lot of communications. We did a lot of communication in our chat channels, um, and, uh, kicked off these S bugs. And these were, these were all S zero S bugs, which we don't do very often. Okay. Um, and, and, and I think this is a good, good to time for Saudi to explain really what an S is. I wish we could play one of our great videos, S S bug
Speaker 3:S so I know I, I mentioned it earlier, uh, you know, in the intro that, you know, our team manages the SPR program for the company and, you know, uh, reach, Lisa just mentioned it as well. So, uh, I spoke essentially stands for security bug. Uh, it represents our program for really notifying business units of security vulnerabilities that the paranoids identify through different identification process that we may have, and it's not just our team it's utilized, you know, across, you know, our governance risk and compliance team, our product security team. So as they're doing different assessments, if they come across a security issue that needs to be remediated, then they follow that Esberg program to get that to the business owner and have them address it. Um, the really neat thing about the program is, uh, it's well documented, well communicated. Uh, you know, there's a severity, uh, uh, level leveling, um, there's severity, leveling details included. And based on the severity, there's an SLA that's dedicated to that severity. So everybody knows if they're getting an S zero, it is you drop everything and you address it within 24 hours. Um, if you're getting an S one that's a seven day SLA, and then, you know, different levels have different SLAs. And everybody knows that, um, the program really allows for a consistent way across the company to, you know, manage security issues that we identify. Um, and I guess one thing I do wanna clarify, sometimes we have non-security engineers or individuals that do identify, but it runs through our program after do the full, uh, evaluation of them.
Speaker 1:Okay. So then getting to the log for show, once in a lifetime, as, as Lisa mentioned, was this something that then even given the spoke program and, uh, BMC ops that was like, we made up along the way to try and get everything crowd, or is this, we had a, already a playbook that we were kind of following
Speaker 3:We would of, I think, I don't know
Speaker 2:Makes us document,
Speaker 3:Um, no, I, you know, we, this is something we've been preparing and we're practicing, I would say on a daily basis, just at a smaller scale. Um, so like I mentioned, you know, we've had our program for a couple of years now, so, uh, as part of developing and maturing the program, I really make sure that everything is documented. There are playbooks that we have for every single thing. Um, so the foundation remains the same. Um, but it's just, we, you know, like we know we have to scope the one, we, you know, we come up with a way to identify, you know, we have to, we're in the cloud, we're on prem, you know, there's in ways of detection. And so the foundation stays the same, what changes is how we fulfill each step, because it can vary from vulnerability to Lisa mentioned, you know, this wasn't on our radar, but we took the lead on this and figured out a way to work with teens and, you know, work with what we have at the time being, um, and then adjusted our, our approach as, as you know, the protections became available. Um, so no, it it's it's, I would say we've been preparing for a long time. Yeah.
Speaker 2:Her, her catch phrase should be, um, can you write up a document and,
Speaker 3:Uh,
Speaker 1:All right.
Speaker 2:Everything is documented.
Speaker 1:Okay. Famous last words. So now, okay. So we've kind of established SBU program. There's a, a running process here, but S zero most critical, um, without getting into how often these happen there, when we issue S zero tickets to our, our partners, like, how do they respond? What does that mean? And maybe Lucy, you can speak to that.
Speaker 2:We we've never done an S zero at this scale. S zeros are usually one Zs and twosies. Um, it might be something that comes from a bug bounty program, or it might be, um, you know, we, we find this, you know, something that was exposed externally that accidentally, you know, it might be something along those lines. They're usually very limited. We've had, I think, uh, um, and a, and another, uh, event, um, couple of years ago, we, we had 120 different teams. And so that was the record before that, but people know that we don't issue them lightly like that S zeros are, we, we expect eminent attack, um, using this vulnerability. Um, and we're usually asking people for logs and, and other things along with an S zero. So they knew that when we is started issuing that this many and our communications, we don't, we don't communicate every vulnerability we're gonna ticket. Um, it's just the really important ones that we're trying to get attention around. And it was so confusing in the beginning because of course, you know, one X is that vulnerable, two X is that vulnerable. Um, and trying to get even people to know, uh, what version to update to and 24 hours later, it was a different version. All of that was really confusing. Um, and our engineers were patient, um, and they were very patient with us, uh, to, to let us, you know, get through that. And everything we were doing was, you know, very open and public. Um, but it, it was, uh, it was really, you know, just that lack of, um, clarity. And I, I remember, um, going to our red team and our bug bounty, uh, uh, operations team members and saying, Hey, this exploits public, can you try it on these servers? They're running one X. Um, they couldn't get it to, to work, but we didn't have anything publicly saying, you know, no one X isn't isn't vulnerable. Right. So we started saying, okay, just ignore a one X for now. We'll come back to that. Shortly after that, we started getting public statements. I, um, on the Apache log for J website saying it wasn't vulnerable. And so then we said, okay, okay, if you're running one X we're, we're gonna downgrade you you're to upgrade, cuz you're running it into life product, but now, uh, but now it's no longer all hands on deck. You can you do about that in a more thoughtful way? Um, is just amazing to me, like how fast our engineers reacted. Like we have about 10,000 employees and hundreds of engineering teams and, and, and most teams were impacted in some way. Uh, and it was just crazy. Um, we, we really had a lot of support, um, from out from within the paranoid, um, to, to handle this. It wasn't just us. I mean, we're a pretty small team, but I mean, it wasn't just us responding. I, I can't imagine that there was any paranoid that didn't respond. Yeah. Uh, we, we had, we had folks helping with communication. We had folks with answering questions, all kind kinds. You were handling all the, the, I can't do this and
Speaker 1:Hey, look, you know, everyone had, we can, and you know, still to give credit back to your team, I mean, you guys are like, kind of like the eye of the storm for everything, but it, it does sound like the engineering teams. I mean, they help find the initial one shot out Jeffrey. Um, so I guess maybe Saudi kind of overseeing the, the program here. How can you talk about like the culture that you guys have helped where people do respond like this?
Speaker 3:You know, it, I keep bringing this up that we we've been doing this for time and it's taken us time and it really has the, the ESPO program, um, has, you know, it, it was developed over 10 years ago and it's, it's evolved over 10 years ago just based on different teams that came in, you know, different technology. Again, the, the risk space has changed, you know, we're, we're headed towards, you know, a, a different area now. So it's, it's, it's evolved with everything that's evolved with Yahoo. And, you know, Lisa mentioned, we had engineers that were stepping in, um, and helping, and, you know, we had, we were testing whether a particular version was, you know, vulnerable, internally, same goes with the mitigation piece. Like even the mitigations that were, that Apache was sharing, you know, we were trying on internally and, and our engineers were trying and then sharing whatever was helpful to them with the rest of the company. Um, and so the, the culture, I mean, it, everyone takes, you know, because we've really changed the program to fit everybody within the company. It's not just, you know, the parent saying this is a necessarily, you must do it. The, the, the receivers of the SBUs truly understand and agree with what we're, you know, uh, we're calling usually. And, you know, if there, there have been cases where, you know, may not agree and we have conversations. Yeah. But we have, I, and, you know, I, I proudly say this where we need to change, you know, lower severity we do after conversations, because we're not the experts on the, and
Speaker 2:Evidence and evidence
Speaker 3:Press by verify. So, so, you know, it, it's, it's taken some time. Um, whenever we introduce change with the program, we meet with the leaders, we explain what we're doing, we get their buy-ins or even approval before we evolve that change out. And, you know, we do thorough communication to make sure that nothing, you know, nobody's caught off guard, uh, with this, but I think everybody, the relationship that the paranoids have with the rest of the company, it it's very, it's very nice. I, I, you know, I don't see that in a lot of other companies. Uh, so I'm, I'm really proud to say that, that we work very well together. We trust each other, um, and that, you know, has really shown in this particular, uh, you know, log for J where everybody came together and worked. Uh, I wanna highlight one thing. The time of this, uh, vulnerability was, is also very important to note that, you know, we're, we're going towards the holiday season. Yeah. People are in that mindset of taking time off, spending time with their family and lock for shell hits. And so even, you know, naturally everybody's in a, but they put that aside and still came.
Speaker 1:It's a, probably an understatement.
Speaker 3:So, um, you know, so it, it really speaks to the culture of the company and how important they really take, uh, security and the data that we're protecting or end users are consumers. And, you know, um, yeah,
Speaker 1:So it sounds like we're very successful cuz we have what a 5,000 person, uh, program, uh, with, with BMC ops, is that right?
Speaker 2:Uh, so there's, there's three dedicated people, uh, and wow. Eugene Cato, uh, on NA J in, I pronouncing this name. Right? Yeah. Um, and myself are the core members and then the other team members on, uh, VC and ops kind of rotate in and out, we have a lot of different work streams. It's a great way to get exposed to a lot of different things. Um, and a lot of like, you know, you find your passion and then, you know, next thing you know, we're losing that person to the product team, product security team, or, and, and that's a good thing. We like the, within the paranoid, we like for people to find their passion and, and, and, uh, move up and, and go where they need to be,
Speaker 3:Which is why I have playbooks. Yes.
Speaker 2:This
Speaker 3:Is why playbooks document, No, Lisa, you know, brings up a, like, are really, I, I feel like, uh, not because it's just my team, but I really feel like it's a really great place to come in cuz we're touching five major different functions. Yeah. And you won't really find that in, you know, um, other places you come in, you learn and then you find your niche and then you, you know, you move on to a permanent role. Uh, if it's within the paranoids, great. If it's, you know, outside of the paranoids, great. You know, we just hope that while you're here, you're really taking advantage of the team.
Speaker 2:Yeah. And, and one thing I really wanna point out is, um, because we have these playbooks and because we've done things pretty much the same way for the past four or five years that our leadership trusts us, nobody in our leadership, in, in the paranoid leadership or, or even the executive leadership was saying, you need to be doing this, you need to be doing that. Why aren't you doing? Like they just said, let us know if you need help, like Sean's at, let us know if you need any help. Like that was the leadership response we got. Um, because they've seen us do this time and time again, maybe not at the scale, I hope never at the scale again, but they've seen us do it. And, and there were just, and, and it wasn't just our leadership. We had, we had people all across. Um, it was, um, EJ Campbell, our, our, uh, who's a vice president who is the lead for sports video and consumer infrastructure. I had to write all of that down. Um, but he's the one who started putting together a knowledge document for us, which wow. We normally do for these big things, but we didn't have time. So he started it and everybody started contributing it. Um, wow. We had, um, within our paranoid, we had Jan uh, shaman, who's our, uh, principal, uh, security architect. Um, he was answering questions. He became our expert and he actually wrote a, um, a ch check for log for J which you can actually get in our open source on GitHub. Um, for a Yahoo, you look for a check for log per J, um, which can look, this is before our, uh, scanners had this detection capability. So it would be able to look inside jars for log for J and determine if the mitigation of the, uh, G uh, lookup class had been removed, which we didn't have have at the time. So he had that, and then we had our legal team help push through the open source, um, to get that published. We had Aaron Salva on our cloud platforms, engineering team. This is outside the paranoid. He wrote, he helped us out. He wrote a script so we could figure out what we had already ES bugged and what we still had left to go. Um, so it was pretty amazing effort and it was, um, it was, you know, at least 1600 people coming together, um, to tackle and stamp this out quickly. Oh,
Speaker 3:It takes a village, makes
Speaker 1:A village. So, so I guess with all that, uh, being said, and we're, we're done with lock for shell now it's, it's completely done.
Speaker 2:I have no common.
Speaker 1:Yeah, no, obviously it's ongoing. There's still, um, more there, but this has been a fantastic story so far just with the foundation of the program, which I'm quite triple dive into on another episode in more detail, and then just seeing how everyone, uh, else played a part. I, I guess, is there any, um, remaining thoughts?
Speaker 2:Yes. Remembered one thing. Um, one of the other things that I was really super proud of is the relationship that the paranoids have built with our bug bounty researchers. Uh, we, and we normally, you know, have an embargo of 30 days on a, on a vulnerability
Speaker 1:And one of the largest bug bounty programs in the world
Speaker 2:Was yes, one of the, and then the best I, um, uh, but we had asked for, um, our researchers help and saying, okay, forget the embargo, go see what you can find. Um, and that gave us a lot of confidence that we had maybe, uh, gotten everything we could find on the outside. Now that is not saying that that is a hundred percent accurate statement. I don't want anybody to take that as a challenge, but that we, we, we don't have the same resources and the tools that our bug bounty researchers have. Yeah. And, and they were enormous help. So that would be my last thought as I don't wanna forget, uh, to, it's very important to thank those, those,
Speaker 1:And that's also something that couldn't have happened overnight also, cuz that's another relationship multiyear to build up that trust with them.
Speaker 2:Absolutely.
Speaker 3:That program, you know, it's phenomenal and all say it again. It's taken time and it it's Evolv and it is amazing
Speaker 1:A village of ex uh, external researchers. Um, so aside any, uh, five final thoughts from you before we,
Speaker 3:I, you know, I'll, I'll just say one thing, you know, our, our, our team is great. Our company is great. Um, I've been, I forgot to mention, I've been with the company for seven years now and have gone through all these different changes. So, um, we are hiring, take a look at our physicians, do join
Speaker 2:Paranoid.com,
Speaker 1:The paranoid.com. Yep.
Speaker 3:Yep.
Speaker 2:You can see all open positions,
Speaker 3:You'll work with amazing peers and leaders. Yeah.
Speaker 1:Come, come be a part of this documentation.
Speaker 3:Yeah.
Speaker 1:No, but on all serious note, I, I know I have an opportunity to work with you guys a lot and you guys make our job, uh, so much easier. Just work with professionals such as yourselves. So, um, look forward to working with you guys in the near future and looking forward to deep, dive into some other foundational things that you guys have going on within the paranoid. So, um, with that, we're gonna close out the episode here. There's our second episode on log four, shell, please stay tuned. We've had Saudi and Lisa of paranoid cybersecurity team. And, uh, that is our episode.
Speaker 2:Thank you very much for having us.