Paranoids' Pod
Paranoids' Pod
The CTO Perspective: Log4Shell
Addressing cyber risk within the business is a challenging task for any security team to manage on their own.
This places a premium on the Paranoids' relationship with engineering teams. An especially necessary one when conducting an expedited patch across the organization for an internet-wide weakness.
Namely, Log4Shell.
In this episode of the podcast, join Yahoo CTO Aengus McClean and Chief Paranoid Sean Zadig in conversation about:
- The Working Relationship (1:00)
- Security Culture (3:10)
- Communicating Priorities: Log4Shell (12:00)
- "Slow is Smooth and Smooth is Fast" (20:20)
- Building Security Into the Process (26:27)
Hosts: Shawn Thomas (FIRE Chief) and Steven Asifo (Technical Security Manager, Governance, Risk, and Compliance)
Guests: Aengus McClean (Chief Technology Officer) and Sean Zadig (Chief Information Security Officer)
This is our final episode in a series about Log4Shell. You can find episodes One and Two on the Paranoids' landing page.
Hey everybody. And welcome back to another episode of the paranoid podcast. And let me tell you, it feels great for me to be back cuz I missed the last one. Uh, I am here with my good friend, Mr. Steven. Ascio
Speaker 2:Glad to have you back. Thank you very much, Sean.
Speaker 1:Oh, I am happy. Happy to be back. And we have another returner tonight in uh, Mr. Sean za, CISO of the paranoid.
Speaker 3:Hey, thanks for having me back.
Speaker 1:Also our very good friend and strong paranoid supporter, Mr. Angus McClean, the CTO of Yahoo welcome Angus
Speaker 3:And I am very D very happy to join this podcast. Amazing. Thanks.
Speaker 2:Cool. Um, so I guess going right in Angus and you are, um, new and people who may not be familiar with, uh, yourself and just how our team is organized. Can you tell us about like what your working relationship is with the paranoid in, uh, you know, Sean ZZA?
Speaker 3:So I suppose I've been, uh, working at Yahoo slash slash Verizon media slash AOL for the last 20 years and, uh, quite, quite, quite a, quite a while as, as you can see. And, uh, over the years, you know, things have happened, uh, there's been run-ins, there's, there's been unfortunate and so on. So from my perspective, as we, especially at Verizon media and, and laterally at Yahoo, where there was actually a CISO, um, that was very active. Uh, I was delighted to partner with Sean because it makes a huge difference to have, uh, somebody actually actively looking at this space and working with this, uh, on these, on these spaces because, you know, to accompany of our size with an audience of our size and with the, uh, breadth and the, and the scope of the products that we offer, it's a, it's critical, critically important that we really support our customers and make sure that they feel safe and they feel that they're an environment that's safe. And, uh, so, so the service that the, that, uh, the paranoids that Sean and, and the rest of the team provide, um, is critically important.
Speaker 1:So that's,
Speaker 3:I I've gotten to know more about it. I will say, as, as, as CTO than maybe as, as the, uh, as a led the, the ads group, um, from an engineering perspective, not because it wasn't as important, but just because I got to know Sean that bit better, and, you know, I've gotten to know Steven a bit better from all, all his time at the open house, as, as an pleasure, the compare, you know, so getting to know you guys, it really does help,
Speaker 1:And that's such a wonderful kind of line. And I wanna focus on that a minute, because you talked about really, you know, how we use this to protect customers and the importance of that. And that's so great to hear coming from like a CTO love, right? So one of the, the themes from the prior two log four shell podcast was the culture and the culture here and how that really helps. So, you know, as you took over as CTO, how have you really worked to maintain and further cement that security culture within the engineering organization?
Speaker 3:You know, when, uh, as, as we modernize a lot of what we're doing from a technology perspective, especially how we operate, uh, as en as engineers, one of the core, uh, tenants, uh, that I have as, as one of my beliefs, is that engineers remain responsible for their code from the time, uh, of inception, right through till its end of life from soup to nuts. As, as you say, in the states, the basically from, from my perspective, you are responsible for what you put out there at every level. And, uh, but you can't know, or be of all of the skills or all of the threats or all of the, the various, uh, uh, idiosyncrasies that, that, that can happen or even the attacks. And you, you, you probably don't even dream of the type of attacks that people make. So partnering with the paranoid really does give us that chance to, to, to broaden the scope of the engineers. So the engineers be come responsible, not only for their code and how it runs, but also for the security of their code. And I know we have, um, we have, uh, set up programs where we, where we train, uh, engineers in cybersecurity. And we try to, to, to push down the whole, the whole knowledge, uh, that, that the, that the paranoids have up over the years, but it's still very reassuring to the engineering population in general, that there is that there is a, an, an organization that is looking at it specifically, that's looking outside their particular code, that's looking at possible. Um, you know, other actors that may, that may have, uh, nefarious, uh, reasons for, um, for trying to attack us at any stage. Um, but as I say, my philosophy is engineers own their, their, their code. They own their products, they own it, you know, uh, from a QA years ago, they used to have QA and engineers would throw code over the wall, as we used to say. And, uh, the QA team would be responsible for finding the issues and, and getting that's all gone. When, when you look at a C I C D type process, each engineer is fully responsible and we need to make a engineer fully responsible for the security as well. And that's where the, the paranoid team can really can really help.
Speaker 2:Thank you. A is, cuz I know we do a lot of work to also, uh, be friendly, paranoid, so everyone feels, uh, part of that security first culture. So I guess maybe even doubling down a little bit more when you're kind of going down and spreading the good word of, uh, you know, security and how that incorporates into the culture, I guess what are like the main points that you kind of try and speak to you,
Speaker 3:I suppose? Um, it, it, it, it's, it's a mixture of, um, security from the perspective of the actual product and how it works and how you protect data, especially consumer data and customer data that, that, that, uh, may be, uh, it may be needed from, uh, time to time or may be used from time to time for, you know, during transactions or whatever. We don't want any, uh, customer, uh, data to, to be out to escape our, our barns and our systems. So that's critically important from that perspective from an everyday perspective, but there's also the challenge that you can put down to engineers and I find engineers really respond to being challenged. You know, how do we make our codes secure as well? So it's that other actors who may have other reasons for trying to, uh, break into our systems or as in the case of, uh, log four J where you, where you have, uh, systems that we use and incorporate into, into products, uh, have vulnerabilities that need to be that need to be addressed. So, uh, I think it, it provides, uh, the engineers with the challenge and that's the, that's what I want to do, make sure the engineers feel challenge that the they're growing, that they're learning, they're learning, um, how to do things correctly. They're learning the best, uh, practices and policies that we can, we can put out there and they're not afraid to implement them in, in, you know, in new systems and in cutting edge, uh, solutions as we, as we really move, uh, the company forward into new spaces. So, um, another thing that that I think is really important is as we move the company into new spaces and we add new products that, um, that we challenge ourselves to find, well, what could go wrong or what could people use or how, how do we make our products secure in this new area that we may not be familiar with? So, um, so it, you know, by challenging the teams that way, I think, um, engineers want to make sure that the, the customer has the right experience at the end of the day. They're proud of their products. They're proud of the reach of their products. You know, a company like hoo is capable of delivering products to hundreds of thousands, uh, millions, and even in some cases, billions of users over to time. So, so really it's, it, it really is something that an engineer can, can be proud of, that they can put on their resume and say, I delivered a product that was used by, you know, uh, a large portion of the population of the planet. And, uh, I did so, and in a way that was both secure and was bug free and, and, um, delighted the customer. And
Speaker 1:For that, our heart grew two sizes
Speaker 3:Larger.
Speaker 1:It really did. That's such a, such a great statement. So, so last one for you, a little harder one, then you get a break as we pivot into that. Um, are there cases where there's individuals across the company that might need a little bit more convincing to buy into this and kinda how do you handle that and how do you approach that, that from that angle?
Speaker 3:Um, sometimes you can come across it. That is true, but, um, I find that there, there are things that happen in, in the, in the general market that remind everybody every so often that security is paramount. And, um, the, the key thing, especially with a company like, uh, like Yahoo, where, uh, customer engagement is hugely important, where we measure it all of the time, we try to make sure that we are delivering. The people are coming to us every day that people, that, that, that we are having the they're having the experience that they want. We cannot get to a stage where, where, um, where we have any vulnerabilities. So generally, uh, where it's happened, uh, we have been able to avoid it, uh, by, by just, uh, uh, or avoid any, any intransigence, if you like by showing and talking about other aspects or other occasions where it's happened to other companies and the impact that it can have. So there's many, many arguments and many threads to the argument that you can make, uh, so that people take it, take it seriously.
Speaker 2:All right. And I guess now for up playing you, ZZA saying that we've had, uh, this champion here as a CTO speaking to us, how do you help support Angus and creating like a security first culture? Do you send them slack messages, like, you know, a fruit basket door dash, or how do you go about doing that?
Speaker 4:Uh, so I definitely send them slack messages. Um, I think we're a, you know, a slack first company. And so definitely a lot of, a lot of slacks, a lot of, you know, gifts and emojis as well. Um, that's kind of my style, uh, but you know, I think, you know, speaking more broadly, uh, we also make him a very active participant in, you know, how we, how we circulate our message internally. And so Angus will star in our security awareness, his videos, and, you know, he will, when he's talking to the entire company at the company all hands he'll, he'll always talk about security to that audience as well. And so, you know, he's a really important sort of, uh, symbol that we can, we can leverage internally to, uh, to spread the message to, to develop a population. Um, and you know, on that message, I think Angus hit it exactly right? Like we treat our engineers like adults and, um, you know, developers wanna do the right thing usually. And so we, I think one of our main approaches is to give people context and, um, and you know, the information that they need to make the right decision. And we do that through training. We do you that through, you know, awareness, um, but we also, you know, provide the background. And as Angus mentioned, there's, you know, incidents that, that other companies experience what we'll talk about those, what are the lessons that we can learn from that? And how do we, you know, make sure it doesn't happen here. Um, and Angus is a really important part of that, that communication plan.
Speaker 1:Excellent. So let's pivot a little bit into the theme, right? So it's great to have the overall conversation, but log for shell, that was a big deal. It was a big deal everywhere. It was a multimillion dollar impacting thing, pretty much everywhere. So, you know, Sean, how do you bubble up that priority and how do you approach something that might have a multimillion dollar impact on us? That's that big?
Speaker 4:I think the first thing is, so Angus actually, before he came, became CTO, he mentioned that he led the ad engineering organization. So he was actually a customer of the paranoids and of our, you know, our processes. And, um, and so, you know, over the years we had built up a lot of trust with him and he realized that if, if we come to him and say, this is a big deal, this and a, you know, five year, 10 year type of incident. Uh, I think he, he, we don't say that very often. And I think he, you know, realized that that, oh, this is something that we really need to pay attention to get all hands on deck for. Uh, so I think the key is to, you know, not engage in hyperbole, but also, you know, really be clear about the, the threat that we face. Um, and then, you know, when it's time to, to, to raise the balloon, like we, we raise it, uh, but only do it when it really matters.<affirmative>
Speaker 1:Excellent. And then pivoting Angus over to you, you know, when, when Sean says it's time to pull that rip cord, we have a real issue, you know, how do you approach that? How do you kind of quantify some of the hidden costs of those decisions and like, what's your approach coming into that?
Speaker 3:So, I mean, Sean is a, a very cool characters. So when he delivers a, a message that, uh, we have a real problem, then I do listen. And, um, I suppose what, we, we look at it from a holistic perspective, you know, because in a company our size, there are, there are things that are going to happen that are, you know, low, low on the scale of things. We can sometimes, uh, things we may need to actually let go on for a bit, if we need to investigate or whatever. So it depends, uh, we work, I work very much dependent on, on, on the advice that I get.
Speaker 1:You're speaking my IR language with that one right there.
Speaker 3:So, um, so I work with yeah, absolutely work with the advice that I get. So if it is thing like log four J and we're going to have, uh, an issue and, uh, it, the imperative nature of, uh, of remedy is outlined. Then the first thing I do is I ping the exec team, the entire exec team. I let them know we have an issue. I let them know that, uh, we have to do certain remedial, uh, items. I informed the business leaders separately. Uh, and in addition to the exec team, uh, that, uh, their roadmaps, which is of course the cost of, of, um, security lapses, be it our own, or be it, uh, systems that we incorporate into our products. But that is the, the, the cost to us, the immediate cost of all engineers, uh, that are, that are impacted or all systems that are impacted all engineers responsible and others who can help out, um, are drafted in to, to, um, to try and make sure that we, uh, ReMed the situation as quickly as we can mitigate at least as quickly as we can, and then remedy as soon as we can afterwards. And, you know, from that perspective, there is a cost to the, to the company because, uh, the roadmaps that engineers have been working on are impacted. Um, in some cases we can do it quickly. In some cases, it takes time. Uh, the log 4g, as far as I'm concerned was, was a huge example of how the, how the company and the engineers can really pull together because literally jumped on it over a weekend, um, started monitoring systems straight away to see had we, any issues worked with the paranoid on those, on that monitoring to, to make sure that we didn't have any exposed vulnerabilities, uh, at this stage. And then, uh, got, got working on mediation so that we could, uh, so that we could continue to, to do the jobs we want to do and not have the impact. But, uh, I have to say that, um, you know, the exact staff, particularly Julie in, uh, Julie Jacobs, Sarah chief council, and, uh, Jim Landon, our, our, uh, CEO are very familiar with what can go wrong and how, how can, how can really impact our, our, um, our reputation in the marketplace. I have never heard anybody complain when, you know, the, the terms of doing business are that you make sure your customer and your business is secure. So I've never heard anybody complain, other than note, that, uh, you know, great job that was done, the number of engineers that jumped on it from a paranoid right through to the development and the core development teams, uh, it was a real, it was a real team effort. It was a great, and it, it was, uh, noted as such by everybody at senior level in the company. So,
Speaker 2:Yeah. And so you've talked about some of the costs in terms of like business prioritization with senior leadership there, but it sounds like there was also another cost in terms of, uh, you know, potential fatigue, or just morale with having to kind of jump in there. So, uh, how do you maintain, you know, engineering in teams or just ICS morale when some products are being pushed back, they're working all night on security issues. Um, so I guess, like what, what were some of those conversations like during log 4g? Just kind of saying, Hey, we understand, um, but this is like a priority.
Speaker 3:I, I, I don't think there was that much pushback from engineers, um, on log 4g, there was a bit of excitement about it actually. Um,<laugh>, you know, it, it, it's, it's a bit, it's a bit of a terrible analogy, but, you know, the way, if you come across an accident, everybody wants to help and, you know, people stop and no matter what they're doing, it's second to, to trying to help out. And it's a bit like that there's things have gone wrong and we're needed, and, you know, we're all in this together. And, uh, so I think that from that perspective, plus the fact that we can learn from this, we can learn, um, you know, the types of vulnerabilities there are, and the, this particular vulnerability, how did it happen? It's always very interesting for engineers. So I think that if it was something that was going to, uh, take months, or probably even, you know, months to, to, to, uh, to fully, um, that stopped everything for, you know, a significant period of time, then maybe there might be a little bit of a, uh, an adverse reaction, but certainly, um, the initial and while it did take us, uh, some time to fully fix the issues, the, the, the, you know, the, the way we could deal with the mediation quickly and, and deal with the fact that there was, um, that there was a problem and, and how we could pull people together, uh, for, for that, for that time, I didn't see any, any negative feedback. In fact, people were sort of high fiving around the place when we got through it without any, without any, um, without any major scars. Okay. So, um, so that, that, I mean, that's the way I would look at it, but ultimately, and especially for younger engineers, it's a learning experience. It's a learning experience.
Speaker 2:So it sounds like you would say yeah. Who every engineer as a security engineer.
Speaker 3:Yeah. That's what I was saying. Uh, you know, CISO and, uh, and C I C D the, um, they basically, uh, you know, we've been pushing down that responsibility. Uh, the, the, an engineer owns everything from start to finish, and one of those components is, is security. So, so every engineer is responsible for the quality of what they do, the security of what they do and the impact of, of, of the product at the end of the day. So, so I think that, uh, that, that has to be a clear goal that we have,
Speaker 1:You're, uh, you're really not wrong about the car crash analogy. I've been an incident response for about 15 years in the, uh, the dirty little secret there is that what we don't want anything bad to happen. We kind of secretly hope for the excitement that comes when things interesting happen. Mm-hmm<affirmative> um, so kinda the hit on that, we've talked a bit about transparency. We've talked about communication, but one of the things that really, really kinda came home in the last two podcasts about this was trust the trust we had from leaders, the trust from not only just like direct line managers, but from, you know, senior leaders and the executive team and all of that, you know, Mr. Added, can you talk a little bit about trusting the team, um, and kind of the importance of allowing them to make heat of the moment decisions that do impact the business, but, you know, for a reason to stop it from a different type of, of impact.
Speaker 4:So, so there's this expression that I, we, we used when we were in, when I was in the government, I still used to be in law enforcement in the federal government. And we would say smooth, slow is smooth and smooth is fast. And that's, you know, I think as security professionals, that's where we would ideally like to be at all times, like we're taking and, you know, we're taking our time, we're making deliberation. We choose the path after considering all the data, but sometimes, you know, the, the ins in the heat of an incident, something that huge is happening, you have to make a gut decision or a really, you know, quick decision. And, uh, in, in those situations sometimes, you know, maybe we, you know, say, okay, we're gonna ticket. This instance we think is vulnerable. And perhaps it wasn't, uh, because of some, some other factor we didn't realize. Uh, and so sometimes we'll get heat from product teams, from engineers who are like, Hey, you wish you had this ticket. And it turns out, you know, we weren't vulnerable, or it wasn't as big a deal as you thought. And in those situations, you know, I think what's really important is that the team that's a security team knows that we have their back and we trust them to make the right call at the right time. And if you're gonna Monday, Monday morning quarterback, yeah. You can always find a few things that you do better, but we use a, it as a learning experience, as opposed to like, you know, trying to, to make somebody, you know, fall in their sword or something. Um, and I have no problem going to teams within the company and talking to my peers, you know, engineering leaders across, across the firm and saying, Hey, you know, with all the information we had at the time, it was the best call we could make. Uh, but, you know, thanks for your feedback. And, and, you know, next time we will, you know, we're learning from this experience as well. So I would say, you know, trusting the team, letting him know that, that, you know, we have their back, um, you know, as much as I could show up in the middle of the night, join the calls, say, Hey, you know, I, I can't do much to help you right now, but like, we're, we got you, we're behind you a hundred percent. That's the importance, uh, to, to, to build and maintain that trust.
Speaker 1:And I actually, I wanna reiterate something here too, both for Angus and, and Sean, just in case you haven't listened to the prior two, but I want you to know that the overall theme of the prior two was that the security teams, the ones that we talked to felt incredibly supported from executive leadership effectively, Sean Angus, both of you. So like, it's really important. I, and I just wanna call out that, like, these folks feel that way. Like, it's not just something that you're saying, like these folks feel supported by the work that you do and the trust that you put in them. And I think that's really important. And I just wanted to say like, thank you to the, both of you for giving us that leeway and that trust to, to solve problem here at Yahoo. It actually means a lot, and I will, uh, pass it to Mr. ACI for final questions, but I wanted to get that out first.
Speaker 2:Absolutely. So as we get to like the end of our conversation here, for folks that are listening and want to create a security first engineering culture, uh, what advice would you guys have have for them, maybe we'll start with you and then we'll go back to UA.
Speaker 4:So I think a couple things, um, you know, first as a security professional, trying to build a culture, like put yourself in the shoes of your customer. So for us that's, well, when I say customer, I mean customer teams. So the, uh, the engineers we're supporting across the company. So, you know, if you're design a solution, design it with one that will work for them, uh, within their workflow. Um, and you know, if we need to make changes to that workflow, do it in a way that is that we're, we can clearly communicate and explain the rationale behind our decisions. You don't wanna be, uh, you know, the Sage on the stage, or you don't want to be, uh, you know, the voice of God and parting decisions down. Like you wanna be sort of like a, a person who is approachable, who you can explain, why we're making these decisions and, you know, ask for input. And that helps build of that trust. And that helps build that, that culture that comes over time, um, when, when the, when your population of engineers, developers, you know, business partners within the, within the company, understand that you, you know, are well intentioned and you're gonna take their, you know, their, uh, their considerations, uh, as you design a solution. So, yeah, I, I would say that,
Speaker 1:Can I actually share an anecdote here before we move on to Angus course? So one of the, the core tenants for us, an incident response actually speaks to that a lot. We work on a, a mitigation strategy of least impact whenever possible. So we consistently try to take the mitigation strategy that has the least impact on the business whenever possible. And we have higher business impacting events that becomes less of a serious issue. And that's one of the way I think that we as IR have been really able to, you know, work well with customers without becoming an enemy, but rather become a partner. So just anecdotally, I thought that was an interesting thing to share along with what exotic was talking about. Cuz it's comes a lot lower than just, you know, Zana hope is CISO that we do this. I think we actually practice it.
Speaker 4:Yeah. I I've been at other companies where, you know, the security team is kind of feared or, you know, looked down upon, or at least, you know, not consulted when decisions are being made because, you know, they've had experiences where they come in after an incident and, you know, shut everything down or, you know, don't care about impact to, uh, to roadmaps or, you know, uptime or whatever. And that's certainly not the, the approach we take here.
Speaker 3:Yeah. And from my perspective, um, just going back to the original original question, I think it's, um, I like to keep it really simple and that is, um, you know, engineers and developers tend to be probably not aware of the issues that can, that can be built into systems, you know, through vulnerabilities, uh, and, and so on. So I think it's really important that we continue the program of training it into engineers so that they get some, some, uh, cyber, um, training, cyber security training, if you like. And I, and then just build it into the process after that, you know, we need to make sure that every engineer knows that they are fully responsible for what they build, if there's nobody else, don't point fingers at paranoid don't point finger at QA don't point fingers at, at customers who just don't know how to work their products, it's their responsibility. It needs to be drill down. So I think once, once, once you have that sort of engineering, total engineering picture in terms of, we own it soup to nuts, it's ours. And if it goes wrong, we're the ones that are responsible, not as John said in a, you don't want people to feel that they have to fall in a knife just from a perspective of ownership. So that it's clear. And that way you, you get it as part of the culture, people understand the various aspects of what they build, um, from everything from the, the cost and how it's deployed, uh, the, the, the, the sort of our response rates, the throughput rates that they need to deliver the, the security they need to, to deliver the quality they need to deliver. So it all becomes part of the report card at the end of the day. And then you get it built into your systems, uh, completely from the ground up.
Speaker 2:All right. Um, so last question I have for you guys is obviously I'm really excited to have only have you Angus one, just to be a part of this conversation, but also as a CTO is that I listen to a handful of like security related podcasts. Very rarely do we have, you know, the CSO and the CTO. So I think people that are listening would be very interested to see, like, how do you guys create such a working broad relationship between one another, like, where you're like, Hey, let's do this podcast together. Right. Um, that maybe other looters can emulate at other companies, or if this is like a common kind of just, uh, relationship here on how well you guys work together. But just, I guess, any thoughts on that? Like, I mean, how often do you guys,
Speaker 3:So from my perspective, what I will say Steven is, is it goes back to that word that, that Sean was describing there, it's all about trust because ultimately, I think was Teddy Roosevelt said, it, you ha you hire the people to do the job, and then you get outta their way and let them do it. And that's really what it's about. You trust Sean and the paranoid to do their job and to deliver. And that way you build a relationship based on trust that, uh, means that they are free to do whatever they need to do. And, and, you know, when they come to you that it's something that's needed and it's something that it's important. So, so, and, and of course you go for a few beers and, you know,<laugh>, whenever you can meet up
Speaker 2:A few door dashes and slack messages and everybody,
Speaker 3:Things like that.
Speaker 4:Yeah. I, I, uh, Angus beat me to the, the sharing of Pinter too. Um, but I would also say, yeah, first echo that, you know, if we are going to escalate an issue, we do it, you know, rarely we do it when it matters. Um, I personally also like to, you know, understand the people I'm working with from a personal angle. And so like, you know, Angus and I have had talks about, you know, his dog, I, I just got a new puppy, he's got two dogs. Um, also, you know, talked about vacations and travel and, you know, getting to sort of understand somebody as a person. I think that's really important to me. So we have that kind of common ground and, uh, and, and makes conversations that have to turn serious, makes it a lot easier to go.
Speaker 1:Excellent. And that brings us to of the end of a, yet another lovely conversation. Um, Angus Sean, I have to say it was such a pleasure to get you both on to talk about it. Yes. Thank I learned so much just from sitting and like watching the two of you interact and I hope everybody else finds this one very valuable and stay tuned for more coming soon from the para it podcast folks.
Speaker 3:Thank you guys. Thanks
Speaker 4:This fun. Thanks a.