Jeevan Singh -- The Future of Application Security Engineers

Jeevan Singh, the director of product security at Twilio, discusses the future of application security engineers. Singh highlights the importance of embedding security into all aspects of software development and the need for a strong security culture within organizations. He also explains the skills required for a senior application security engineer, such as application security, software development, and teaching skills. Singh underscores the importance of empathy and influence, emphasizing that soft skills can significantly affect adequate application security. He also discusses the impact of AI, particularly OpenAI's GPT, in supporting the work of security engineers by providing valuable insights and information. Singh concludes by urging application security engineers to broaden their skills, particularly in software development, to ensure they can effectively handle the industry's evolving demands.

Connect with Jeevan Singh:
→ Twilio
→ ChatGPT

Mentioned in this episode:
→ Twilio
→ ChatGPT

Follow the Application Security Podcast:
➜ Home
➜ X
➜ LinkedIn
➜ YouTube
➜ Instagram
➜ Facebook

Chapters:
00:00 Meet Jeevan Singh: The Future of Application Security Engineers
03:25 Would you describe the future of application security engineer
07:18 Yeah, I feel like there's, I mean, maybe 5,000 AppSec engineers
10:24 Hmm. Yeah. And you're talking to, to two owas lifetime members
12:43 Of those three, you know, so AppSec skills, software development skills
17:16 About the teaching skills
21:01 You tell stories about, you know, things you've done in, in
23:17 Yeah. Yeah. And I like, I like the, the, where you
25:03 I like that. I'm gonna use that going forward. That multiplier
32:38 Does it scale
35:26 Does that play, uh, in, so you talked sort of around
39:23 We just got a, there's a, there's a mini episode included
43:59 I've been thinking about this quite a bit, lately, as far

Transcript

Copy Transcript Copied

Chris Romeo 0:00

Jeevan Singh is the director of product security at Twilio where he is embedding security into all aspects of the software development process. Jeevan enjoys building security culture within organizations and educating staff on security best practices. He's responsible for a wide variety of tasks, including architecting security solutions, working with dev teams to resolve security vulns and building out security features. Before life in the security space, Jeevan had a wide variety of development and leadership roles over the past 15 years. Jeevan joins us to talk about the future of application security engineers. He explains the different required skill sets and the path that anyone can take to become a senior application security engineer. He also gives us some perspective on how this works in the startup world when you're not an enterprise, and as a bonus, there's a mini episode within this episode when Jeevan covers the democratization of vulnerability management. He uses that as an example. If you're in AppSec or you want to be in AppSec, this episode will provide you a roadmap of ideas for the future. We hope you enjoyed this conversation with Jeevan Singh. Hey folks. Welcome to another episode of the Application Security podcast. This is Chris Romeo. I'm the CEO of Kerr Ventures, and also joined by my application security partner in

2:07

crime, Robert Hurlbut Hey, Chris.

Chris Romeo 2:08

Robert.

Robert Hurlbut 2:09

Hey, Chris. Yeah, Robert Hurlbut and I'm a principal application security architect and threat modeling lead at Aquia. and, uh, again, as you mentioned, uh, really excited to be here as another application security podcast gets away here.

Chris Romeo 2:25

Yeah. And, uh, a topic, a new topic. Often we're talking about things that people could be saying, Hey, well it feels like you kind of talked about this one before, but in this case, we have something new. And so we're super excited

2:36

to have Jeevan Singh back

Chris Romeo 2:38

with us for a second visit to the podcast. And so, When we have a reoccurring guest, we don't ask them to tell their origin story, but I do wanna share a pointer. If you want to go

2:49

and hear Jeevan's origin

Chris Romeo 2:50

story, it's in an episode entitled Threat Modeling Based in Democracy, which I mean, come on, talk about just an excellent concept of what he explained to us the first time around. So if you didn't hear that episode, you, you really should, uh, go listen to that. It's pretty, pretty different than what we're gonna talk about today, but still was, was an excellent

3:08

episode on Jeevan's approach

Chris Romeo 3:10

to threat modeling inside of a, a, a pretty good sized organization. But with that, um, Robert, I think you wanted to kick off this, this first topic or this topic we're gonna talk

3:21

about with Jeevan So Jeevan, So Jeevan, uh, how

Robert Hurlbut 3:24

how would you describe the future of application security engineer? And, and just to give a little context, I remember you wrote a, a, a post on this, on LinkedIn and it really captured our attention. And here you are again on our podcast. So, so welcome. Uh, you too. Anybody listening? If you put something really interesting on LinkedIn, you might be on our podcast as well. But, uh, let's dive into this. Uh, what's the future?

Jeevan Singh 3:51

Yeah.

Robert Hurlbut 3:51

what's a future application security engineer?

Jeevan Singh 3:54

I think one of the problems is I'm very opinionated on things which, uh, lands me in, uh, hot water sometimes. Uh, but, uh, the, we've seen a lot of changes to application security over the last few decades. Um, initially people need to just have, understand what is security and help developers solve security problems. As companies started to grow, we needed application security engineers to also be really strong as software development. Uh, a lot of, uh, application security teams are small, um, and we can't scale, uh, unless we have, we build our own programs, products, features to really help ensure that we're covering, um, the entire ecosystem. And the third iteration, which I think we're slowly getting into right now, is that. where there's an expectation that even with both, uh, helping developers with their security problems as well as software development, that's not enough. Uh, we're not, we're not winning the battle. We're falling behind. There's too many engineers that we need to help with. So the expectation is now, um, we need to actually teach development. Um, so that third skill that we need to have is being able to be teachers within our, uh, environment. And we've sort of seen it, uh, a while ago with. when developers were just doing software development and when they were done with their software development, they would throw their code over the wall to the QA folks and say, Hey, I've done the development. It's your problem to make sure that you have high, we have high quality for this feature. Um, but, uh, the quality engineers and developers start to work, uh, really closely together. developers started to write a lot more tests, uh, with respect to the features that they had, and now the developers sort of own that sort of experience. And we're gonna see that with security, where developers are now gonna be a lot more responsible for security. And since they're now gonna be responsible for security, there's an expectation that they should also know how to do security. And as application security engineers, we have to help 'em on that journey.

Chris Romeo 6:08

So when you think about the universe of application security engineers that exist today, do you have any idea how many people, can we even take a guess at how many people exist in this role today? Any any idea or thoughts?

Jeevan Singh 6:24

Not enough. I think it's the

Chris Romeo 6:26

know not enough is definitely the answer.

Jeevan Singh 6:29

Um, I, they always point back to like a million unfilled headcount for, um, uh, security jobs in general. I. And I feel that over the last few years, AppSec engineers must start taking a bigger and bigger bite of that million unfilled roles. Um, application security is getting way more important than finally getting the due that we've deserve for a long time. So, um, I, the space small, uh, I know that, uh, um, when I go to our biggest of events, global AppSec, uh, put on by olas, um, Best case scenario, there's 800 or a thousand of us. So I, I, I still feel that the space is small and we, we need a lot more support, uh, in our industry.

Chris Romeo 7:17

Yeah, I feel like there's, I mean, maybe 5,000 AppSec engineers that exist in total. Um, not everybody care, maybe 10,000 at most. Not everybody can get out to the big conferences and stuff, but I feel like to your point, this is. Something that is about to explode from a demand perspective. And I was joking, uh, in, in the talk I did at RSA on the state of the union of application security and also joking with some other friends, um, around the RSA conference in different settings. But the year of the application is really, I. is, is upon us, and I don't know if it's gonna be 23, 24, 25, like what year it's actually gonna hit, but think about all of the investments that have been made, like an infrastructure security in information security, in all of the things that are happening from a cloud security perspective, like a lot of, a lot of the, the core levels have gotten a lot of attention and a lot of investment application security is not in that same boat. Not anywhere near the amount of, of thought. That's, and, and, and resources and budget that have gone in that, that, that we've seen in like infrastructure or cloud security. And so I, I'm predicting, and I know it's dangerous to make predictions sometimes, but the year of the application is, is coming soon. And I think that's why this topic is, is resonating with, with me so much thinking about the future of these folks because we're gonna need more application security engineers to be able to support. The, the drive to secure the applications. When, when people kind of finally wake up and say, well, we've got everything else kind of covered, but this application thing we really haven't done a lot with. There, there's gonna be a, a gigantic demand there. And so from your perspective, why, why is this topic important to you? Why does this even matter from your perspective?

Jeevan Singh 9:05

Yeah, it matters because, Uh, this is the need of the industry. Um, I, I've been in a bunch of roles now where I never felt that we were fully ready for battle. Um, and when I've made battle, it's really partnership with engineering. It's not like we're, uh, fighting against them, but it's like battling the threat actors that are attempting. We all see the noise in our application logs. We always see either bug bounty research well, or what I hope are bug bounty researchers attacking our systems. Uh, but yeah, we see it. Um, we know that. Our bad actors are out there. Um, and a lot of our companies that we've been at, uh, have been, have had incidents, uh, due to threat actors. I know that, uh, the current one that I've been at, uh, we had an incident in August. Um, and we just need more people to come in and be able to help us out and support it. Um, I'm extremely passionate about it, so much so that I've been a part of o OSP for the last five, seven years. Um, Helping out, trying to build out the community here in Vancouver and making sure that the future application security engineers ha have that CAPA capabilities that we need when they need to come in and be able to hit the ground running.

Chris Romeo 10:23

Hmm. Yeah. And you're talking to, to two owas lifetime members here on the line. So, um, we're, we're also big supporters of o osp, and we don't, I mean, I would say I, I'll speak for myself. I don't always agree a hundred percent with everything that's happening, but I agree with the mission. and I think Andrew and the rest of the team there that are part of the OWA Central group are, are doing a great job of trying to push stuff forward. Um, there, there, there's a lot of constraints that exist and, but they're doing, they're, they're, they're, Andrew's really been good at Andrew Vander stock who's taken over those that may not remember Andrew's taken over as the, the, um, one of the leaders of the, the O osp. Uh, Organization and he's really done a lot of good stuff to push things forward and, and, uh, I think that's, there's just a lot of, a lot more good stuff that's gonna happen in the future. And when I think about the future of application security engineers, um, one of the things I've been telling developers that I meet with, like, especially in the, the development of security champions kind of programs, I keep telling folks like, you wanna talk about a unicorn? If you're a se, if you have senior dev skills and mid-level even AppSec skills to senior app sex skills, you wanna talk about being a unicorn, you're a unicorn. And you can write, you just write a number and push it across the table, and a lot of people will pay you that Whatever you write on that piece of paper, they're gonna pay you because that level of of connection of skills, there's just not a lot of people, I mean, I can, we can start naming people that we all know that, uh, kind of fit into that category, but there's just not a lot of people with that skillset, so I'm totally with you as well. Like we've got, there's, there's a lot of room to grow people into this industry and prepare them to do these things.

Jeevan Singh 12:05

Yeah. I, I don't expect, um, every single application security engineer to be good at, uh, the OPSEC skills, like threat modeling, bug bd, F testing, and all the. Up as well as software development as well as being able to teach, but they'll be stronger in some areas, but they should have a, at least foundational understanding for all of those areas. So, but you're right, the, there are unicorns and you're right, I can count on the number, uh, individuals I know that are good at all those areas on my hand. Uh, it, it is, those are hard, hard skills to acquire and it takes a lot of time being in the industry to be able to build up that skillset.

Robert Hurlbut 12:43

So of those three, you know, so AppSec skills, software development skills, and teaching skills, um, could you describe some of those that, uh, and what you mean by those in terms of those particular skill sets?

Jeevan Singh 12:54

Yeah, that's very fair. Um, and so op application security skills. Fall under the traditional AppSec, uh, uh, behaviors and roles. So I do expect people to be really good at security views or threat modeling. Be able to dive deep, partner with engineering, make sure that they understand how to design their features. Um, but they also need to be able to run a bug bounty and know how to build a community of researchers so that they want to continue testing your, um, web application. Um, and you have to know how to deal with vendors, uh, when you're procuring a pen test, um, and be able to get the most outta the vendors to do a real strong pen test. Um, um, so all those sort of traditional skills, you know, how you have to know how to use the tooling, SAS test, uh, um, all sorts of s e a, all those, all those sorts of tools, uh, in general and how to integrate and all that. So, um, that in itself is. It's quite difficult and requires a lot of experience. And even on the tooling side, uh, I see a lot of, um, folks, uh, that uh, um, partially integrate tools as well. They don't fully integrate tools. Um, they don't have coverage or metrics and. They don't even know how to, or they don't rather, um, burn, use the tools to actually burn down risk. So I, I'm asking a lot for, um, future AppSec folks, when you have to be good at just application security skills and just be able to dive very, very deep in all those domains, uh, in itself. Um, with respect to software development, I, I have a great example. Um, there's a member of my team, Alejandro, that, uh, Um, as we started to really integrate tools on our ecosystem, um, we want to make it in a way that we are able to pull the. Data out of the tools, uh, present them into dashboards for our security leadership, but also engineering leadership. We don't, one of the thing philosophies that we have is that we don't want to drown our engineering team with tickets. We wanna be able to do the crawl, walk, and run methodology first, let's present them dashboards so they actually know how good or bad their posture is. Um, then we'll analyze the data and work with them and burn, um, the critical vulnerabilities, uh, as much as we can. And then we'll, eventually we will start taking them. Um, Alejandro not noticing that we're doing this quite inefficient. Um, why are we, um, integrating one tool? they change our APIs, it breaks all these other things. Um, there's gotta be a better way that we can actually, um, integrate all of our tools in a more standardized framework level. So he built a, um, what, uh, tool that he calls MultiTech, where, um, it will, uh, it has a AW w s lamb. It will call the API of the tools. Suck that information and it spits it through our systems. Um, and it'll eventually land in our data visualization tool. Um, but the good thing is that, um, as a application security engineer, we only have to build that lambda and just everything else will take care of it. Um, it'll, um, suck in the appropriate metrics and put it into the, uh, data visualization tool. It will let you know if, uh, the end API endpoint is down. Um, if there are errors as a part of it, you get all that for free. and that's the expectation where I have for application security engineers. When we're building these tools. It didn't make sense that we would build this one tool, takes us maybe two months to integrate or whatever amount of time and do that eight times for all of the tools that we have in our ecosystem. Rather we just build a framework and then we just build out the lambdas that just pulls in the data and the rest will be taken care of. So that's where I'm looking for software development. It really helps us with skills. But on the flip side, I also look at, um, if we are analyzing the tooling data, um, and we say, notice that, hey, you know what, we can probably burn down 80% of our critical vulnerabilities if we do X, Y, and Z. And if we partner with the engineering team, we can actually sit, sit down with them. we can code it out together and we can actually eliminate. So there might be multiple different ways where your software development skills as an application security engineer will help you with the ultimate um, goal, which is burning down risk in itself.

Chris Romeo 17:16

What about the teaching skills? So we didn't, uh, we didn't get to that one. I'm, I'm, I'm curious what, what do you mean by the teaching skills?

Jeevan Singh 17:23

Yeah, so it, it sort of goes back to the democratization of threat modeling. Um, but beyond that as well, where we, we have to do a couple of things when it comes to being teachers. One is we actually have to develop the content and we have to develop it in a way that. The engineers can actually learn it. Um, so, uh, the threat modeling content itself, um, it took me a month or so to develop, um, and really refined. I had beta engineering teams. We sat through it. I realized how my first iterations need to get better and better. Um, I iterated a couple of times and it was pretty solid after that. But you have to be able to deliver the content that even a junior engineer. Gain something out of it as well as a staff or principal level engineer. So, um, developing a content is a particular skill. It takes a lot of just being on the dev side to have, uh, knowledge of how they think and how they will actually respond to it. And then the other side of it is actually teaching them, so you actually deliver the content as well. So developing the content is one part, but delivering it as well. And, um, it, it's hard, it's hard to deliver the content in a way that, um, it will make, allow for the engineers to actually understand the content you have to. Be empathetic. Um, you have to be forceful. Um, if you notice some blank stares, uh, you have to call up people and say, Hey, what, what's on your mind? Why, what are you not understanding? Um, and you also have to be able to, um, build up those examples that you need so that it provides interaction amongst the teams as well. So, I know a lot of people learn, uh, a lot of people learn by just, uh, listening. A lot of people learn by doing hands-on. A lot of people learn by doing group work. So you, you have to be, as a teacher, you have to be able to understand what type of behaviors will allow the individuals who gain the most of your, uh, trainings. Um, some of the challenges that we all had, um, I'm sure going, uh, And through the AppSec, uh, uh, building our own application security skills is that we go to these trainings that would be a day's worth of content and it's really hard to retain any of that. So you have to be able to build a training in itself where hopefully they can retain 75, 80% of the training itself and the rest they'll learn, um, while they do the hands-on work, uh, for threat modeling or beyond. Um, Threat modeling was just one of the categories that, uh, we're building out for our trainings. We also have, uh, secure coding training where we actually show them what are some things that you need to look out for when you're doing code reviews, um, as well as like, think like an attacker. What are some of the things that attacker, how is it gonna exploit your systems? So there's a number of, uh, trainings that we can actually provide our engineers so that they, they can grow, um, and learn security themselves.

Chris Romeo 20:25

I am curious to get your take on, uh, when I think about these three different things, ABSEC skills, software development skills, and teaching skills, for in my mind, two of those three, I can teach people how to do. One of them requires somebody who has some I, I think of teaching ability personally as something that someone has natural ability. To do, to be able to, and I, and I think, you know, and I'm looking across the three of us, I know we've all taught classes at different perspectives. I think we're all, I think we're all people that have that ability. Because I know, I've seen Robert teach

20:59

classes and Jeevan, I've heard

Chris Romeo 21:00

you tell stories about, you know, things you've done in, in teaching classes and things and teaching developers how to do certain things. Do you think you can teach, can you teach someone to be, to have teaching? Are teaching skills teachable? Can you te can everybody, can all of your ssec engineers teach? And is, should they be, should they have to teach to be at this level that we're talking about?

Jeevan Singh 21:24

Oh, that's a great question. I, I really feel that everyone can teach and everyone has something to teach, but I do know that folks on my team would be, there would be folks that shy away from it because I. It is, it is hard. It, it is really hard to, uh, teach and um, and that's how I sort of feel that I would build out a team as well. Um, I would have a core team that mostly focuses on AppSec stuff and another team that mostly focuses on software development. Another team that mostly focus on content and content delivery. But, um, I would allow folks to be able to do all, all of those if they want to grow their careers. So, um, I, I do expect that as you mature in your, um, career, um, the senior staff, principal, architect level engineers, they would start gravitating towards the instructor, uh, sort of, uh, uh, role within the team now and may do a little bit more of that. Um, cuz they went through it, they've been in the trenches, they understand things and, um, I, I assume they're like me where I don't want people to make the same mistakes that I've made in my career. I want them to learn and grow and, um, be better than me. So I see more of the senior folks, uh, sort of gravitating towards that. Um, and then I do definitely see a lot more of the junior ones focusing on the AppSec, uh, skills and potentially the software development as well, just cuz those are the, um, skills that are a little bit more well defined and easier to grasp, uh, earlier in your careers. So that, that, that's where I sort of see it, um, um, as a career progression standpoint, but also like a team side where, uh, I wouldn't want. I want people to be able to sort of align their um, roles to where they feel that they can make the biggest impact.

Chris Romeo 23:17

Yeah. Yeah. And I like, I like the, the, where you landed there because I would, I guess I would take it a step further and say a staff engineer that cannot teach is not a staff engineer. That's part of, part of reaching that pinnacle in the en in the engineering or the security community or anything, is being able to, like a sta in my mind, a staff engineer should make 10 junior and mid-level engineers, two x. From what they do, because that, because the, one of the junior or mid-level folks should be able to turn to a staff engineer and say, okay, I've tried this, I've tried this, I've tried this. I'm stuck. The staff engineer should be able to say, well, let me just take you through a little bit. Let me give you five minutes on what I, how I would approach that. Boom, you've, you've unlocked that person now. You're not going to do it for 'em. You're not, look, you're not pair programming and they're watching how great you are. You've, you've unlocked their ability to go to the next level on, on a problem that you've already, to your point, you've already been, you've already experienced it, you did it wrong. You it didn't work, it had a vulnerability. Whatever was wrong with it. Right? Like, you, you're able to then instill that knowledge and, but that, to me, that's teaching, like there's classroom teaching. there's coaching. These things are all part of being a, a senior, anything in my mind. And so that's, that's kind of where I land

Jeevan Singh 24:38

yeah. Whenever I do performance reviews, uh, those at the staffing principal level, what we call it is the multiplier. You have to be well beyond just Your level of impact, you have to be a multiplier and make o other people be able to do your level of impact as well. So the programs, the role itself, um, they have to choose them or focus on them where they can actually multiply the effort that they put in.

Chris Romeo 25:03

I like that. I'm gonna use that going forward. That multiplier effect. I think that's a good, that's a good, very positive way of, of describing what we're expecting people. If you want to be called a staff engineer, you better already be multiplying people and then we promote, you know, I guess I'm somebody who, I always promote people to what they're, you better do the job you want the promotion. Great. You wanna be a staff engineer. I'll watch, and when you act like a staff engineer for a while, then I'll go file the paper and make you a staff engineer. That's maybe I'm old school from that perspective, but that's how I approach the world. Let's, uh, let's tackle, uh, an issue that, uh, givin you brought up in your kind of initial thoughts about this, that some people are gonna, are gonna be like, there's gonna be some groans behind the scenes, but, uh, I, I know where you're going with this and I'm super excited for you to answer. Our soft skills a requirement for the future application security engineer. How does soft skills play into to being a successful AppSec engineer?

Jeevan Singh 25:58

Yeah, that. that's a lesson that I learned very, very early on. Um, so, uh, security engineers, we, we could only make a certain amount of impact. Um, and our job is to actually influence the business to do the right thing. So, um, What I've learned is that the most important scale, or actually the second most important, the most important scale, I feel that for application security, engineering gen in general, is empathy. Um, and you need to be able to empathize with your stakeholders, with your clients, with your partners, and be able to deliver the functionality that you need just in general. But the second most important skill is influence and. it has to be directed influence. Um, so what I mean by that is that if you are establishing a new program, um, while it's important to influence the individual comp, compute contributors on your team or on, uh, the engineering teams, um, that won't get you the amount of, uh, return of investment as you would for, uh, directing that influence at the leadership level, making sure that you're focusing on, um, engineering leadership or the execs and making sure that they understand why it's important to do whatever you need to do. So once you've actually established that influence, um, once they understand data driven, um, how well the security program is doing. then I, I would recommend that you start focusing on the IC level so that you build a culture within the organization. So, um, In order to do that, you actually have to have influence skills and those are hard skills to gain in general. It's something that you need to do through practice. You have to realize your team isn't the AppSec team, it's the business in itself, and the folks that are actually doing the work are likely gonna be the engineers. So in order to get them to do the right security work, You shouldn't be doing it. You shouldn't be fixing vulnerabilities yourself. You should be really guiding the business to fix vulnerabilities and being able to influence the, uh, business to fix vulnerabilities at scale. So yeah,

Chris Romeo 28:12

Yeah, for me,

Jeevan Singh 28:13

Yeah.

Chris Romeo 28:14

I I agree 100%. And you know, when I think back through my career, like I'm never the smartest person in the room. I'm never the best security person in the room, but, but for whatever reason, I learned at a pretty early age how to use influence, how to collaborate and get people thinking in the same direction and then going in a direction. And it really is a, a, a, a, a super blessing to, to just be able to, I, and I don't even, it's not like I think about it, but it's, it's soft skills are just so important to just even know how to talk to people. because as you said, we're trying to influence, we're trying to, we're trying to go to, um, somebody and, and we, they don't report to us, so we can't use positional power to say, you must do this. You don't have any positional power. All you have is that influence. And so you have to be able to show them what is in it for them. What are you gonna get, what, what's the reward for you as a result of going down this path that I know, I'm telling you that, that, you know, in the short term's gonna be more work. So I'm literally asking you to do more work, but in the, in the end, it's gonna make your life better. And it's, it's about being able to describe that. And so I I think we still don't focus on this enough. And whenever you talk about it, like if I did a, they'll say I did a class at, um, I put in a, a, a training for O os global and it was called Soft Skills for ABSEC Professionals. Nobody would show up. It would be empty, unfortunately. Like, and which is sad because like if you, if if someone, if people took that class, they would vault their career like multiple levels because those are the things that you, you can't really read a textbook about. About this stuff, and so, yeah, I'm with you too. I think soft skills are so crucial and, and they're so often neglected.

Jeevan Singh 30:04

It all ties back in as well. Like, uh, influence soft skills teaching. They all are all part of the same sort of ecosystem. Um, and if you do well in one, you can do well in others as well. So, um, I, I, Soft skills is out of those AppSec and, uh, software development skills. Uh, you will find that soft skills is gonna be a much more important part of, uh, toolbox that, uh, uh, every application security engineer needs to have.

Robert Hurlbut 30:37

So you may have already touched on this about a junior, uh, developer, but, but are they also expected to have all those skills or are they looking towards sort of a, a. A path to get to where they do have those skills as we've talked about, a staff engineer or somebody like that.

Jeevan Singh 30:54

Yeah, at the junior level, it, you're right. Like it, it'll, it'll be a path to get those skills. Um, whenever I do hire at the junior level, um, junior, Level, the most important thing that I look for is, um, that hunger, that desire, that passion for security, and. that sometimes is hard to teach as well. Um, uh, so those that are really hungry, that really want to consume knowledge and grow, um, uh, that's what I look out for. And then I know that I can, those, uh, technical skills, no problem. We can, you just need experience. You'll get that. And then we slowly get them, um, as part of, maybe they'll start, uh, joining some of the training sessions and understand how we deliver it. then they'll actually shadow one of us, um, and do it. And then what we like to do is also reverse shadow. So, um, they will, uh, lead it and, um, maybe, uh, an individual on the more senior individual on the team would be a part of it. And sort of jumping in if, uh, there are some little gaps, uh, as the training that they're trying to deliver. So, A lot of different ways for those, uh, junior folks to sort of scale up. But, uh, usually the junior folks, we're not, we're not looking for all three of those skills to be there, but more of that, uh, desire to gain those skills and how strong that desire is.

Chris Romeo 32:19

So if we change gears a little bit and think about now the team, instead of just individual AppSec engineers, how do you scale a team based on all of these things we've talked about with different skill sets, different seniority levels? How does this come together in a team?

Jeevan Singh 32:37

Yeah.

Chris Romeo 32:38

and how does it scale? How does it scale? Probably the most important part.

Jeevan Singh 32:41

Yeah. I think the biggest problem that we have as application security engineers, or even the security team is that our engineering counterparts grow way faster than we do. Um, and it'll never, we never have a ratio that will. always stay the same. Um, they will hire several hundred before we make it a couple of more folks on our team. Um, so the things that we have to sort of realize is that, um, through there's various ways to scale. So, um, those application security skills that we talked about, um, threat modeling, uh, and, uh, well doing it one at a time is not gonna scale. And if they're hiring hundreds of engineers, they're gonna be pushing out so many new features. we're not gonna be able to review. Our queues are gonna get longer. So, which is why we have to rely on the teaching aspect and allowing them to do the heavy lifting there. Um, similar sort of deal with, uh, code reviews. So, um, it's very, it's not bad in a smaller company. Um, we can sit down, actually do a manual code review. But as we start adding hundreds or thousands of engineers to the team, you're not gonna do any more manual code reviews. It doesn't make sense that you really have to put the tools in place where we actually can scale. So, um, every sort of program, the one by one, but, um, it, it always goes back to really the teaching aspect of things. And I think, uh, maybe even a topic that you really like talking about Chris is like, building up those champions. Um, I, the, I really want to make sure that those secur, the engineering champions for security are as strong as maybe a junior or an intermediate, uh, security engineer, where they're able to have the basic skills of, um, adding your repo to a SA or s e a or be able to understand what are the results coming out for your team for those security vulnerabilities. Being able to come up with a remediation plan. Um, being able to work closely with the security folks and understand our guidance, um, and be really that first level of, uh, support for the security team and answering all those sort of questions so that when there are. More difficult and deeper questions than we pull up in the security engineer. So, um, really relying on our teaching capabilities, our app skills capabilities and software development capabilities to help us scale, but also building up programs that really make it easier for us to, um, extend our team in itself and work with, uh, other folks, uh, within engineering.

Robert Hurlbut 35:25

and how does that play, uh, in, so you talked sort of around maybe larger organizations, but how does that play into a small organization that maybe only has one or two AppSec engineers?

Jeevan Singh 35:36

Yeah, it's a different ballgame because you have to be very focused with your plan. Um, you can't do everything, uh, with a smaller team. You can't do everything. And, um, fortunately I got off a call very recently with someone. On a very small team, and they're like, okay. Um, there are at least a hundred different fires that I have. How do I decide which fire is the most important? And, uh, funnily enough, um, my first question was, um, does the business really care about security? Um, so sometimes you, they hire security engineers, um, for like, that's what the industry is doing. Maybe that's what I need to be doing, compliance reasons, whatever. Um, but uh, they're not as, um, Informed about what security can and should be doing. So, uh, my question to them was that, should we focus on building influence first? Um, let's start first reporting on let's get the data that we have currently in the, um, in the company and sort of figure out how well are you all doing with security. Um, they mentioned that. A lot of P one vulnerabilities are over sla. There's no, um, incentive to actually fix things. I'm like, yeah, that very common in smaller organizations. Unfortunately, still common in bigger organizations too, but uh, in smaller organizations you have to incentivize folks to actually fix those vulnerabilities. So you team up with your security leadership, uh, and engineering leadership. Have, uh, provide them the actual metrics of what's going on. Say that, uh, this is how much security technical debt that we are accruing and this is how we need to address that security technical debt and also loop in the executives so that they are aware, um, for it as well. Um, one of the more, uh, memorable programs that we put out is a democratize on the building management, um, where. We actually decided that security should not be part of the X L A extension process. We don't own those vulnerabilities the engineering team does, so why are we a blocker or why are they coming to us to ask for extension? So what we did is we actually put the engineering leadership as a part of it. So P four, P five s uh, engineering managers can extend and self, uh, uh, go through that self-service process. P three s go to directors P two s go to the VPs. P one s can go to your executive if you're at a small company. And what would happen is when the executives actually understand there are, um, P one s that are going to them for extensions. They're gonna ask more questions. Why are we extending this? We've extended two times already. Why aren't we addressing this? Um, my favorite story is when we had, uh, a VP of engineering. He extended it once and then engineering team came back to him and saying, Hey, we need to extend this again. He's like, okay, let's, let's hold on one second. I Alfred extended it once. If anything happens because of this vulnerability, they're gonna point fingers at me. because this is my vulnerability. So before I extend it, I need to know who's gonna work on it, what's the plan, and how long it's gonna take. Uh, by the end of the day, they got that. They said, okay, this is the plan. This is the individual that's gonna work on it. It's gonna take six weeks. Um, it was fixed in a week and a half cuz no one was going to go back to that VP of engineering and ask for another extension. So once you are able to involve the executives and the engineering leadership as part of it, they will influence them to be a part of these programs. They will start, um, exerting their influence on the engineering work as well.

Chris Romeo 39:23

Well, we just got a, there's a, there's a mini episode included right within this episode. The democratization of vulnerability management was a little, a little bit of a carve out here that just happened, but I, I love that. That's, that's, that's, uh, such a neat way to approach it, you know, put it, it's this, I love the, what you just said, like this is not securities. Thing. This is engineering's thing. Why are we getting in the middle of it? Establish the governance from an SLA perspective. Set up the systems and processes to make it work, and then get the heck out of the way. And let let the, the, the, uh, organizational chart and their comfort with risk. Like to that, to the story you just shared. That's so excellent that VP of engineering was like, wait a minute, they're coming for me. if I extend this again. So you're basically, so like that changed his, he or she's wholes perspective on what the, you know, how, how to approach it. So I love that as a, as an approach. Okay. We can't talk about the future of AppSec engineers without addressing generative AI chat, G P t, Bard, whatever, whatever, however you want to get to it. How does generative AI impact the future of this role of AppSec engineer?

Jeevan Singh 40:33

Yeah, it's already impacting. Um, so what I've seen on my team today is that we get pulled into a lot of security views and threat models. and sometimes we don't really understand the technology behind it. Um, and what I've seen some of my junior more intermediate engineers is like, they will ask chat g p T, Hey, what is, uh, Kafka? Uh, what are, how does r d s, uh, what are some concerns that I may have with r d s? Um, it doesn't provide you the full picture, but it at least points us in the right direction of some of the things that we can think about. So to even today, uh, chat, g p t can help us and. in the future. I do hope, uh, they can, it can help us a lot more, but, um, The flip side of it is we're gonna be seeing a lot of AI in our organizations as well, and how do we protect the data going into ai? Um, I know that Microsoft has put out great, uh, I think a top 10. Threat modeling, ai, sort of, of risks that we should be concerned with. I know that, um, members on some, uh, our teams security teams have started already diving deep and performing actual threat models on some of our AI models that we have in itself, ML AI models. So yeah, this is, this is. Here this is today, we can leverage it to help us to do better threat models, and we also need to make sure that the data that we are collecting and hoping to actually make it easier for our customers doesn't get polluted or corrupted by outside adversaries, uh, for these sort of purposes as well.

Chris Romeo 42:16

Yeah. Yeah. And I, when I think about generative AI in in general, which is kind of a weird thing to say, I just realized, but it's, I don't see, like AI is not gonna replace AppSec engineers. It's not gonna replace developers. It's not gonna replace. There are some people that believe that are from that school of thought. I'm not one of 'em. I believe that generative AI is going to two x to five x. What an AppSec engineer can do to your point here where you already have folks who are saying, who are using it as a resource to quickly gain knowledge about something. That they would've had to, you know, Google in the past and, and read, decipher various webpages to try to figure out what is the, you know, well this one says this, this one says this. Now you've got a single source of, of truth kind of happening there. And so I, I see this as more of a tool that can help those AppSec engineers do more. So we know we don't have enough ABSEC engineers, both in our industry and in most companies would, no one's gonna say, well, we're a little, we're, we've got a few too many ABSEC engineers. They're just sitting around. They don't really have, they hate drink coffee. They don't really, there's just not that much for 'em to do. Nobody's gonna say that. So anything that we could do that would, that would help them to even 1.5 x their output is a, a, a positive step forward for. Changing the business, uh, helping the business to think more about security. Um, I think it's all, it's all all good stuff.

Jeevan Singh 43:48

Yeah. Agree 100%. Uh, it will. Supplement us and it'll help us be better at what we need to do. Uh, put it very succinctly. Thank you, Chris.

Chris Romeo 43:58

I've been thinking about this quite a bit, lately, as far as how to, and, and listening and, and like all of us, you know, learn, I'm learning a lot from a lot of different people. I'm, I'm l reading things, listening to things like, it's such a new area and there's so many different facets to it. I'm just trying to absorb as much knowledge from other folks that, um, have, have, you know, Real backgrounds in this and, and how all this stuff comes together. So if you were to give us a key takeaway or a call to action about the future application security engineer, what, where, where would you go with that? Is it a call to action? Is it a key takeaway? How do you want to land the plane here?

Jeevan Singh 44:39

Yeah, I, I wanna first say the feature is bright. Um, it's exciting. It's, um, where all of us really want to go. Um, we're gonna see even stronger partnerships with our engineering teams. Uh, everyone's starting to care more and more about security year, right? Maybe 2023, maybe 2024, maybe 2025 is gonna be the year of the application. Um, so we'll see better and stronger partnerships. Um, what I'll ask of, uh, application security engineers is that, um, make sure that you're, um, having more breadth and depth to your skills. Uh, so, um, if you are very heavy on the application security side, a threat modeling by count and all that, Learn how to do a bit more software development. Think about how you can scale your programs. Think about how you can ensure that the programs that you build will last longer than you and those that are good in both of those domain. Start thinking about how do you build that next generation, either of junior engen, uh, junior security engineers or engineers themselves, where that you can impart your knowledge on them. So, um, let's all get iteratively a little bit better, uh, at what we do, and then that will just make our companies a lot safer and much safer for our users as well.

Chris Romeo 45:57

Excellent. It's always such a joy to have you on the podcast because you have so much knowledge and wisdom and I'm just, I'm sitting here taking it all in. Just thank you so much for sharing, being willing to share all these things. From inside, you know, kind of some inside experience and, and how we bring this all together. And it's just, it's really helpful. I mean, we got a mini lesson in democratizing vulnerability management right in the middle of this. Like, that's, that's the thing I just, I love about your approach. You're just willing to provide that and share that with us so that we can all learn from it. And I know I, I learned many things already just in this conversation. Many things that I can apply. So thanks for taking the time, and we look forward to the next. Next time that you join us here to bring some more wisdom for our, our audience.

Jeevan Singh 46:41

Thank you so much for inviting me. I love talking about all of the mistakes that I've made in my career, so that other people can not make those same mistakes. Uh, really appreciate, uh, always chatting.

Chris Romeo 46:54

All right, thanks Steven.