Dr. Jared Demott -- Cloud Security & Bug Bounty

Chris and Robert are thrilled to have an insightful conversation with Dr. Jared Demott, a seasoned expert in the field of cybersecurity. The discussion traverses a range of topics, from controversial opinions on application security to the practical aspects of managing bug bounty programs in large corporations like Microsoft.

We dive into the technicalities of bug bounty programs, exploring how companies like Microsoft handle the influx of reports and the importance of such programs in a comprehensive security strategy. Dr. Demott provides valuable insights into the evolution of bug classes and the never-ending challenge of addressing significant bug types, emphasizing that no bug class can ever be fully eradicated.

This episode is a must-listen for anyone interested in the nuances of software security, the realities of cybersecurity employment, and the ongoing challenges in bug mitigation. Join us for an enlightening journey into the heart of application security with Dr. Jared Demott.

Links:

Microsoft Security Response Center MSRC: https://www.microsoft.com/en-us/msrc

FOLLOW US ON SOCIAL MEDIA:

➜ X: @AppSecPodcast
➜ LinkedIn: The Application Security Podcast
➜ YouTube: @ApplicationSecurityPodcast
➜ Instagram: @appsecpodcast
➜ Facebook: Application Security Podcast

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Transcript

Copy Transcript Copied

Chris Romeo 0:00

Hey folks, we're thrilled to welcome Dr. Jared DeMott, a cybersecurity veteran who started his journey at the NSA and has since made significant contributions to the world of AppSec. From teaching at renowned conferences and universities to leading successful startups focused on malware, monitoring, and pen testing, Jared's done it all. Currently, he's making waves at Microsoft, leading a team dedicated to safeguarding products and customers by transforming bug bounty reports into crucial fixes for cloud services. We'll dive deep into Jared's role at Microsoft, explore the intricacies of the bug bounty program, and journey through the ever evolving landscape of cybersecurity. Gear up for an insightful conversation with a passionate individual who's not only making the cyber world safer, but also believes in building fun, inclusive teams with an entrepreneurial spirit. Note, Dr. Jared Demott is not officially speaking on behalf of Microsoft during this interview. Hey folks, welcome to another episode of the application security podcast. This is Chris Romeo. I am the CEO of Devici and a general partner at Kerr Ventures. Also joined by my good friend, Robert Hurlbut, the Sultan of threat modeling. This is a new, I'm trying out new, new names, new kind of fun things. I don't know, Robert, does that sound like something you want to be?

Robert Hurlbut 1:59

I don't know yet. but, but definitely an, an advocate and a promoter of threat modeling for sure. Uh, yeah. So Robert Harbutt, uh, I'm a principal application security architect and threat modeling lead at Aquia. And really glad to be here today.

Chris Romeo 2:14

Yeah, it's, it's, I feel like we're getting the band back together. And people that are looking at this are going, how is this the band? We've never met Dr. Jared DeMott, but let me tell this quick story. Um, in the early days of security journey and around the 2017 ish, I can't remember the exact year. Uh, but Robert and. Jared were gracious enough to join me in what was effectively the lobby of a video studio, a video production company, where we set up a little studio environment and we recorded the first ever Security Journey White Belt content. Um, and these two gentlemen were, were, awesome on camera and provided all their expertise and we had a lot of fun doing it. And so you guys were there at the very beginning of, uh, of my startup journey. And, uh, that was a number of years ago. So, uh, Jared, let's get into your security origin story. Um, I really do want to know this. Like I don't, I don't know kind of how you got into security and you have a little bit of some of the details of your path, but tell us that story if you would.

Dr. Jared Demott 3:19

Yeah, sure. Interestingly enough, uh, you and I both grew up in Michigan, so a little, uh, fun fact there. I grew up in a small town, uh, on the west side of the state, Reed City, if you know where that is, and, uh, was, uh, was good. And my wife and I met at, I was there at Ferris State University, got married young in 98 and 2000, you know, was ready to enter the workforce. And, uh, you know, pretty much at that time in Michigan, it was like the big three for auto and there was other manufacturing and different types. So there was some consulting and different types of healthcare and professional support, but there wasn't, certainly cyber hadn't even become a word until what, probably 2008 or something like that. So much later, you know. for that whole field. And so I was about to actually accept a job at State Farm down in Bloomington, Illinois, as a Unix administrator, when I got a sort of mysterious call from somebody from Baltimore saying that they wanted me to fly out and interview. And they said they were from the NSA. And I was like, okay, do you guys make. Tires for Ford or like, I had no idea. I just, I had not been introduced to that world. I mean, growing up in the eighties and nineties, like three letter agencies, you just hadn't heard of them. Like you did. And I was before nine 11 is before Snowden. It was before. And if you were, you know, from a rural area, like that whole intelligence collection scene in DC, like I literally had never even like heard of that whole like scene, there weren't like the. Number of movies there are now and stuff on it. And so anyway, long story short, I ended up going out there, flying out their interview and they hooked me up to the polygraph and you're like kind of rigid on the thing with the heartbeat, you know, and all. It

Chris Romeo 4:44

How early in the interview do they hook you up to the polygraph?

Dr. Jared Demott 4:47

right away. They're like, you know, you go through some like interviews and then they're like, yeah, we need to do the poly if you wanna be considered. And it's like you're kind of, it's like an all day thing and like before you know it, you're in the chair and you hear like. You know, somebody else being grilled in the next room and you're like, what is going on here? You know, I think sometimes they kind of like, with young people, it's a pretty short process. Cause like, especially back then their hiring strategy was like, you know, ignorant rural kids that have never traveled overseas was kind of like their target audience. Cause they knew that you wouldn't have any like. Foreign connections, right? And now I think they broaden their perspective on how they hire and you know, the whole DNI and everything is a different world we live in now back then, but back then they had a very focused strategy on like rural kids with all A's was like what, basically what they were hiring for. And that was me, you know? So anyway. It was a fun experience. I ended up hiring on there. It was just a great way to start a career. I loved it. I loved the environment. It was like this interesting mix of kind of academic, but also like you were working with military people and, you know, kind of, you know, I did a master's at Johns Hopkins while I was there and you kind of just got exposed to a lot of the stuff, you know, that became the early seeds of the cybersecurity field before it really blew up and became a And so I started going to the big conferences, you know, like DEF CON and all that sort of stuff way back in like 2001. And so I've been going to those for a long time. And that's kind of what really launched my career and probably how we ultimately met as far as, you know, like you were talking about, you know, working on some training videos and stuff, because I had a real heart from that early on. and I, you, you, if you went to these conferences and you watched enough talks, you were like, ah, I feel like I could have gave that talk better at some, you know, once you've been going for a few years, you know, you'd see one that was really bad. You'd see somebody. You know, back then, the stuff that was on stage, thank goodness, some of the things in our field have changed, you know, you'd see people drunk on stage swearing and like, it was just like the environment wasn't the professional environment that it is now. So I think a lot of things have changed for the better in regards to conferences and stuff. And, and I kind of just said, you know, I should try that too and got into that. And that was a fun way to jump into that. So. My wife and I decided as a child, we wanted to move back to Michigan. You know, remote work wasn't really a thing in 2005, but I found a way to make it work somehow. And there was some other defense contracting company that, uh, was like, eh, no, nobody works remote. Nobody, you know, you have to be in a SCIF and, you know, do all the things that people do when you're in cleared work. And I was like, But, you know, and they, but back then, especially, you know, if you had a master's degree and, you know, five years experience in the top, you know, TSSCI, they call it top secret security clearance, you could pretty much negotiate, you know, whatever you needed. And I'm like, well, this is, this is the deal, you know? And they were like, all right, fine. But, you know. You got to be in your office eight to five. I'm going to call you three times a day to make sure that, you know, nobody in your family's bugging you, like the whole like environment of like, like flexible work arrangements. It wasn't a thing back then. Like it is now, like they didn't care about your personal space and like, so my boss would call me like three times a day and like make sure I was at my desk. Cause people didn't trust people working remote back then. It's so laughable in hindsight. Now, of course, I manage a distributed team at Microsoft, but I'm skipping ahead. So I kind of stayed in that field, decided to do a PhD at Michigan State, decided to rep the, rep the gear a little bit since my, uh. My older son is now, he's a third year data science major at Michigan State in the Spartan marching band. So we go to, we go to all the games and so is my senior at Rockford High School. He's in the marching band too. So we do a lot of football in the fall as far as watching the bands and all that. Um, but, so I got into kind of teaching and training and did a bunch of, you know, courses and taught at Black Hat and kind of got into that whole, wrote a book, which is, I don't know if you can see it somewhere on my wall. I don't know if the video shows it or not, but. You know, and that was great. And then, uh, enjoyed the academic scene and all of that, but I really felt like, oh, this startups thing sounds kind of cool. Everybody in our field started getting into startups, and it was kind of a hot thing to do. And so I worked for a couple startups and had a lot of fun, learned a lot, um, and ultimately started my own consulting company and sold that and. You know, that was kind of, that was probably the first moment in my career, because all the rest of the time, I kind of felt like life was dragging me along, you know, it's like, especially for me, because I was married young and had kids, so you kind of felt like, got to provide, get a house, get a job, get a better job, get a master's degree, get a PhD, you know, there's all these like, like, you kind of knew where the next rung is, but after you sell a business, and you've kind of, you got a PhD, wrote a book, sold a business, had a family, my kids are almost grown. It was the first time in my life, I'm 45 now, I was like, I don't know where I need to go from here. Like, I'm kind of like feeling a little bit lost, actually. Not really lost, but you just kind of like, what's the next thing that you do? And, uh, one place I hadn't worked was in big tech and ended up, uh, I did a short stint in Amazon, which I liked. I was in Prime Video and AppSec, but I was missing the management piece. I was back in an IC role, so I decided to get back into the management. Uh, part and got a call from MSRC, which is the Microsoft Security Response Center. That's where I work now at Microsoft. And I really like it. It's a great organization, great mission, great group. And I can, yeah, tell you more about that if you want, but that's kind of my long, probably too long origin story.

Chris Romeo 9:51

No, it's good. It provides perspective and, you know, I learned a couple of things that I can apply in my hiring process. So polygraphs for everyone. As soon as you come in the building, it's like, hook yourself up to this thing. And we're going to check and see if you're, uh, if you're, so did you really work at this company? And we're checking, I've got the little scribble thing and I'm standing next to it. Like imagine that scene, like nobody would work for you in this day age, if there was a polygraph attached to it. But yeah, it's great to hear how you've, uh, how you kind of moved your way through and, and just, what you learned about remote work is, was excellent.

Robert Hurlbut 10:29

a day.

Chris Romeo 10:30

You may have been the first person to ever work remotely. Like that might be a distinction that you could, you could, I'm going to look at Wikipedia and see if that's your face. you're in there as first person to ever work remotely was Dr. Jared DeMott in, in the Western part of Michigan. So

Dr. Jared Demott 10:45

Nice.

Robert Hurlbut 10:46

Excellent. So, uh, Jared, tell us about, you know, what you, you mentioned the group you're working with, the team you're working with, but what do you do at Microsoft these days?

Dr. Jared Demott 10:56

Yeah. So as you know, Microsoft, huge company and, and kind of an interesting company to work at. Uh, these days, I would say, I don't know if I would have said that 10 or 15 years ago, cause they were, you know, doing office and windows and it's kind of, you know, business software and stuff, but now they're a security company. And they're the kind of one of the forefront runners on AI and kind of, it's a pretty exciting place to work for a big company. I would say, at least in my opinion, I definitely enjoy the current culture. The culture there is good too. I think our, our, uh, Our VP, Anshal, she sets a really professional vibe for our entire organization. And so we, we're in a pretty big and complex security org known as the Microsoft Security Response Center. We have a lot of different elements to that, um, not the least of which covers all of what happens at Microsoft and security. When you think about security at Microsoft, it's just huge, right? With 230, 000 employees, you know, six different SOCs, I don't know how many different red teams. I mean, it's just a huge company with a huge amount of different security orgs. So at least within ours we have all that too with threat intelligence and SOC and engineering teams that make software for our portal in particular. But one of the things that we do, the team that I work on is uh, we do a couple things. One of the main things is bug bounty. So you've heard of that program. Security researchers all over the world can get paid for their work. They can submit vulnerabilities through our research portal if they find, you know, different types of vulnerabilities whether it's in You know, Office or Windows, or Azure Cloud, AI, you know, whatever it is. And so our team kind of splits out into different pieces depending on what the vulnerabilities are. So in my case, I'm managing the team, what we call OLS, online services. So if a researcher submits a vulnerability and say like Azure or something like that, or one of our different, you know, um, portals or, or different types of software associated with, with the cloud online, um, Whether it's, you know, Dynamics or Power BI or whatever it is, um, that type of stuff will end up coming in through us and the folks that report to me will spend half their time, uh, doing, uh, basically repro of that if it's, so it comes into like there's a layer of automation, then there's a case manager, and if it looks like a good case, the case manager will case it and connect it to us in the engineering team. We'll do the repro and we'll look at it. Yep. This is a bug. And then we'll do the severity assessment. Like this is low, moderate, important, critical, whatever it may be. And depending on that, we basically only track from an MSRC perspective, important and critical, the low and moderates get tracked by the engineering teams, but we don't track them in quite the same way because they don't meet our, our, essentially our bug bar. Cause we get, you know, quite a few things that we, we do assess and, uh. Yeah, that's the whole process. And then the other half the time they spend, uh, you know, looking at, uh, research, security research, novel mitigations. When you think of things, I don't know if you've heard of things like ASLR and DEP and all that kind of stuff that came out for Windows on the, on the platform side. So that's kind of like my sister team. They deal more with the memory corruptions and all that. They were, you know, creating mitigations for many years and we're doing the same thing on the cloud side. So,

Chris Romeo 13:46

yeah, that's a couple of acronyms I hadn't thought about in a while. ASLR, address, space, layout, randomization, DEP. I can't remember what DEP is off the top of my head though. Like.

Dr. Jared Demott 13:57

Data execution prevention, I think.

Chris Romeo 14:00

the, that's the thing. That's where you use the processor to control, memory, basically.

Dr. Jared Demott 14:06

or non executable. Yeah.

Chris Romeo 14:08

Yeah. Okay. Yeah. So yeah, those are, those are flashback, uh, to my time, uh, also working at big tech. So, um, I've had a, a, a similar path to you, just in a different order. I did my big tech stint early in my career, which led to the, to the startup world. So, um. Let's catch us up on the, on just bug bounty program in general at Microsoft because I've certainly heard about Microsoft's bug bounty program and what was happening in the early days with MSRC, but this is going back 10, 15 years ago. Um, give us kind of a, give us an updated view of, of how this thing works and how researchers interact with it and, and just take us through that if you would.

Dr. Jared Demott 14:52

Yeah. I mean, it's a big program. Uh, not big in terms of what Microsoft does for security. So when you think about, you know, threat modeling and SDLC, and AppSec and red teaming and CICD and, you know, Microsoft spends a fortune on trying to make sure that software is secure as we know how, but there's that, you know, after the release, kind of after pen tests and all that on the SDLC, you would add bug bounty as kind of like a, Final operational, sort of like for anything else that's fallen through the cracks, you know, we want our partners all around the world to help us identify those things that they might notice that our security teams, for whatever reasons, didn't notify. And it's, but in terms of that, it's, it's the largest, uh, privately run bug bounty program. So we don't run it through like bug crowd or anything like that. We run it ourselves at Microsoft and we pay out around 15 million a year in bug bounties. You know, and some of the bug bounties are substantial. So there's people that make a living, you know, um, bug bounty hunting. And some do it, probably, probably many do it on the side, you know, in conjunction with whatever else they do. But there's definitely people that are full time bug bounty hunters. And, uh, it's a thing and it's really super great that they're, you know, willing to submit to our program. We're happy to... to pay him for that because a lot of times, obviously those are interesting things that somehow slip through the rest of, you know, the processes to secure software. So that's kind of like the quick overview of the pro of the program.

Chris Romeo 16:13

From a. Just to kind of look at Bug Bounty maybe a little bit closer as far as how it's, how it's working in the modern perspective. So, With Microsoft's program, how many, like, how many total researchers do you have that are interacting with the program? Like, how does it stack up against some of the other, I know there's some other big, big bug bounty efforts that are happening in our industry. Like, you know, do you have 1, 000 researchers, 10, 000 researchers? Like, what's the scale this thing?

Dr. Jared Demott 16:45

Oh, I was just going to look it up. Actually, we're giving a talk at GERCOM where I've got like more specifics on our program as far as like how many different countries and researchers are involved. I have all that data. I don't have it up. Let me, let me see if I can pull up my other laptop here in a second, but it's quite a way, it's quite a lot. Long story short, um, we're going to be talking about our program and specifically some of the different bugs that we see on the online services side, like one in particular that's a gnarly bug that everybody's dealing with, all the cloud security and every, every online thing is SSRF, we could maybe talk about that later too, but that's definitely one of the bugs that's kind of concerning. And so, um, Yeah, we'll be mentioning that. Let me see how many researchers are in our program. We have, of course, it won't come up at the moment. 345 researchers were awarded in the past year. 1, 200 eligible vulnerability reports. Largest, uh, was 200k. Over 17 different bounty programs. So, interesting thing about Microsoft's bug bounty program is it's not like, hey, just pentest anywhere you want in Microsoft, and then if you find something, send it to us. That's not exactly the scope, which the scope is basically kind of like what you're, you know, asked to do and legally allowed to do, almost like a pentest scope or something. Some companies do have that, kind of a more totally open bug bounty. Microsoft more runs specific programs, so you have a different one from Office and Windows and Azure and different pieces, you know, of, uh, of the thing so that it's kind of a way to channel the forces of researchers to the areas that we feel like are most critical. And they each kind of have their own little nuance as far as what type of submissions and you can. You can go read on that's what I would suggest actually to bug bounty hunters that are interested in Microsoft's program is just you know Google for like MSRC bug bounty or Microsoft bug bounty or something You'll find our page it shows off what the different programs are what the different payouts for different severities within different programs are and how to write a good report because that's kind of like one of the keys that we See that sometimes researchers may be goof is it, you know, a lot of them are maybe English isn't their first language or they're young and it's their first time submitting or whatever. And so they, they submit kind of a report that maybe isn't super high quality. And, and that's makes it harder to understand what they're trying to say and reproduce the bug and, and see if there's really something there that we want to, obviously we want to, are interested and want to take care of the bug. But, you know, it can be hard if the data is not presented well.

Chris Romeo 19:10

So what, um, you're certainly, you're describing a bug bounty program for a company that is highly mature when it comes to security and just a side note for, for our audience, um, now that you've got some perspective on what Jared's describing about where Microsoft is today, we did an episode with Steve Lipner years ago. Where we went all the way back to pre Trustworthy Computing memo. And he told us the story. He brought kind of the before story and then how they went through that. You're kind of describing 20 years in the future after that event. And they're 25 years now, 2003. Yeah. 20 years, I guess. Um, you're describing kind of the, the future event, but does, is bug bounty like when, what, where do you have to be on the maturity scale? for a bug bounty to be useful, like can a, can some company that's brand new and they got nothing from a security perspective, like when is the right time for bug bounty to play out in a company's life cycle?

Dr. Jared Demott 20:13

Yeah, that's a good question. I think it could vary a lot because I suppose there's no reason why a startup or any other company couldn't offer some sort of rudimentary bug bounty type, you know, program or something, but in general, I would agree, you know, if you haven't done the fundamental, especially if you have a, if you're an older company and you've got a lot of legacy and you haven't even done your first pen test or you're in some disastrous situation, I don't think starting with bug bounty would be the best place. I think more of a security engineering, you know, Roundtable or Walkthrough or, you know, those, those different kind of like one on one. There's actually a lot of like different like cyber one on one type stuff you can go and find on the internet and look up and kind of like what are the first moving principles for companies to consider, you know, NIST or whatever it is. And that would be a good place to start, I think, rather than starting with a bug bounty.

Robert Hurlbut 21:05

So you mentioned a moment ago SSRF and I'm sure there are some other vulnerabilities, but what are some maybe key ones or types for cloud and online services that you've seen? Okay.

Dr. Jared Demott 21:17

Yeah, so at Gercom, we're actually going to present our top five. So it'd be cross site. This is of important and critical, not considering like low and moderate, but of the ones that we get that meet the bar. Um, XSS, we get a fair number of those. Those don't tend to be critical. They tend to be more kind of in the moderate or important category and it's something that we definitely address. So it's not, even though it's like the highest by volume type of vulnerability we, we get, it's not necessarily the most like severe that we get, I would say. Um, and then CSRF or SSRF, it tends to be more serious because you know, rather than kind of, you know, running code that's kind of more based on a phishing attack or something like that, that you might get with an XSS. With SSRF, if there's some kind of interesting bug in a service that would allow an attacker to then touch some kind of back end of the cloud or something that they really shouldn't even know is there or ever be able to get to, that's definitely worse from a criticality standpoint. So that tends to be something that We're currently pushing hard to address. Um, a couple other things that come to mind in terms of vulns that we see a lot are, and it's funny how terms change over, over the years. I don't know, you probably heard AuthN and AuthC, but it used to just be called authentication and authorization, but now for some, we have to like shorten and acronym everything. So AuthN and AuthC type bugs, uh, which, you know, that might be anything from, you know, a missing, missing Authentication on an API endpoint to maybe a role that's over permission. If you have like guest, guest access in a tenant, for example, that shouldn't be able to do anything, but you can see everybody's files that they've shared, which is, you know, that something like that would be a nightmare, you know, things like that would be like AuthC. Um, And so it's, uh, as you said, Chris, you know, we're in a world that's now so much more mature, especially at a big mature company like Microsoft compared to 25 years ago or whatever, but we're also like every other organization on the planet, you know, rolling out platforms and, you know, the amount of technology and cloud and AI and everything else that's coming out as, as such a large and rapid scale that it, it kind of like, You know, and the importance of cyber. I mean, 25 years ago, yeah, there was like, you know, code red and whatever and all these different kinds of like blaster and all this stuff, but it was like, in some cases, it was like people screwing around or like it hadn't become such a serious, like, you know, it wasn't the Russians like trying to disable the Ukraine's power grid or whatever's going on now. It's the, the amped up capabilities of cyber now is so much more key in terms of what we see in the world. Lastly, there was a code injection and I could talk more about Different places where you might see that in the cloud and stuff, but those are our top five anyway.

Chris Romeo 23:57

Well, uh, can you take us through an example of an SSRF? Um, something, a real life example. I still feel like SSRF is the OWASP top 10 item that people just don't get. They don't understand it very well. Like, I mean, cross site scripting, it's been, and SQL injection, these things have been around for so long. It's hard to find somebody, even in AppSec, that can't give you a good... Three sentence definition of either one of those, but SSRF, I still feel like people are not 100 percent there. And so I'd love to walk through a live, an example of something that has been reported that you have triaged through and let's see what SSRF looks like in real life.

Dr. Jared Demott 24:39

Yeah. Yeah. And, uh, there's probably a lot of reasons for that. For example, as a bug class matures, like we kind of started off by talking about, you know, memory corruption and buffer overflows. And so there were generic mitigations like ASLR and DEP and many others that tried to address that bug class. Although an interesting thought, you know, that I have is that basically you can never really fully address any significant bug class. Like some people will try to tell you that you can, you know, just try to tell you that if you just. Like, XSS should never be there. It's a simple thing that developers should know about. They should just never put it in. And so just try harder. Like that doesn't, we've not seen that work in 25 years for buffer overflow. Like they're still there. So unfortunately, unfortunately the idea of like, can't we just do better? Isn't really a good strategy. Um, you know, because these things are significantly difficult. So I think we see that in every bug class to your point. And so even. Uh, with mature bug classes that are better understood, that have legitimate mitigations in place, like with XSS you have CSP, uh, the content security policy, which is a legitimate, you know, mitigation that can work effectively against those if it's rolled out correctly and implemented everywhere and tested and, you know, everything else. So, I think to your point about SSRFs, one of the reasons it's maybe not as well understood, and actually it's true for the other ones I mentioned as well, because it's kind of trickier. It's kind of hard. There's no like generic mitigation for that. Same thing with AuthC and AuthN. There's no like, hey, if you just put this framework in place, then you'll never goof. You know, you know, RBAC again, or something like that. It's not quite that simple. Um, code injection, I would say, we don't really get too much SQL injection in our stuff. So, like, that's, that one is one that we pretty much whittled out. But there's other newer types of different kind of code injection. Like, you think about different kinds of, like, uh, run operations that can happen in GitHub, automations, and there's different, like, takes on that that I think we're starting to see more of. So, um, just to kind of round out the conversation of like why, you know, certain things are harder to like address and understand and resolve. And then sort of to answer your specific question, I don't have an exact case in front of me that I would, you know, could share at this moment, but. You can imagine, you know, um, some type of product and you just go to like, you know, look, just go out and go to Azure and look at all the different products that are inside Azure. There's so many different products, right? And they all have an interface, they all have different capabilities to run some kind of service or code or VM or whatever it is. Connect to something, display something, database something, AI something, you name it, right? And there's just a, there's just a plethora of how this could all work. And so there's a lot of different technology and communications behind the hood to make that work. So sometimes a finder may identify what they think is an SSRF, and sometimes it's a little hard to tell if it is, so maybe it can like reach out to a third party site and they notice that by using burp or some other man in the middle proxy. They're like, oh I have this, you know, tool set up to catch that and I can identify this ping coming back out. Or maybe they try to, a lot of them will try to show that they can do some kind of port scanning if they can. Change a parameter, so there's some kind of request you send in and they realize there's a URI in there so they can change the URI to like, you know, instead of whatever it's supposed to, it's inherently set to something, they tweak it to be 127.0.0.1 or something, and they realize they can hit something internal to your network, which is really the problem that shouldn't, shouldn't be allowed to happen. Then they show that they can do a basic port scan and stuff. But I think, you know, one of the things that we're going to encourage researchers to do when we give our talk at GERCON is to try to take that a little bit further and realize there's probably more you can do with that. There's probably some sensitive backend server you could actually hit and force it to do something and, and leak a cookie or whatever it would be. And that would be a lot worse. So I think that part of it is why you said what you said. It hasn't been. For one thing, the test environment isn't as easily accessible. Like, when you think about trying to create, like, a buffer overflow attack against, like, an office file or something, well, you have the whole operating system, so you can, like, open it up in WinDebug and look at the crash, and you can tweak all the stuff and figure out what's happening in memory, and you can kind of create your exploit like that. But with this, it's like you don't have the private part of the cloud to, like, play around with, and it really don't want researchers necessarily playing around with that private part. So there's a, there's an element that the, like. Training and learning on the debugging environment is a little bit trickier to understand in some cases. And as is true of every CSP cloud service provider.

Chris Romeo 29:10

So what do you think, like, when you think about SSRF as a class, and this, this be a, a, a panel, a panel port, panel discussion portion of this. Um, but Jared, give us your thoughts on this first. Like, what's, what's the overall solution? Like, do we need framework input validation libraries that are processing any URLs and are just, just removing these, you know, when you think about the, the paved roads and guardrails of what we provide for developers, like what's your take on the solution to this, about this problem of SSRF specifically?

Dr. Jared Demott 29:52

Yeah, that's something that we're going to cover as well. And there's a bunch of different options. Unfortunately, it's not a. Totally, you know, just do this and everything's always perfect. But for example, you could either use whitelists or blacklists and allow lists or deny lists as far as like, you know, there, there should never be a, you know, internal address in those parameters. For example, that would negate the internal port scanning. Um, unfortunately, maybe there's a service that has to touch internal resources. So then you can kind of whitelist to make sure it only touches a certain internal resource. Uh, you can also insert different types of headers that you could check for. Um, and so, and just kind of have a standard. library, that you use to do any of these types of calls, that has some of that filtering already built in, and rather than forcing every product, you know, owner, to try to like roll their own crypto, so to speak, and this isn't crypto, but you know what I mean when I say that, kind of roll their own mitigations. They can use a centralized secure version of a Certain type of call. That's, that's definitely one of the better. And then we've seen that in other fields too. We see that with the same thing again. If you go back to the analogy of like memory corruption, like use a safe version of string copy or something like that, you know, kind of a similar concept of a, of a better safe API.

Chris Romeo 31:10

Yeah, sounds like a library would be the right place for some type of SSRF protection to proliferate. When we think about the impact of libraries, even just with CSRF, I know you mentioned CSRF initially, um, But I really attribute the, the downward, uh, reporting and finding of CSRF problems. It is really the frameworks that have brought us along. I mean, CSRF used to be in the OWASP top 10. It's not gone. It's number 13 on the priority list. So the people, I always look at it and I get, ah, it's gone. Well, no, wait, it's not gone. just, it just didn't make the top 10 anymore. But that's an example of where a framework improvement. caused a lowering in a particular bug class because people are just, developers are using the easier framework provided options which are preventing CSRF in the things that they build. I think SSRF, we could be, with a lot of these bug classes that you're talking about, eh, I mean, not, to your point, not AuthC or AuthN because they're so unique. Um, but with something like SSRF, it seems like a library framework component should be able to, Insulate us from this problem so that we see it disappear in the future.

Dr. Jared Demott 32:27

Yeah, and that's really the goal. That's why I said the folks that report to me have two pieces. They have the intake, you know, analysis of, you know, casing work and kind of our, you know, the cadence of what we do. And then we also have the proactive like variant hunting and fixed validation and mitigation work. And that's what we're trying to do is basically come up with different types of, whether it's a library or some other type of broader mitigation that can try to take care of these things at scale and the whole, rather than kind of whack a mole.

Robert Hurlbut 33:00

Is there any other, um, uh, case study examples for bug bounty that maybe, um, you can think of they're not in that group?

Dr. Jared Demott 33:10

Yeah, I mean, there's, there's definitely all kinds of different bugs, you know, that maybe didn't get mentioned in that top five, you know, kind of one off bugs that come on, come in through. You know, there could be like an Xbox online thing, or there could be, you know, some kind of mobile app thing, or there's all kinds of different, Microsoft's a huge company, that's one thing I realized when I started working here is the amount of products and, and services is staggering, you know, um, so yeah, there, there could be a lot of different types of, of bugs that come in, but those are definitely kind of, you know, the ones that are kind of more prevalent, I would say.

Robert Hurlbut 33:49

Do you ever see, um, there was an announcement I think yesterday about, uh, dropping Visual Studio for, for Mac, which I know that for a number of years they've been trying to get that to work, but, uh, any, any related to other systems like that? I know Mac, uh, a few times I've been at Microsoft, you see more and more people that were running Macs as opposed to, uh, Windows machines. Uh, anything like that in terms of Mac products?

Dr. Jared Demott 34:16

Oh, I can't comment on that. I wish I could run a Mac, but I'm on a Surface. I miss my Mac. I shouldn't say that. I'm not allowed to say that. I don't think, but, uh, no, it's a, it's a, it's a thing. I do know for sure that obviously like Office for Mac, you know, any kind of, if there's a fix for a bug that happens to be, especially if it's for the, What we would call more the on prem solution. That's kind of the term for like what software used to be. Now, basically most software has either like a cloud or a, or an on prem version of that same software. Then there would probably be a need to fix it for the client side version of Mac, for example. And there's probably other examples of that client side software that might run on a Mac, but in general, no, there's actually not a lot of Macs at Microsoft, uh, as far as running any key infrastructure, even people using them too much. I haven't noticed a lot of that, but. It, I'm sure there's some, somewhere, especially if it's like a some small company we just got, just got bought that, you know, was a security, a lot of security companies mags, so you probably see like pockets of little orgs maybe that do, but not as a corporate wide thrust for obvious reasons.

Chris Romeo 35:25

All right. Well, we were going to talk a little bit about more about history, but I feel like we've kind of, we've kind of covered some, some historical references throughout here. And so Robert, why don't you take us into the lightning round here with

Robert Hurlbut 35:37

Yeah, sure. So lightning round, uh, is just three questions that we ask, uh, have been asking, uh, different folks who've been joining us. So first one is, uh, what's your most controversial opinion on application security? And why do you hold that view?

Dr. Jared Demott 35:53

Well, I think it probably is a controversial view and kind of a hot take, I guess I would say in a sense, but, you know, you've been hearing preached to you for at least the 23 years I've been in cyber that there's this massive millions of jobs that go unfilled every year. And just an absolute, you know, dumpster fire of staff shortage and all this kind of stuff. And I actually don't think that's true. I know more smart folks than ever that are looking for work right now that are having a hard time finding work, especially certain types of, uh, of demographics and stuff. So I actually don't think there's a huge labor shortage in cyber right now. Personally, I know that's a little bit of a hot take, but, um, I think it's pretty balanced at the moment. There were companies actually laying off. Quite a few security companies cut staff over the last year. I don't think we see a lot of that. I think a lot of them are hiring again and kind of, you know, making sure that they're appropriately leveled based on macro economics and things like that. But I think, um, You know, I've, I've seen more, especially sort of like folks our age, you know, sort of what were senior level CISO type folks, out of work, looking for work, uh, you know, having a hard time finding a role that they used to be in that they can't find anymore. Maybe looking at having to take like a, you know, security engineer one type role because they can't find a comparable senior role anywhere. Like, so. I mean, I'm sure they'll find something. Most people I know that looked long enough and hard enough did find something because there are still opportunities out there, but I don't think there's any. Universities are cranking out cyber graduates like crazy. There's a lot of small universities that like realized that that was a thing and they got on the bandwagon of doing cyber. So they're, they're just cranking out all these graduates, which I think will largely fill the need. I think sometimes companies say there's a shortage because they have a misunderstanding. They're like, I want to, I want to hire 30 stock analysts for 6 an hour each and make them work midnight to, to 7am. And they can't figure out why they can't find staff. Well, that's a different problem. That's not, that's not an industry shortage problem. That's a hiring problem. Nobody's going to take that job. Right? So anyway, I don't know if you agree or disagree with my hot take, but that's, that's a little bit of a hot take I have on that, on that whole topic.

Chris Romeo 37:57

I mean, I'll comment on it a little bit just because, uh, I mean, seems like there's just, it's a, it's a hot news item. Like it's a, and I felt the same way for the last couple of years. It's, you know, 750, 000 people are, we're short 750, 000 people. And then the next article is 2. 2 million and the next one's 200, 000. Like, it's like, well, you guys, everybody just picking these numbers at random. Like what, what are you using to generate this number of, of jobs that are open? And, you know, I think to your point, like we've, we've. There's been a lot of improvements in the education process and attention and focus on cyber security degree programs. We're still not doing what we need to do from an AppSec perspective, but we are doing, we are bringing more people and I think it's really exciting, we're bringing a much more diverse range of people. into our industry, which we needed that. We all, we would all, we all knew that we knew that 10 years ago, and we're starting to see some of the fruits of the, that labor happening now. And I would say there are still pockets in our industry where, like, for example, uh, I play specifically in the AppSec space and that's where I've played for the last 10, 12, 15 years or whatever. AppSec. I don't know anybody who's unemployed at the moment. Nobody in my circles is unemployed because there's still the opportunity to leave one job on a Friday and go to the new job on a Monday, but this is a very specific segment of our market. This isn't the SOC, this isn't GRC, this isn't audit, this isn't, you know, any of those other pieces. This is a, a very unique part of the market and over time as we have more. More new grads and people that come in and enter this space. I think that'll change like the rest of the environment has. But, um, yeah, I'm with you. I'm not, we're not sure where these numbers are coming from. I'd love to see somebody do an actual study that proves their data that has data that they could, that we could then go trace the data and go, yeah. Okay. All right. Now we understand your number because we see where it's coming.

Robert Hurlbut 40:01

Yeah, agree. Uh, so second question is, uh, what would it say if you could display a single message on a billboard at the RSA or Black Hat conference?

Dr. Jared Demott 40:12

Um, yeah, I think for me, I mean, I love cyber. I've been in this field, you know, field my whole career and like everything about it I'm passionate about and love it. But on the other hand, I also, you know, just have a different take as far as what really matters in life as far as faith and family and all that. So if I had, if I had the chance to put one message there, I would probably say something like, Jesus is coming, are you ready? Or something like that, you know, just to try to like recenter people's minds on what really matters in life.

Robert Hurlbut 40:39

Excellent. And number three, what's your top book recommendation and why do you find it valuable? Okay.

Dr. Jared Demott 40:47

bit of a hot take on this too, which is, I don't really, even though I've written a book in this field, I really don't read books too much in this field anymore, and or advocate for the writing of them, because I feel like there's so much content that's so available, like, just jump on Pluralsight, or LinkedIn Learning, or whatever, like, any class you want to take, if you want to learn how to reverse engineer, there's a, there's an eight class track on Pluralsight you can watch over the next day and learn how to do that, like, It's so much more efficient and effective and impactful than reading a book. That's like, I have all the old books, all the old, you know, the, the white coding book that in the Chris Siegel's IDA Pro book, and I'll have all these books from like, you know, 20 years ago that just sit on my shelf and collect dust because they get out of, they're out of date in like a year. And so my hot take is that. Printed books are good, but probably not the best way to train, I guess. Unless you're just a reader and you love to read, I think there's still a place for them. I think there'll always be a place for them. I actually do like to read, but not so much tech books. I prefer to read, you know, other types of books, more leisure wise or kind of broader, you know, wisdom literature or something that's other than just like tech book. I think you can train faster with video content.

Chris Romeo 41:55

All right. Well, as we conclude our conversation here, Jared, what would, what would you want to leave our audience with? As far as a key takeaway, a call to action, perhaps a homework assignment that they're not going to do, but that's okay, I mean, they're at least not going to report back to you on it, but like, what would you leave our audience with?

Dr. Jared Demott 42:14

Yeah, I mean, I think the one thing that we've been preaching in our field forever is probably still an important message for you know business owners and companies and software makers to understand which is kind of the whole idea of I don't want to say shift left because that sounds like such a political statement as far as like that words been used so much But basically that like making it an integral part of your development from day one So all the things from proactive design, you know and mitigating risk, you know, they minimize costs, you know, from not having as many bugs and as many fires down the road. And you just, uh, it keeps you more compliant. And so all the things that are in the SDLC, like security training and code reviews and automating and CICD and, you know, security champions, like I feel like on any level, whether a company is small or large, you can find a way to do that in a way that makes sense to you. So just being thoughtful about it and at least having. Even if you're a tiny company with like five engineers, you can have one of the engineers spend half their time on making sure the design looks good and stuff like that. So that's kind of the, probably a message you've heard before and are tired of preaching. I think everybody in our field has mentioned that, you know, if you're in AppSec at all, but, um, I think it's still an important thing to help. business leaders understand that maybe in a more practical way to maybe gathering data around it or something because I think sometimes business leaders now they understand that they've heard about it they've heard about all the breaches and they're concerned about cyber but they still don't really know like what's the appropriate level investment like how do I get you know people to really be excited about this and make sure that it's kind of part of our core ethos you know around the product that we make.

Chris Romeo 43:51

Well, Jared, thanks for joining us on this episode, sharing your experiences, and... giving us an update on our view inside of Microsoft security posture. Now, I thought that was, that was fascinating just to, to get that update after hearing Steve Lipner tell us that story about how things started in the early days. It's, it's great to hear how Microsoft has matured. And I mean, I kind of knew that already, but you just provided us with some additional details that helped us along the way. So thanks for sharing that insight with us. And, uh, it's great to finally get you as a guest on the application security podcast.

Dr. Jared Demott 44:26

Yeah thanks it's been too long appreciate it.