Ep. 26 | Impacts of AI Social Engineering

From creating a profile of a target at rapid rates to implementing sophisticated deepfakes, AI is reshaping the effectiveness of today's social engineering attacks. But while it's being used for nefarious purposes, it's also being leveraged to combat these threats. Join BlackCloak Director of Product Innovation Swai Dhanoa for an in-depth look at these practices. 

If you're interested in learning more, you can request a demo with BlackCloak here, or visit the BlackCloak website. 

Transcript

SPEAKER_02 0:11

Welcome to Uncloaked, a podcast series brought to you by Black Cloak, the pioneer in digital executive protection and leader in personal cybersecurity. I'm Ayla Fitzpatrick, and today we're talking about AI social engineering and how it's changed the way threat actors target victims. Joining me for this discussion is Sway Danoa. He's our director of product innovation here at Black Cloak. Thanks so much for being with us today.

SPEAKER_01 0:36

Happy to be here.

SPEAKER_02 0:37

Okay, so let's start off. Social engineering has kind of been around forever. And when we talk about the tactics from a cybersecurity perspective, it's often centered on unsolicited emails or messages trying to get users to click on malicious links, provide credentials, or even send payments. The Nigerian Prince example is one of the most common that people kind of bring up when we're talking about this. And you know, we kind of laugh about the bad grammar and the nature of those emails that come through. But how has AI kind of changed the effectiveness of some of these attacks?

SPEAKER_01 1:10

Yeah, so I think like originally what you'd see with some of these like targeted attacks, right, is like someone was working on a multitude of things, right? So they needed some technology to send the attacks out. They needed a little bit of cultural intuition, right? Of like what really resonates here and there, and then a bunch of trial and error. So I think like AI has probably changed like three things, right? One I think is like the speed at which attackers can start deploying attacks, right? So like you don't have to sit there and go through like 20 test cases of like, does this attack work? Does it not? Um, the other, I think, is more on the personalization end, right? So like the way I would expect someone to speak um from you know the United States versus from India, like having lived in both places, is very different. And it's really easy for these models to be able to localize and personalize. And then on the speech side, we actually went from this thing where like, you know, you would kind of expect like a little bit of disfluency in speech when you talk to a person. And then the models got like a little too perfect, right? You'd kind of like see these messages come in, you're like, whoa, that feels a little robotic. And now it's actually crossed over to the other side where you can essentially throw any data at them and they'll figure out how to make it sound relatable. They can play different personas, they can pretend to be an executive, they can pretend to be a friend, they can pretend to be a colleague from uh a previous job, right? And you really have like a harder time discerning, and that's just kind of more like the natural evolution of what the space is demanded from like model companies where you know we're using them for good things, we're using them for products. Go to Amazon now, you see a summary, and you want those to feel a little more natural, but the same tools have kind of gotten the bad actors, so they kind of get the same like efficiency, personalization, um, and polish on them in any direction they really want to go.

SPEAKER_02 2:55

And when it comes to building a profile of a target, like you mentioned, how is AI being leveraged to kind of increase the efficiency of that process? You kind of touched on that a little bit, but I'm sure it's really helpful in those things that you said before were extremely robotic and now they've got a little bit of help.

SPEAKER_01 3:12

Yeah, I mean, I think like so context is king in the LLM space, right? So, like, you know, using some of these models, like this technology, the problem we're facing isn't really new anymore. It's more of like how can you apply it at a larger scale? So collection and aggregation, right? There's been ways to aggregate and collect this data. You can go to a data broker and purchase information about someone's family, about where they worked, where they used to live. Um, so that's always kind of been there. What we've really seen, I think, is like the speed at which you can do other types of collection, right? Like, can you deploy a social media bot to go add your kid on Instagram and know, like, okay, they were vacationing at this place this weekend based on the post we saw. And you can use that into context. Um, from like a practical perspective, I think there's also like, or I guess more of like a tactical perspective. We use AI models to be able to do more work, right? Like we want to hand off things to an agent and say, hey, go do this for me. Um, that's the same thing that bad actors have. They can go and deploy multitude of experiments, multiple collectors. And once all that data comes back, right, like you would imagine it's like a little bit overwhelming to have a bunch of people's information that you're trying to synthesize. Synthesizations also just become like a complete snap of the finger type of task. Um, so I think like you see kind of the same like technology, same trends that you'd like apply for like someone like us in the commercial space, right? Where we're trying to build with AI, we're trying to build AI systems. The bad actors are the same thing, it's just kind of like you know, the mindset and how you apply them. Um so I think definitely like the the collections become a lot more efficient, the synthesis has become a lot more efficient, and then like as models have improved, what you're putting into the context window, right? So like you send like some general instructions of like, hey, your job is to do this, fit this persona. You can change it on the fly now by throwing in as much context as you want. And it can handle it, it can process it, it knows how to exactly start personalizing. Um, so it's really a kind of end-to-end. They've gotten like a lot more efficiency, a lot more speed, the breadth and depth of the attacks have gotten a lot more robust. Um, and they tend to like, you know, you kind of start seeing these things emerge when you see papers coming from some of the research labs saying, like, hey, we stopped this cyber attack, we stopped these bad actors. Um, they're referencing things that like we kind of see on the research community, right? Like we see as like people trying to build models where they're doing these like clever, you know, reinforcement cycles of like, okay, go send this attack out, see how it works, correct your prompt, correct your tone, update anything you need to update, update your context window, collect more data, and then attack again. Um, so sophistication, I think, from bad actors, you know, that's it's really gotten um a lot more robust, and I think it's made uh made privacy and kind of your digital safety a lot more uh a lot more critical.

SPEAKER_02 5:56

And I think part of the sophistication comes with also like sometimes who is that attack coming from, right? Before we, you know, you talk about the Nigerian prince example, but now with AI and creating deepfakes and you know, mimicking the voice or um the video of maybe a loved one or a coworker or something like that to kind of build that trust before I guess the ask kind of comes. Um, how is the precision of those deepfakes and the way they're engineered really upping the ante for these threat actors?

SPEAKER_01 6:32

Yeah, I think deep fakes has really evolved, right? Like maybe even in the last year. So if you go back like kind of pre-Chad GPT era, right, you had a lot of deep fakes were coming on these like neural architectures where you would have these like really big training jobs and you'd have to really understand the math and what's happening under the hood. And it was like a specialty skill, right? And like you would see platforms that made it a little easier to do, do like D pick attacks and software that you could probably acquire to make it a little less like, you know, you really have to know your stuff to be able to use this. Now I think it's become like both a commodity, right? Where like you can throw in, like, you know, you have funny examples of this, right? Where I'm like, oh, I'm gonna throw like a photo of my niece in here and like maybe have her like you know wearing like a commander's outfit for the game. But you can kind of put people in compromising positions. So I think the the transformer architecture, which has made like ChatGPT, Claude, all these other models like hyper effective and easier to engage with, um, they've kind of done the same thing for deepfakes. Like now you don't have to go do this like long, thought out, like, let me go collect 30 images of someone. You can just take their profile picture and be like, okay, let me show them doing something malicious, right? Um, and then the modalities have changed. So, like, you know, deepfakes was like kind of like synonymous with uh images, right? Like, oh, it's a photo of someone that's not actually them. Is that real or is it not? Now you've kind of gotten into this like almost like a holistic deepfake thing, right? Where you can do someone's face, their image, you can generate audio that sounds just like them, baking the same disluencies, right? Like I say um a lot. And someone might say, oh a lot. And it's like, oh, okay, use this for this person, use this for this person. Um, and then video as well, right? Like that's just blown up over the past year, and it is becoming a point where like, you know, you'd kind of see these like AI videos, and you're like, well, it's obviously, you know, like someone's walking and all of a sudden they just do a front flip, and you're like, that's not real. Um but now even there's times where I'm like looking at something, I'm like, wait, what just happened? I was like, oh my God, wait, this person did that? I was like, oh wait, this has to be fake.

SPEAKER_02 8:31

Um and you're an expert in it, so if that tricks you even for a second, that just shows how good it is.

SPEAKER_01 8:36

Yeah, and you know, we we kind of I think we're are a little more uh trust like trusting, right? Like we've kind of spent so much time in like the social media space where we're kind of used to seeing something, we're like, oh okay, like you know, you trust on the source. Now you can even like start faking sources, right? So like you can build a website that looks hyper realistic. Um you can build social media profiles that I mean there's influencers now that are like fully AI influencers, and good luck trying to figure out if they're AI or if they're a real person. Um, so I think that's been like another in challenge and thinking like kind of the like almost like the deep fake ecosystem of sorts has evolved and it's a super lucrative space, right? You see like some of the play people that are building this for like maybe application use cases, they're doing well, right? There's a huge demand there, and there is an incentive for them to make these very hard to detect, right? Is it like are you talking to someone on the phone and it's a support agent or is it AI? They have an incentive to make that technology better, but the risk comes with like how that technology is actually applied. Um I think that's that's kind of like uh I think some of the evolution there on like deepfakes.

SPEAKER_02 9:43

Yeah. Um, we talk a lot about how threat actors use AI to you know trick us or do whatever malicious things they want to do. But then on the good guy side of things, we can also use AI to combat those threats. So could you just touch on that a little bit?

SPEAKER_01 9:59

Yeah, I mean, that's a I think you know, one of the most obvious examples, like we kind of talked about the context engineering space, right? So the ability to go and find any information about anyone, pay a couple of dollars, and now you know everything, right? Where they've lived, where they've worked, family names, all this like crazy stuff that's just available to you. So I think one of the applications on like you know, combating AI generated risk, right, is like using AI to actually go do like a digital footprint cleanup. Um and that's again a technology that's like advanced a ton uh in maybe the last eight months that's made it a lot more realistic, a lot more uh robust, a lot more deterministic to be successful using AI technology to do kind of a digital footprint cleanup. Um on the deepfake side, I think you know the reality is like there's not a great solution yet, right? This technology is starting to go pretty early. There was a lot of work done during kind of the pre-LLM era, pre-transformer model era, right? Where there were, you know, you could essentially poison uh training sets. So you throw like an image out there, it'll completely throw like a uh GN model, though GAN like completely off oops. Um, but now we've had this like new and really fast evolving space with transformer models uh and some of the bigger research labs, where it is a little bit harder to keep up. Um, but you start seeing things like you know, the same kind of trends emerge, right? Like that have always been true with machine learning and data science and AI, where it is a lot of pattern recognition. So you start observing like, okay, there is like a pixel level progression that's very specific to AI. And now we're starting to see more deep fake stuff be reported, AI generated content reported, um, AI generated attacks reported. So we'll start having a bigger training set to be able to say, like, okay, we know what like the pattern is for an AI generated attack because we know how these models act, we know what these models say. Um so I think it's kind of like the same cat and mouse game that's always been true in cyber. Like it's kind of the same thing we're seeing here. It's just maybe you know a little less in like the typical way you'd see it. It's kind of more of like someone in a lab is trying to like crack this problem, we're trying to crack this problem, and hopefully we can find like the right pieces of the solution to be able to provide kind of the total coverage. Um, but again, as it evolves, new problems arise and problems to solve.

SPEAKER_02 12:15

And finally, what are some best practices that you would recommend to you know some of our clients or the average person out there to avoid falling victim to some of these scams that could be either AI generated or using AI to come off more convincing?

SPEAKER_01 12:33

Yeah, and I think the first thing you should keep in mind, right? And this has you know been true with the attacks historically, right? Sense of urgency. It's the biggest thing. Like, hey, we need you to send money right now. And I don't know, that's never been true in my career. I've never gotten a call from any executive saying, like, hey, we need$100,000 in my bank account to pay off a vendor that you know we're behind on some bill. So like they create that sense of urgency. Um, so if you kind of see that signal, right, and you see things that are maybe like if you wait a little bit and they respond again and again, like you kind of start feeling like they're trying to try to apply pressure to you. Um I think the other thing is like having kind of systems right in place of like, okay, like I'm not sure if this message is real, what should I do about it, right? Like have a way to kind of be able to do that communication and say, hey, is this legit? Um, if you see a message come in, right, think about it a little hard, maybe Google it, right? See, like, do you have like any um like do you see this as like a common attack that's occurring? The whole toll road thing was a nightmare, right? I mean, I was like apparently delinquent with every toll road for every 50 states, and I was like, oh, I don't remember going to Alaska. Like, what what's happening? Those messages started looking real, the website started looking real. I started questioning myself, I'm like, I go to Alaska.

SPEAKER_02 13:51

Wait, was I was I there? Like, and I just don't remember it. Yeah.

SPEAKER_01 13:54

So I was like, this has got to be a thing, right? And it's like, okay, well, this is a thing. And I think, you know, the last thing is uh, well, I guess there's two things, right? One is also having like experts to be able to work with, right? Know who knows what's going on, have people that you can call and ask to say, like, hey, I've seen this. Like, do you think this is real? What should I do about it? Um, so always have a little bit of suspicion if something feels weird. Trust your instincts. Um and then the last thing is like cleaning up your digital hygiene, right? As you think about like what you're posting on social media, what you're leaving out there, um, how you're setting up your Instagram or your LinkedIn, etc., your family's Instagrams and LinkedIn. Um, be diligent about like what you're posting, right? Like if you say, like, oh, I'm a I frequent this coffee shop every day, I mean, they could get me with a Starbucks, you know, attacks. I'm like, oh well, that actually does sound like I do think I have a free coffee at Starbucks now. Um so just kind of manage your uh digital footprint, manage the breadcrumbs you leave out there, and just be a little more conscious about you know, not putting too much information out there that'll just eventually get fed into a model to make those attacks feel hyper-realistic and personalized.

SPEAKER_02 15:02

Absolutely. Some great tips there, some great insights. I'm sure these will keep evolving as AI continues to evolve. And uh, we just really appreciate your insights and your thoughts on this topic sway.

SPEAKER_01 15:14

Yeah, thank you. Thank you for having me.

SPEAKER_00 15:16

You can listen to all episodes of Uncloaked at blackcloak.io slash podcasts or on your platform of choice. And if you're interested in becoming a member or want to learn more about how to protect your digital life, visit us at blackcloak.io. Thank you for tuning in, and we'll see you next time on Uncloaked.