Episode 19: AI Data Poisoning: How Bad Actors Corrupt Machine Learning Systems for Under $60

Clint Marsden breaks down a critical cybersecurity report from intelligence agencies including the CSA, NSA, and FBI about the growing threat of AI data poisoning. Learn how malicious actors can hijack AI systems for as little as $60, turning machine learning models against their intended purpose by corrupting training data.

Clint explains the technical concept of data poisoning in accessible terms, comparing it to teaching a child the wrong labels for objects. He walks through the six-stage framework where AI systems become vulnerable, from initial design to production deployment, and covers the ten security recommendations intelligence agencies are now promoting to defend against these attacks.

The episode explores real-world examples of AI systems gone wrong, from shopping bots buying drugs on the dark web to coordinated attacks by online communities. You'll discover practical mitigation strategies including cryptographic verification, secure data storage, anomaly detection, and the importance of "human in the loop" safeguards.

Whether you're a cybersecurity professional, AI developer, or simply curious about emerging digital threats, this episode provides essential insights into protecting AI systems from manipulation and understanding why data integrity has become a national security concern.

Key Topics Covered:

• Split view poisoning and expired domain attacks
• Data sanitization and anomaly detection techniques
• Zero trust principles for AI infrastructure
• The role of adversarial machine learning in cybersecurity
• Why defenders must learn AI as quickly as attackers
  
  The PDF from CISA etc al: https://www.ic3.gov/CSA/2025/250522.pdf

Transcript

1:22

Hey everyone. Welcome back to TLP. I'm your host, Clint Marsden, and today we're gonna talk about something that's getting a bit more airtime and dunno if it's a bit more scary or if it's just becoming kind of the norm. But if you picture this, AI systems are essentially everywhere now, right? They're in our phone, they're in our car security systems, protecting critical infrastructure. They're everywhere. All of these systems are learning AI from some kind of data. The AI has not come up with this on its own. It has like all learning has been given this as a, a data set. Of course, if some evil doer, some bond villain. Has gone and messed with that data. If they've modified it, put things in there that are inaccurate. Then basically this controls how the AI thinks. I'm sure you remember a few years ago when AI or automated robots online started to get some airplay. I think there was a researcher who created an AI bot to go and buy things. Online and very quickly it started to just go and buy drugs on the dark web. It was kind of a, kind of an an advanced shopping bot and it shows how quickly things can turn without the correct, fine, fine tuning and without the correct guardrails. And there have been other instances of early AI systems where when they went onto the internet. The likes of people from four chan or the likes of people from Reddit have decided for one reason or another that they want to try and take this down and just fill it with junk and to fill the training model with junk data. Maybe they're anti ai, maybe it's just for the lulls. Who knows? But that is what happened. And most recently we've seen a report that's been created in collaboration with a lot of intelligence agencies. I believe it's been spearheaded by CSA as well as the NSA, the F-B-I-G-C-H-Q, Australia's a SD. These are all intelligence agencies, including other intelligence agencies from around the world. Was also cool to see some representation from the New Zealand Cybersecurity Center. They're now reporting, this is becoming a lot more prevalent, the hijacking. Of AI systems also known as poisoning, is, is becoming the new norm. It is not expensive to perform. We are talking the cost of, say, a laptop. In some cases we're looking at, so if a cost of a laptop might be two and a half thousand dollars, then there are also cases of this being done for under$60 or under a hundred dollars. As little as a little, as little as 60. The training model of of AI is like learning as a child, and it's like teaching a child how to recognize things. What we are doing while we're learning to learn, we look at, sometimes we look at pictures of things, and if we're teaching a child, uh, or if we're learning another language, you might receive an image and you show them a picture of a cat or a picture of a dog, and you showed them this and you say, this is a dog. And over time through repetition. That cognitive enhancement, they learn the difference. What we're talking about with this AI poisoning is someone kind of coming in and, and flipping that around. The label being this is a cat when it's actually a dog, and vice versa. The technical term here is, is actually called data poisoning. What we're really doing is, well, we're making a rod for our own back by by doing this, but we're just lying to, we're lying to robots. Because the AI is in this impressionable phase, it doesn't have any idea that it's been lied to. It's, it's developing, it's learning these capabilities. It is designed to learn what it has been taught. This goes into the memory, and then unfortunately over time these issues compound because it is retaining this incorrect information that it has been taught. And then it's making decisions and delivering output based on that poisoned information. The report that CSA have released has got six stages where things can go wrong with, with this data poisoning. And in a, in a brief summary, it starts off where, when organizations are planning the architecture and the design of their AI systems. And then it finishes at when the system is running in production and it's making real decisions and it's, it's informing real decisions made by humans in the loop and humans in the loop, if you don't know, is just a step where you can run an AI automation and at some point it stops and waits for someone to a human to physically verify that what it is proposing to do is okay. So this might be useful in a, the con. I mean, I don't have any context to give you for something in a nuclear power plant of anything, anything like that. But what I'm looking at at the moment is content creation using ai and before it goes out, I wanna make sure that it matches the tone and structure and has information that I would publish if I was writing it. And if I need to, I will then get in there and adjust it. Accordingly and make sure that it is delivering the message that I want. Because the last thing that I'm trying to produce is what people are calling AI slop. I don't want to produce content that's AI slop because it goes against everything that I'm trying to do. I'm trying to produce content that is helpful for the reader or is helpful for the listener. So that's what Human in the Loop does that gives you that, that kind of last chance before it goes live. To verify and make any changes and then you can publish it and to get around these perceived problems. And I say perceived because I don't have any physical examples of them right now, to give you, not to diminish the seriousness of, of what they are. The document has provided us with some frameworks that we can use. The first thing is we need to know where our data's coming from. So we're going back to some real fundamentals here. Like if you download software on the internet, you see there's an option to download the cryptographic hash as well. Just to go into a little bit of a segue, say you're running Microsoft have retired this now, but say you're running a red forest, you're running hardened domain controllers in a tiered environment in a T zero environment, which might contain your domain controllers and other T zero systems. Like exchange servers. Remember those before we went to office 3, 6, 5. Before you install any software in this environment, once you've downloaded the installer package, or even if you've already got the installer package, you need to make sure that it passes the hashing algorithm to ensure that it hasn't been modified in transit or has been modified repacked, loaded with some malware, loaded with something dodgy in the time that it's been sitting on your systems. We won't get into, but how do you make sure that the cryptographic hash that's being provided is correct?'cause that just creates a bit of a loop and I dunno, honestly, dunno how to solve that one. So what they're suggesting is use a cryptographic hash to verify the data and you can still hash the contents of documents. It's not only reserved for binary files, they also talk about using blockchain tracking. And if it fails, these checks don't use it. This isn't so much of an issue. If the document is not a huge tome, that will take a long time to view. But I recently purchased a book on AI and red teaming, and specifically red teaming the AI platform. The book is a thousand pages. It's gonna take me a long time to get through that book. It's gonna take me months. That is where the, using a cryptographic hash to verify that the contents has not been modified before, using it as training data is gonna be quite helpful. Second, they're talking again about maintaining the data integrity during storage and transport, and this is, this is where I kind of disagree in how they've done this. It's, this is kind of one point. Yes. If we're downloading the content of the information. Sure. We need to verify that the contents is what we expect it to be, and then if it's being stored well before, it's then used in the training data. Yes, hash it. Again, similar to forensic practices where we might be running. Every time the data is transferred or moved or used, you might decide to run a hash to make sure that it is maintaining that integrity. So moving on to number three. Again, same kind of concept here. Employing digital signatures to authenticate trusted data revisions, using a CA to ensure that things are not being modified in transit. And I'm just gonna move on to number four, which is leverage trusted infrastructure. Remember those words? Zero trust. If you're still trying to get your zero trust program up and running, I understand it's a bit of a beast. The point here is that they're talking about providing a secure enclave for storing your data. Again, I feel like we are still on the same point. Uh, I can't believe that we have four points of this recommendation that are essentially bundled into the one concept. Keeping it in a secure location and hashing it and then verifying it, and then making sure that it can't be modified in in transit. This could be some real strong recommendations. This could be from areas of the government that we might not normally hear from, where they're operating in such a secure environment that these are the measures that they have to take. I kind of feel that these are some super secure areas. Or some super secure recommendations that they would follow, maybe in top secret networks. I've never worked in a top secret network, so I'm only guessing, but listening or reading the amount of rigor that's needed for these types of scenarios, it seems, it seems pretty plausible. Then of course, we're talking about user access control, role-based access control, RAC. Making sure that only the right people are allowed to access these things. That comes down to principle of least privilege, something that I spoke about very, very early on in the, one of the, I think the first four podcast episodes when I was doing that NIST series. Making sure that only the people who are going to be working with the system, the, I guess the digital librarians are the ones that have the access. You don't wanna be training the AI again, once you do it, once you get this first hurdle out of the way. So you wanna make sure that you're training it with the right stuff. Then of course, encryption, that's a bit of a no brainer these days. It's, it's so easy to ensure that things are encrypted and that covers not just data at rest, but data in transit. So data at rest, sitting on the drive, sitting in a database, data in transit, going across a network, going from cloud hosted platform where your AI might be stored, even if it's private cloud. Or even if it's just across your network, these may not be problems that you need to deal with right now, but as the company expands, these are things that you will need to ensure that they're covered. As this system becomes a bit more of a mission critical system for you, it's something that you'll need to have anyway. And the point is to do this now so that you're not dealing with technical debt later. I've seen technical debt so many times and over so many years at all these different organizations. And I think it's easy to try and avoid doing things right initially from a place of we need to get this out. We're behind on the project. We want to implement it. It needs to happen. Now there might be some downwards pressure from above to make things happen. Maybe there are other reasons. Competitive advantage. The sooner we can get this out, the sooner we can take more market share. Great. Let's do that, but let's do it securely. Let's follow these basic fundamental principles right now. Hopefully it's why you're listening to this particular episode because you wanna know what do we need to do now to protect us for the future? Again, number seven, talking about storing data securely kind of done to death by now. Talking about the use of cryptographic modules and encryption. It's quite interesting actually, because this is a TLP clear. So you've got TLP, clear green, amber, and red for something that's A TLP Clear. It seems to have such a deep resonance with things that may not be in a TLP clear environment or maybe in documents that you might generally not have access to. So it's pretty cool that the government is sharing these types of things with us as private citizens. Moving on to number eight. Leveraging privacy, preserving techniques, and then doing things that will make your privacy officer very happy and very proud of you. Doing things like data masking that is removing sensitive data with and replacing it inauthentic, but realistic information basically means when you look at the data set, it still looks normal, but instead of Clint must, and in the record, it's. Tom Jones. Of course, it needs to be names that do not exist in your customer base to make it effective. It's not enough to just move the rows around and move someone's name from row 2000 to row 500. It needs to be completely unique. Number nine is deleting data securely, working with physical hard drives, and then it's time to dispose of them because they have failed. This is important even if the drive appears to have failed. Or if you are at end of life three years down the line, four years down the line, and you decide that it's time to upgrade these systems, obviously the best recommendation is to physically destroy them. Putting them through a shredder is a common favorite method of doing that, engaging a third party company. I used to drill holes in the platters of hard drives before we threw them out, and there's a lot of discussion on this. And Rob Lee from Sands. Talks about it and the, I don't know if it's a myth, but there's the knowledge of an electron microscope being used to recover data. And this was back in the days when hard drives were only 20 gig. Uh, now I've heard that the density of data is so large that even an electron microscope is not gonna help anyone trying. I guess it depends on your threat model. Uh, if you are worried about people using an electron microscope to recover data from the drives. Melt them down, shred them, do whatever you need to do. But physical destruction is probably the best. Talking about data wiping methods, the DOD three pass, or the eight pass, or the 16 pass type wipes, from what I have read in the research, not necessary, just a single proper. Once a once it is wiped, it is wiped. Not a quick format. Quick format just marks everything available for deletion. That's why it's quick. The data is still there. It's not physically there. Doing a zero of the drive flips all the bits to zeros, and in the case that it's already a zero, I believe that it flips it to a one and then back to a zero to ensure that it's fully gone. And then lastly, number number 10 or step 10 here is to conduct ongoing data risk assessments. So using frameworks like the NIST SP 800 dash three R two haven't heard of the three R two before must be a a latest revision. The idea here is to evaluate the security landscape, identify risks, and then prioritize the actions that are relevant to your organization to minimize breaches or security incidents. They then move into some different types of risks and these risks. The first one that they're talking about is a curated web scale data set. They make reference to a couple of curated AI data sets, and the risk here is called split view poisoning. They're saying that the risk arises because these data sets often contain data that is hosted on domains. They may have expired or no longer actively maintained by their original owners. In such cases, anyone who purchases these expired domains gains control over the content hosted on them. And this situation creates an opportunity for malicious actors to modify or replace the data that the curated list points to potentially introducing inaccurate or misleading information into the data set. And that is a great example that's represented by. When you see that a domain has expired and goes into the 30 day grace period, someone comes along or someone might be waiting, they may have put their name down on the waiting list to purchase that domain, and they can use that domain's reputation and that provenance to do some content injection later on because people are still referencing that domain. It's the dangers of the internet that have not gone away, but we've just become accustomed to and kind of forgotten over time that that's, that's a risk that. That is present. Then doing things like web scraping that presents its own set of risks because collecting data on mass like that, there's no real quick and easy way to verify that the data you've captured is accurate. And the main point that they're trying to drive home here is that it's not particularly sophisticated. This is, these are some basic attacks and whether they are done by. Advanced actors who still favor using really simple stuff, using really simple techniques to achieve their objectives because why spend more time or why spend more money if you have to? These are things that we need to pay attention to. Coming up, we have the adversarial machine learning threats. This is where we're talking about the four chan groups or the Reddit groups where people are deliberately trying to deceive, manipulate, or disrupt the AI system, and the malicious actors are employing data poisoning to try and corrupt the learning process, which is compromising the integrity of the training data set. There are a few mitigation strategies that you can use to try and avoid this from happening. There are some algorithms that can do some anomaly detection, so if they start to detect that something's not quite right, it could be a pirate copy. There's also data sanitization, which is sanitizing the training data by looking at using techniques such as data filtering and normalization, and that is looking for outliers in the data and it's trying to grab those high quality results. And sanitization needs to happen on a regular basis. And that is prior to and after each training session, I guess we could call it, or when fine tuning is occurring or any other process that might change the model parameters. So the model is the, the AI engine or the AI brain itself. So going through this entire document, it is a, a very interesting read. I would say that it is probably more aligned to data scientists and AI information architects. It's not really suitable for the standard consumer. And the reason I say that is because the standard consumer, like pretty much myself, I am making some AI apps and doing some, some development there. I'm still using off the shelf tools. I did use an alama build. To build an AI platform at home. I kept it really, really simple and I haven't looked at that for about 12 months. And the other tools that I'm using are Clawed and Jet GPT as my, as the model. I haven't used Deep Seek, and so a lot of the recommendations in this report are not really relevant to the way that I'm using ai, even though I've been building custom GPTs and chat bots. I'm still referencing a model that is a commercial model that's available, and I haven't started to fine tune those models either. And so I'm relying on the fact that I'm paying for those services, that they will actually be performing the risk mitigation tactics to prevent my models or my ability to use these models from being poisoned or providing junk data. And it's still the case that I do. Get a lot of junk responses, and what that comes down to is my prompts not being specific enough, and it's a little bit more nuanced than what should be talking on a forensics podcast. I suppose what needs to happen is if you're finding that the results that you're getting have been poisoned. That is probably part of a bigger problem if you're, especially if you're running an internal LLM large language model in general. If you're still using commercial LLMs like Claude or Google Gemini or chat T or one of the others, try and go back and do some better prompt engineering. Be a bit more specific with how you were doing it. That is all I've got today from the summary of the AI data security paper. Best practices for securing data used to train and operate AI systems. And as you've probably seen on LinkedIn, I'm starting to make a bit more of a, a move towards talking about AI and while still having a very strong digital forensics incident response focus, I can see that. The use cases for AI will be huge to help us process data. I also feel that it's quite necessary for us to maintain an AI skillset because threat actors are utilizing AI to attack us more and more. The ability for AI to be used to weaponize attacks and to cause more damage and require more digital forensics and incident response. Is going to be greater than ever before. And despite using some automated or some best practices for triage techniques and using some great digital forensics acquisition tools, the processing of the mountains of data that we are going to need to do means we must get better at learning how to use AI apps and the game of cat and mouse between red and blue teams or threat actors and defenders. He's not going away. And I would encourage everyone to get into AI as much as you can and start learning because it is not going away. And the only thing we can do to try and keep defending against the bad guys is to learn as quickly as they are learning to attack us. So I hope this was informative for you today, and I'll see you in the next episode. Bye for now.