Vibecoding Meets Regulation

Last week we explored vibecoding and how AI is making it possible to build software faster and cheaper than ever before.

This week we look at what happens after you've built it.

Many businesses assume AI regulation only applies to large technology companies selling AI products but the reality is more complicated.

Under the EU AI Act, an organisation can become a provider of an AI system even when that system is built for internal use and never sold to anyone.

That means the AI tool your business created could carry obligations your business didn't expect.

In this episode of Glitchd, we break down the difference between providers and deployers, explains how organisations can unintentionally become providers, and explore the situations where a simple internal AI tool can trigger significant regulatory responsibilities.

From high-risk AI systems and provider obligations to governance, accountability and compliance, this episode focuses on one of the most misunderstood aspects of the EU AI Act.

https://skspsl.com/

Transcript

SPEAKER_00 0:00

I'm Sharon, AI governance professional and founder of SKS Professional Services. I help businesses turn AI from theory to reality responsibly and profitably. This is Glitched, the podcast for businesses navigating AI with speed, direction, and with purpose. A new episode drops every Tuesday and welcome to season two, episode two. I'm really, really excited, and if you haven't yet listened to season one, I really encourage you to go ahead and listen. We had some really amazing episodes, and last week we talked about vibe coding. I explained that essentially vibe coding is the ability to build software simply using plain language, describing what you want, what you're trying to achieve, and letting AI write it for you. This is changing the game. If you don't know about vibe coding, welcome. We are talking about vibe coding today, but today we're talking about it from a regulatory standpoint, the impact regulatory requirements and obligations have on businesses. Last week we talked a bit more about the foundation, the capabilities, and the opportunities, so do not miss that. Right, let's go and get into it. So the EU AI Act, which I'll refer to from now as the Act, it has different obligations for providers and deployers, and this is really key where a business wants to vibecode or incorporate some form of vibe coding into its systems. Okay, so it might seem really obvious, right? Provider, deployer, it sounds like you know it defines itself essentially, right? But actually the application can be a little bit complicated. So under Article 3 of the Act, a provider is any organization that develops an AI system or has one developed on its behalf and then places it on the market or puts it into service under its own name. So for most business owners or you know leadership hearing this, they might hear this and think, well, this sounds like your Google or your Microsoft. Actually, that's not quite correct. So where it's put into service, it doesn't mean that it has to be sold, it doesn't mean that it has to be licensed externally. Actually, this means you are simply deploying a tool and that can be purely internally, it doesn't have to be used externally, and where it is used externally, it doesn't have to be paid for. Okay, so this covers internal use and where anything is built or developed and it's given away freely. You could then be caught by that definition of a provider. Now, not every AI system carries the same weight, so it's worth me just pointing out that the act is risk-based. So there are four different categories of risk, but for today we are talking about high risk categories, and these include employment decisions, HR screening, credit assessment, access to education, and a number of other areas that touch on people's rights and opportunities. So if a business is vibe coding, they need to be aware of these risk categories because obviously, with high risk, um, there have to be more processes put in place to reduce that risk as much as possible. And then obviously, there are obligations that are more onerous compared to your minimal or your limited risk category. In terms of what a business would need to start considering this, I've got a few questions that your business could be asking. Conversations you could be having with leadership or your teams, which I'll get into shortly. But first, what a provider status requires is or could be a documented risk management system and technical documentation that covers how the system was designed and what its limitations are. It's key to understand how this system works. Can it explain itself? Do you know what it's doing and how it's doing this? Also, data governance is important to show the information it was trained on or tuned on meets quality standards. So it's key for you to incorporate your DPO, your data team, and just to inform them of this vibe coding that you're looking to use so they can be aware of where the data goes, how the data is being used, where it's being stored, so on and so forth to make sure they're happy with that setup. Also, human oversight is important. This though has to be built into the design, it shouldn't be added later as an afterthought. It's really key that human insight or human oversight, human in the loop is built into the design. So, for example, where someone is vibe coding, okay, and you build that system, are they able to um continually review and are they able to challenge and are they able to um check themselves for hallucinations or mistakes and be able to correct that as well? Or is what you're building something more autonomous that it makes decisions for you? These are very different things. Also, you should be testing for accuracy, robustness, and then making sure that you are happy on the security side of things. If this episode is helpful to you so far, please do follow the show and share it with whoever owns AI across your business. Leave a rating if you have a few seconds to spare, and let's stay tuned. So, I've talked about the initial thoughts or considerations that you should have, but it's really really key. It might sound really boring, but it's very, very important because with anything, there's always a hype, isn't there? With AI and tech, there usually is a really big hype and excitement, and then sometimes a lot of businesses leave it too late to start thinking about these things about the processes and the guardrails, um, the risks and the regulatory requirements. Also worth noting that there's another way that businesses can fall into this high-risk kind of category with vibe coding. So this is under Article 25, um, and it states that organizations can shift from uh simply using an AI system as a you know deployer to becoming a provider in different situations. One being putting your own name or brand on a system uh that someone else built, two, substantially modifying a system in a way that changes the risk profile, or three, and most relevant to what we're discussing today, taking a general-purpose AI model, the kind of model widely used for many uses, and repurposing it for a specific high-risk task like job screening or scoring credit worthiness. So if this happens within your business or a team and you're essentially vibe coding it into something more narrower or more specific for one of these purposes, then that repurposing can be exactly the moment that provider obligations attach. And it's very key that you have someone on board who can support you and give you the direction uh as to what should be happening and when, and whenever things switch. So it's key to also know that you can be a deployer and a provider at the same time. They don't have to be, you know, the application isn't one or the other. A business can very well be a provider and a deployer at the same time, and then also things can switch. So you can start off as a provider, then things change, now you're a deployer, or vice versa. So it's key to have that support. And if you're not certain about any of this, but you feel like it's something your business should be thinking about, or you need to assess whether this is already happening or soon to happen, this is exactly the kind of work that SKS Professional Services does. So please head on over to SKspSL.com and get in touch. Whether you are just trying to stay ahead, understand, and get more information, we are here to support with that. So to close off, then here is some practical guidance. If you take anything from this episode, it will be these questions that are starting points, starting conversations for you to have. So before your team builds an AI tool for anything, especially high-risk categories, ask these questions. What decision is this AI system or tool influencing or making? Does the decision affect employment, people's employment, education, finances, access to services or other significant rights or opportunities? Are we building something ourselves or are we substantially modifying an existing AI system? If things go wrong, who would be harmed and to what extent? Next, do we have documentation explaining how the system works, its limitations, and importantly how it was tested? Who is responsible for human oversight and accountability once the system is live? Do we know who is responsible for other areas? And do these people know what action must be taken when that kicks in? And then have we assessed whether our organization, our business, could be considered a provider under the EU AI Act? And also, do we know what we need to be doing if we are simply a deployer? And lastly, do we know what we need to do to shift from a deployer to a provider? These are really, really important questions. And it might sound really overwhelming at this stage, but I promise you it's not as bad as it may sound right now. It's simply that the AI Act, the EU AI Act, is very broad, it's a really big document, but that's the good thing because it's supposed to support businesses at different stages of their AI use, and it also does cover vibe coding as well. But like I said, you're not alone. Do visit skspsl.com, get in touch, and we can support you to take this further. And if you haven't listened to the first episode, please do listen to that. It gives you a really good foundation and understanding of what vibe coding is. And today we're just looking at it from a regulatory standpoint, kind of an overview, a high-level view. We could go into a lot more detail, but for the purpose of today, this is the detail that we'll cover. As you can see, glitched moves at the pace of AI. Things are changing very quickly when it comes to vibe coding, and we are here reporting, we are here updating, and giving some practical um suggestions on what you should be thinking about, what you could be thinking about um in the near future, and some actions for you to take away. Every Tuesday, a new episode drops. I'm Sharon. This is Glitched. I'll see you next Tuesday.