Using Vibe Coding in Real Data Projects

In this episode, the team gets hands-on with vibe coding and what it actually looks like inside real data work. From building API connections to generating synthetic datasets, they walk through where AI is genuinely speeding things up and where it still runs into friction.

The big takeaway: writing code might be faster, but everything around the data still matters just as much.

Blue Margin helps growing companies make better decisions with their data. From building data pipelines to creating reporting and dashboards, we handle the technical side so teams can focus on using their data—not chasing it.

Transcript

0:05

Welcome to the Dashboard Effect Podcast. I'm Brick Thompson. And I'm Landon Oakes. Hey, Landon. So we thought today we'd talk a bit about vibe coding using AI. I've been doing some personally over the last few months, especially since about I don't know, Thanksgiving, Christmas time last year when I think it was Opus 4.5 came out. It was pretty darn good. All of a sudden Claude Code was really good. I don't do any coding really professionally at all. Haven't for you know a couple decades at least. Not probably a couple decades. But all of a sudden, I'm doing all kinds of coding. And it's really fun to be able to do that. And then when Opus 4.6 came out from Anthropic, I think it was February 6th. Uh same time GPT 5.4 came out. Anyway, I was I was still doing the vibe coding there using uh cloud code, still fantastic. Now, just yesterday we got GPT 5.5 out. Codex, you know, the the OpenAI app is updated. I'll probably be spending my weekend doing that. So anyway, I thought we were just talking about it. We're doing uh not really vibe coding here at Blue Margin, but we are using AI tools to help us build our stuff. I know you've been experimenting quite a lot. Yeah. Um maybe maybe we can start there. We just talk about what we're doing, what we're finding is helpful, where it's working well, where we still have to exercise a lot of care. Yeah, definitely. Let's see. So and we've we've done several things recently from all the way from uh you know automating, not necessarily automating, but vibecoding a uh API notebook that that pulls data out of out of a source, um, which has been interesting. We've you know created synthetic data sets and demos, demo data sets, because one of the struggles is you want a data set that replicates a business, but it you also want one that's updating and new. Yeah. Um so you can kind of create your own now without a ton of effort, which has been really cool. Um Yeah, several more that I'm probably not thinking of. It is cool. So you you manage our platform team, and so you're you and your guys manage all of our clients' data lake houses, but a big part of that is pulling in new data sources. So creating uh you know a really robust data pipeline that can run daily or multiple times a day and has a graceful fail mode if something goes wrong so it doesn't leave the data lake in bad shape and sets off alerts and you know makes it easy for the platform engineers, well, easier, to go in and fix it. But also right up front, when a new data source comes from one of our clients, we've got to make that connection. So we've got a series of Jupyter notebooks, depending on what type of data source it is, so we can make that connection. In some cases, um, I think I don't know, somewhere like a hundred data sources. We have something we've used before, so we at least something have something we can start with. Um sometimes we get brand new stuff though, like uh like a totally novel API. It's not uncommon for API documentation to not be great. Yeah. And so, yeah, that's a perfect place to use vibe coding. Um, but we want to start with our standards. We have best practices, we want to do all those things I was talking about. So how do you how do you start? How do you get uh your I don't I think you guys are using VS Code with Claude. Yeah, if I'm if I recall. So how do you get it to know what your target architecture is and make sure it does a good job and doesn't go off and do something kind of crazy? Yeah, yeah. So it's still a work in progress. We're still refining it, which I'm sure all things AI are going to be that way. But um right now we're using skills. So we've you know kind of trained skills to uh learn what our current best practice template is. So what some of our best Python engineers made. Um we gave it that and we said, hey, this is the stuff you need to follow. Here are the things you shouldn't do, here are the things you should do. Um it's actually quite interesting because we we ended up trying to put everything in one skill. And that so that includes like how do I handle all these edge cases? How do I go out and find documentation? What do I do if I can't find the documentation? Yeah. Okay. Yeah. And we found some weird behavior with that. It's almost like we just gave it too much. Yeah. So one of the things we're experimenting with now, which seems a little bit more promising, is having discrete skills that can call each other. Oh, that's okay. Do you have like an orchestrator skill that knows which skills to call? So, like, for example, I'm imagining you can tell me if this is what you're doing or not, but um you have a new API, you can have a skill that goes out and reads the API documentation. Exactly. For example, that's actually what what led us to the idea because we are having a lot of lot of trouble with the API documentation, because not all documentation is AI friendly. I just can't read it because they put it in JavaScript or there's some weird backend on it. So Yeah, interesting. And some of these APIs have sort of secret off-the-menu stuff that you have to know to really make it work well. Yeah. Yeah. Yeah. And then sometimes the APIs don't work well, so you end up going to the old ODBC or something. Yeah. Anyway, yeah. So um I know you've got goals around bringing down the amount of time it'll take to create one of those. One of the interesting things uh to me that uh you told me you found out was that it's not necessarily the hands-on keyboard time, but there's a lot of time around that, getting access to the data source, getting permissions set up right, getting the OAuth set up, you know, uh the system accounts, whatever you need, um uh as well as you know, looking at code and and doing reviews and all that. So the hands-on keyboard time can come down significantly, but you can still end up spending days. Like may maybe hands-on keyboard is half a day now. Yeah. But you can still spend days or even a couple weeks getting it done because of all that ancillary stuff. Yeah. And I sort of wonder if that's going to be the case with a lot of business agentics stuff for a while. Yeah. I think it probably will, actually. Yeah, I would I wouldn't be surprised. You know, it's whenever you need to coordinate people, really. That's where a lot of time a lot of times our roadblocks come from. So Yeah. Yeah, it's true. It's gonna it's gonna be interesting to see how all of that plays out. Like if you tried to build a skill that said, okay, contact the client, get them to set up the author, it's just not gonna work because you need to talk through that very uh variable thing that happens, I think. Yeah, yeah. And it's you know, we we've seen just hundreds and hundreds and hundreds of sources, and it's just always so completely different. Yeah. You know, sometimes there's nothing even online about it, right? Yeah. We've run into that before. People just don't talk about it online. So the AI doesn't know, doesn't isn't trained on it, it doesn't know where to go find anything about it. Kind of runs into a dead end. Yeah. Do you think does Claude Code do a pretty good job of following our best practices? It's I don't know. I'd say like 80-20. I might my team might say I'm being too generous to Claude. Really? Yeah. Um they uh they have some of their times where it's like, oh well, it just went over here and edited that thing that I didn't want it to edit, right? Right. And so that's part of one of the reasons why we actually switched to VS Code. Um from just using the terminal. Yeah, exactly. Because one nice thing is that as long as you don't tell it, yeah, just go crazy, edit whatever the heck you want. Um, you say, Let me approve edits, it will show you a side-by-side comparison. Here's your version, here's what I'm gonna change, and it highlights the changes for you. You know, like those of you who might have used um Git compare. It's that essentially you're comparing what the AI did to what you have. Yeah. And so makes it way easier to make sure it doesn't go off the rails and edit something completely different, along with sneaking in, you know, the editing you asked for. I hate that. Oh, and that was the problem, you know, six months, nine months ago, whenever I would play with vibe coding, because I have been for, I don't know, the last year or so. But not that consistently, maybe for the first half of last year, till we got up to the holiday season, because it just wasn't that good. Yeah. You know, it would produce something that was close, and then you'd risk r request fixes, and it would maybe make the fix but break something else. And it was just sort of it was like playing whack-a-mole. And then all of a sudden, around Thanksgiving, Christmas time, it stopped feeling like that. And then around February, it really stopped feeling like that. Yeah. Yeah. It's gotten so much better. Yeah. So uh well, you're gonna have to try GPT 5.5, the codex. Yeah. Uh and uh and let me know how that is. I'll be playing with that too. Another thing I know you were working on, so we've got I'm doing work for clients to connect Claude and other um agents to a data lakehouse, to their data lake house, and be able to do ad hoc queries for executives to be able to do, you know, get quick answers on financial data or operational data. And it seems like that would be easy, but um it's easy to get wrong answers, actually. And so we've been building something called a platinum layer, which is to take the gold layer out of the medallion architecture and then uh do some things to it, denormalize it even more, rename columns, rename tables, make sure all of our joins are done in a consistent manner, and then actually stack a bunch of markdown files into the data lake as well with context. But then in addition to that, um, we started building custom MCP servers for different clients that that have a lot more instructions about how to go and get an answer for the agent. Um, it's been really interesting, and you've been doing some work to build a demo for that. Yeah. And part of that was sort of vibe coding a data set, right? Because we don't want to use our data or any of our clients' data, obviously, um to demo. So we had to come up with completely synthetic data. And there's things like um AdventureWorks out there or Contoso or whatever, you know, the Microsoft thing. So there's there's a lot of stuff. But but you actually started from scratch. Why don't you talk about that a little bit? I thought it was pretty cool. Yeah, definitely. So as the as the idea to start from scratch actually came from AI itself. Um, I I I just started asking it like, how do we get around this problem? Help me brainstorm this, please. Got that idea from it, and then was like, yes, let's go for it. Um so I basically vibe coded it's it's nothing crazy, but it's a it's a SQL script that builds out tables, generates random data, generates customer sales reps, products, et cetera, targets, um which then you just kind of run it. It takes like 20 minutes because you know, it it worked worked at first. And then where I got myself in trouble is I started trying to tweak it to try to make it seem more real. It's really hard to make like seasonality. Well, seasonality actually is easy. That's that's not hard. Okay. But like a customer leaving and then maybe returning a year later, right? Stories like that, really, really hard to find. Yeah. Um, so I uh was trying to prompt it to do that, and I ended up destroying pretty much just wiping out the thing. I mean, I didn't wreck it completely, but it made zero sense. Like I'd go ask, because I use Claude to connect to our MCP server to talk to the data, and I'd be like, hey, tell me our you know sales on this customer versus their target. And it's like, well, they are 2,000% above their target. You should talk to the person who set these targets. I don't know what they were thinking. Like, oh, oops. So that was when I learned that I really want Git to back up my code base when I'm Oh, you weren't you didn't have that in a repository? I had to start from scratch. Oh, that sucks. Yeah, it was painful. And we use we use uh Azure DevOps, right? Yeah. Are you s are you starting to use Git now that you're doing more Vibe code? I still use DevOps. I mean it's Git on the end. Um Yeah, I would like to get more into Git, but we have we have DevOps licenses. It's kind of it's free for us, right? So it is Git on DevOps. So I just I guess I mean GitHub. Yeah. Yeah. Yeah. That's all my hobby stuff I've been doing on that. Yeah. All right. So then how did you get it? So you started over, I guess. Started from scratch, essentially. Okay. Pre-prompted, just just wipe wiped it all out, started from scratch, connected it to a repo, um, so I can commit changes and roll back. Yeah. But yeah, it was a good learning experience for sure. Aaron Powell So one of the problems with sample data sets is that they're static usually, almost always. And so as you're using them to test and so on, it's for data that's six months old or two years old. So have you figured out a way to sort of keep adding fresh records? Yeah, yeah, definitely. So that was one of the key requirements I gave it. You know, I that's one thing too. Like when you Vive Code, you gotta start with really clear. This is what it needs, this is what the end product needs to look like. Here are the things that it needs to do, et cetera. The clearer you can be, the better. It created a pretty intense, I mean it's not intense in terms of like sloppy, it's actually really good uh procedure that you call and it goes through several different conditions. It has seasonality built in and has customer targets and segments and kind of weights where their purchases go depending on attributes on the customer. Um and you can tell it, I want to generate a thousand orders a day, I want to generate 10 orders a day, et cetera. That's cool. And so it was it was pretty cool to see because you know, with Python, I immediately attached to this is so easy to, you know, like coding typical coding. I know SQL's coding, but kind of in a different bit. Um immediately attached to how you can do that. With SQL, it's always been a little harder to figure out how do I get this thing to write good SQL because you need that data context of it. Um data and you almost you have to have a business context too, right? Exactly. So you know, here's here's the conditions I'm trying to simulate so that I can have an interesting demo. Yeah, exactly. But basically it it's uh you know, I consider myself a really strong SQL developer. I've been doing it for years and years. You are truly an expert, yeah. And yeah, the stuff in there was was insane, you know. It was stuff that I knew, but the way it put it all together and in creative ways was was really cool to see. Oh, that's cool. How much debugging did you have to do? On that proc? Yeah. Yeah. I went back and forth with Claude. I don't have a number, but like probably three hours or so back and forth. But the thing is, you know, I'd have to go go in there, then I'd have to run everything, um, wait for it to populate, and then check the results. Sometimes I'd export CSVs and send it back to Claude. I'd ask it like, hey, write me a query that will summarize this so we can make sure this makes sense. And I just paste the query in, you know, and run it. Did you have it create a test uh test suite to run? I did not for that one specifically, mainly because I need to still figure out how to connect AI to Well, I guess we do know how to do it now. To the data later. Which right now, yes, I can do that. So that's a great idea. Next time I'm gonna have it just run its own tests, get its own data back, adjust it. Yeah. It's a great idea. So what's next for you with the team here in terms of uh AI coding? Yeah, so you know, we're just trying to get as efficient as we possibly can with uh our day-to-day work, right? So a couple of things, you know, we're redesigning our standard templates, um, we're making it a bit more modular so that AI has less to edit. Right. Okay. Um there's a couple of driving forces on that. You know, obviously the less we need to edit, the less context it has to use on it. Yeah. Um more targeted it can be. And then also, you know, if prices increase like crazy for some reason, we're not we're not completely underwater. Right. Um and then the other one that I really want to do too that I've heard success about is um getting just automated bug fixes, right? So pipeline for a client fails overnight, it's going to automatically go look at the code in our repos, um, make what it thinks is the fix, put a push uh pull request in, and then in the morning we just look at it, right? We'll still want that human in the loop for for now. Um that's pretty cool. But I know some you can do a code review too with a recommended, you know, I've I've reviewed the pull request. Yeah. I recommend that this one be approved. Here's why. Yeah you still might you still might look at it, but it might speed you up. Yeah, if we got trusty enough with it, yeah, maybe we have a reviewer that reviews all the requests. Yeah. If it's ends up being good enough, it can just automatically push it and rerun it. I'd be crazy. I think these tools are getting really good on coding where you can almost get there. I mean you have to obviously be super careful around anything that has to do with security, exactly. Or that's super important um operational data. You don't want to put bad data in the data lake in the lake house. Um but because it's funny, I was reading an article about why do LLMs hallucinate so much just in free text conversations or summarizing dictation or whatever, but in code it really nails it. And and some of the theories were that, well, there's so much code out there that you can train way, way more than sort of random um prose that's out there. And it's way more structured. And the languages are very confined, like the the total vocabulary and the syntax is not that big. So they can get really good at it. And so it's not like the problem they're seeing in law where you know 75% of big legal briefs written with uh with AI have hallucinated stuff in them, made-up cases. And I mean it just happened again to another big law firm, amazingly. Um it's it's different than that. Not that it can't do things wrong or start going down a bad path or introduce some some bug, but it's amazing how good it's gotten this last, I don't know, six months. Yeah. Just as really complete step change. Oh, yeah, I completely agree. I mean, it's got public repos out there all over the place. Stack Overflow is a gold mine of info now. Yeah. Yeah. I've heard they're actually um selling their data now as training data. Oh, that's why models are that's where you used to go. If you had a question, that's where you went. Yeah, exactly. I used to go there every single day. I'd always have Stack Overflow open. I haven't opened it in four or five, probably more than that, probably like six months at this point. Yeah, I'm not surprised. I just get answers faster, you know, less digging. I think next time I'd love to talk about sort of culturally what it's been like to get the team on board with that because there's a there's a wide range of reactions to from oh, I can't wait to do all write all my code that way to gosh, one of the things I really enjoy about my job is crafting code. Like it's almost an art. So I don't really want to give that up. So anyway, we'll talk about that next time. Yeah, absolutely. But uh good discussion and uh talk to you soon. Yeah, thank you. Appreciate it. Talk to you later.