What's New In Data
A podcast by Striim (pronounced 'Stream') that covers the latest trends and news in data, cloud computing, data streaming, and analytics.
What's New In Data
Joe Reis on Staying Grounded in a Fast-Moving Data World
Joe Reis joins us to reflect on life after Fundamentals of Data Engineering, what makes data content worth consuming, and why good taste matters as much as technical skill. We talk about burnout in big tech, the myth of AI replacing everyone, and how Discord communities, DJ sets, and a sense of humor are helping shape the future of data. This one’s part industry pulse check, part real talk.
Follow Joe:
What's New In Data is a data thought leadership series hosted by John Kutay who leads data and products at Striim. What's New In Data hosts industry practitioners to discuss latest trends, common patterns for real world data patterns, and analytics success stories.
Well, we have the famous Joe Rice here on this episode of what's New in Data. I'm putting you on the spot. How are you doing?
Speaker 2:Joe, I'm doing good. I'm just slightly jet lagged. I've just been getting over that, but otherwise I think I'm good yeah yeah, I'm just in. German. I think net is nice, so yep. How are you?
Speaker 1:I'm doing good. Every time I catch up with you, I know I always like to ask you where you've been, and it's always some exotic answer. I'll just ask you. I mean, we're recording this in mid-March, you know, joe, where were you last?
Speaker 2:week spent some time in Munich, then Zellemsee, I think that's how you pronounce it in Austria, which is in the Alps, so a winter data conference put on by, actually, chris Tabb. Yes, great data crew. Yeah, my kid drew a Chris Tabb mug for me, so, yeah, it was a great one. I think you were at Skid last year over in.
Speaker 2:Gerbier. Yeah, that was a lot of fun. This is the new version of it with probably a more official sounding event name Not skiers and data, but a winter data conference instead. So it was a lot of fun. Great turnout, you know. The Alps are lovely, as always, so, yeah, but now I'm back in salt lake city or it actually snows more here than it does in the alps, apparently.
Speaker 1:So yeah, yeah, I remember last year when we had berbier. You were mentioning that. You know you're from salt lake city and there's like 50 berbiers in salt lake city, so yeah it was trying to kind of Utah.
Speaker 2:It seemed to be cocky about that, huh, but it is true yeah it is true, so it's not cocky.
Speaker 1:If it's true. Everyone knows that you put out this book, the Fundamentals of Data Engineering, and it's received such great acclaim. It's one of my favorite books. I recommend it to every data engineer because it's both theoretical but also very practical in how data engineering is applied. And now you know all your world travel and events you speak at and companies you work with. You know it's all kind of rooted in. You know the success and how people have really been captivated by your book. What it, what's it like just uh, continuing to to work with people who've read your book or know you through your book?
Speaker 2:it's kind of surreal really. I mean, yeah, I mean it's coming on three years old now, which is kind of crazy. It was published, so it's. So it's interesting in that way where you know there's definitely you know, I think we knew it was going to be successful just because the you know, on amazon it was a number one new release, a bunch of categories before it came out and then it obviously still does really well. But it's definitely a bit of it's still a bit surreal, um, you got to see posts about it every day on social media, like literally every day there's a new post about it still. So that's that's pretty cool.
Speaker 2:You know, I'm definitely grateful for the success it's brought um, and I feel like, um, you know, just just thankful that at this point, you know, a book will be successful on its own merits and through word of mouth, not because of marketing. So apparently people liked it. So I think that that's pretty cool. It's definitely opened up a lot of opportunities. I think changed, um, you know, definitely my trajectory and stuff. At the same time I'm always on to the next thing, you know. So I don't want to be like uncle rico from napoleon dynamite who, uh, I remember that character. So he's just so for the audience. Uncle rico, if you haven't watched the play in dynamite, he's the uh kind of washed up high school football player who still reminisces about his glory days when he almost won state championship.
Speaker 2:If coach just put him in he'd be in the NFL, he'd have mansions and women and all this stuff, cars, yeah, and I mean that's, that's a danger though, right, because if you, I think, if you rest on your laurels, um, you know, and aren't working on the next thing, and then, uh, you know that I think you do risk becoming kind of uncle rico in that way. So, once the book was done, I I kind of, you know, definitely gave a lot of talks and you know I've been very grateful for the success, but I was always just on to the next thing working on that. Uh, we're next things, I guess, but you know, it's definitely cool, but it's, it's weird. You know, um, at this stage I think I'll first look through the podcasts and being more visible, um, you know, going to conferences and, um, definitely getting noticed or some cases, you know, I hate to say it, but probably mobbed, and then, you know, even in public, just walking around, getting noticed in public. So that's always interesting.
Speaker 1:I've witnessed it. Yeah, I mean, people are like, hey, dude, get out of the way. Joe Rice is here. I want to talk to him.
Speaker 2:Yeah, it's funny.
Speaker 1:Yeah.
Speaker 2:It is what it is. I mean I'm thankful for it. I mean it's it's. It's definitely takes a bit of getting used to, but by this point you know that's what it is.
Speaker 1:So, yeah, and you know people, ultimately I mean it's, it's, it's one of those books that you know it, it is, it's in the title, it's, it's very fundamental and you know, now data engineering is is changing in a lot of ways, uh, but also going to stay the same in in others, like a lot of the core principles of data management and pipelines and data modeling.
Speaker 1:I don't expect that to change too much, but the way people write code for data um is is definitely going to change, just because code generation is also getting disrupted in its own way.
Speaker 2:Oh yeah, it's ubiquitous. Yeah, that's how it goes. But I always wondered our book came out right before ChatGPT hit the scene. No, not right before. It was like five months, four months actually. But what's interesting with that is, you know, I kind of called it. You know the data engineering lifecycle is not going to change. The undercurrents won't change. You know you're still going to get data from source systems. You're still going to need to make it secure, your system secure, and I would say, more than ever. Actually, data management and data operations and governance and modeling is actually more important than ever, essentially because of AI. So I think it's been driving a lot of interest in data engineering. At the same time, as you point out, data engineering itself, at least the way we do it, it certainly is changing. You're not handcrafting code anymore. Code gen is becoming more and more of a thing, which I think is a blessing and a curse, but it's reality. You're not getting away from it. So you know. Yeah.
Speaker 1:It's interesting, yeah, and you know, but I also want to ask you. So you know, we talked about fundamentals of data engineering. Being, you know, just over three years old now you know what are you working on now.
Speaker 2:Well, I have a new company, kind of hinted at it, you know, it's publishing, it's education, it's media. It's a lot of things that I've been doing. So you know you want to talk about like, definitely, I think, jumping into what the book provided. It provided me a lot of opportunities to, I think, become more involved in media and education and so forth. You know, I did a course with Andering and the deep learning fine folks there at deep learning last year. That's on Coursera and that's done really well. But you know, I think doing that and a few other things you know reminded me of like I really do like, and I want to focus more on that.
Speaker 2:I think one of the biggest gaps we have in the industry is, you know, we have great tools. I think our ability to use these tools to its fullest potential is really hampered by our skills and knowledge, and so I think helping upskill a new generation of data an existing one of engineers will help push us forward. That plus AI, great skills, and AI will help you do great things. But AI on its own, without having great skills, I think is could be a recipe for, you know, pretty interesting outcomes. So that's that's one thing I'm focusing on right now.
Speaker 2:So, yeah, my new book is going to be out on this new company. There's a few of the books. That's one thing I'm focusing on right now. So, yeah, my new book is going to be out on, uh, this new company. Um, there's a few of the books being written as well, but yeah, but it's, it's an interesting thing, um, and I'll ask you. I'll answer a question you didn't ask, um, which is, uh, why do publishing in an age where you could seemingly just make a book in an hour you know, probably a half hour and get it all published on?
Speaker 1:Amazon, that's what I was thinking about seeing you so thank you, yeah, just perfect for that.
Speaker 2:Thanks, john, great question. So you're welcome. But I think with AI, people are you're already seeing it people are getting tired of AI-generated content, especially AI-generated slop, as it's called, which is just low-quality writing, low-quality images and videos and so forth.
Speaker 2:I think that, more than ever, there's going to be a really strong need for people for really strong human-generated, human-created ideas that are going to help push our industry forward. This is something that I don't think AI can do, at least not yet. But at least for now, I don't want to write technical books per se, like how to do AI engineering with a python or something. I don't think that needs to be a book. These days, that's something you could ask claude or chat gpt and get a pretty good answer, uh. So I think the tactical work and the tactical types of books I don't really want to do, but I think the big idea books, the ones that are really going to push our industry forward, those are the ones I'm interested in. So so for the audience, yeah, I have an idea like that. You know, hit me up, uh, but yeah and that's what's so important these days.
Speaker 1:You know there's, you know, as you, as you mentioned, ai slop is definitely, uh, a strong label that's been attached to a lot of the you know, not to sound redundant, but slop that's been put out there. And you log into linkedin and you know there's like so many posts that are like clearly ai generated how many?
Speaker 1:they all use the same uh, like they all use the same kind of writing style.
Speaker 1:Uh, you know they'll, they'll, uh, because it's trained on. You know very, very specific types of media and you know you can just tell, like, within you know five seconds of looking at a post, whether it's generated by chat gpt or not, and and uh. But I think what people are gonna be craving now more than ever, is hearing from experts, right, and hearing from people who, like you said I'm, big ideas that will actually have a big impact within an organization, because people are kind of more scrutinous about you know, just things being built, because we know ai can build things too. So you know what's to say that you know someone just didn't tell chat gpt to build something for them, but, yeah, they're doing it in a way that's grounded in fundamentals and, you know, based on experience and really solving business problems well, absolutely, and I think the litmus test is going to be, people gravitate towards like, really good books, really good writing, really good podcasts, right, these are, you know, human exchanges, um, and in person.
Speaker 2:Right, I think this is increasingly going to become more and more viable and what I think is actually just doing a podcast. Today with Jordan Morrow, we're talking about public speaking and it's interesting because I think, increasingly, giving talks is going to be one of the differentiators between people who write real stuff and can talk about it. Um, you know, in a public setting, especially during the q a, like, I think that's going to distinguish the experts, the people who wrote books and who know what they're talking about when they write the book, versus just having chat gpt generate a bunch of garbage. Um, they probably couldn't speak to you know, or understand um, so I know, with my book I mean, I counted probably over a dozen, you know, throughout, throughout the you know a few years, probably a dozen different books titled fundamentals of data engineering, uh, you know, which are all kind of knockoffs of that nice book and um, I download the samples of them, they check them out and kindle and it's exactly what you think it's.
Speaker 2:It's, there's no thought put into it? Um, there's not. You know, if you're to ask these people to give a lecture on data engineering, I'm pretty sure none of them would, because they're probably not people. They're, they're uh, they're bots under pen names. There's one guy, on linkedin, I think, who wrote a book under my name, but you know, I'd love for him to give a talk on the topic and let's see what he can do.
Speaker 1:Well, yeah, I mean, yeah, there's a lot of people who generate a lot of AI content. They just want to publish it. But if you ask them about any of those topics, ask them one hard question about it. Unless they have Claude in front of them, them, uh, won't be able to tell you much. That's just it. That's just it.
Speaker 1:I mean like, yeah, yeah, and I, I deal, I deal with that a lot too, because if, being in this industry, you're constantly researching, you're constantly learning, you're constantly like trying all the new um patterns and frameworks that are coming out.
Speaker 1:Like you know, model context protocol, for example, was a uh, you was kind of an opinionated way of writing your AI apps to fetch external context. And they say it's like having a USB-C input for AI and there's so much slop generated around it where, like the first two or three times I I read about it, I was like this is garbage, mcp is stupid. And then, finally, like I found people credible talking about it and the way they talked about it, I was like, oh, okay, now it makes sense to me, right, right. So you're almost it's it's, it's almost hard to not throw the baby out with the bath water when, when there's like all these people generating a I slop about things that are actually valuable. Yeah it's. It's hard to get around that, so it's good to like like with the company you're building you're, you're gonna sort of have those lists of like curated minds that's just it, they just have.
Speaker 2:You know you get the best minds of the planet. You know producing the, and I think magical things would happen and treat them well. That's the thing. You've got to treat people with respect, especially authors and creators. These days, I think that, more than ever, coming up with an idea is one thing, but then you've got to help market and distribute, which is something most traditional publishers don't do. The sad thing is, if you write a book, you're also responsible for marketing it, and that's something that's. It's a difficult skill set for somebody to have, right, and so that's. That's another thing. Where get the ideas out there? Get good ideas from great people out there, and I think the rest will take care of itself. But you know, if you, if you market a book though right, I mean, marketing is only going to probably push a book or you know, a piece of content for maybe a week or two, right, yeah, it has to be good to center on its own.
Speaker 1:And I guess what's the definition of good?
Speaker 2:People will tell their friends about it. People will tell their friends like this is worth your time, you know yeah.
Speaker 1:It's really interesting because it's similar to the music industry where, like music industry has you know what are called tastemakers and they're the ones who influence. You know the big playlists, you know who the big music publications will talk about what plays on the radio, even though radio is kind of uh, lost in the space yeah you know, okay, the big spotify playlists, right, uh, the big apple music playlists, etc.
Speaker 1:So, like you see, these djs who, who basically decide, kind of like you know they, they have the curated mindset to say you know what's what's good and bad, and, uh, and and joe, you're a big uh music aficionado yourself. You're, you're, you actually are a dj, right, so you do have incredible taste. Just because you know, and I've listened to your sets, they're good. I think just being able to, to curate a great like dj set already shows that you know, you understand what, what people want to hear. You know one way or another, I think. I think it's the same thing as uh, understanding you know what, what, what data patterns make sense, or adoptable, adoptable and and practical, um, and I don't know when was the last time you DJ'd actually in public gosh.
Speaker 2:That must have been maybe a year ago or something we both did. Chad's event that was November it was November 23 actually, so that was a while ago, right oh yeah but yeah, I always carry my usb stick around with me if I travel.
Speaker 2:You never know when you're gonna find turntables and it's kind of fun. But, um, yeah, I've just been pretty booked. I haven't had time. I mean, I've been, um, there's a, there's a very good chance I might be doing some some local gigs here just to go dj at um some clubs and probably do some more live sets too.
Speaker 2:Like I have a whole uh live um, you know kind of hardware setup that I like to play on, uh, but but curation, right, I mean, I think djing taught me a lot about kind of how I approach you know what I'm doing now, right, uh, like, I think you just sort of develop a sixth sense for where things are going or, more importantly, need to go in our industry and and so you, you think at a certain point you can help shape the discussion, you can help shape where things go, and that's pretty cool to do. I'm doing that with data modeling right now. Um, that wasn't really a cool topic for a bit until I mentioned, you know, start writing a book on it. I'm not saying you can write it for all of it, but I think it definitely helped push that back into the foray, right? So I think at a certain point you can just start. You can kind of determine where trends go, or you can at least see where they go Right. But, um, or you can at least see where they go right, but you have to be in it for a long time to to know the patterns that are hidden behind the patterns and uh, you know it's um, but everything you know, I think, a lot of things in our industry. They definitely move in cycles, they kind of go on a pendulum and uh, you just you know.
Speaker 2:I think once you understand the rhythm of it, I can't say you can predict the future, but you can certainly tell where things are likely going to be going. So even with AI right now right, I mean, that's, it's new, you know might shake things up a bit, but when you get down to it, there isn't much different from when you saw other. You know, significant technology patterns emerge in the world. Of course there will be differences, but it's not like this is brand new out of the scene. It never happened before.
Speaker 1:We never had a new technology it happens a bunch, yeah, and you know, like when I, when I look at the way you approach it, I mean you have both. You know a STEM, technical, academic background and actual industry background building. You know analytics and data science and data infrastructure. You know for, for, for large companies, intuition about kind of. You know what's good and what's bad and that's always subjective, right, people debate it a lot. You know who's to say one person's right, one person's wrong but I think what's sort of indisputable is that you know when you put pen to paper and you put out that book with Matt. You know Fundamentals of Data Engineering and you know it was just widespread acclaim. People agree with it. And when you generally go talk, people are captivated and interested.
Speaker 1:And then, even coming back to DJing great DJ sets we were just talking about you were putting me on to FIAC and speaking of misconstruing names. I call them FJAC and it kind of reminds me of this quote I actually saw it on X from Sam Lambert, the CEO ofo of planet scale. It's a database company. Yeah, he says do not work for anyone who doesn't love music. They will never build anything that that humans want, and I think you were one of the first people I thought about there because I was like because that's a, that's the ceo of a database company saying that so, like, what does a ceo of a database company care about? Like, like people having musical taste? But then kind of, when you connect the dots and you meet people you know, uh, across the industry, like there there is actually some relation there.
Speaker 2:Oh tons, I mean, you're a musician, I would say you're. You're a very classically trained musician, right?
Speaker 1:Yeah, yeah, I uh, you know, yeah, definitely, uh, my days at san francisco conservatory music, and then I was, I, you know, I had my own artist project. I go under a pseudonym, though it's very, it's very secret and you know, uh, I try to keep it separate, right, so people don't kind of, because I, I always, I was always kind of insecure about that too like, okay, people know about, like my music. You know that they, they might not take my uh, uh work in data and engineering as credibly, because I'll be like, well, how do you, how do you do both? Like, because I feel like, yeah, confuse people I think it's.
Speaker 2:I think those walls are diminishing though, right, like social media sort of broken down a lot of the barriers between, you know, public and private persona. Um, I mean, for god's sake, you know, the ceo of golden sacks is a dj. Um, you know, I think a pretty popular one at that, diesel david solomon, right, yeah, like, yeah, I don't think he has any you know qualms about. Oh, I'm the ceo of golden sacks. Nobody can know that I dj, you know, I think he uses that to almost to his advantage in a bit. I mean, we're talking about it. Um, yeah, you know, and it says, I think this is the walls between public and private, whatever, I mean it's, you know, depending on what you do, obviously, but I don't see any.
Speaker 2:I think data needs more of that really, where it's kind of a boring industry, we sell database systems. Woo, I mean, that's's cool, it helps make the world go around, but so does music, right, and I think, you know, and I meet a lot of interesting people, as you do too, in the industry and I think, um, you know that because you see, especially at conferences, conferences are hilarious because you meet people and they have to put on the conference face and then where you know, where the conference you know, attire usually a suit of some sort, but then you hang out with these people after the after hours of the conference. These people are nuts. So I think that's I kind of hope that a lot of these walls get broken down. I just think that I'm much more impressed when I think there's there's a, there's a level of credibility to somebody and a level of authenticity.
Speaker 2:I, I think we're. If you know, if the, the barriers are pretty indistinguishable, then I think I can trust you a lot more for one right, because I know you're not like hiding, I know you're not, you know, full of crap. So, um, you know, yeah, yeah's me, but that's how it is. That's the trajectory of the world. The world is going though right, like it or not, it just is what it is. It's John Coutet. You can either change your name to your pseudonym or unleash the music into the world, into your real name, but I think it would benefit you.
Speaker 1:I tease a bit on your on your discord, which is something else that you know I I want to talk to you about. So you, you launched a, a discord channel called practical data. Tell me about that practical data is a.
Speaker 2:It's really an extension of the subsec that I had. So I'm releasing, uh, early sections of my book, usually in draft form, on practical data modeling. That's sub stack, uhcom. If I had half a brain I would have just called it practical data, because then I can have more books that I'll be writing on there as well. But here we are and so we had this chat group going for a bit on sub stack. But sub stack chat, it's pretty mid, as the kids say. I was like okay, so let's make a chat community. I was like okay, do I want to use Slack or do I want to use Discord? I joked, you know we'd better use Teams, but I'll just keep it really corporate.
Speaker 2:Can you bring Skype back back? Skype like yeah, bring skype back. That makes it pretty funny. Uh, we these webex, um, that's amazing that'd be hilarious actually.
Speaker 2:Maybe I should do that for april fools. Um, I'll write that one down actually, but anyway, so I think it's. Yeah. So I started a discord group and, uh, you know, soft launch into about 100 people and that was really fun, just to see how it would go right, is this even worth pursuing or not? Um, and then, you know, open it up to a few more people and now we're almost like 1100 people. It's not not huge, but I think the quality of the conversations I've heard from quite a few people that it's definitely their favorite data community to be involved in because it's just very candid.
Speaker 2:Um, conversations you know about, about our practice, right, and I have other channels as well so you can talk about. If you want to talk about politics, you can. There's a politics channel. I don't care, right, as long as you keep it civil. There's an unhinged channel. You want to talk about just the craziest stuff you can think of? Go put it in there, I don't care. Um, you know, we opened up a new one about sports the other day, so we can just talk about whatever sports you're into. So I think it's because what you realize is there's when you start a data community. Most people want to talk about everything except data, and again this just comes back to you know um djing and working in clubs for a while too, or I think you just get to know human behavior. At the end of the day, people have a lot of facets to themselves. They want to talk about a lot of things, and a lot of times the last thing they want to talk about is work, especially in a chat setting.
Speaker 1:That's one reason I didn't want to do Slack or anything, because it's like that format is too familiar to you, because you associate that with your job yeah, I, I think just having yeah, like and just like, you understand people better and you know, because all this you know, talk about like data infrastructure, like everyone has a as an opinion on it, and unless someone has like a, really you know uh is really grounded in something. That's just absolutely proven. It's all up for debate. When you get people together to chat about stuff, it's the same way Like, oh, I think this football team is going to win the Super Bowl this year, based on my opinion, people just chat about it the same way. It's just open-ended and casual. I really like the, the kind of the discord, because people both talk about data the same way. They just talk about other stuff right right yeah what you realize too, is it culturally.
Speaker 2:Data is one of these topics where they're all cultural nuances to how we work with data right, how we architect systems and how we build them. It's not like. So what I realized, you know, especially in the travels as I'm sure you have around the world and talking with people is like they're you have to take into account the geographical situation that you're in. You have to take into account the business culture that you're in. Nothing is monolithic. When you're trying to sell, for example, you, you know, say, stream into a company, it's like there are other factors you need to consider besides just typical. Am I talking to the qualified buyer? Am I, you know, am I hitting a pain point? It's like that's basic stuff, but it's also, you know, how do they get to the situation they're in? You know what's the team like that's going to support this and a what's the team like that's going to support this? A lot of this comes down to the nuances of the culture that they happen to live in.
Speaker 2:In our Discord group we have people from all over the world, even people in Europe. Eastern Europe is different than Western Europe. It's different from the UK. It's different from Nordics. It should be obvious, but I got called out on this. The other day I was in Munich and I asked one of my friends what is her thought on AI innovation or something like that in Europe? And she's like, well, but you've got to understand, we're in Germany, that's different than it's not Europe. I was like thanks for the reminder. You're absolutely right, but you realize the data world is a pretty big place and everyone's got their own opinions on how to do stuff and you can't just. I think all too often, especially in America, we try and paint it from the broad strokes of well, this is how it's done in the Bay Area, for example, so therefore everybody is the same right, or this is how we do it in New York or wherever, but that's certainly not the case.
Speaker 1:So yeah, yeah, and I think, coming back to taste, you know a lot of the status stuff does and it's not new necessarily. Like when you look at, like, the role of a data architect in the enterprise, like usually, like the you know some domain specific team, a software team or a business team will say, hey, we want to do these things. Uh, it'll technically work for us. Data architect, what do you think? And the architect will shoot it down just because they think it's a bad design. And you know, there might be some, some technical principles that they're applying there.
Speaker 1:But a lot of times when I see it, it is sort of like a form of taste making, like I just like, oh, I don't know that pattern, like I've never heard of it and you know it's, even though it technically might work that you know, I don't know if it'll be resilient or scale. They might shoot something down because of that. Right, and it can be, it can be a little arbitrary, so it's it's. It's super interesting to see how that also correlates with what you're doing with the Discord, where I see people talking about announcements of new data tools and open source frameworks and what sub-vendors are pitching and they haven't tried it yet Some of them have tried some of the open source stuff. But even with stuff they haven't tried, they're just inserting their tastes and their view on just what they think about it broadly yeah, yeah, I always get a kick out of that.
Speaker 2:Um, yeah, sort of having an opinion before trying it. Um, yeah, that's something I personally always, always want to try and avoid. I uh, I, uh, well, I very rarely will talk about vendors specifically too, just because I, I mean, for better or for worse, if I say something, it does have an impact on things sometimes, so I usually keep quiet. But other people, you know, I mean, they definitely post things and it's cool to see the enthusiasm and sometimes lack of enthusiasm, maybe warranted or unwarranted, but uh, um, but that's how it is. I think, you know, people bring a lot of biases to how they evaluate tools and technologies and, um, but that's the fun part about our field, I suppose, is, uh, you know, if a vendor is liked or disliked, that could, that could change. Um, uh, I'll give you know, I'll give you an example, I will call it a vendor. Um, uh, microsoft. They I'll give you an example, I will call it a vendor Microsoft.
Speaker 2:They were sort of, I think for decades considered sort of the great Satan in the tech world by developers. Now they've made a huge change. I remember 10 years ago one of my friends, she went there to go work at Microsoft as a Python advocate. I was like what are you doing? This is nuts. They're making some of the biggest strides with Python and Guido's there, and so you know they have amazing tools right.
Speaker 2:I mean VS Code is widely used, so I mean things could change right, and so I never write anybody off. And you know, and at the same time I don't really get too hot on things because it's like it's good for today, who knows how it is tomorrow. I take a very, I guess, zen-like approach to all this stuff because, as you know, you're around long enough as things kind of flow like water and that's how it is. So you know. But I think that's a good example, you know, of what Microsoft did. I mean they've developer to the non-Microsoft people using it. I think that's awesome to see. I mean I used WSL on my Windows laptop and I think it's fantastic. So kudos to them. They did a great job.
Speaker 1:Yeah, and Power BI is ubiquitous and Azure is everywhere.
Speaker 2:It's everywhere.
Speaker 1:And, speaking of BI, one thing that you mentioned with AI BI is that old-school BI dashboards are kind of following the Lindy effect. Do you see them being disrupted by this concept of AI, bi or AI in general?
Speaker 2:a bit ago and that spurred a lot of discussion. I think it was a podcast I did about old school versus new school BI, and you are seeing this. I think that there is chatter about how we can just put a chatbot in front of everything. You don't need dashboards anymore in the old school sense you can chat with your data. So I guess the questions I had was you know, did the data change along with this new interface, or is it the same data? Because if it's the same data, most of the questions that you need to run your business are probably already answered, I'm guessing, in a dashboard that's been there for probably quite a while. Right, like what are the trends of my sales? What's my? In a dashboard that's been there for probably quite a while, what are the trends of my sales, what's my operating margin and so forth?
Speaker 2:Whatever user statistics you have in your app, whatever you're into, those probably have been, I'm guessing. I hope, answered in a dashboard of some sort or a report. But who uses these? I think by most accounts, bi tool adoption has never hit above 25% in most companies. So my question was if usage has been pretty low across the board or at that level, what makes you think that, given the ability to chat with your data, that you're going to ask different questions, and the ability to ask questions is the only thing that stopped you from not looking at your dashboard before? And my answer to this is it's going to probably be both. I think old school BI, it still accomplishes 80-90% of what you need. Those other 10-20%, sure, if you want, that'll certainly um make it faster than trying to talk to an analyst who might take weeks to build a report for you, um.
Speaker 2:But so I think the question is both, because I kept seeing these very um binary reactions where it's, yes, you know you need new school, ai ei, and the old school stuff is done for. Or you know the other camps, like no, that's all garbage and it's has to be the old school stuff is done for. Or you know the other camp's like no, that's all garbage and it has to be the old school stuff. I'm like maybe it's both, I don't know. I mean, like I said, I always try to be flexible in these ideas because I don't believe in absolutes. When it comes to technology, things move too fast and things improve, like, I think, the people who say that it's always going to be old school bi and like large language models are useless. It's like, yeah, they're just looking at today's hallucinations, right, it's not not accounting for the fact you know these will improve.
Speaker 2:They have to yeah um, it's like you know, so um yeah, we're like.
Speaker 1:I mean, we were chatting about this or a text and like, for a long time people were ragging on including me, ragging on llms for being bad at math. Right, like, okay, if you give it like a, uh, like a division problem, it'll get it wrong. Uh, I think, I think I sent you a passage from a chip wenz book. Uh, yeah, you can just give ai a calculator, you know, or it'll write some python code you know, and suddenly it's amazing at math, right, yeah, so ai just gets better and better, with, with, with tooling or improvements. Uh, you know, reinforcement learning, you know better models, better inference, time tools, rag, getting better. So, like you said, I mean, yeah, it's like, yeah, you can't just look at how it's working today and see it hallucinate once and be like, oh, you know, it's never gonna work oh yeah, I mean.
Speaker 2:The classic counter example is like how many times you've asked, you know, maybe it's never going to work? Oh, yeah, I mean. The classic counter example is like how many times have you asked, you know, maybe your teammate or an employee about something and maybe they get it wrong? You know, yeah, what are you just going to discount humans Because one of them got a question wrong? You know? Are you going to fire that person? Probably not, I mean, that'd be pretty heartless. Fire all humans fire. I mean. Yeah, I mean, some ai accelerationist thinks it's what you should do.
Speaker 2:Um, but it's a, it's a classic, you know, upton sinclair quote what is it? It's difficult to get a man to understand something when his salary depends on his not understanding it. Um, so I think that's one of the truisms in our industry, and in every industry really, it's. You know, if you're incentivized to, um, you know, to ridicule or or promote something, that that's what you'll do, often at the expense of having a? Um, uh, you know being able to hold two opposing viewpoints in your mind at once, which, uh, apparently, is a sign of intelligence if you're able to do that. But that's the paradox we're in right now is more and more. Vai tools are getting pretty good, and now this is supposed to be the year of agents, and so we'll see how that goes, but it's only March, so we have some time.
Speaker 1:Yeah, I expect a lot to happen between now and the end of the year for sure, especially with with agents and just the. You know there's so much going on with ai engineering as well and we were, you know we, like I said, we were chatting about that over over text and uh, uh, especially as it applies to data. You know the. If people can just kind of chat with data, they're going to ask different questions than. Okay, I get this pre-canned report with some filters and dimensions that I can apply to it. But if I can just say, hey, really colloquially or casually, tell me how my business is doing or show me which customers you know, show me which customers are happy, show me which customers are unhappy, you know there has to be this person's sort of interpretation of what the user means, right, and then has to go kind of pre process that against the existing you know data models, which comes back to data quality.
Speaker 1:But first, looking ahead, right, you have to come up with the right SQL query. Like this is a classic Texas SQL. So like understanding what types of questions people ask and how it relates to your data and then understanding is that something, is that answer what they're actually looking for or not right, and this is just you know. Naturally, like an agentic problem, right, it requires reasoning, it requires a chain of thought. So do you see agents becoming a big part of BI as well?
Speaker 2:I suppose. So I mean, you got me thinking, though, about something where it's, if you think about dashboards, sort of as we used to think about the three big TV networks back in the day when people would watch TV. I mean, you brought up radio, too, earlier in the conversation, right, so it was that famous LA station KROQ, I think it was. People used to have limited sources of information, right, and in some ways I think that was good, where there was a common set of beliefs and common set of the quorum and standards of ideas. Internet happens we're able to search for whatever we want. Social media happens Now we get into filter bubbles, and so if you draw these same conclusions to being able to chat freely with data but you don't have, I I guess, a deterministic way of knowing which queries people are looking at, the sql queries, that is right, um, I I think this, this could be a very interesting outcome in businesses, because, john, you'll, you'll be looking at your reports and say, well, I asked this prompt here and I got this answer. That's nice, john. I asked this question here and without us knowing SQL, do you think that that's going to improve the situation where BI is already messy?
Speaker 2:People have a lot of questions about data. So this is an interesting scenario you just brought up to me that I was thinking about. Like that, because it reminds me of what happened with um uh news and information in greater society, where now everybody, there is no sense of truth anymore, there is no sense of ground truth anywhere. You can't agree on what. What's a fact right now in in in the news, it's whatever, it's whatever your social media feed tells you. And I think it's interesting because I do think that this actually could happen with these bots, unless there's some sort of governance on the queries, making sure these are consistent. So what does this do for data quality? I mean, you better hope that your data is good quality because bad quality data, with even SQL queries that are slightly different, but different in very important ways, uh, that'll be just very interesting outcomes, um, ones I would shudder to think about if I was running a business yeah, yeah, absolutely, you know it.
Speaker 1:It could just become like an evolution of like the existing bi products. I mean, oh, I didn't even get a chance to look at what open ai put out. I mean we're oh yeah, it's mid-march and they just put out like their first uh, data analysts, uh, analysis agent, uh, let's see, yeah, and it's, it's gonna be. I still don't really think it's possible for ai to really, especially with just how messy data is and, like you know, the relationship between, like what the data says and what's reality within a business. Like you know, even like if you look at like data in salesforce, where you have you know 500 columns that have like similar semantic meaning, but you know you have to know the sales ops uses actually this column, not this one. When they say you know lead source, right, they have like five. You know they have like dozens of columns for lead source, but the one that you Lead source one, two, three, four, 12, yeah.
Speaker 1:So, like an agent, being able to understand that right is almost impossible because everything kind of relies on yeah, that person in sales ops to explain it to me, Right?
Speaker 2:Right. So yeah, I mean you've seen this before, I mean because you live in reality. So it's like this is the kind of stuff you get to deal with all day. I'm a data integration vendor. I'm sure it's just how you see databases. You're like what exactly is this that we're looking at right now?
Speaker 1:Right, that's reality though, yeah.
Speaker 1:I see all these LinkedIn posts I'm sure you've seen them too where it's like oh, bad data equals bad AI. It's really popular to say I see all these LinkedIn posts and I'm sure you've seen them too where it's like oh, you know, bad data equals bad AI, and you know it's really popular to say you know, and it's very true, but I don't know if that's like a and every vendor claims they're the solution to bad data. Of course, right, all you need is one more vendor, a data quality vendor, and suddenly all your, all your problems will be fixed. Uh, but, uh, but I don't.
Speaker 1:I don't know if that's a surmountable problem, right, because just when you see, okay, like the last 30, 40, 50 years of data infrastructure and even these new sophisticated startups, tech startups, startups coming up, I mean the thing that's always the least you know, sanitized and at least structured is like the analytics process, right? So, yeah, you get these curated reports that solve, like very specific burning business problems or questions that you know the data team knows the CEO is going to ask, but just having this you know broad data lake that you can chat with and answer questions, no one's cracked that yet, but it requires a lot of other problems to be solved which just take a lot of brainpower and, honestly, people who have like intuition for how the business actually works.
Speaker 2:Yeah, it turns out, and a lot of these problems that you see in see data sources, for example, or whatever, right, if you look at again the classic continuum of people processing technology.
Speaker 1:Yeah, that, exactly, that's exactly it.
Speaker 2:The culprits of a lot of bad data models, for example. It's not because you don't have technology to do it Right. It's like often it's at least these days that the common thing I hear from people that want to model their data and want to put some cycles through it is I don't have enough time to do it, I don't, I'm not given the time to do this, I don't have, I'm not given the budget, I'm not given the support to do data modeling, and so I just need to cram some stuff into a database, and we'll call it that. It was interesting.
Speaker 2:I was talking to someone recently who runs engineering at a fintech company. This company started out as a startup and now they're publicly traded. He described his job as basically buying more bubble wrap and duct tape to put around the data systems, and that's pretty much what happens on a weekly basis. Decisions were made early at a startup, which at a startup. That's how it goes. Things move fast, but there wasn't the attention to say well, is this thing going to scale when we're successful? And now that we're successful, should we? Maybe not? Maybe we should figure out an alternative to putting bubble wrap on this thing. Nah, just keep putting bubble wrap on it, right?
Speaker 2:So it's always about incentives and outcomes, like if you don't have everyone incentivized to do something, the outcome is well, I guess you get to be a master of bubble wrap and duct tape.
Speaker 1:Yeah, I mean speaking of that, I mean you even brought this up in your community, your Discord and your pod which is that there's a lot of people feeling more burnout in the past in the data industry, more burnout in the past in the data industry, and people in big tech who just feel burned out because there's too much going on, too much. You know bubble wrap, duct tape and you know not enough incentives on a budget.
Speaker 2:You feel like it's worse now than it was in in previous years I think it is, I think it is, and it's not just in tech, really. It's like I think it's just this general sense of malaise I'm seeing with people, no matter where I go in the world. I think there's sort of a Maybe a bit of reflection, I suppose, on what exactly are we doing. When I talk to people at tech companies, for sure, it's definitely a bit of a meat grinder right now you get paid a lot of money, but that's no guarantee that that will be around tomorrow, uh. So you know, I think, when you see, you know, it's such a contrast to where we were in 20 and 2020 and 2021, when, uh, well, around this time actually was about the uh, uh, 50th anniversary of COVID lockdowns. But back on that time, right, like everything is kind of falling out from underneath. Everybody Gave it a few weeks and then all of a sudden, it's like okay, let's hire, like mad, and so that's what happened, and it was a euphoric time. It was like the sky's the limit. I would hear people say that, you know, nothing's going to stop, this is going to keep going, you know. And then it did.
Speaker 2:But what's interesting is a lot of companies are posting, you know, pretty decent profits, sometimes record profits, and these same companies are letting go of workers, and I think that just creates a tremendous amount of insecurity and burnout for sure. I mean, I've done several podcast episodes on burnout and it's a real thing. I mean people who are, you know, making great money by any standard, or just you know they're table flipping, so to speak, and so it is interesting. And then, you know, with the rise of AI, I suppose there's the speculation like well, you know, do we need to hire more engineers? Are we just good doing AI now, because it's really amazing? Yeah, so it's a fascinating time, but you know this is actually happening in, you know, non-technical fields as well. So I just think there's a lot of trepidation about what is? What does AI hold for for the workforce, you know, in the next 10 years, for example? Nobody knows.
Speaker 1:So yeah, and that's why I think your discord is so important, because it just kind of gives people a place to feel community and and chat and you know, just chat with people and and that does make people feel better about burnout it. And that does make people feel better about burnout. It's like do you feel like you're overworked? Yeah, I feel overworked. Why? Because you know there's all this AI stuff around us that you know I have to research every day and answer questions about. You know, while you know the stuff that we built 10 years ago, I'm still maintaining it Right. So it's like, how do you go?
Speaker 2:It's hard. It's hard for people who are at stages of burnout in their career. I mean, I've been there before. It sucks. I mean I decided that I probably wasn't fit to have a proper job, so just never haven't had one for years. But yeah, but that's not for everybody, right? And I think some people find satisfaction in the community that a job brings them and the sense of purpose.
Speaker 2:So, it's definitely interesting right now, for sure. But I think one of the reasons I wanted to start the discord, uh community was just, I think, to provide people a place where they can talk and, um, you know, again, kind of reading the room, right. I think that's what people wanted. They wanted a place where they could have conversations, I think in a very candid way. I don't jump in and censor anybody. Really, I'm like, if you want to, you know, whatever you say is on you, as long as it's not like grossly, like terrible, um, that I might have a problem with it. But I, you know, I definitely believe in setting up a free speech zone for people. They can just say what they want I don't care.
Speaker 2:Yeah, yeah, but uh, yeah, community is important. I think more and more and more it's going to be probably the most important thing there is.
Speaker 1:Yeah, that human connection in the time of AI is going to be much more unique and desired by people. It'll definitely be interesting to see what part I mean. The lazy trope these days is that AI is going to replace everybody. Anyone who's actually practical knows that AI is not even close to doing that, but you do have to figure out how is AI going to come into your workflow? Because I think what will happen is people that refuse to use AI the right way and we haven't decided what the right way is yet but people who refuse to use AI will right way, and we haven't decided what the right way is yet but people who refuse to use AI will probably get replaced to some extent.
Speaker 2:I mean it's a cool person. In the 80s and 90s I remember there were people who just insisted on using paper for everything. I had an old boss who would have all the emails printed out, you know, and he was boss too, so whatever. But I mean the whole point is like you know some uh, his boss too, so whatever. But I mean the whole point is like you didn't. You know, some people catch up with the time, some people don't, I don't know.
Speaker 1:But yeah, and he's all the time. You know it's gonna just create like a new category of productive people, right? So you know, like uh, anthropic ceo dario amode, you know, famously said oh yeah, I'm like you know, within 12 to 18 months, ai is going to replace engineers. Why do you have all these job postings to hire all these engineers that you're supposedly going to replace?
Speaker 2:If you're applying for that job, you'd be like, okay, I have a job, for how long?
Speaker 1:Yeah, writing the code to replace myself forever. The reality is that AI engineers will, will, will be, will, be the future, right people? Engineers who know how to use ai and you know, uh, fin ops. People who know how to use ai and data people that know how to use ai and those those are efficiencies. Or people who just come up with a really good reason why they should never use ai. Um, and they're credible enough where their word is essentially bond with their organizations and they trust them enough to believe that we can't have this process run through an LLM, because it's too critical. But I think the best way to follow along with it is just being active there and thinking about how you can solve real problems.
Speaker 2:Absolutely. Yeah, I mean, lean into it, there's. It's awesome, there's this. The cool thing was, what's happening now is just the rate of innovation, the rate of releases. Is it's mind bending Just how much stuff is announced every, every week. Basically, it's awesome. I mean it's jarring. I can't keep up. I don't know if you can. I mean, I'm just like I'll just find what's interesting to my use case and go with that. I don't have time or the energy to expound on every single possibility of this stuff, but maybe that's the whole point. You just pick what works for you, for your situation and go with it. But there's no shortage of great tools and offerings out there. The models get better every week. So cool, yeah, yeah, you know it's like in the internet, in the internet age.
Speaker 2:It would be like if you had, if you went from like copper, fiber optic that whole transition in like an hour across societies, like that's about how fast things are happening right now. I just made up. Yeah, maybe do the math on it, but it's like it's about that quick. I mean, I grew up on copper modems, I think what 1200 odd or something like that, maybe, maybe less, but it was like that's what I started on on copper line now it's yeah, now I have like eight gig google fiber at my house up and down.
Speaker 2:Yeah, it's like that's that. If you were told me as a there's a uh scrappy teenager hitting bulletin board services in the early 90s that I'd be using eight gig of fiber, I'd be like that's that's. It's impossible. Look how it's like science fiction, right.
Speaker 1:Yeah, it's moving incredibly fast. You just got to find the best way to make it work for you. I actually tried this stuff out with MCP. Like I was saying, that was the big craze last week. It didn't really hit me why it's useful until I went through Matt Palmer at Replit. I ran through his example code and I was like, okay, I could see how this could be useful now. And you know, you ultimately have to just, you know, wade through, you know kind of the AI, slop out there, talk to people in the community, you know, go to that curated list of experts who really can kind of discern this stuff. And you, you know, I think that's the best thing people can do right now and it ultimately will save them time oh yeah, embrace the AI yeah, why not?
Speaker 1:yeah, well, joe, always good catching up with you, man. I'm sure by the next time I see you would have circled the globe four more times. Who knows?
Speaker 2:If I see you next week, then probably not, but who knows? There's a few trips in the docket coming up, yeah, but I'll probably actually see you. I'm going to be in San Francisco soon, so release the area.
Speaker 1:Yeah, let me know when you're here.
Speaker 2:Yeah, we'll do.
Speaker 1:Yeah, a lot going on in San Francisco of course.
Speaker 2:There's so much happening there. It's so cool.
Speaker 1:Yeah, it's definitely. San Francisco is definitely. I mean, I left when it was dead and now I'm kind of back and I can. It was really dead when we left in 2020, like in the middle of COVID, it was like a zombie ghost town, basically. I mean things like in the middle of covid, it was like a zombie, it was like a zombie ghost town. Basically, I mean things were boring. It felt like that I was there, it was nothing going on. Now, now you can definitely feel the like, just like the, the ai and like the builder energy for sure. So I I do mention that to people that it's, it's palpable and you know, I see it every week yeah, you know it's.
Speaker 2:It's so many events going on, just so much action happening, and um, it's, it's awesome. I'm inspired every time I get out there. I also don't really want to live there, so I'm glad I live in salt lake city where I can just take an hour 20 minute flight there.
Speaker 1:Um, but it's uh yeah, exactly yeah, same same story for me, you know, living in la on weekends and san francisco during the week, and yeah, that's kind of the balance that we're that we're looking for. It's a little bit right, but yeah. Well, good, catching up.
Speaker 2:Joe, likewise See you around.
Speaker 1:Yeah, see you around. And thanks everyone for tuning into this episode of what's New in Data. We'll have links out to Joe's Discord and all the other places you can follow him down down in the show notes. All right, joe, see you around. All right, take care.