Your VP Is Doing a Rogue Analysis in Cursor Right Now — with Nell Thomas

Nell Thomas has spent two decades in data — from equity research to the DNC to Facebook to leading a 400-person data org at Shopify. She walks Michael and Wes through the modern data stack role by role, gets honest about what AI is and isn't changing about data work, and admits the semantic layer has been her greatest leadership failure. Plus: Sneakers gets the respect it deserves.

Episode Notes
What does it actually look like to run data infrastructure for millions of merchants while the entire industry reinvents itself in real time? Nell Thomas (VP of Data, Shopify) talks vibe-coded dashboards, political campaign data scarcity, blameless postmortems, and why no one should be locking in on an AI strategy just yet. Recorded live in Times Square.

What’s Inside

• Mapping the modern data stack, role by role
• Why data quality is still the #1 problem
• What "good scrutiny" looks like on a data team
• Vibe coded dashboards and the trust problem
• Shopify's MCP for their data warehouse
• The throwaway tech problem in political campaigns
• Why the semantic layer is so damn hard
• Sneakers!

Transcript

0:02

Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Dig into what makes them tick, plus the insights, experiments, and OMG moments that shape the field. On this episode, we sit down with Nell Thomas, who has led data teams at Etsy, Facebook, and the Democratic National Committee, or DNC. And now she's VP of Data at Shopify, which has mandated that AI usage is a baseline expectation for every employee. She leads a team of over four hundred, and she'll be the first to tell you nobody knows the right way to do any of this yet with AI. We talk about the modern data stack, building trust in data organizations, and why the smartest move right now might be resisting the urge to lock anything in with AI, but embracing exploration and discovery. We also, as a bonus, talk a little bit about the movie Sneakers. So I feel like all you sneakerheads, the movie, rejoice. This one's for you. Alright. Nell, welcome on to the test set. So I should say, you're Nell Thomas, and vice president of data at Shopify. But I I also admit, I'm totally enchanted by your your career I don't know the word for it, record choices. Thank you. So you were vice president of data, or you are at Shopify. It sounds so official when you say vice president. I'm so sorry. I didn't know. It's like when you meet Yeah. So I was I just get on one knee. And before that, you were CTO of the DNC Democratic National Convention? Committee. Committee. Committee. I'm so sorry. No worries. Got this. We're good. We're good. And before that, you were at Facebook? Yep. Yeah. Before was Yeah. Nice. Before it was cool. Yeah. Exactly. Or when it was cool. I don't know. Was it ever cool? One of those. We'll we'll figure it out. And then Hillary for America and Etsy. So I know I know it's a long I know. And there's like even, you know, time for that. I'm it's a nice way of me saying I've had a long career. Yeah. Lots of interesting choices. I'm excited to talk about them. I'm excited to be here with you guys. Yeah. Thanks for coming on. I That's all to say. I'm enchanted by your weaving between sort of like political impact and industry work. So maybe just by way of introduction, I'm Michael Chow, and joined by my cohost, Wes McKinney, principal architect at Posit. And we're in Times Square, so there's a very large glowing billboard just rocking behind us. Just to make it more complicated for you distracted. It means I have to up my game so that you I can compete with the Yeah. You're competing with some kind of ad for something. Yeah. But thanks for coming out. Yeah. I'm so excited to talk a little bit about your career, and this really interesting path that you've woven, and also about your work as a as a leader of a a data team and a data group. Yeah. And I think in twenty twenty six, that often involves the emergence of AI and and what that looks like for Absolutely. Yeah. But but maybe to start, maybe you could just tell us a little bit about yourself. Yeah. And yeah, I'm very excited to be chatting with you both, so thanks for having me on. I've been working in data now for two decades, which is a crazy amount of time. And I'm probably could start the conversation at like any point of that journey, but let's say like, from the start, the the most important thing, like, I love working with data. It's always been something that's motivated me. I love finding interesting hard problems, and I think I've been very lucky to have this opportunity to sort of haphazardly, like make my way through a series of different choices. But before we get to the queer stuff, like I'm also a human. I live in New York. I live in Brooklyn. I am a mom of two little girls, which is an important part of my story right now because they are relatively young and take up a huge amount of energy and time, which is exciting and awesome, but also real. Yeah. Yeah. And at Shopify, I run a team right now. It's a little bit over four hundred. It's data infrastructure, data engineering, data science. Some analytics in there. And it's a really fascinating place to be working, especially I joined three years ago, and really been following the arc of of the advent of LLMs and AI, and how that is rapidly evolving, and how thinking thinking about how I run and manage a large team. Yeah. Happy to touch. I can go back on the career in more detail, but Yeah. Yeah. Well, I think I think what's interesting is that, you know, you've been been working as a as a data leader over, like, essentially many, like, mini eras of, like, big companies figuring out how to do analytics and data science in a way that's, like, sustainable and scalable with good best practices, engineering, like engineering mindset. And so, like, I remember, like, you know, I I first got involved in the data ecosystem in the twenty tens, and so that's when people were starting to talk about big data, and then Yeah. It's like, you know, we we heard about, you know, large data prac data orgs, and data data problems at companies like Google, and Yahoo, and Yeah. And Facebook, and and so forth. But then, all the rest of the companies in in in the world were like, okay, we need data teams too. Exactly. How does that work? What software should we be using? Who should we be hiring? Like, what is a data scientist? Like, what does it mean to, you know, to do analytics and to build analytics teams? And so, I think you've had a front row seat to doing that inside, you know, both like, you know, like think really interesting e commerce, you know, businesses Yeah. And as well as, you know, tech, and and you know, tech and politics, and so, I think you have like a pretty interesting perspective on like, how that, you know, that that understanding of how to build an effective data organization has evolved. And I'm sure, you know, with each passing year, it's, especially now, is is changing rapidly. It is funny because it's a weird combination of things changing rapidly and still the same problems at the heart of things. Right? Like, the quality of your data is like always the number one problem. Right? And now maybe more so than ever. But, yeah, it is this interesting both, you know, two different lanes of things. Some things are just consistent backbone of like core issues while we're seeing the tooling and the skills kind of rapidly move. But yeah, when I when I so I graduated college in two thousand five, before like the word data science was even like officially coined as a thing, and I started work. I'll just take a very brief note here. So I graduated with a degree in cognitive neuroscience and psychology, and I worked in psychology labs, running experiments on humans. And that involves obviously learning a lot of stats and doing a lot of work with, and analyzing the data that comes from those experiments. When I graduated from college, all I wanted to do was move to New York City, you know, obviously close to Times Square. And so I took basically the first job that hired me, and the thing I had that was marketable was my stat skills. And so I took a job in equity research, which is a variant of finance, where I was doing large scale data analysis to predict the movement of stocks, basically. And what was the job called? Like what were they called? Yeah. Was called a research analyst. Nice. Yeah. Okay. Yeah. Yeah. But, you know, I as an undergrad, you know, I'd, you know, used SPSS and Stabba, and like, you know, basically, when I when I started at this job, I they basically handed me a book on, this is how old it was, the book, SQL for the workplace. Right? And that was my first job. Was learning SQL. Python came much later in my life, but it was very much on the job learning. And this is what it means to extract value from large data sets and make it meaningful for some outcome. And that was way essence, there's a lot of still what happens and what the goal is, but way before we had some of the terminology we have for now, and right before big data kind of became this craze, and you have this emergence of the crazy data industry that cropped up, I would say, within four or five years. But yeah, but it set the ball rolling in this sort of unexpected way, where I actually found something that I loved to do unexpectedly, and you know, then it took me a few curves to get to the rest of the journey, but I never would have anticipated it when I was an undergrad that that's where it ended So not Yeah. Not to try to go too deep into it, but I'm really curious about when you So you did You mentioned you did social psychology and neuroscience. Yeah. And I'm really curious if, when you started that, if you were planning on becoming a research analyst. Absolutely not. Like, I mean, I I studied I studied those topics undergrads they just were, like, really interesting to I think in my head I always assumed I would back to grad school, and so I was like, oh, out of college, I just wanna get a job, and be out in the real world, and live in New York. And I wasn't too fixated on the topic as much as I was getting out and experiencing Yeah. Working. And so it was pretty random is probably too strong of a word, but wasn't premeditated. I'll say that. But it again, what's funny is the the through line now, in retrospect, is I can really talk about how I've always been really interested in human behavior, understanding it, using data to understand it, which is honestly at the core of a lot of psychological experimentation work. Oh, cool. And so, yeah, there is there is more of a through line than I probably realized at the time. Yeah. I would I mean, I'd imagine, like, I'm curious to hear some of this journey, but yet, running a four hundred person team is maybe very psychological Yeah. In a lot of ways. Oh, a hundred percent. I mean, management and leadership is obviously a lot of, like, dealing with humans. At least right now. There's a lot of yeah, there's a lot of understanding how to help people do their best work. You know, I think a lot about my job right now is creating scale and leverage. So how do I create systems that enable people to do phenomenal work? And that means making sure that no one is limited in what they can get done, but that we're also holding a really strong quality bar for everyone at the same time. And so it's kind of, you know, it's actually, it's a little bit when you think about it from a data perspective. It is a little bit like law of large numbers. Like how do you make sure that you keep the raising the standards while not putting caps on that would reduce quality for anyone? And, yeah, it was a lot where where I spend my time is also just making sure that, people feel like they are energized, they're motivated, they have the tools they need, they have the context they need, and that there is good scrutiny on the work. Because actually I think that's the number one thing I've noticed, is in organizations where people aren't actually paying attention to the data work It's very hard for it to be amazing. Oh, interesting. Because you need, like, a great audience to appreciate it Yeah. And ask questions, and to push it, and deepen it, and challenge you. And so that's like another thing that I always wanna make sure is happening. Yeah. That's I feel like the idea of scrutiny is such an interesting one. Like, don't know in an org what does scrutiny look like. Like, what is You know? Well, and it's also kind of a bad word. Right? Like, I think you hear the word scrutiny, you're like, for real. Like, it feels Yeah. It feels like it feels It's like, check my work, but don't look too closely. Exactly. And like, no one I mean, no one wants to feel like their work is constantly being questioned. Right? Like, that's not a good feeling either, because it suggests a lack of trust. Right? Yeah. And like, people who are have amazing skills, and are curious, and are great problem solvers, you know, they don't want to feel like someone's coming in and being like, but did you add those things together correctly? And, you know. So it's not that type of scrutiny. It's more it's more like people who can appreciate it. Like, maybe thinking about having a really great audience for a play. Yeah. Where doing a play with no one there watching it, like, you gonna do your best work? Right? But it's not just people clapping, it's also people engaging with the material, and being receptive to it, and contributing back to it in a way that has the ability to make someone feel like they want to do better and better work. Right? And they want to keep staying in that conversation. So I think it may be scrutiny I should think about a better word than scrutiny, but it I think it's about attention. Yeah. Anyway, will you ask me what it looks like though? I would say it's making sure that that there are the right forums or rituals. It could be meetings, it could be async things, but where people's work can be like reviewed and discussed and debated, and Yeah. And it's part of the conversation. Yeah. Mean, feel like what's what's part, you know, partly so interesting about modern modern data teams is how how essential data has become to every every aspect of the business and how businesses function. And so, I remember fifteen years ago at conferences, there was this discussion about like, you know, every company needs to become a data company. I think today, you know, the companies that are left are all effectively data companies. And so, using data, making data, both collecting the right data, and then processing and organizing it, and making it accessible in ways where it can be available to all of the people need it, and get the results, you know, the analytics, and the the, you know, the insights from that data in a way that is timely and actionable, so that it can influence decision making, it can be a part of product development, and like all the decisions that get made in the business. And so, there's probably never been a time, you know, greater than now that businesses have been more more dependent on their their data teams to to be able to function. But that also, like, creates this tension where, like, the entire organization is like like our data team, our data team like, you know, they they don't have something they need, like Yeah. They're they're not able to get the answers they need fast enough, something doesn't work quite right, like the tools that they're using to analyze the data, you know, the the dashboards or the query interfaces or whatever tools they're using. If they're not, you know, working in the way that they need to, that does make the business run worse. Yep. And so, that that that does put a lot of like, you know, pressure on the data organization to to deliver and to continue to innovate, and and do better and better to make the, ultimately, the better the data team does, the the better the organization can operate, and that that yields more success in the future. And so, I'm curious, like, how you see that playing out and Yeah. How how you manage kind of those, like, the expectations that the rest of the the organization has on what your your team is building. I mean, that was incredibly well said, and I I one thousand percent agree that fundamentally, like most of the great companies right now are data companies. And how well they leverage that data and the systems they build around it are one of the things that differentiates. So I completely agree. Yeah. I mean, some you know, I like to think about the kind of the value chain of work of data from from raw data creation all the way through to how that data's being used. And one of the nice things about my current role is that, especially leading the data infrastructure team, I get to be pretty far left on that sort of My spectrum is left. From raw data creation, like how that data is ingested, how it's processed, how we make sure that it is kind of prepared for downstream users, both for analytical purposes, or like kind of data science purposes, and for production purposes. So for building ML models or any ML use cases. And that so being really far left on that spectrum is a place of immense leverage, because the better we can create really well, a, make sure we're collecting the right data, which is always a fun problem. Again, one of those age old problems that's been true for my entire career. And making sure that it's captured in a clear way, and making sure that we're actually being cost efficient. That's increasingly an issue now that we see how quickly costs can increase around usage. All of that creates this ability for the whole company to operate better, like you said, whether that's the ML models being built or the analytics that are being pulled out of it. I think in terms of managing the appetite for that, which I think is a harder question, I see that as an incredibly exciting challenge. I'd always rather be working in that environment than trying to convince people why data matters. Right? Which at various points in my career, that has also been part of the job. I think that right now we're in a moment where I don't need to convince anyone that they need to care about that spectrum of data. I think it's more managing the reality of what is possible. Managing the expectations around how quickly we can get what we need, and making sure that we have the strength in our foundations on that left side of the spectrum so that we can trust the outputs. Because increasingly, as you see, it's very easy to work on the right side of the spectrum with vibe coding and lots of ways that you can very quickly do an analysis. Underlying data quality, underlying foundations make everything, it's only positive benefits downstream from that. And so I'm really orienting myself and my team around making sure that is where we have the trustworthy work so that we can make it self-service, we can make it accessible. We have an MCP for our data warehouse, anyone can very easily access that data, making sure we have the right privacy and security controls in place to make sure that's all guarded in the right ways. Those are where I'm really passionate right now because I think it creates it does manage a lot of those things naturally when you focus on trust and quality and systems. And are you saying it sounds like your your side of things is very, like, data engineer heavy, is that right? And analytics engineer heavy? Yeah. Yeah. Yeah. But all of the above, like, yes. Yeah. We have a relatively sizable analytics engineering team that has traditionally focused on the pipelines that we use for all of our data science work and kind of business analytics reporting work, dashboarding, etcetera. We also have data engineering problems that are much more about using that data for some of our production use cases. And again, that's a fun thing about being a company that's at the scale of Shopify is that you have great use cases on all sides of this, whether that's like Yeah. How we're using data that we're getting to understand to predict the likelihood that a visit is a bot, and whether we're using that either for routing that traffic on the edge, or whether we're using that on how we are counting visits. Right? Those are a wide spectrum of use cases. One is for infrastructure purposes, and one is for business reporting purposes. But it's actually the same data. It's the same model. You don't want to duplicate all that. Right? And so that's where having, you know, having that holistic view of the sort of data journey and the types of data work happening creates a lot of value. And I've always thought like data is a very ambiguous term. You'll say, I'm a data scientist. Like, what does that mean? Like, what part of the data stack do you work on? And so I'm more and more seeing the there's a lot of differentiation in that spectrum. But I think that people who are really good have a can kind of run that spectrum in their head at least. And they can see the connection between, hey, this is the raw signal that we have over here, and here are the myriad use cases. And they can create kind of unified solutions that help enable all of them. And so it's it's one of things I I definitely look for when I'm like looking for new issues to the team, like someone who can They don't need to be a specialist in all of them, but they can Can they see that? Play it through. Yeah. Yeah. I mean, I think like one of the one of the problems that I, not to steer the topic, and you know, still wants to kind of stay on this to this kind of platform kind of architecture problem a little bit, but a problem I've been thinking a lot about lately is like, you know, we think data teams do all this work to build, kind engineer this whole pipeline of, like, data collection curation, ETL, data cleaning, data quality, and then engineering the data warehouse, like designing the storage, the the databases, the data access layers, and then the metrics engineering, like semantic modeling, you know. So you get further and further up the value chain, and eventually eventually you start reaching the users and you can start thinking about like, how are they querying the data, like what their dashboards look like, are they using Tableau, are they using some other business intelligence tool? Are they using Python? Are they using R? Like Right. What's their interface? And so, I feel like right now what's really interesting is that we're seeing AI pop up and people using LLMs like in all of those different parts of the value chain. Yeah. And it it also, it's it creates a lot of opportunity, but also, like, there's, you know, in in every place there's an opportunity for like, you know, agents gone wild a little bit. Yes. And so, you know, I can imagine, you know, there are gonna be instances where the end user who One thing that people are talking a lot about these days is this concept of, you know, you're probably well familiar with it, of like headless BI, so like like who says I have to use, you know, the Tableau's UI, like, just give me the endpoints and I'll use Claude Code or I'll use I'll use ChatGPT to vibe code my own, you know, custom dashboard and not knowing that what's inside might be some very large SQL queries that the person building the building their their personalized dashboard, like, you know, they can't read the SQL or they can't judge whether this pile of hairball of vibe coded SQL has errors. And so, they get a dashboard, they're like, oh, it looks looks right. Yeah. It actually contains errors. And so, it's kind of a little bit of a disruption of like, you know, the more heavily curated, like, actually having, you know, an analytics engineer or somebody with data science experience, like building the dashboard and then writing the SQL by hand and making sure that it it, you know, what what you're seeing is like what the reality of like counting things correctly. Yes. I agree. But I'm also gonna challenge that a little bit because I think that I mean, I've observed at a lot of companies, a lot of like overly complicated verbose SQL with business logic business logic embedded at that, like, last mile delivery of a data dashboard as well. Like, you know, it's very easy for someone to I don't wanna throw any particular BI tools under the bus, so I'm like, I'm not, like, saying I'm not naming names, it's very easy for a lot of these tools for you to like actually push your Instead of doing your semantic modeling in the right place, and doing it in a way that's code controlled, and doing it in a place that you can really audit, they're they're putting it like in basically for the presentation layer. And that becomes very inscrutable. It also creates, like, common problem at a company is we have one thousand definitions for this core metric, and you have to you actually have to ask the person who knows the right SQL query to write to get the correct value. And again, that can all best practices, you're handling all that. But you know, lot of companies, there's slow creep, and all of a sudden you have, like, which is the right dashboard to use for x? And this person gave me this number, that person gave me that number. So I don't think the problem that you described of, like, can you trust the dashboard is new with the emergence of AI. That's totally fair. The AI tools can get very creative, I guess, with the They can. SQL they generate. Yeah. And I think, I mean, it's I think it's a not uncommon, but suboptimal value prop that sometimes the data scientists have, which is like, I know this table the best to write the right It's not even like knowing the It's not knowing the code. It's like, I know the nuances of this particular table in a way that's arcane, and I'm the only one who can properly write the where clause that suppresses the types of things that shouldn't be counted in a way that's like You know? And I think that that's not ideal, because then you're encoding a lot of really important information in like one person, as opposed to making it something that a whole team can benefit about. Right? Like, how do you get the system? How do get the scale? So I think everything you described about like how to make sure that you have, you know, really great, you know, ETL, really great data, you know, pipeline orchestration, really, you know, really strong modeling, have semantic layers, how do you have, you know, the right tests to ensure that all those things are running well, like, so so so so important. You know, I think that humans and LMs might be almost just as likely, though, to kind of have some challenges and making sure that you have that consistently trustworthy presentation. Not that I trust LLMs more than humans, but it's more like it's a common trap that everyone falls into. I almost wonder if at this point it'd be useful, because we've talked a bit about like data engineers, analytics engineers Yeah. A little taste of BI in this kind of like big stack. Yeah. Would you be up for almost trying to lay out who's involved in this Yeah. Stack as you see it? Yeah. Yeah. Sure. So I mean, think, you know, all the way on the left, I'd say you really have your, like in my mind, this is just my you have your production engineers who are writing the code that hopefully generates the instrumentation that gathers the data you need, right, whether that's writing to a database or firing an event that is sent off to some sort of log. Then you have your infrastructure, your data infrastructure engineers that are ensuring you're ingesting that data correctly. Right? So whether that's how the events are being fired into whatever Kafka topics, or whether it's you're replicating data over from a database. And then doing the work in building the data platform of frosting that through a whole bunch of storage and compute layers that are doing some sort of modeling of that data. Right? And like usually, you know, at some point you're handing it over from like what is more of a data platform engineer of like creating the house to like the data, the analytics engineer who is then ensuring that the raw ingested data becomes model data, that is hopefully some sort of canonical data asset that then the data scientist can query. Yeah. And just to is it at that point, like, once it hits the analytics engineer, you're in something like BigQuery or dbt. Writing you're writing SQL or writing dbt, maybe you're using Airflow, you know. That I mean, that's, like, the stack we currently use, but, yes, that's very common. Maybe you're using Snowflake instead of BigQuery, but obviously flavors vary, but I think that sort of those big buckets are pretty universal. You have the team that's responsible for maintaining the data house, and then those people are working inside the house, I like to say. Yeah. And then obviously at the very end, have your presentation layers, whether that's Looker Studio in Google or Looker or, you know, there's some fun new ones, but you know, whatever flavor of how that looks. They all have their limitations in my opinion. The other the other use case here that I kinda mentioned in the beginning is also the diversion of some of that data out of the data science use case into sort of more of the production modeling use case. So like, hey, we're gonna model this data to use it for recommendations, for search, for some sort of prediction. Right? And those cases, usually the handoff isn't to an analytics engineer. It might be to a data engineer or to a machine learning engineer. Kind of the the path of coding for the machines as opposed to coding for human input, which is where I think usually most data science and analytics work ends up you know, it's something that a human looks at to review and make a decision about, as opposed to a machine is taking it in to decide about how to rank something or how to display it. Are you saying, like, the analytics engineer is modeling the data for, like, an analyst or people downstream to kinda, like, pull it, but once you get into, like, special cases of, like, modeling, where it kind of requires, like, domain expertise or, like Yeah. I mean modeling skills. I think I think the two to me the differentiation there is a little bit like, one is latency. So like, analytics engineers often working in batch, you know, might have, hey, it's totally acceptable to have a eight hour, twelve hour, or twenty four hour delay in your data being fresh, because you're counting things to do an experiment analysis or do a user behavior analysis of usage of a product or a feature changes over time, but usually have a longer scale. Some of the production or operational use cases I'm talking about, have very low latency requirements. Right? Like, wanna be able to process data very quickly because I wanna see what a buyer's doing on this site to be able to change what I'm showing them on the site. Right. Right? And so that has different requirements for the data infrastructure than you would for the analytics use case. You know, I think what's interesting, I'm trying trying this fork in my head, I guess one of my points here is that further left you go on that, the more unified it should be. And then it kind of starts to fork over time, but you don't want those forked the whole time because you end up with a bunch of data replicated, a lot of costs involved, and probably a lot of lost knowledge and context and value, because you're not getting like great reusable **** learnings between those two streams. It's so helpful I think to lay out the people, like I know it's a lot of work, but it it's like so helpful to hear it Yeah. Kinda laid out. I feel like I feel like oftentimes there's a lot of data talk, and it's hard to like keep track of the roles. Yeah. I feel like it's really helpful to hear. But the the other thing I'm really curious about is we just talked with Tristan Handy from dbt Labs recently, who just tried to break down like, why do people use dbt to someone like Hadley Wickham. Okay. Who's thinking a lot about data analysis with R. Sometimes it's worth asking the question, like, why are we using this tool again? Yeah. Oh, it's always usually good to ask it. People are afraid to it, because they're like Right. They're like, I'm cool. I get it. But I know dbt's relatively recent, like in the last seven and eight years. Yeah. And I I'm really curious how that's kind of like, how you've seen that shift, like, I I'd imagine a place like Etsy, which was I think Yeah. Pre d much so, yeah. What what, like, what was that world like versus Oh, gosh. World today? So that's a so that's a good really good question. And now you're gonna test my memory too, because Yeah. Sorry to I mean Send you back. Etsy's an important part of my, like, I think my evolution as a human, but also specifically as a data person, because it's where I learned to be a manager and grow a team. So holds a very special place in my heart. Very quickly on the laying people out, I think my number one recommendation to any person who's new to a company is to get your hands on an end to end data flow diagram. And what are the technologies being used at every step, and who owns them? Yeah. Really I don't know. Think it's to my point earlier about just having that holistic view, I think it really helps connect thoughts about like where you're Yeah. Like, it's kind of like a food chain, like what are you what are you eating? And are you are you telling people to Google the term modern data stack or something? No. Don't want to divert you from Etsy, but I feel like you brought us there. But I mean, it's you know, because every company has their own flavor of it, you know, and it's like you're gonna you're gonna end the teams that own it are gonna be a little bit different, the terms they use, like Yeah. All that. But if someone's bought, they're like, now I'm in. I'm ready to look at this diagram and end. What are they typing in? Well, mean, you I think this is you should ask someone internally. Like, a new company, be like, hey, like, do you have this? I Yeah. Bet half the time people don't have it. But if they do, like, hey, we're like or or they only have part of it. They're like, oh, yeah. Well, the data's in BigQuery, and then we use this. And you're like, well, how did the data get in BigQuery? Like, know, and then it's kinda like asking that infinite set of questions. Like, well, where did that come from? And where did that come from? Yeah. Where did that come from? Like, it's just tracing every data point from like, you know, somebody clicking on a website to like, how does it end up in this Yeah. How does this event get fired? Like, I literally, like, can see events get fired. Right? Like, oh, this event got fired here. Like, what how long does it take to go through different hops? Where does it go through? I don't know. I just think it's a really interesting, Yeah. It's a way to, like, make tangible. Also, of these things we talk about that are super abstract, like, the data was, like moved from here to there, and then this team does this, you'd probably find like, I don't know, you'll learn something about the data you're using along the way. Yeah. Well, guess like a closely related question to that is like how you've seen like, you know, not only Shopify, but also, you know, kind of other teams that you've led just like navigating technology choice. Because I feel like as time goes by, as time goes by it gets harder and harder, and like the the landscape, I feel like each year, like, you know, it's like we gotta zoom out a little bit, and so now we're at a place where like the modern data stack, you need a microscope like look around at like all the different thousands of technologies that exist and I know. You know, there's open source projects and there's companies and products and things, and you know, I remember there was like a famous blog post years ago, like choose boring technology. Right. That's probably like the right answer most of the And a lot of, oftentimes I can imagine like engineers are enthusiastic about like some new open source project that they discovered on the internet and using that thing. But, you know, often like adopting new technology, any new piece of technology you adopt carries like risk and maintenance, and you know, institutional knowledge and all that. So it's a big topic for sure, but but yeah, curious. Oh, and it's a huge topic. I mean, the the question of of There's the classic like, build versus buy part of this too. And there's the deprecation of legacy tools, which is really important. Often doesn't happen. And there's the sort of fragmentation of the technologies people are using internally, and how much do that do you encourage and permit versus how much do you clamp down on that. I remember when I was at Meta, was people used all sorts of different personal data stacks. Mean, data sources were the same, but then some people were using Tableau, and some people were using Mode. Like, was kind of a Oh, interesting. Wild West of, like, what you know, so I always thought that was kind fascinating. But I'm gonna attempt to sort of answer your question by connecting back to your question on Etsy. Yeah. Yeah. Which is, when I was at Etsy, it was Etsy was very big on like, do it yourself. I mean, kind of per Etsy's Yeah. Was a It's very DIY. A very DIY company. It was founded by Carpenter. It's very much like about, you know, people building craftsmanship and other artisanal work. I mean, like, our our database were all like on prem. Like, it was very it was very like, you know, build build your own stack. So that was true of our data as well. And we were writing I mean, I was not writing, but our data folks were writing like scalding. Like, was Yeah. I haven't heard I haven't heard that name in a long time. Nope. You haven't. Yeah. It had a moment, though. It did have a moment. It did have a moment. We hired, I'm liking his name, but we hired the guy the guy who like was a big Spelding proponent at SU Wish Oscar? No. Was a there's some different ones. Abby, maybe. Yeah. But yeah, we we had a Vertica cluster. This is where we did our yeah, which is also an old technology. Weirdly, I've done two Vertica migrations in my life. Oh, nice. One on Etsy and one at DNC. Like older data stores. So it was a lot more homegrown, like a lot of the tooling generally. It was a little bit deeper in the stack, which actually, again, don't mind because I think it does make people generally When you have less sanitized data, as a data scientist, you're gonna be getting your hands a little dirtier. Think you're gonna learn a Yeah, obviously pre-dbt, you know, people were writing a lot of and some people were writing R, some people were writing Python, you know, people were writing obviously SQL. But a lot of the challenges we had there not that dissimilar to what I see now in terms of where is the data for x, and am I using the right data? And I think with LLMs, those are still the same questions, right? To your point earlier about it, can I trust what the LM is generating? Because I can interrogate what data source they're using, and are they holding it correctly? And there's so much I mean, there's unfortunately, there's a lot of art still in, are you kind of, like, using your dataset correctly because you understand the, like, nuances and arcane details in a way that means you can leverage it correctly? And I think that part is, you know, it's still gonna be, it's an interesting evolution of the challenge for us to figure out how to make sure we interrogate the way LMs are using it, and create kind of visibility on the choices an agentic data scientist is making, and how they use their data, and make that more transparent, and get more testable, and, you know, we can quantify some of their you know, errors, etcetera. Yeah. I mean, one of the things that I, you know, things I'm seeing discussed a lot lately is is agents putting a lot more load and stress on on data platforms, because essentially, like, you know, used to be like you would write a SQL query, you run a SQL query, you look at the results, and by the way, so essentially, like, often, like, the, you know, human in the loop using the data platform, or like using a dashboard, kinda limited by like, human in the loop doing stuff and thinking and using their mind to think about things. And that that kind of creates like a natural, like, kind of rate limiter in terms of like the number of queries and so forth that are being run. But now, you can have, you know, a bunch of agents like interrogating a data platform, and so, you know, in a way, it's almost like a not not differentiable from a DDoS attack in some cases where it's just like Yeah. I run this SQL query over and over until I understand the problem. And so, probably that, I imagine that's gonna change like the way that the data platforms are designed and and with with guardrails to to help the agents like, not DDoS the data platform Yeah. But also like, still to be able to like, be compatible with their needs in terms of like, an efficient agentic loop where they were able to like efficiently get access to, you know Also, LLMs are not good at looking at large amounts of data, so you also like, can't give them very much data to look at at any given time. Yep. So, yeah. Well, again, I think this is where some of the best practices that the industry has been talking about for decades are still the most important best practices, which is like, you know, work out in the open. Like, when you're a data scientist and you're writing code, write first of all, write code. Don't do your analysis in Excel. Right? So write code. Check it into GitHub. Make it observable by others. Have it code reviewed. Store the output of your analyses in a notebook, have it all be visible and accessible so someone can walk through your chain of logic. Like what is the data used, what are the assumptions made, how is the data changed or manipulated to reach some conclusion. And I think that, you know, we need LLMs to be able to follow that same path so that we can follow that, you know, we can follow their reasoning just as much. Like, I will I do get I get shared on screenshots of Cursor outputs all the time, mostly by non data scientists. Right? You know, it's like, you know, I have a I love one of my fellow VPs of the company, but he loves to do a rogue analysis and send me a screenshot and be like, I found the number. And you know, my first question is always like, what data sources are you using? Like And you know, and I think this is where the more we can get, you know, this is all emerging so fast, but the more we can get people on workflows and have their agentic toolkit include some of those best practices that we all know and love, the more we can trust the output. Yeah. It's interesting you made a point earlier about like, you have the value the value of an audience, and also here it's like, it sounds like you're talking about also the value of like an internal like audience, or like putting your work out so people see it. Yeah. I mean it's it really builds a lot of trust. Right? Like when you can show I mean, I mean, to kind of the point you made earlier, Wes, that I kind of like, a little bit challenged you on where I feel like there are a lot of people, you know, a lot of people will just like produce an answer. Like, we ran the experiment, and here is the result. And you get like a slide in a deck. And it's like, then you're forced to just rely that you're okay, trust this human is like doing everything. But hey, actually if we can build a system and a set of tools where it makes all of that experiment work like really easily audible and you can see it and interrogate it, and you can like that inherently builds trust in the platform, and it also allows the data person to do, I think, higher order thinking a little bit. So I feel like that sort of when transparency in your work is the default, it just automatically breeds trust in the results of that work. And it kind of frees people up from that loop of like questioning, like what assumptions did you made when you did this analysis? Or like, what was the time period again? Or like, you know, what ways did you filter or not filter this audience? It bypasses all that so you can just actually have a conversation about what what to do with it. Right? Like, okay, now what do we do now that we have this answer? And so I I find that part of best practices really compelling. Because it's not just governance for governance sake, or dogmatic best practices for practice sake, but it actually speeds up the rate of kind of idea to answer, and getting us to like insight faster. Yeah. I'm so curious. Do you do you find that people do you find so you you have a group of four hundred, which is a huge amount of people to to track, but do you find people are pretty quick and and willing to like be transparent or do these things, or is there some resistance? Like what do you see It's a good question. In people. I mean, I would first first thing I'll say is that I have a huge amount of selection bias in what I know about. Like, this is one of my it's it's one of the hardest things about being and overall, it's a joy and a delight to do my job, and I feel like it's an honor. So it's hard problem is a wrong thing to say, but the challenge of being in a more senior role is that people don't always want to tell you the truth because they feel like they should be managing up in some way, which is a reality. And I also I talk to people who, you know, work for me directly the most, and I talk to your average entry level data scientist the least. And so I'm always careful to how I answer a question like that because I'm like, I would much rather you had, like, you know, someone from my team here to, like, talk about how they think about it, as opposed what I think they think about it. Yeah. That's a very transparent answer, I feel like. You're like, I'm I'm looking at I'm like, I don't know if I trust the data I have. Yeah. I mean, think that I mean, I think one, just in terms of how to encourage a culture where people feel comfortable sharing and working transparently, I mean, to me, the number one rule is making sure that when people share their work, they aren't the dialogue is healthy and productive. Even if there's an element of critical review, it's from a place of good human interaction. People don't want work with people who are jerks. They don't want work with people who are tough or demeaning or being critical of work just to be critical of work, and they want to do it if it's making their work better. So I think there's actually human interaction is really important, and the culture of making it feel like people get excited and have exciting and positive interactions when their work is up for debate, as opposed to feeling like it's only an opportunity for them to get criticized. So like an obvious thing, but like I think it's probably the most important thing, is that you have to like, to be kind of vulnerable in sharing your work openly, you have to trust that it's gonna be received with good intent, and like Yeah. I could see that being a huge factor, like how how it rolls out, and like, is it an environment where people can be Yeah. Transparent, and like, put their work Yeah. Out. Yeah. Yeah. I mean, and, you know, it's easy for me to say that. It's hard for that to happen, you know, at scale too, but I I mean, that's That is definitely one of my corporate beliefs, is that like, you know, you want that sort of like radical candor where people can be honest and say like, you know, say the hard thing, but also do it from a place of like good human trust. Yeah. Yeah. I can't even Honestly, I can't even imagine, at the scale of four hundred data workers, like, it kind of boggles my mind. Like, I'm not sure my mind's ready to wrap around an org of four hundred people cobbling I mean, I I think the to me it seems like, you know, the challenge is is, you know, for for leadership is creating like psychological safety and where people Yeah. You know, feel like, you know, don't don't punish the messenger, like Yeah. People feel people feel comfortable, like, they if they see a problem, they see something's not working well, like, that they, you know, they feel they like they can share her critical feedback about, you know, something they they observe, or something that's not something that's not working without it being like, it's just ultimately the goal is, how do we have a successful organization? Like, how do we do how do we do better? It's not about like trying to point fingers or Yeah. Blame people or say like, oh, this solution that we built six months ago, it worked then, but now it's awful, and like, we need to fix it. And sometimes, like, you know, there's this retrospective looking at at past work, and saying, well, it's it's not working. It worked then, it's not working today. And somehow, sometimes people feel like that is a, like, a judgment on the work that was done before to say like, that was bad work because it didn't scale, or because like it didn't sustain us to the present, and now it's creating a problem that that like, you know, that's a negative judgment on the work that was done before. Sometimes like, you know, if if, you know, you allow kind of, you know, anyone who makes anyone feel that way in an organization, I feel like that's that's bad, like, you know, because again, like, teams have to continuously improve, and like, whenever something, you know, was good was good then, not good now, like, let's figure out something better. Let's make it better and and not be too attached to, you know, you know, sweeping things, you know. I think some, I've seen companies and teams, like, allow things to, like, like, rot and fester, and like, you know, code that bit rots, or like systems that outgrew their usability, or that start to become liabilities, and don't get torn out and replaced because it's like, oh, well, you know, we can't do that because it's gonna hurt somebody's feelings. Yeah. Yeah. And these are a couple of things that really resonated. One, psychological safety. Mean, it's little bit of well worn term, but I think it is the core of your question earlier too, and how you kind of make that as possible for for teams. I think also sometimes people do get really, things become precious. Like you kind of start having these tools that no one can touch, right, or decisions. And I think, one thing I actually really appreciate about Shopify, because it has a real focus on like, our our CEO founder Toby is very focused on like first principles, and like that's what comes first. And that it means that kind of every tool and technology and choice is kind of always up for discussion. And there's no kind of sunk cost fallacy of like, well, we can't talk about that because, you know, we've already invested x amount of time. So it's like the orientation around not like specific emotional choices around tools, but around just like, hey, these are the guardrails of what we want and how we want to get really I think opens up more room for that freedom for people to not feel encumbered. Know, so many so much of organizations is is like you're always dealing with the years that came before you. Organizational design is always a product of past choices that often are random Not random, but they're often unintentional. Like you kind of get into the weird patterns of why does this team report over here? Well, four years ago someone was reporting to someone, they got into a fight, then moved it, or whatever. There's always these weird historical accidents in organizations and in systems. And so the more you can de risk reinterrogating those, I think the better. And I think it creates a much healthier conversation internally. Have you seen big differences since I know you've worked at, like, really different Yeah. Organizations. Yeah. Like, what what's that willingness to, like, interrogate and kinda, like, switch up been like it, some of the different kinda like in the different contexts you've been in? Yeah. I mean, it's a great question. I think that it is I think that it is a culture that is kind of it is a cultural artifact, so I'm kind of meandering around. So a lot of As a technology leader, you spend a lot of your time thinking about the tools you choose to use, but that cultural vibe is actually, I think, almost as important because you're gonna make a lot of wrong tooling choices. But if you can get the right culture, and you can get the right people, you can make a lot of mistakes, and it's me okay because you can switch them out, you know? And so I think, you know, I think I've worked a lot of interesting cultures, I don't wanna call anyone out more than others, but I think that there have been places where there's more freedom to interrogate, and there's less. I think one thing Etsy did very well in terms of its psychological safety was they had a They're very very big on blameless postmortems. And that idea of like, hey, we're gonna do a retro, and we're gonna make a point of going overboard about never ever having this to be about an individual failure. It's always about a system failure, and how did a person get caught up in that system failure? You know, that spread around the company as a way of just thinking about mistakes generally. Right? And so that was a really important cultural touchstone. You know, I think that right now at Shopify, like Toby's investment in just like hiring people who are super high agency, who have like a high curiosity, a willingness to learn, and teach learning like a muscle, like not something you do once, but something you do like every single day, and that that's actually the most important skill set, I think that creates also a culture where it's like, okay, we're not here because we're experts in Rust. We're not here because we're experts in Python. We're here because we're smart people who can like solve hard problems, and like use all the tools that are disposable to do so. Yeah. Know I so I will say I like heard on a podcast one really interesting thing that that you gave. I can't I I think it was about the DNC, but I Alright. Maybe you can correct me about this challenge of kinda like almost like the opposite challenge of like political campaigns where they make a lot of creative use of technology, and then they kind of like wind it all down Oh, yeah. Overnight, and like throw it away. Yeah. Which seems like almost the opposite side of this problem, like innovation just gets like Yeah. Thrown in the garbage. Oh my goodness. It's a huge it's it's like this is yeah. You're I'm sure I said that once in the DNC. It's a really big issue in in political technology. Yeah. So just like to pivot for that for a second, campaigns are these like ephemeral moments. Right? You have a campaign, it gets spun up, it lives as an organization and as a set of tooling choices for, you know, maybe at best, if a presidential campaign and you make it through the primary, maybe it's eighteen months. But like Yeah. More often it might be three months, maybe nine months. Right. And so, yeah, it's a lot of it can it can be a lot of quote unquote throwaway work, and throwaway org structures, and throwaway, like and that is really hard on the — the overall it's really hard on the outcome of wanting to be able to learn continuously, and like leverage those learnings, like cycle after cycle after cycle, so the next campaign can be better at talking to voters than the prior campaign. And that's actually so it's one of the things that you know, there are a lot of great organizations in the political space that try to create that, like, consistency across those cycles. Oh, interesting. DNC being one of them, where it's like, hey, this is like a we can make an investment in our technology and our infrastructure that creates, like, a really strong foundation that these campaigns can build up on. And instead of the foundation being like this, the foundation's actually like this, where every year the foundation's getting better, and so the starting point's getting higher for every campaign. And that, ideally, we take the best of the campaigns and build them into the foundation, and create this really positive feedback loop. Yeah. That's the dream, you know? Yeah. So interesting because, I mean, hearing what you said about sunk cost, and being willing to investigate sunk cost, and then this where like, you're sinking. Exactly. That's the problem. You're like so underwater. Yeah. And the problems in political data, I think, are much harder than the problems in, I would say, like, a lot of the online data companies, whether it's online shopping or online advertising or social media, like, it's a it's a different problem space you're working in, and I think the margin of like error in terms of lost learnings is different as well. And are you do you feel like you're bringing in the same kind of like playbook into those situations as a like data worker? That's a great question. So as a data leader, for sure. Because like a lot of what we talk people are people, and they fundamentally need like a few things, or they need to be able to have a good flow state, they need to have like a bare minimum of good tools, they need to feel like they're respected in their environment and have that psychological safety, you know, they need to feel like they are connected to something that's greater than them. Right? Like, those are usually the core things that people get motivated by. And that's true whether you're working in like tech industry or working in political campaign. I think, you know, obviously political campaign, you probably have a higher likelihood to be super connected to the mission, and then because it's usually you've made the choice to do it because you're really motivated by the candidate or the party or the political topic. So my learnings and expertise, I even it's weird saying that, but my experience managing people I think translated back and forth between politics and tech pretty fluidly, and I think, you know, it always makes me better. I always mess things up, and then I learn from them, and then I get better. Yeah. I think on the the data side, like, it's it is a little I mean, there's obviously a lot of similarities. You need strong data quality foundations. I think the biggest difference is just scarcity of data. So in political campaigns, mostly the outcomes you care about are getting people to vote. Voting is an offline behavior, whether you're in person or whether you're doing it by mail. It's also a very sporadic behavior. Maybe you're getting it once a year, if you're lucky and you have a super motivated voter. So it's usually not the type of voter you're trying to talk to. Right? Yep. Or you're getting it once every four years, or maybe not at all. So that's really hard. Maybe you sit maybe someone's like, oh, but you can obviously just connect them to like all of their online behavior, and obviously magically will know which news sites they're reading, and how they're or like, I bought UGG Boots, and then But you know what turns out, like most people don't spend a lot of their time online. First of all, connecting those things is really hard, and second all, most people don't spend their time most people are not that politically engaged. Unfortunately, you know? And usually the people that you most want to reach are the people who are least likely to be politically engaged. You know? And they are what they're doing is buying, you know, UGG boots, and they're clicking on an ad, and they're watching a dog video. And that gives me very little information about like how to convince them to like some candidate. So anyway, it's it's, I mean, it's a really challenging, super fascinating problem. We like, are some really smart people that are working on it, but it is, you know, in comparison, being at some place like Etsy or Meta or Shopify, it's like us, like, I'm just like spoiled by data. Right? Like you get online, you get so much data about, you know, every little thing someone's doing. And it makes you know, creates different challenges. How do you manage all that data? How do you make sure it's high quality? How do you sift the junk from the the good stuff? How do you, like, leverage in all the right ways for impact? But and the feedback loops are much, much, much faster. Yeah. But, yeah, it's a it's a little bit like, yeah, it's it's a different beast in terms of the type of data you're working Yeah. I also realized this might be a good chance, maybe maybe I'm coming in too late to explain a little bit about who what Shopify is. I think I think a lot of people know Shopify, but I'm realizing we really Shopify is one of those those companies or technologies, like, probably Everywhere. Like, everyone uses it, maybe not a lot of a lot of people don't realize they're using it. Yeah. But also A hundred also put people in the ballpark of when you say data Yeah. Just roughly, what like What company I'm A hundred percent. I know. So, great question. Shopify is it's a platform for merchant, or for a platform for businesses. Primarily, we're selling goods, you know, online or in real life. We also have a retail offering. And it has millions of merchants using the platform to reach customers and consumers. So there are some very big brands that everyone would recognize that use Shopify, like your SKIMS and your Alo Yoga and your Allbirds. But there are millions more of these amazing kind of like long tail merchants that might be in your local neighborhood, like my local yoga studio is a Shopify merchant. Nice. Yeah. They have an amazing little store, and you can buy from there, as well as ones that you might know for subscriptions, things like that. So lots of small businesses and large businesses on Shopify. And it's really the infrastructure. It's the pipes underneath. It's how stores can build a storefront, but also manage their inventory. How they can email and talk to their customers. Right now, there's a whole bunch of work, obviously, to make all of this agentically really easy, so you can like, very, you know, you you can tastefully take the Shop online. Yeah. Yeah. Yeah. And it's really a play of like, hey, if you're you're an entrepreneur building a business, you're passionate about what you're building your business, you're probably not passionate about writing the code to run a website. And so how does Shopify take that off the plate so the entrepreneur can do their best work as an entrepreneur? So the data we have is a lot of online shopping data. Right? So it's millions of buyers making purchases from merchants around the world, and it's an amazing view into online commerce. It's it's one of the major engines of online commerce. So, yeah, data about people are buying is basically the very long winded answer to your Yep. And I love your it sounds like you do data infrastructure for the companies to is infrastructure for shopping. So it's like infrastructure all the way. It is. Absolutely. Yeah. No. So it's it's a lot of infrastructure. And and again, it's this is the first place I've managed this large data infrastructure team. I I did have it at the DNC. I started my career on the, you know, analytics and data science front, so it's kind of like I've migrated down that spectrum to get more and more closer to the Just infrastructure. Out of morbid curiosity, were you like Python or R or what? Python more than R, but like, I mean neither really that well. Honestly, like, I'm just gonna be honest. I mean, I I write I mean, I would say my coding skills were never like my strongest technical I'm sure you were a contender, you know. Though I like to say my very first credit report was VBA, which really dates me because, yeah. Horrified look over here. Oh yeah, was a little VBA. VBA, yeah. A little too much VBA in my past unfortunately. Yeah. Terrible memories. I mean, way I always describe Shopify to people is I think, you know, part of the reason why Amazon got so popular so fast is that people didn't like typing in their credit card information, their addresses into, you know, slightly different ways into every website. And so now, like, whenever I find whenever I shop from an online merchant that uses Shopify, I'm like, oh, like, I don't have to type in all my stuff. Like, it just remembers my my credit card information, and, you know, my my it has my addresses, and it just kinda makes the whole especially, like, makes me feel better about, like, not giving all my money to Jeff Bezos, you know? Fair enough. But speaking of, you know, speaking of agents, I mean, Shopify was in the news last year from, you know, its founder founder Toby, you know, published an internal memo around adoption of, you know, incorporating AI into to everything that the the company does. And so I think we find ourselves in this really interesting moment in January twenty twenty six where, like, I think we had, like, I feel like last year, maybe like March was the Well, there was this inflection point with the release of ClotCode and like the first coding agents, but something in the has like happened in the last sixty days, which has triggered like, and even like I feel like it was like, kinda going like this, and then going like this, and now it's just, like, vertical. It's, like, not even a hockey stick, it's just like a straight line up. And I'm really curious, like, you know, I think there's there's, you know, navigating overall, like, what does it mean to incorporate AI into all the different ways that that a team like yours works. But now, with like, essentially, you know, engineers going into overdrive Yeah. Using coding agents to build things, Also, like, I can imagine that that could create a lot of like, you know, chaos with like a million, you know, agentically engineered side projects, and essentially people wanting to like reevaluate every piece of technology they use, and whether they could replace it with something that's like more bespoke and more specific to like their particular needs. And so I feel like it's completely upended the whole, you know, not for all things, but for certain kinds of things like the buy versus build, you know, calculus. Yeah. Like, you know, could we spend a month and let somebody loose with a fleet of agents, and build something that's a better solution, but it's just for us. Yeah. Yeah. No, it's mean, it it was a pretty big shift when, like, Toby wrote that memo, it was released internally, and then, you know, it found its way outside of Shopify, and I would say I feel very lucky that I was at Shopify at this moment in time because that mandate created a lot of I say it took away a lot of the noise that was happening at the time of like, should one or shouldn't want? Is this like a direct assault to one's craft, one's discipline? Like, what does this mean for the future of work? And I kind of just said, the future is here, like, figure out how to use it. And that's actually fun. When you shift from the fear based mentality of how is this encroaching on my job to a this is another tool, and how do I use it to its fullest potential to do my work? I don't know. It just kind of really shifted the, I think, the feelings around it. And I've seen such incredible work happening internally. I'm continually humbled by how smart and creative the people I work with are. I was one thing that I was incredibly proud of, like, I think Anthropic announced the model context protocol for MCPs in November of twenty four. And my team had built one when I say my team, but it was like, I had nothing to with it, which was like the smart team did it. By like February we had one for our for our data warehouse, which is like So very very quickly, people could start just like doing all of the fun analysis. And I think that was again like a good six months before a lot of other companies were doing something like that. And if a lot of companies still probably don't have that. And so that's an example where I think it's it's creating a lot of play. And I know it's a scary thing, and it feels uncertain, and we don't know where it's going, but it's also fun. And I think that it's kind of creating a lot of opportunities to just try different things out, and that is sometimes rare. And it's, I mean, it's, again, it's part of the cool part of work is when you get to try something and learn something new. So I'm still grappling a little bit as a leader of a large team of how to like watch all the fun and tinkering, but also make sure we're creating scale and leverage because I wanna make sure everyone can kind of be along for this journey, and like, you know, there are a lot of variants right now. So I'm just one of the things I think about the loss the most. I don't have a crazy like, answer right now of how I'm, like, walking that line. I am thinking a lot about how to converge on, like, one default wave. Yeah. Leveraging it. Yeah. But, yeah, it's a hard question. Yeah. I'm having a lot of fun with with with agentic coding. Yeah. Like, was just watching, you know, Peter Peter Steinberg's the creator of, you know, the now viral Claude Bot. Yeah. You know, he gave a talk late last year. The title was, You Can Just Do Things. And Yeah. So, you know, it's kind of like, and I wrote a blog post that's called Why Not? And that was kind of like my new motto of like, oh, you wanna build a thing? You know, why not? Like, try. You know, build something, it doesn't work, it's not good, you can just throw it away. And code, code now has, you know, much less value. You know, it used to be that like code artifacts would be like, oh, this was the product of human labor, and like we can attach a cost to this, and like, you know, it's always funny, like the the code counting pools like Slock and Clock and SCC, They're like, oh, this codebase would cost three million dollars and three years to build, and be like, I built it today Exactly. Yeah. With with my agent. And so it's I think part of what's fun is just the the, you know, not knowing what's the best, you know Yeah. Or like what's what's the right and proper proper way to do things and encouraging like that experimentation and creativity. And I feel like for me, like the almost like, for me it's like it it gives me what 's happening right now, it gives me the same feeling that I got when I started to do Python, which was, you know, like in the late two thousands where Yeah. It's like, I can just can just write code and do things, and so now I feel like fifteen, you know, almost twenty years later, it's almost like that same feeling again of like, you know, I can just, like, can just do things. Yeah. And, but also, like, can imagine in a in a, you know, in a company setting that that that, you know, trying to channel that enthusiasm and that, like, zealotry towards productive, you know, productive ends is gonna be like, you know, in a sense, like, yeah, I guess, probably, you know, maybe the challenge will, or the strategy is to, let engineers enjoy tokens and Yeah. Try to understand, well, you know, that was a good idea, maybe that wasn't a good idea, but we ran a lot of experiments and some of them worked and, you know, probably maybe three or six months from now, some of the like, fervor that's going around, like the zealotry will dissipate and people will be like, okay, this is the best way to use agents to build stuff, and we won't have, you know, Steve Yeggie rolling out, you know, wacky, you know, orchestrators with pole cats and deacons and whatnot. But it's, yeah. I think the uncertainty is part of what makes it so fun. Yeah. No, I mean, also kind of feel like the most of the best things in life, it's like they get more interesting the more you engage with them. And I think that's true of AI and LLMs right now. Like, The first kind of couple of times you do something with them, you're like, oh, is magical. But then it's like, the more you use, the more almost the more magical they feel. And that part of it is actually really exciting, because you kinda wanna keep going down the rabbit holes, employing those those threads, which again, to your point, can be a little bit distracting if you're like, hey, we also need to all do our jobs, and you know, do a back of work. Tickets waiting or Exactly. But I think those two things can go hand in hand. I think that there's ways to I think most of what I see, because also Shopify is a pretty fast paced company with a pretty, you know, rigorous set of milestones to hit, people are mostly doing all that creativity for the purposes of impactful work. And so that, right now, I worry too much about it being a bunch of side quests. I think that they're all pretty aligned. But I do think it's uncertain. I also think, you know, if if six months ago we like locked in and we're like, alright now this is the way to like AI, we would have made a bunch of bad decisions. So I think it is probably premature to be like, hey, here's the right way to do it. And I think also, a lot of people are still just figuring it out. And so I do also think like demystifying the idea that there is a right way to do anything right now is important, because no one knows. Like we're all just stumbling around, and you know, the next Cloudbot's around the corner. Know, it's like, you never know what's gonna emerge like in two months or four months that's gonna change how we thought about things a year ago. Yeah. It's interesting to hear. I think because it reminds me like Charlie Marsh also mentioned in another interview, like, this sense of uncertainty, like, a leader, the immense sense of uncertainty with AI, and Yeah. And I think where you you mentioned like, who knows what'll happen like three, six months from now. Yeah. Like, how do you Yeah. How would you like quantify Not quantify, but, like, if you had to describe how much uncertainty is there, you know, is this, like, a level of uncertainty that's, like, you've seen before, or does it feel like kind of one of one, like That's a great question. I mean, I do think this is, for me, the most uncertainty I felt about how a technology might evolve. I was also, like, I I emerged like from college like post Internet. Like, don't know, people I think some people compare it to like early Internet days, which was like, I didn't quite experience that. So Nineteen ninety five. Yeah. I was still on Wii One then, so. But I think for for my for where I am micro, I think this is the moment of the most uncertainty. What I will say though, a fun fact, is I studied the history of science and technology, and there certainly are moments throughout history where there was immense periods of uncertainty about emerging technologies. So I don't think that this is like one for one in the history of how humans have used tools. I think it's a very big one though, and certainly it's the biggest in my lifetime so far. But it's a pretty common pattern, actually. When a new technology emerges, whether it's the car or the fax machine or whatever, there's a period of misuse of the product usually. Right? I mean, that's a classic example of email, but where technology is changed by the users. And actually if I can quickly do a quick detour into history of technology. I mean, look at this whole thing now, history of technology. That's a crazy credential to drop. You're like, I dabbled in a little bit of history of science technology. This is a fun fact. That was one of my career, yes, side quests. I think like the the sort of like simple narrative of the emergence of technologies is usually like the founder myth of like, hey, this one person invented x, and they knew exactly how it should be, and how it's being used, and it's like, they get all the credit for inventing, like, the thing. But in many, many, many cases, the technologies get fundamentally altered by the people who use them. And that usage of those things changes the trajectory of them in unexpected ways. And that it's very much like a coevolution once technology has emerged between the technical practitioners, the users of the technology, and often regulation. So that's also not unusual, where you get people like how do governing bodies start to play into how the technology can or can't be used. So we're at that moment right now where all of those things are a little bit up in the air, and I think what's very cool is AI is something that is so clearly not like founder myth. It's not like there's one person who's like, and yes, now you all can benefit from my great tool. Like, is and we're gonna see all of these, like, little, you know, mini founder moments of people being like, I just developed that, or this, or that, but it's it's such a it's such a collaborative moment, really, when you think about it, between like all the way different ways people are like sharing what they're doing, and being able to kind of inspire each other. So I I think that part of it's very exciting to see the kind of co like the evolution of this technology. It's so inspiring, think, and like, yeah, for real, like fun to hear from that perspective, like, versus a force that's like threatens to crush you, the like co designing, kind of like exploration. Yeah. And it's an it's a really interesting point too that Yeah. It is interesting so many people are sharing like what they're building and stuff Yeah. Right now that it's I don't It's a really helpful perspective I think to hear. And it is it is. There's a lot that is scary. So I don't wanna, like, gloss over that and be like, oh, it's everything's just, like, all in the F and F. Like, obviously, there's massive, like, existential questions at play. But I think purely thinking about it in the bubble of how do we use this to do better data work, like I think it's a lot of there's a lot of fun. I mean, I think in terms of, you know, being able to create, like in the past, you would you would say, oh, well, we we can't justify staffing a team to build, like, internal tools to help with, like, I don't know, certain types of, like, you know, very particular, like, platform observability, or, like, you know, essentially internal systems which are used, you know, used used for development. And so, you know, think now, like, that that cost equation's, like, you can, you know, there's a tool which can make like these ten people more productive, and you know, in the past, it'd be like, well, we can't have three engineers spend six months building a tool that only makes these ten people more productive, like, they can just get by the way that they way that they are, but now, you know, it is possible for small, you know, there might only be a small audience of people that benefit from creating some some new tool, but if it saves them an hour a day, they spend, they might spend a day building the tool, but then if it saves them an hour a day going forward, like, that's that's material material savings that can direct that direct that that saving, time savings toward, you know, something else that could help the company. I realize I've been speaking a lot in, like, generality, so actually, on that topic, if it's useful to give, like, one more concrete example of, like, how I'm seeing this play out right now, like, a very common data science problem, analytics problem is, why did that metric go up or down? Right? It's like, what happened? And usually the answers to that are like, one, seasonality not being properly controlled for, two, a data pipeline issue, three, there's actually some meaningful movement here that we need to understand. I think that there's so many ways that automations slash AI can really help run through the set of initial hypotheses and jump start the investigation into further into the interesting problem space. Right? And that sense, there is a little bit a playbook most data scientists use of, okay, here's the ways I'm gonna go about investigating. Like, can you get those reps in of faster, kind of like in a less costly way from yourself, you know, some automation, using some AI, and allow you to get faster to the point of the interesting work. And I think that part, that's really cool. We're doing a lot of work right now to try to put better best practices in place and governance in place around defining metrics. It's like basically your good old semantic layer stuff, but defining metrics. Segmentations, reusable segmentations or dimensions that you care about, so like making sure everyone's cutting by the same things. Building better benchmarking tools, so how do you use Prophet in the best way or whatever you're gonna use for time series. And then how do you build alerting and automation on top of that? Like, so that's kind of a good example of like there's a kind of interconnected set of things from the like, how do you define the metric in code, and then how do you, like, get to understanding how that metric is moving in the most meaningful way as possible, and, like, using good practices across all of it, and AI where where it's where it helps get to us a better outcome faster. Yeah. It's interesting to hear sounds like you're saying, like, there's when a metric changes, like, a data scientist kinda has, like, a list they can go through to kinda pull that thread. Yeah. And as it turns out, like, if it's a needle in a haystack, like, AI can be pretty good at kinda like God. At least at least taking a pass at it. At least taking a pass at it. Like, you know, and and especially, like, usually it's gonna be like, alright. Okay. Is it is it like moving in all regions or just a couple regions? So we're gonna look at that. Okay. Oh, it looks like it's mostly moving in, you know, the US. Okay. Like, is it based on the age of the cohort or whatever? Okay. We're gonna like, look at that. Okay. It looks like it's actually primarily coming from this segment. Or no, it's widespread. Like, those are the type of questions that like, you have to have the right model in place, but going through those permutations I think you can use automation to do. So that it's not hand exploratory analysis is so fun and important. You don't want to automate all of that away, but it's like, can you handle can you help get your it's like having a metal detector. Can you get like, okay, it's over in this area. We want to start over here. And that's where you kind of like to jump start so that you're not just doing the macro wrap. And a lot of times people, don't think to cut by, I don't know, x y z's dimension, and so they look at it, that's actually where the interesting nugget was. Oh, interesting. They spend all their time over here. Again, data modeling is still the most important problem in that though, because you have to have, can't do any of that fun automated analysis if you don't have the dib modeled properly. So you mentioned the semantic layer, and that's that's been a hot topic at at various times. I know I don't know. We we can totally skip this, but I also wonder if it's worth just saying briefly what a a semantic layer is to you, just to kinda Yeah. I mean set the stage for who might be listening. Also, we probably shouldn't use this bit, but like, I would say semantic layer decisions have been my greatest failures later. Because yeah, there's anyway, yes. What is spend to failure? I mean, basically it's a way to translate between a I mean, it's my dumb way to explain it. Between a defined metric, or like defined business logic, and the model data that you have. And it's reusable across like the company, right, so that you can always reference the same thing. Some like Looker has a semantic layer that's like built into it, LookML, right, and that is like the magic sauce that you're stitching together, the data sets, and then the sort of like business units on their side. Many companies use different versions. It's Circling around the semantic layer, what the right tooling choices for semantic layer has been something I've been grappling with internally. And I had a lot of conflicting input from my team, and I had a really hard time complicating Yeah. So it's something that we we're re, like, gonna interrogate. Honestly, it's so so helpful to hear, feel like, that Like, if you're grappling with it, I'm sure everybody else in the world is grappling with it too, you know, like Yeah. And it's also a good example where just sometimes I know this is one of the challenges. Like, have a lot of different stake team I consider my team a stakeholder. Like, different components of my team have different needs out of what they want out of it. I have a team that just does what we call executive insights. They're looking at cross company trends. And then I have other teams that are deeply embedded in the product areas. They're working specifically on our our storefronts, or on our checkout, or our shop app. And how what like, their use cases might be different, and people get attached to what they've been doing. They're like, don't give me some new thing. I have my semantic layer I like. I use it over here. And so that's where unification, I think, is really important of getting everyone on the same thing, but it also comes at the cost of people migrating, which is never fun. And so that's where some of my job is to have to make the hard decisions, just be like, yep, we're all doing this, and yes, I'm I'm sorry you have to do it. And but I don't have clarity on what the right tooling choice is, it's the hardest. Because I and then I'm not confident that the juice is worth the squeeze. Yeah. What is there any way you can describe like the difference between some of these tools? Like what Like the difference between different semantic layer up? Yeah. Like what's the variety of flavors you can Yeah. I mean, of them I I know, and I some of it is like how how homegrown your you are versus not. Like there are versions that we could use that were more dbt like reliant versus not was one example. I mean, look at like whether to go on and look her was another decision we had to make. So like there was what is like how much how how much dependency do want on a third party tool, and how much investment do you want to make in your own your own tool. We were also like, we were hitting a limitation with some BigQuery some BigQuery, like there's some notes on how BigQuery was handling, oh my god, I'm gonna like, changes in datasets over time that was like was like we were running afoul on. So there was like, we were hitting technical limitations, we were kind of debating the future of our commitment to Looker, which we are not really committed to, and we were also debating whether or not we wanted to get all these people who were on separate systems onto the same one. So those were we have one internal tool that uses a different one, but it's not like itself, it's a very small amount of code. It's not like it's a major It's not like it's a huge investment, it's just more Yeah. The users had different perspectives on what their most important priorities were. Yeah. Mean, Looker is Yeah. I mean, I would say Looker is probably the most successful semantic layer company. Well, essentially, it's a semantic layer company that that, you know, looks like a b like a business intelligence company, kind of Yeah. Business intelligence and semantic layer in one, and the create, you know, creator of LookML, Lloyd Tabb, it has a new semantic layer called Malloy that I've been Malloy that I've been following, you know, quite quite actively, but then there's, you know, there's other companies like Cube and other ones that people are people are using, and probably others that I'm not familiar with because I'm not I'm not doing, you know, enterprise data modeling Yeah. You know, very much. But but it seems like it's important to have the team invested in whatever, you know, whatever you're using, and that I can imagine there's also, like, the need to have, like, good, you know, telemetry about, like, what's actually being used, and so that, you know, because you could have, like, blind spots where, like, this part of, you know, the semantic layers being maintained really well, and we have good confidence that like the queries that are being written based on it are the right ones Yep. And and and the, you know, the analytics are correct, but then, you know, there might be like dark corners where it just hasn't received the attention, or maybe it's like drifted because there's underlying changes in the data warehouse or things like that. And we've invested a lot in data quality metrics for data warehouse, and so we The analytics engineering team I mentioned earlier is responsible for this. We have a monorepo basically of our data warehouse, so Shopify is also very big on monorepos, so it's a monolithic development And so that team is really focused on the ergonomics of working in the data warehouse and the quality of the data in the data warehouse, and then actually you have to meet quality standards to be in the data warehouse. And there we have a big effort around measuring documentation, completeness, measuring obviously like freshness and timeliness, measuring usage, and kind of a bunch of tests as well about like completeness of data. So that's it's kinda again, that's to me is like that's age old problem, that like it's really important and you can't like, you always have to be investing in. I feel like we've talked about so much from like data cultures to you you kindly laid out the whole stack for us, which I feel like is a Herculean task. And I think attempting to tackle a semantic layer is a special form of health, so I really appreciate it. I didn't expect us to talk about the semantic layer. Yeah. It's it's a tough But I do think, like Would it would it be if we can't mention the modern data stack and not talk about semantic layers? So hot. Yeah. So hot right now. I do think this conversation wouldn't be complete though with the fact that you did mention your last podcast was to talk about the movie Sneakers. You came on as a Sneakers expert. Is that like, on the side, you do a lot of data stuff, but also you you also consult, it sounds like, on podcasts. That's right. I have a very specific movie knowledge about one movie from the nineties. You know, I do love the movie Sneakers. I am not an expert, but I am a, you know, a lover and champion. If anyone has not watched it, it's my people often ask me, like, what advice would you give to someone earlier? Did data occur? And I'm like, watch the movie Sneakers. It's incredible. Yeah. Incredible. I mean, it'll teach you nothing about data, but it's just a good time, and like, it'll give you a good laugh. It'll make your day better. Is that It Yeah. It's like, it's, know, you get good warm feelings, you'll like admire, you know, great actors Is that Do you tell people if they're getting into data work, they should watch Sneakers because that's the right answer? Because that's your answer to everything. You know, like, what's That's a good question. That's the data scientist in me, you know, I'm like, what's this model doing? It's it's a good movie. I I only I only saw it in the last twelve months, and so I made it until now, and it's still it it holds up. I mean, it's it's definitely got some nineteen nineties charm, but for me, like, the nineteen nineties movie that I I've seen the most is Groundhog Day, and so like I have very like Oh, yeah. Very very good knowledge of of that movie, but yeah. I think I would I co endorse Sneakers. I haven't seen it. Groundhog Day is kind of like a data science movie too, because it's like what happens with repeated No, apparently, I didn't realize that there's a whole like, there's a whole like, of cult following of Groundhog Day, and a lot of like alternate theories of like what the movie is really about, and certain things. I was like, really? Like, I think it's, you know, I think it's like a nice it's like a really golf. Yeah. It's like a nice romantic comedy, like who doesn't love? Like, the cast is great, you know, yeah. I love those corners of the internet though. Like, when they're positive. You know, when it's like, oh, this is we're gonna have fun with, going as deep as we can into the mythology of, like, this particular movie. Like, that's that's like people at their best. I do feel like it's closely related to the activity of trying to connect cinematic universes, like Yeah. What if Groundhog Day and Sneakers occurred in the same They were released very close together. Right. Coincidence? Great movie taste. So you need to get it on the ninth I gotta get into Sneakers. Yeah. And I feel like now that we know you're a cool person, maybe maybe to just wind down. How yeah. How do you like to unwind? What's your when you're not watching Sneakers? Flip on Sneakers. I flip on the podcast when you're talking about Sneakers. No. What do I do when I unwind? Well, I mentioned at the top that I have two little kids. So I wouldn't say that's unwinding, but I Fulfilling. Fulfilling. And my actually my recent unlock with them, my youngest has turned three, is that I've introduced them to Mario Kart. Oh my gosh. I'm by no mean like an excellent Mario Kart, you know, I'm an enthusiast, but not an expert. But it's so fun to play with them. They When I say play, I mean, The three year old just like holds the remote, and it like auto drives And my five year old treasure As you're blasting them with shells, is that I know. Well, weirdly, she usually comes in like, eighth, and like my sweet five year old who's trying really hard always comes in the top. Yeah. But it's that's been like very fun to introduce them to like a little nostalgic throwback, and also, yeah, get to, you know, just have some silly fun. It's underrated, just like fun. Yeah. That's so sweet. Yeah. I really appreciate you coming on, and I feel like such Yeah. It's so inspiring to see how you've like woven between different jobs, and and balance like impact, and industry work. Yeah. Really appreciate you weighing in on the the modern data stack, and AI. And I think it helped me wrap my mind around what it means to to lead a group of four hundred people, which I I'll still have to spend some time marinating on. Yeah. I'm still marinating on it. Yeah. All I'm three years in, so you know, we're all still figuring this out. It's that's my biggest thing. It's always a little bit haphazard how we make our way through these traces, so. Yeah. Really appreciate having you. This was fun. This was really fun. The test set is a production of Posit PBC, an open source and enterprise tooling data science software company. This episode was produced in collaboration with creative studio, Agi. For more episodes, visit the test set dot co or find us on your favorite podcast platform.