Introduction

Vanessa Lee | Shopify 00:00:00

There has to be a moment in your career where you say, "Screw it. What is the worst that can happen? They're going to fire me." My philosophy has always been a bit different. I believe PMs provide "courage as a service" to the team. They are the people who make the hard calls and say, "You know what? We worked on this for three months, but I don't think it's the right thing. We have to change course and throw away that work."

Carlos Gonzalez de Villaumbrosia | Product School 00:00:22

There is nothing I like more in a work context than someone pushing back on me—especially if it is someone who isn't "supposed" to be in the room, or someone who is willing to risk their job because they believe in doing what is right.

Vanessa Lee | Shopify 00:00:40

That I'm like, I knew in my gut that we should version the API from all of my experience in Shopify. I was like, I know that this is the right move. That's a pretty magical moment, and I think the people who are successful here, when those moments hit, that's a high, you will never get back. It's like we built just this thing that is so imaginative and creative.

Carlos Gonzalez de Villaumbrosia | Product School 00:01:01

Hey, this is Carlos, CEO at Product School and your host on the Product Podcast. Today's guest is Vanessa Lee, Vice President of Product at Shopify. Shopify is the engine of global commerce, powering over $1.1 trillion in sales and standing as the second-largest online retailer in the US, right after Amazon. They have a market cap of over $165 billion and recently grew revenue 30% year-over-year while remaining profitable.

Vanessa is a two-time YC founder who rose to become the most senior product leader at Shopify, reporting directly to the CEO. During our conversation, Vanessa dives deep into their playbook for transforming from a traditional SaaS company into an AI-native one. We discuss their recent partnership with OpenAI to make e-commerce stores discoverable by agents, how to apply an AI-first mindset to hiring, and what it really means for a product leader to maintain a builder’s mindset. We even get a real-time demo of how they use "evals" to grade AI output. Finally, she shares how she developed the courage to follow her intuition—like the time she convinced her CEO to version Shopify’s API. Let’s get into it. Welcome to the podcast, Vanessa.

Vanessa Lee | Shopify 00:02:12

Thank you for having me, Carlos. It's super fun to be here.

Working with a CEO

Carlos Gonzalez de Villaumbrosia | Product School 00:02:15

You are the most senior product leader at Shopify and you work directly with Tobi Lütke. That is very cool, but I imagine it can also be pretty intense, right?

Vanessa Lee | Shopify 00:02:27

No, it is a walk in the park! I barely even come into the office. No, I'm kidding—it is intense, but it has also been very fun. Tobi and I have worked together for almost a decade, and over the last six or seven months, we’ve worked directly together. It is intense, but it’s gratifying.

Carlos Gonzalez de Villaumbrosia | Product School 00:02:56

Some people in my company say I am "lively," which is a very nice way of saying I am very intense.

Vanessa Lee | Shopify 00:03:03

Exactly! It’s like saying, "You have a lot of energy" or "You’re very passionate." Those are all very kind things to say.

Founder Background and Early Career

Carlos Gonzalez de Villaumbrosia | Product School 00:03:13

You have a really cool story. You went through Y Combinator with two startups before joining Shopify as a Senior PM and individual contributor.

Vanessa Lee | Shopify 00:03:24

Yes, I studied robotics engineering back before robotics were cool, in the early 2000s. As a robotics engineer back then, you learned a bit of everything: software, chemical, mechanical, and electrical engineering, plus sensor work. You come out as a jack-of-all-trades. You aren't necessarily a "great" software or mechanical engineer because you didn't specialize, but you understand the system.

In the early 2000s, there weren't many roles for that. Most teams were siloed into specific engineering departments. However, because we had to context-switch so much in university—going from a material science exam to an operating systems exam—we learned how to learn. We had a lot of grit.

Many of us went into startups. While living in California during an internship, e-commerce was just starting to get popular, but we had a hard time receiving packages. For our thesis project, we built an Amazon-style locker delivery box. We turned that into a company called Buffer Box. I initially accepted a job at Microsoft, but Buffer Box went to YC and was acquired by Google about 18 months later.

That gave me some financial freedom and a bit of a chip on my shoulder. I was 21, working at Microsoft, and I decided to move back to Toronto to do another startup. We went to YC again and spent four years rethinking how senior care could be more efficient. When I wound that down in 2017, I joined Shopify as a Senior PM.

Transitioning to Large Companies

Carlos Gonzalez de Villaumbrosia | Product School 00:06:54

As a founder, I'm curious about that transition from founder to a large company, then back to founder, and then back to a large company again. Why have you stuck with Shopify for nearly a decade?

Vanessa Lee | Shopify 00:07:12

I wasn't very good at sticking with Microsoft; I was only there for about 18 months. I think it depends on the company. When I took the Microsoft job, I was a new grad with student bills to pay—very normal reasons to take a job.

Microsoft was going through a lot of change then. It was Steve Ballmer's last year. I went from a scrappy startup to one of the largest technology companies in the world. That was hard for me. I missed the pace of a startup where you can solve problems directly. At a massive organization like Microsoft, you have to be careful not to "break Wall Street" or compromise the security of airlines using Windows. I worked in Windows security, and it was a bit of a shock to go from building things quickly to writing 40-page PRDs.

However, I met amazing people. I worked down the hall from the original creator of the kernel and talented "white hat" hackers. They were so humble and gracious with their time. That taught me that there is no room for jerks in this industry. No matter how successful you are, you should give back and teach the next generation.

Shopify was a whole different vibe. When I joined in 2017, I was the 21st PM. It was much smaller than Microsoft. I actually didn't think I would stay this long. Healthcare tech was a slow, difficult industry, and I was tired. I wanted a break. I realized I didn't know much about the world outside of my bubble, so I started cold-calling people to learn about different businesses. I met an entrepreneur on Queen Street in Toronto who had started a brand on Shopify. She had put so much blood, sweat, and tears into her business—painting the walls herself and fighting for funding. I admired her courage. I'm an engineer, so I offered to help her with her website. That was my first experience with the product. I reached out to Satish Kanwar, who was then VP of Product, and I joined the following Tuesday.

Founder Culture at Shopify

Carlos Gonzalez de Villaumbrosia | Product School 00:13:25

Working with other founders is like hitting the jackpot. I recently spoke with a GM at Rippling here on the podcast and she said that rippling employs over 100 former founders. That defines the culture.

Vanessa Lee | Shopify 00:13:58

We have many former founders here, whether through acquisition or simply because they joined after their own startups. Shopify's culture is unique because of the level of autonomy that is possible if you know how to tell a story and make decisions. Tobi is so involved in the details that you can get more done here than anywhere else, and at a massive scale. That is addicting for founders.

Being in the Details vs. Micromanaging

Carlos Gonzalez de Villaumbrosia | Product School 00:15:02

Let's talk about being "in the details." Shopify is notorious for that. It sounds better than micromanaging, but tell me how that helps build better products.

Vanessa Lee | Shopify 00:15:22

It's a nice way of saying we definitely micromanage sometimes! The challenge as a company grows is that you can lose the forest for the trees, and many companies build processes to fix that. We've stayed at around 10,000 people for a while, which is deliberate.

Being in the details is difficult. I'm sure some PMs wonder why I care about the specific shade of a shadow or the namespace structure in an API. But for people who were taught that PMing is about being the "CEO of the product," that ego can get in the way. The people who excel here shed that ego. They realize that when smart people tear down and rebuild ideas together, the result is better than what any one person could do alone. Those are the magical moments we fight for.

Finding Leverage Through Sprints

Carlos Gonzalez de Villaumbrosia | Product School 00:17:37

I want to talk about leverage. Being in the details means you have less time for other things. How do you find the leverage to stay involved in things that, on paper, a VP shouldn't be doing?

Vanessa Lee | Shopify 00:18:02

First, we have phenomenal directors and PMs. Second, I do what I call "sprints" with teams. If I have an inkling that something isn't right, I'll suggest we meet every day. It’s very unorthodox, and it can catch teams off guard.

Meeting daily allows me to get into the weeds, ask questions, and fill in my data gaps. I did this recently with our inventory team. I dug into the bugs, the table structures, and the UX. By the second week, I understood the lay of the land well enough that my future involvement could be high-impact but low-effort. It also lowers the ego in the room. By day four, the fear of "meeting with the VP" disappears, and we just focus on the work.

Carlos Gonzalez de Villaumbrosia | Product School 00:20:15

I love that. It defines the culture. You set the tone for what you expect, even when you aren't in the room.

Vanessa Lee | Shopify 00:20:34

I can't do that for every team, but I keep a running list of important areas. As a leadership team, we also meet monthly to review projects. We see every project at least once every two months. This includes Tobi and the entire R&D executive team. That is how we get the signal on where we need to perform deep dives.

AI Native Transformation

Carlos Gonzalez de Villaumbrosia | Product School 00:21:45

Let's do a deep dive into AI. Shopify has been a pioneer in this space, from the CEO's famous memo to building AI-native products. How are you thinking about AI-native products and agents?

Vanessa Lee | Shopify 00:22:12

That memo set off a lot in the industry, but for the team building Sidekick, it wasn't a surprise. Building Sidekick taught me that the job of a PM is shifting. It’s no longer just about writing rigid specs. In a non-deterministic product like an AI assistant, users can ask infinite questions. How do you ensure quality when you can't define every interaction in a PRD? We've had to adjust how we build. You aren't just building the plumbing anymore; you can't just live in a document.

Product Strategy and Storytelling

Carlos Gonzalez de Villaumbrosia | Product School 00:24:04

You’ve said before that you are a "framework speaker" but prefer to tell stories instead.

Vanessa Lee | Shopify 00:24:13

The goal of a PM is uniform: help the team build the right product at the right time. But the how is different for every team and every problem. Frameworks try to codify what worked for one specific team at one point in time to give people a sense of security.

My philosophy is that PMs provide "courage as a service." They inject courage to call out when team dynamics aren't working. They make the hard call to throw away months of work if it isn't right. Stories do a better job of illustrating that than frameworks because they acknowledge that every situation is different.

Demo: Internal AI Quality Tools

Carlos Gonzalez de Villaumbrosia | Product School 00:25:53

I love the focus on courage. I’d love to see you roll up your sleeves and show us how you actually build with AI.

Vanessa Lee | Shopify 00:26:08

This is a perfect segue. I’m starting to do a roadshow of this internally at Shopify. This is a sneak peek at the internal tools we use to build non-deterministic products like Sidekick.

The challenge with AI is knowing if your changes—to the architecture, the prompt, or the integration—are moving the product in the right direction. In SaaS, you have unit tests with expected outputs. With AI, we use an LLM-based evaluation structure. We nicknamed our judge "ACE." ACE looks at every conversation and gives it a score.

The problem is: how do I impart my product opinion to the judge? A PRD used to define what "good" looks like. Now, we use this tool so PMs can define quality by grading conversations over and over.

[Vanessa shares her screen]

This is a portal with simulated data. A PM can look at a conversation between a merchant and Sidekick, see the tool calls the AI made, and annotate it. They grade it on dimensions like grounding, merchant sentiment, and engagement. The PMs actually defined these dimensions themselves. We have two different PMs rate the same conversation to ensure alignment on subjective goals. Once we have this "ground truth" set of training data, we use it to prompt and create an automated judge.

Carlos Gonzalez de Villaumbrosia | Product School 00:31:54

So this, this seems like an internal evaluation tool to improve the, the quality of this non-deterministic feature.

Vanessa Lee | Shopify 00:32:03

Exactly. So this is our internal tool where at this point it's like me and you working together to impart our product sense and our intuition and opinions into these conversations, right? So we'll generate these across different domains, different topics, and then we'll also sample like real live conversations, sanitize them and do the same. And then we take all of these and what we do is we can generate, so let's say I actually, 'cause this is test, I can actually approve this. They get approved by, um, an even more, like, it's usually the lead pm uh, I'm pretending to be a fake user here, which is Charles Babbage. But once you approve it. Then you have a new, you have this subset of what, what's called like training data, ground truth set. This has all of me and you and all of the other PMs on the team. It imparts all of our opinions on what good looks like into this data, right? 'cause we have all that grading and then we basically run what we, we have an automated way of creating and prompting a judge based on this. This is gonna take a while, so this probably won't finish while we're here, but we basically use this to create a judge.

Applying AI Quality Frameworks to Existing Products

Carlos Gonzalez de Villaumbrosia | Product School 00:33:11

Love it. So this makes a lot of sense for, call it an, uh, AI native, uh, feature or product. How, how would you go about, let's call it a product that already existed before ai, where you are also infusing more AI capabilities?

Vanessa Lee | Shopify 00:33:28

Yeah, that's a, actually a really good question. Um, so. Sidekick, let's take flow for example. Okay. Flow is like our, if this, then that our Zapier inside of Shopify. So you can, like say if I get an order coming in that's more than a hundred dollars, tag it as, uh, you know, expedited so that my fulfillment team knows to accelerate it, right? So flow also has, you can manually create flow. Flow existed for many years. Um, this is an existing product before even 2023, right? What we wanted to do is we wanted to give the ability for users to basically generate flows. I wanna be able to say, Hey, I wanna have an automation that tags, you know, tags, orders, and then it'll just generate the flow for you. Right? Which is a lot of ai. It's like an auxiliary AI helping you to use flow. We do the same thing. So this is the part that is, is challenging. It's like, oh, well that works for conversations. It also works. For something like flow. So in this case, we have a judge for flow as well, because there's always an LLM there taking an input and then turning it into a bunch of output. In this case, I think we actually use, um, we actually have a language that defines a, uh, a flow. In this case, you would do the same task of judging, uh, a particular, you know, input. Is the output correct? And then you would say yes or no. And the sub team, in this case, the flow team would do that over and over again. And then they would also create a sub judge just for flow, because all of this is generational, uh, generational, right? So like in segmentation, you're generating. A particular, um, shop ql like you saw in the first conversation that like query language, we have based on an input theme gen, same thing. So we actually do the same thing that you had just seen with conversations, but anytime there's a generation, there's always a judge in place and so we'll just do the same thing, but just in different mediums.

Technicality in Modern Product Management

Carlos Gonzalez de Villaumbrosia | Product School 00:35:21

Got it. Well, there are multiple things that came to mind. Well, first of all, um, obviously as a large company, you in this case chose to build and have your own solution, which obviously makes sense at that scale. And I think. Some people might be thinking, well, but I'm in a smaller company. Maybe we'll have the resources to build something. Well, you can buy maybe not exactly percent what you, it is a hundred percent exactly what you need, but like works good enough. There's so many no code and other tools out there SaaS that can help PMs to get to certain level of sophistication. My second observation is, is like how technical do you need to, you come from a robotics engineering background, but like based on what you demoed here, you don't need to be a robotics engineer Right. In order to make sense of all of this data.

Vanessa Lee | Shopify 00:36:07

Yeah, exactly. Because that, that, that's the part that I think, I mean the concept of building a LLM judge using the ground shoe set and then that judge. Then leading to whatever the engineers are building for sidekick, they test it against the judge. Right? So they're using the judge to, to kind of be like, uh, the unit test, you know, remember test driven development where sometimes people would build tests. That's your judge. So yeah, like I think that it is something that is difficult. We've built this for ourselves and, and for teams across Shopify, um, to, to basically. Accelerate this, but it is something that is, it takes a little bit of time to get used to, but I think that we, we started doing this even before we had this interface, is if you can build a judge, however you can build a judge, right? Um, you can actually start to build using ai. We just scaled it with this interface.

Case Study in Courage: Versioning the API

Carlos Gonzalez de Villaumbrosia | Product School 00:36:56

Exactly. And I'm going to connect this to your previous point around courage because, well, first of all, what you just did takes not only courage. Knowledge. I'm like, you proved it. And I don't know, there's a lot of group VPs out there or CPOs out there that are able to go to this level of detail. Um, but I also want to learn more about how you're thinking about courage, right? I always define this as taking action in the presence of fear. Mm-hmm. And I can imagine, especially for people who maybe don't report to you directly or even for you in some situations when you need to report up, you know, you also have to bring that courage and, uh, find common ways to, to get to a resolution.

Vanessa Lee | Shopify 00:37:39

Yeah, I mean, it, it's not just even for PMs, it's, it's for, I think everyone who has a challenging career, cur courage and self-confidence is the start of everything. And, and that is actually really not acknowledged and not talked about enough in my opinion. Here's like a, I remember the first time I ever met with Toby in a small room, which at this point I think I was maybe a lead, like a senior lead. I think it was like 20. 19, maybe 2018. And I had like, you know, I had talked to Toby in, in larger meetings, but I had never met with him one on like small, in like a four person meeting. And during that meeting, my goal was actually to convince him that we should version our API. Because at that point I had owned our app platform. I thought that in that point, um, I call it. Like the, like I call it kind of your like screw it moment there. There’s gotta be a moment in your career where you’re like, screw it. What’s the worst that can happen? They’re gonna fire me. That’s better than me. Not, uh, me not, you know, making this call and you kind of, for certain problems, you do have to hit that point. So visionary API is a pretty big move. Right? Like that is not something that you do lightly. It has ramifications for internally do are we are teams breaking the API and especially for me being a junior PM at that point, a more junior pm. Um. It was pretty scary. So I, I think it’s, it’s just about knowing so much information that I’m, like, I knew in my gut that we should version the API from all of my experience in Shopify. I was like, I know that this is the right move. So much so that I’m like, ask her like, what’s the worst that can happen? They fire me. Like, it’s like a way to just, just like come out of that. And I think in that 45 minutes we, we decided to version the a p.

Carlos Gonzalez de Villaumbrosia | Product School 00:39:31

So just so for for context, basically you are the up and coming lead PM so you don't report to the super CEO yet. The CEO has a different opinion. So here you are trying to convince your CEO to change his mind.

Vanessa Lee | Shopify 00:39:45

Carlos, he literally told me when I sat down, he’s like, he’s like, what is this meeting about? And I was like, oh, we’re gonna talk about versioning, the API. And he’s like, oh yeah, this is gonna be a short meeting ’cause I’m, I’m not doing that.

Carlos Gonzalez de Villaumbrosia | Product School 00:39:56

So let’s break it down. What, what, did you do in that situation in order to get to a, a resolution?

Vanessa Lee | Shopify 00:40:07

I mean, this is, there’s like a few things. Um, I actually use this story a lot because I do think it, it represents a lot of what earlier PMs struggle with a lot. So one. What I don't say when I, you know, when you see that just, oh. The decision was made in one meeting is I had been working with teams across Shopify who were inadvertently breaking the API for the last 12 months. So I knew more about our APIs, what, how they were breaking, how they were teams were contorting themselves to not break. I basically knew the domain better than anyone else in the company. And I knew that version of the API was correct because I had more knowledge there than anyone in the company. ’cause I spent so much time working in that platform. Um, and so when I went into that meeting. I remember I did two things. One is I already had all of my proof points of like, yeah, you think you’re not breaking the API Toby by not versioning, and that teams are like perfect, they’re never gonna break the API. Here’s four examples of breaking the API that I can, that I worked directly with teams on in the last year. And the other thing that I did was I drew, I haven’t framed, I think at home somewhere in the depths of not like my. I think in my storage. But um, I actually drew out our, the versioning structure. I was like, okay, we, we, I wanna version the API based on calendar quarters. Um, it will continue to be a rolling, uh, uh, a rolling API version, uh, with one year backwards compatibility so that, you know, we can continue to evolve as a platform. All this stuff. So I’d done all this homework, but the way that I drew it, I literally sat down in Ottawa in that morning ’cause I’d flown there and then I drew it on like a piece of paper. ’cause I was like, I don’t. This is an in-person meeting. I was like, I don’t, I don’t have a drawn out. So I drew it. I have it like, it’s like in 2021, you know, oh 1 20, 21 line. And then I had like timelines. It was like very simple. And I did that somewhat on purpose because I wanted Toby to know that I have all of this data, I have all of this stuff. But that the versioning structure exactly was still up for debate. So it’s a format that like, I didn’t spend a ton of time on. I wasn’t precious about, um, but I was convinced we have to do this. But how we do it, I kind of gave him a little bit of room to give him, give me his opinion. ’cause it was just on a piece of paper. And so there’s this piece of like lowering, lowering the, um, the barrier for even him in that moment to give his opinion, right? If I came with a very polished presentation, it would’ve felt like I had. Done all the thinking and that I had finished everything. But by kind of just drawing it out, I kind of gave myself a bit of courage. Say like, oh well, you know, he didn’t like it. It’s okay. Or he had some notes on it, that’s fine. Um, but I also invited him. It is like an invitation for him to also give his opinion ’cause it’s, it’s just a piece of paper. So there’s all these small things that you can do.

Scaling Courage and Product Culture

Carlos Gonzalez de Villaumbrosia | Product School 00:42:57

Yeah, I mean, there’s nothing I can like more in a word context that someone pushing back on me. And I know it can sound counterintuitive, you probably can resonate with this because that, that shows care. Especially if there’s someone who is not supposed to be in the room, you know who Yeah. Giving levels to go and say, you know, I’m willing to get fired because I believe in this and I care about this being Right.

Vanessa Lee | Shopify 00:43:24

Yeah. And I think it, it is actually something that when you scale, it’s hard to encourage that on mass. Right. It’s like how do you, you can find like one or two or three people and yes, ex founders generally do have some of this grit because even just being a founder in and of itself, um, is often a bit of an unorthodox path. And so, you know, they had some courage in there, depending on some years, were very sexy to be a founder, and so maybe not so much those years, but you know

Carlos Gonzalez de Villaumbrosia | Product School 00:43:56

only the first one.

Vanessa Lee | Shopify 00:43:57

Only the first startup. Yeah. By the time you get to the second startup or second year, you’re like, oh, this person had the grit. I actually tell my team, I’m like, look for people who were. A founder, but did it for at least a year. I don’t want the one who like gave up on the third month. Right. So yeah, I think that there’s just a bit of that as like how do you scale that courage and how do you build the culture where it feels like. Oh, Vanessa is gonna always give her opinion, but the way she does it invites me to be like, Hey, can you, like, I’ll be very f if I feel like something’s very firm, I will tell you. But otherwise I’m like, can you like tell me where I’m wrong here? Like what? Because it is actually very hard to scale that.

Influencing Decisions at Scale

Carlos Gonzalez de Villaumbrosia | Product School 00:44:37

Yeah. I, I, I agree with you and I, and I think just getting to the mechanics of this, uh, the tapping into. Like deep emotions for someone to truly pay attention. I think it’s critical. Like obviously we’re talking about a CEO’s pet project, they’re going to care for sure. Or your own pet project, but what if there is something else that is not that as sexy? You know? And, and I think at least when I hear that this is. Going to generate either fear or greed. I tend to pay more attention, right? Like if we are going to die, if we don’t fix this, I’m going to pay attention. Or if we’re going to make a lot of money, if we do this, then I’m also going to pay attention. But kind of meeting in the middle tends to be like, well, I have a thousand other things to worry about.

Vanessa Lee | Shopify 00:45:17

Yeah, you’re you’re right. And it is not just, I mean, that’s how we build products, right? Is like, what about the, what’s the ugly human truth that your user is going to want because users are the exact same. Users are also like, yo, this something’s gotta hit. And speak to me in the first two minutes or I’m, I’m done. Right. But the same is for, um, how you talk. I mean, we’re all humans building for other humans, I like to say. So like even how you think about how do I present information, right. Is, is. Not just internally, but externally. That's a practice that we do a lot as PMs. It’s like what’s the, like what’s the unshakeable thing here for why I care about this so much? And that’s actually really hard to do is like, for example, I care about versioning, the API, because if we don’t our future of Shopify, we cannot evolve our models. Like we cannot change our inventory system, we can’t change our location system. Shopify will stagnate and we will be slower. If we do not give teams a proper release valve to evolve the underlying models of Shopify, and, and that’s, that was my, like, that’s why I care so much about this thing and that’s why I’m willing to like, ah. They can fire me.

Closing Remarks

Carlos Gonzalez de Villaumbrosia | Product School 00:46:28

Vanessa, it’s been a pleasure to have you on the pod and I love seeing your passion, not just your intensity come through in an extremely uh, productive way. So thank you for your time.

Vanessa Lee | Shopify 00:46:40

Thank you so much for having me, Carlos.