3000 Customers, One Bold Pivot: Building the First Generative AI Copilot for Lawyers | Scott Stevenson, CEO of Spellbook

Scott Stevenson is the cofounder and CEO of Spellbook. They launched the first generative AI copilot for lawyers. It's used by more than 3,000 law firms and are growing really fast. They most recently raised $20M Series A from investors such as Inovia, Bling, and Moxxie. 

Scott's favorite book: Zero to One (Author: Peter Thiel)

(00:01) Introduction
(00:06) The basics of legal AI and its applications
(02:57) How lawyers use AI for contract review and drafting
(03:22) Inspiration from GitHub Copilot and AI’s role in legal work
(04:19) AI in litigation vs. transactional legal work
(06:37) Why AI adoption in legal work accelerated
(07:07) The launch story of Spellbook
(07:45) Finding the right problem to solve in legal tech
(10:11) How GitHub Copilot influenced Spellbook’s early direction
(11:42) Spellbook’s first prototype and early traction
(13:11) The moment Spellbook realized it had product-market fit
(14:31) Measuring actual product usage and customer adoption
(15:27) Early learnings from new customers
(17:59) Growth experiments: what worked and what failed
(20:48) Discovering unexpected customer segments
(22:50) Company building philosophy and structuring Spellbook
(26:18) Avoiding over-optimization for structure in startups
(30:21) How to decide what to ship in a startup
(32:21) Pattern-matching vs. narrative reasoning in product development
(35:39) Lessons from AI contract review and LLM usage
(36:14) Rapid-fire round

Transcript

Prateek Joshi (00:01.369)
Scott, thank you so much for joining me today.

Scott Stevenson (00:04.398)
Thanks for having me. It's great to be here.

Prateek Joshi (00:06.891)
Yeah, yeah, it's a legal AI. It's heating up so much and I think the customer adoption has been phenomenal. So let's start with the basics of legal AI. It's a pretty broad term. Legal work has so many facets and components. So can you talk about what are all the different things AI can do within the legal world today?

Scott Stevenson (00:32.782)
Sure, yeah. think the first thing to understand is legal work can broadly be split up into the transactional category and the litigation category. And those are very different. It's almost as different as like civil engineering versus like computer engineering or software engineering. We are primarily focused on the transactional side. So that's, know, business agreements, corporate transactions, &A, real estate, you know, employment.

all of these sorts of areas where there's some kind of agreement or a transaction happening between parties. And that's what I can speak most intelligently about. Of course, we have lots of litigators using our software too, and I can speak a little on that. on the transactional side, a lot of what Laura's working on is drafting and review of documents. you want two clients or two parties want to come to some kind of agreement with

usually some kind of money being involved and they want to make sure they come to a good agreement that protects both their interests in a fair manner. And to do that, you usually are drafting a contract or a set of agreements. So, you know, if venture capital fund invests in us, you know, there's maybe a set of 10 different agreements that we sign as part of that transaction on all sides. And yeah, it's usually not

simply like just generating with AI this agreement from scratch. How lawyers work is they tend to work based on precedent. So if you're doing something like a VC financing transaction, well, there is a set of really common documents called like the NBCA, North American Venture Capital Association docs, which are like preset with all these like different variables and different kind of flavors you can add to them. But then it's still these are like 60 page documents that you have to like edit very meticulously to kind of get the result that you want. So

It's not just going to chat to BT and asking for a contract to be made from scratch. You're usually sort of reviewing and manipulating documents in a very surgical way to get to the result that you're happy with. So the things that you might do on the way, and so this is the main thing that Spelbook helps with is these sort of surgical reviews and edits of documents. And that's a lot. You might have 60 pages and one clause in this document could, if you're at a law firm, you know, it might totally screw over your client. So

Scott Stevenson (02:57.182)
It's a needle in a haystack problem a lot of the time or mini needles in the haystack. I don't know. It's like, you you have to be really careful that you're not letting something through that's going to hurt your client. And then you might want to make modifications, draft new language. Maybe you decide you need a late payment interest rate because you're worried that the client's going to be late to pay. So you draft that and shove it in the agreement. it's some.

Prateek Joshi (03:02.516)
Hahaha

Scott Stevenson (03:22.998)
And all that is a lot. It's a lot like coding. So I come from an engineering background and our inspiration when we launched spellbook was really GitHub copilot. I think working with agreements is a lot like writing software where, you know, one bug can just bring the whole system down and be catastrophic. It's you're dealing with logic. I think the other thing that's interesting in legal is that creativity is actually a negative. Like lawyers don't want a

to create a creative agreement with clauses that have never been seen before, they actually all just copy and paste from each other and they're totally fine with that. They don't want unfamiliar language. They wanna see the common late payment clause that's typically in a commercial lease and that makes it easier for them to review. I code is pretty similar. Engineers are not super picky about writing original code. They wanna write code in a concise, reliable way that works.

Prateek Joshi (03:54.159)
Right.

Scott Stevenson (04:19.354)
I think AI really helps in drafting and review these documents very similarly to how it helps engineers using tools like Cursor or Windsurf, things like that. So that's the primary thing I would say it does. On the other hand, you have deep research problems. So in litigation, you're coming up with a court case and writing a brief. There's tons and tons of case law out there that you have to be able to search and reference to help build your own case.

And that's more of like, yeah, like OpenAI's deep research kind of problem where you have this huge corpus of data. You're trying to find bits and pieces that you can put together to build a great case or a great brief. We don't deal as much with that at Spellbook, but it's another great use case. And the last thing I'll say, like what makes Legal.AI so exciting, I think is, you know, when we started, we were not an AI company. JNI wasn't around. We were

We were a templating company and a document automation company. And our thesis was, everyone's drafting these same agreements over and over. We should just use templates and then we don't need to pay lawyers anymore or lawyers can work 10 times faster at least. actually people have been trying to templatize legal work since the 70s. And there's hundreds of companies that we were competing with and it just didn't work.

And I would say largely like software did not really help lawyers in the practice of law really for the last 20 years. look, like when we first went to raise money, VCs would turn us away and they're like, we just don't think lawyers want to adopt software. They're a bad customer. They're behind. They don't adapt, adopt these templating solutions and things like that. But I think the problem was like,

The software of the past 20 years was all about structured data and databases. Like at the end of the day, most software was database software. So yeah, companies like Clio, which are helping in the business of law of keeping your database of all your bills and your clients and business records. But the practice of legal work is dealing with unstructured texts fundamentally and software was just really, really bad at helping lawyers deal with the reading and the writing of unstructured texts.

Scott Stevenson (06:37.122)
So you had decades where software companies servicing law firms just weren't doing that well. And then all of a sudden there was all this, know, pent up potential energy. Lawyers wanted help. They wanted an escape from the drug jury. And when, you know, LM models, know, like GBD3 came around, all of a sudden you just saw this huge shift in sentiment from lawyers. Holy cow, this can actually help me in my day-to-day work, which is bespoke manipulation of free-form text.

Prateek Joshi (07:07.331)
That's amazing. And I love that journey. Okay. So now, Spellbook, in the last couple of years, you've been on a tear. The speed has been spectacular. So I want to go back to the launch of Spellbook. You mentioned you are running the document automation company. It was okay, not, it was okay. And now on the day of Spellbook launch, how did you decide to launch this? What...

did the MVP look like? And also, what questions did you ask your customers before launch to make sure that this aligns with what they're thinking?

Scott Stevenson (07:45.196)
Yeah. So we first, we were really set on this problem. Like we really believe in like the zero to one thing of like, you have to find a spiky problem that hasn't been cracked yet. And I think that's really important. And we were convinced we had found a spiky problem that was hard to solve that needed to be solved, which is like, from my end, I had received a small business before I received a, invoice from a lawyer and it took out half the cash from our small business bank account. And I was like, Holy cow.

This is horrible. No other service I get feels like the ROI is this bad. And I was like, I think there's a much bigger problem to solve than what I was working on before, which is musical instruments. we were really, really, really convinced that there was a huge problem to solve here, and it was spiky and hard to solve because no one had figured it out yet. So there's this big pent-up potential energy that no one had figured out how to tap into.

And at the time, a little before I started the company, read zero to one. I tried to start things before, and they were too easy, or they weren't spiky enough. And because we were convinced that we found an important problem that was spiky and hard to solve, we had a lot of persistence over a period of years to keep trying stuff. And so probably over three-ish years, a little bit more, we were trying to sell the dock automation product to law firms. We sold it to maybe like 100 and 150.

law firms and we had launched over a hundred landing pages with different variations on this product, different ways it could be sold. And we were ultimately convinced like, yeah, people are buying it, we can sell it, but it wasn't like selling like hotcakes product market fit. And we were like, we're not, you know, spending a bunch of money until we're convinced, like we have really exceptional product market fit where people are pulling the product out of our hands faster than we can keep up with. That was our bar. It's like, we do not have product market fit.

until people are holding the product out of our hands faster than we can keep up with. And we knew we could sell the doc automation stuff to lawyers, but we noticed the time to value was really bad. Like, and time to value was the problem. Like they thought they wanted the product. We deliver what they asked us for. And then they like wouldn't build the templates. They wouldn't use the templates. And like the actual usage was really mediocre. And so we obsessed a lot over time to value. And we noticed like getting lawyers even go to a web app and log in is like,

Scott Stevenson (10:11.468)
ridiculously hard thing to do. You're just not gonna do that for half your users. And so, yeah, we were trying a bunch of things. We knew time to value was a big issue. Lawyers are so busy, they're jumping from deal to deal to deal. They don't have time to set anything up. And then I tried GitHub Copilot really early on and I was like, shit, this is what our customers need because it works right out of the box immediately. It can start providing value on...

on minute zero, you start seeing value. it works basically, it's unconfigured. It works out of the box with any kind of code that you have. And so we saw that, and we knew the big reason lawyers didn't like templates is they just don't work in most situations. Most of the time, lawyers are working on a doc someone else sent them or some old doc they have, and templates are not that useful. And so we put all that together, and there was this spark of, OK, I think

GitHub Copilot for Lawyers is what needs to exist. And this was before, there was no chat GPT at the time either. This was the summer of 2022. And yeah, we actually built a prototype in about two weeks. It was a side project. It wasn't even in our board deck. It wasn't in our plans. We were like, maybe this will be like a cool marketing thing. We'll get some marketing leads by being the first to bring like GPT-3 to lawyers in like a copilot forum. So we basically did like a hackathon, really fast prototype over about two.

two weeks and we basically shipped autocomplete. If you remember GitHub Covalent was just autocomplete originally. And we did that for lawyers where you could start typing a clause like late payment and then it autocomplete and it will like draft the late payment clause for you. And we shipped it with in two weeks, we built the prototype and made a landing page and we made the landing page totally separate from everything we built. So we didn't even advertise any of our other features. It was just on Spellbook, this new prototype we had built.

Prateek Joshi (11:42.477)
Yeah.

Scott Stevenson (12:05.138)
And we started showing it to our existing customers and immediately it was just like a completely different reaction than we had ever seen before. Their pupils were dilating. NFX has this great article called, Find the Fast Moving Water. We might have chatted about before where, you know, the author talks about the first time he saw, think, Cabulus, which was a precursor to like Lyft and Uber. And he just remembers this visceral feeling of like this neurochemical reaction of his brain rearranging to be like,

Prateek Joshi (12:18.403)
Yeah. Yeah.

Scott Stevenson (12:33.872)
this is the future. This is what the future of transport is going to be. And we saw that in our users. was leaning in, pupils dilating. Lawyers are like, holy cow, my entire career is about to change. I don't know what's going on, but it's big. And the fact that we had run 100 landing pages before and seen the other reactions so many times made us be like, wow, this is really, really special. And we've never seen anything like this.

Fast forward three months, we had 30,000 waitlist signups. Every single slot on our sales team's calendar was full. And we went to the board and we were like, yeah, the revenue of this is about to surpass the revenue that we've generated, the ARR from our past three years. We think we should just pivot the company. And everyone was like, yeah, let's do that. So that's kind of what happened.

Prateek Joshi (13:11.151)
Thank

Prateek Joshi (13:20.555)
Right. Right.

Scott Stevenson (13:32.154)
The thing that is most, I think you're back to your more specific question, the big thing that we really care about so much is actual usage. We realize if you're good at sales, you can sell anything. You can sell it to law firms. They'll sign a three-year contract and it'll look like your revenue is retained, but no one's actually using it. The thing we care about really deeply is how many people are using this daily? How many people are using this hourly?

And that's the thing that shocked me the most because you can sell stuff to people, but you can't force them to like use your product every day. They have to genuinely love it. And it's really hard. I've worked on a lot of products. Usually you launch a product, people buy it, they say they like it. And then you see them use it for a couple of days and then they never come back with this. Most things I've launched are kind of like that. And then seeing them actually use it day one and then use it again, day two, use it again. Day three was like, holy cow. They're actually escalating their usage very quickly.

And that's when we knew I think that Spellbook was going to be something pretty special.

Prateek Joshi (14:31.439)
Yeah. And this reminds me, there's a great, I think Orhan Pamuk, he says it, something to the effect of a great artist. They don't just affect you with their masterpiece, but they change the way you look at the world itself. It's basically like your entire brain rearranges in a different way to see, the world is now this. That's amazing. So now, so you launched the signups went crazy and you're getting early customers. Now,

After the new customers who are not with you before, they looked at Spellbook, they came to you, and they started using them. What were the key learnings from the first 10, 20, 50 net new customers who just hadn't interacted with you before? And how did they react, and what were the learnings, and how did that impact your V2, if you will, the thing you shipped next?

Scott Stevenson (15:27.286)
Yeah, sure. People loved it, so it was a lot of just very positive feedback.

I mean, people love that it was embedded in Microsoft Word and that it was there in their own workflow. And that's something that we heard a lot. In terms of the feedback that drove us towards RV2.

Scott Stevenson (15:55.19)
Yeah, this is going to sound like the answer you're looking for, but like. We felt like we could see the future and we were delivering the future to our customers and that they didn't know what was coming and we like. I would say our V2 was not driven by like customers asking for new features. It was like as soon as we had that moment we were like, then we can do this. Then we can do that. Then we can do that. And I don't even think our customers knew what.

was possible or what was coming. They didn't know that AI was so advanced that we would be able to do the next thing. like, I would say our big V2 feature was contract review, where a lawyer could load up a contract and you could be like, hey, negotiate for this for my client and make sure their data is kept secure. And it will actually mark up the whole contract for you in Microsoft Word using track changes, just like a lawyer would. And like no lawyer knew to ask for that. So.

Prateek Joshi (16:49.465)
Right, right, right.

Scott Stevenson (16:49.888)
I think we were just very inspired and we just kept, we felt like the path was very clear. There was a very clear roadmap and we just kept shipping things and the lawyers kept loving it. And I wish I could say we were like customer driven, but I do think, I think people underestimate how much like inspiration, internal inspiration can be important because in these inflection points, it's very hard for customers to imagine what's possible.

Prateek Joshi (17:18.489)
Yeah, no, that's amazing. And that itself is actually a great, great point because when something is as new as LLM, because the entire world was swept up, taken by storm. So they don't even know what to expect until you actually show the magic. That's amazing. Okay. So going from the early days to today, you're standing at 3,000 customers, insane growth, lot of usage. So during this time, going from early days to today, you must have run a bunch of

growth experiments. Some would have worked, some didn't. So maybe pick the one that worked really well and pick one that's supposed to work well. Most people think should work well, but no, it doesn't really work that well.

Scott Stevenson (17:59.886)
Cool, yeah. And do you want me to focus on strictly growth and distribution or combined product innovation as well? Yeah. Yeah, because I think they go hand in hand. So back to your previous question, I'd say the way we find out what's going to work next in the product is we think our thesis is we should be able to show something to a customer and then have this crazy visceral reaction where their pupils dilate. It's a literal like.

Prateek Joshi (18:07.991)
No combined, yeah. Yeah, combined it, yeah.

Scott Stevenson (18:28.446)
neurochemical drug like reaction. so like, the first thing is in product and product marketing is like we keep showing stuff to our customers and seeing if we can elicit that like, totally wow response. And if we don't, then ads aren't going to work, know, landing pages aren't going to work. So like the core is like, do we have a core feature or offering or value prop and

And we don't think about like sell the benefits. Like we don't like selling the benefits. I don't think you generate that reaction by selling the benefits. You generate that reaction by showing with like a little animation or a little demo and they look at that demo and within five seconds they're like, wow, I need this. And like, you don't even need to explain it. Like if your demo is, or your animation or your graphic is crisp enough, I think you don't even need to sell the benefits anymore. Cause I think consumers have become very smart. Users have become very smart.

And so our goal is like, can we put an image or a little bit of copy in front of a customer? And then, and someone would be like, wow, I need that. And we'll, we'll ship those all the time. We test them through ads. So often in product development, we will ship the ad before we ship the feature. So we will, we will make a very small ad and we will put a very small amount of budget on it. And it'll be like an early access landing page. And, you know, we'll have the, an image of the thing we're thinking about building with a very small headline.

And we'll see if anyone clicks it. Most of the time they don't. Most of the time they do not care. And then we don't build the feature, which is really like we might've just saved months of engineering work by doing that. So that's a process that we think about. And if we don't run an ad, we might just show a mock-up to customers and see if we can elicit that reaction. And that's our general approach, I would say, to a lot of product marketing. And that's the core of really everything.

In terms of more growth hacks, I don't know if I can get into the secrets. I don't know if can divulge all our growth secrets. Yeah. I won't get too deep into really specific growth tactics, but I will say the other thing we look for is where we get a momentum where we're not even trying to get momentum. What channels are working when we're not even

Prateek Joshi (20:26.703)
Right.

Scott Stevenson (20:48.782)
trying because it's really hard to get anything to work at all. Our motto in growth is like nothing works. Yeah, what's the other part? Yeah, it's just nothing works. That's what I say to the team. Don't be disappointed when it fails. almost, almost, or no, it's almost nothing works. That's how we think. And almost everything fails. And so if you see someone that's working at all, especially if you're not trying,

Prateek Joshi (21:03.471)
Alright, bye.

Alright.

Scott Stevenson (21:16.174)
you should really investigate it. So an example of that for us was the in-house council market. We saw coming in through our ads and landing pages a lot of in-house council teams. We did not intend to build for that market. We're like, focused. We're for law firms. And so, but one growth experiment for us was like, well, let's just let some of these customers in and see what happens. And it turns out they love the product. They were expanding in the product. And so we slowly scaled out a team to sell to that. And now,

in-house revenue is like 35 % of our revenue very, very quickly. that's an example of not really a specific growth tactic, but opening up to a new segment. Because the thing we notice is they're coming in without us even trying. And that's pretty rare. If you see something work and you're not trying, there's probably something there.

Prateek Joshi (22:03.587)
Right. Yeah, no, it's amazing. The almost nothing works philosophy is basically anything working is a miracle. So treat it as such because most things never work. So I think the goal is to keep trying. And if you see a little nugget of something, you jump on it because as you said, it's almost nothing works. That's beautiful.

Scott Stevenson (22:13.144)
Mm-hmm.

Scott Stevenson (22:22.218)
Yes. Yes. This is like the heart and core, the core of our philosophy because we were wandering so long without that, that much traction. And yeah, anything working is, a total miracle in my, and like the other thing we say is like, you know, executing the plan is 10 % of the work and 90 % of the work is what you do after the plan fails. So that's also how we think about everything. Yeah.

Prateek Joshi (22:38.446)
Yeah.

Prateek Joshi (22:47.439)
Right.

I want to move

Scott Stevenson (22:50.744)
Startups are grim, but once you get climatized to it, you just love it. Yeah.

Prateek Joshi (22:53.943)
Mm-hmm. Yeah, I want to move a little bit to company building and you know your posts on an X LinkedIn It's just they're almost they get philosophical. I love the edginess of it So I want to spend a little bit time on just a couple of different aspects of company building especially coming from from you You've been in the early days didn't work. There was a bit grim and suddenly you are you're on this insane trajectory so I think you have a good appreciation of both phases of

startup life. all right, let's start with structuring a startup. today, how is the team structured in the sense that there's so many things to do, product, marketing, sales, and engineering. So how have you structured Spellbook?

Scott Stevenson (23:41.294)
Good question. I'll start with the philosophical meta answer. Am I allowed to share my screen on this or should I not do that?

Prateek Joshi (23:52.623)
No, go for it.

Scott Stevenson (23:56.298)
I don't know if you normally do that or not. so I'm just pulling up like, there's this like incredibly influential blog posts that, we might've chatted about before that, like really whenever we talk about structure and like how we build a company, I refer back to, and, yeah, it's this, ribbon firm posts, big little idea called, called legibility. And, it talks about how like,

humans tend to over-optimize for legible systems and very organized feeling systems, and then those systems tend to fail. So he gives the example of, you look at a complex and confusing reality like an old city, fail to understand how all those subtleties and that messiness works, attribute the failure to the irrationality of what you're looking at rather than your own limitation to understand it, and then come up with this idealized blank slate vision of what reality should look like.

and then argue that the simplicity and the orderliness of that vision represents rationality and then use authoritarian power to impose that vision. then you watch this rational utopia fail horribly. I think this is like, I think this is what makes most startups like fail is that it's this seeking of legibility and how are you thinking about your engineering and your product process and how you gather customer feedback. think most folks, think, especially if you're coming out of like a lot of

founders are coming out of the academic environment, which is maybe a little bit more legible. You get grades and if you do the tasks properly, you're going to get a good grade. And if you stay organized, you're going to do well. But I think a startup is a very messy thing. And there's a high, high, high cost to creating the sense of structure and sense of legibility. And I think big companies have to do it, but they pay a very high price.

Embodied in our culture is the idea that like actually that the messy organic forest is better and more healthy in a lot of ways The challenge with it is it causes people to feel stressed if they're not used to it like that Honestly, I think a lot of companies create structure because it makes people Emotionally feel like things are under control and that they're better but there's so many startups that you know have perfect board decks like they You know, they they do a board update at exactly the right time every month

Scott Stevenson (26:18.086)
And on paper, they look amazing and they have these really crisp processes that everyone's running, but they just fail and they run out of money. And I think this is so deceptive in a startup that thinking that like making something nicely structured out of Lego bricks is going to like perform because usually it doesn't. And I think if you really focus on the main thing, the main thing is getting the product market's fit, getting this thing out to customers.

And if you empower your team to do that in a more organic way and let your team know also like the mess is okay. We're supposed to be letting fires burn. Like we're not here to like check all the boxes and put out all the fires. If we do that, we've probably lost because we've stopped growing. Like a growing company is meant to have lots of messy, slightly broken things and that's okay for a large number of things. And building that culture of letting people feel okay with that is, you know, I think

the core of how we structure the company comes back to that of very little structure and creating this cultural understanding on the team that we're not here to create a sense of structure because that's often creates this false sense of security and false sense of progress. Another instance is like, you might have this product roadmap and you might feel really good about checking all the things off your product roadmap, but if you're in the middle of the quarter and you find another opportunity,

Like OpenAI dropped the O1 model. We found that using the O1 model, we were able to do whole new things that we just couldn't do before. So we just threw a roadmap out the window. And in 48 hours, we had a new feature using O1 that delivered immense value to lawyers. And that's very messy, but incredibly effective. So teaching everyone to think on the fly, being able to be really agile is really important. So that's how I think about it in a meta way.

a more pragmatic answer. Yeah, we do actually have some team structure. We have a product team. We have a product lead at Semback who's an incredible designer. We have a lot of generalists. have a lot of what I would call product engineers who enjoy having both product taste and building prototypes and also enjoy the deep engineering work.

Scott Stevenson (28:42.062)
Product lead is a designer as well as a product manager. So we have a lot of generalists on the team who are able to sort of shape shift depending on what the time needs. So again, helping us be really agile. We have a sales team. So my co-founder Dan runs sales, have great VP sales, also Dan Wardle. And yeah, we have all the typical.

We have an SDR team, we have an AED team, now we have a mid-market and in-house council team. Nothing super surprising there. then we have marketing, customer success and operations. I don't don't think there's anything too special in those to note. But I should say they're all very loose and with very little management structure.

Prateek Joshi (29:19.533)
No. Yeah.

Prateek Joshi (29:27.599)
All right. Now, I think your first half of the answer, that's the meat. That's what I think we're going for. It's just like what guides your decision-making framework. Obviously, the actual implementation, as you said, it's a standard, but the thing that guides it is important. Now, I've seen many young early-stage founders struggle with just deciding what to ship. Clearly, as a founder, technical founder, they can build a lot.

And if you build in a vacuum for like four months, and then they ship it, and just dead silence. Nobody cares. On the other side, you said if you ask a customer everything that they want to build, many times they don't even know what's possible. So you personally, Scott, the CEO, what are all the inputs you use to decide, hey, what do need to ship? So obviously, customer input is one part, but clearly that's not all of it. So what are all the inputs?

Scott Stevenson (30:21.422)
Yeah, sure. Yeah, I mean, this is a huge question that we think about all the time. First off, as an engineer, I definitely spent a lot of time in the horrible void of like putting my head down and building stuff that I thought was really cool that just completely flopped. I think you have to be incredibly careful of the pattern of like, engineers are really interested in building these like kind of cool modular systems that seem like they can do a lot and this sort of like castle in your head of

if I build this, this and this, it's going to connect together in this really interesting way. You know, these sort of like nerds, nerd sniping type problems. And I was devastated by many building a lot of things like that. It just seemed so cool in my head and no one cared about. And what I, and yeah, I also think that going to customers and asking what they want generally, I think it worked in like 2007 or like whenever the lean startup was originally written.

like it worked because there was this huge new wave of SaaS and so many things to be built that weren't obvious, that hadn't been built yet. And maybe it's happening now again with AI, but there was this drought in SaaS where everything obvious had already been built. And if you go to customers and ask what they want, I don't know, that soil has already been harvested from. And I think that stopped working for a while. Now with AI, there's probably more obvious things that you could ask for your customers to build. But I think most importantly for us, what we think about is we have this

framework we talk about, which is called pattern matching. It's the same as an investing where it's like, I wrote this blog post called pattern matching beats narrative reasoning. it's like, talks about how like why good bets usually are not explainable in like a single sentence. And, know, like here in Canada, know, marijuana is about to be legalized in Canada, therefore I should invest at the ground floor in this new industry and buy, you know, these marijuana stocks and like lot of people got

you know, did very bad and these investments in these companies did very terribly. And these like simple stories tend to get over bet on and they're too one dimensional when the world is actually very hyper dimensional. And there's a lot of reasons why something might end up being good. And, you know, at Spellbook, you know, early on, we thought, you know, let's launch Shopify.

Prateek Joshi (32:21.423)
No.

Scott Stevenson (32:43.224)
for lawyers because, you know, hourly billing lawyers care more about getting business and efficiency. So let's help them sell their services. you know, these things are very contagious lines that you see them in a board meeting or say them in a group and people knob their head and like, yeah, that makes sense. But we found that like thinking about things this way really doesn't work. And what instead we started to think more about is like, really develop developing kind of like features or like patterns that we notice would ship a lot of stuff.

And then note what is in common with the things that are successful. And we did this a lot over a long time. And we saw a lot of things fail and a lot of things succeed. Most things still fail, to be honest. But we've improved our hit rate. And what we noticed is the characteristics of the features that succeeded often were very interesting characteristics that didn't actually have that much to do with the customer. This is an example of a list of features that we shipped.

kind of scores on like the ultimate fit and whether they were successful. Or no, sorry, these scores are generated based on the patterns, but here is kind of based on, here is the success score based on whether the feature was successful. And we noticed things that like a lot of the things we shipped that were really successful felt trivial to the point where like the team would shoot them down because they almost seemed too easy. And it reminds me of like ChatGVT, I think when Sam and the...

OpenAI team was trying to ship chat GPT. I think some of the team I've heard was very skeptical about it, almost seemed too easy. basically like, I think if we built a version of chat too, which is like, just take chat GPT or take GPT-3 and give it a little chat structure and now it's a chat bot and it really wasn't anything. It was actually very trivial to build chat out of the raw GPT-3 completion model. That was one of the most successful things we ever built.

Yeah, we also noticed that a lot of the things that were successful were built in V1 was built in less than two weeks. We noticed that a lot of the things that we built were successful had no pre-configuration required. There was zero configuration required by the user or very limited controls. It's like you hit a button and it works. so every market is different, every user base is different. But I think this is the technique I would use if I were to start a company again. It's like start shipping features, notice what works, what doesn't.

Scott Stevenson (35:09.782)
and then log the common characteristics of the things that are really working. Because this also helps you go against the grain. I noticed a lot of things that worked for us are things that the team actually pushed back on a little bit. A lot of things that succeeded were we felt uncomfortable trusting LLMs to do it. When we did AI contract review using GPT-3, everyone was like, what the hell? This isn't going to work. How is this going to work? And actually, the first version did suck.

Prateek Joshi (35:28.607)
Alright.

Scott Stevenson (35:39.256)
But then GPT-4 came along and it was awesome. So, you know, yeah.

Prateek Joshi (35:39.887)
All right. Right, right. Yeah. Now, this, I really, really like this framework. Like you mentioned, like, narrative reasoning, there's so many fallacies, and you go into this rabbit hole, like, why don't you just, like, do a bunch of stuff, the activity generates data, and just look at the data. I think that's a very good, like, a strong philosophy, so I love that. All right.

I know we're at time, so let's enter the rapid fire round. ask a series of questions and would love to hear your answers in 15 seconds or less. You ready? All right, question number one. What's your favorite book?

Scott Stevenson (36:14.478)
All right, let's do it.

Scott Stevenson (36:19.886)
yeah, this is a trite answer, but I got to say zero to one. yeah, I, I tried investing like active investing my own portfolio over like 10 years before I read that book and, you know, slightly negative returns. I read that book and like immediately my active portfolio did better. started to start startups that were actually successful. yeah, I don't know. It was a life changing book for me.

Prateek Joshi (36:40.399)
Yeah, I know.

It's a great book. Yes, I really, really enjoy that too. All right, next question. What has been an important but overlooked AI trend in the last 12 months?

Scott Stevenson (36:54.542)
I would say the expansion of the context window and using that space for kind of individual user preference tuning. I don't think enough people realize that now that you have this big context window, you can use that space to tune your results for individual users based on their past responses. So you don't need to like fine tune a model for every user. You can start stuffing your context window with sort of different types of preference data and you know.

For something like contract review, contract review is more like a YouTube recommendation than a yes, it's right or no, it's wrong. It's a preference problem. So I think that does not get talked about enough.

Prateek Joshi (37:33.871)
No, I think that's pretty unique. And I agree. think the context window is going to open up so many things. right. Next question. What company do you admire the most and why?

Scott Stevenson (37:44.792)
Wow, this is a hard question.

Yeah, I don't know. I do. I admire OpenAI a lot. I would say I think they know a secret that a lot of people still haven't figured out. And they're really product focused. I think they do this kind of jujitsu with their products that people haven't fully picked up on yet. And they keep innovating ahead of a lot of the other companies.

I think, again, what I talked about in the pattern matching of doing things that feel trivial and that seems kind of stupid and simple, like Chatjibiti was kind of a trivial leap, technically, at least the early version of it. I think they're really good at doing that. yeah, whereas a lot of other technical people are too romantic about, it's not hard enough, or it's not interesting enough.

Prateek Joshi (38:25.678)
Right.

Prateek Joshi (38:43.351)
Right, right, Yeah, that's right. And then they've somehow figured out like B2B, B2C, B2Developers. think it just, and it works equally magically well across many different ways. think it's, yes, I think there's something that's truly working. right, next question. What's the one thing about legal AI that most people don't get?

Scott Stevenson (39:14.592)
I that contracts and legal documents are intentionally ambiguous. this whole idea of smart contracts turning everything into deterministic code, it is a lot like coding, but the real world is too complex for deterministic logic to capture all of the situations that could arise. the ambiguity and the

reform text nature of contracts is actually essential to their functioning and They're meant to you know, eventually be brought to a court or something like that So a contract is really not this like program with like a true false output and and in many ways a contract is also just a snapshot of a point in time of what the spirit of the deal was at that point in time it's not it's not actually as You know Binary as people think

Prateek Joshi (40:10.903)
Yeah, I that. Next question. What separates great products from a merely good one?

Scott Stevenson (40:19.95)
Scott Stevenson (40:26.636)
You know, I think this changes based on like the era of software that we're in. think right now the biggest thing is, are you simple enough to pick up and use? And I think the world is noisier than ever. And I think 50 % of making a product successful right now is like, can you just like wedge into a user's workflow in a really seamless way where they have to do minimal habit change? And I think, I think

habit change and figuring out how to actually change someone's habits to use your software or minimally change their habits to use your software is like the most important thing right now because everyone is just so overwhelmed and distracted.

Prateek Joshi (41:07.981)
What have you changed your mind on recently?

Scott Stevenson (41:13.486)
Not enough, I don't know.

Prateek Joshi (41:17.292)
You

Scott Stevenson (41:22.466)
I think...

The importance of eating well, this is going to sound stupid, but like I took a very utilitarian approach to eating food. And I was like, as long as I'm fueling myself with like enough calories, like I'll be okay. More recently, I've been eating much more healthy and like trying to get the, you know, the macro nutrients and everything. And I'm like sleeping two hours less than night. Like I used to able to sleep nine hours a night and now I feel refreshed after seven hours.

I don't know, I never appreciated just, I've always, I've never eaten like super unhealthy, but I under appreciated just how good I could feel and I've changed my mind on the importance.

Prateek Joshi (42:07.191)
Right, yeah. No, that's a great one actually. It does make a difference. All right, next question. What's your wildest prediction for the next 12 months?

Scott Stevenson (42:20.366)
Scott Stevenson (42:24.11)
I don't know, 10 % of knowledge workers will actually have an AI colleague that they're working with in Slack and in their email, just like a human.

Prateek Joshi (42:35.213)
All right, final question. What's your number one advice to founders who are starting out today?

Scott Stevenson (42:42.162)
yeah, I wrote, I wrote this blog post. It's called Lego mindset versus woodworking mindset. And, it talks about how like, yeah, basically nothing works. if you were ever thinking about the world as a set of Lego bricks and, know, if you hire the right people and snap them in and you use the right dev tools and snap them in that everything is going to work, you were wrong. It's not going to work. Legos are not real. to actually make something work.

takes this sort of relentless sanding. It's more like woodworking. It's relentless sanding to kind of try to force the pieces to fit together. And everything is like that. And when you understand that that's how it's supposed to be, it's not stressful. It's very stressful if you think the world is not like, if you think that the world is supposed to be like Lego bricks, it's very stressful because all your Lego castles will keep falling apart and you'll wonder why. So the world is not like Lego.

Prateek Joshi (43:33.549)
Right. Scott, this has been a brilliant, brilliant discussion. I had like two hours more like worth of material to talk about it. Obviously, it's a separate episode, but I do think I think your philosophy of building, shipping, just worldview, I think we can talk for like three to four hours just on your philosophy of building and shipping and worldview. So thank you so much for taking the time. This has been brilliant.

Scott Stevenson (43:58.286)
This is really fun. Thanks for having me. Appreciate it.