%20(150%20x%20150%20px)%20(4).png)
Invest with AI
Brett Caughran and Khe Hy lead a deep dive into AI for investing, joined by guests at the cutting edge of the field. Our goal is to be your Sherpa through a rapidly changing landscape by distilling what's working, what isn't working, and how you can leverage AI in your own process. Follow along as we tackle AI's biggest challenges and opportunities, one episode at a time.
Invest with AI
Deploying AI on the Buyside: A 20-Year Engineer's Playbook
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
What’s the biggest hurdle in deploying AI on the buyside? After 20 years of building software, PragmaNexus Founder Matt Stockton has found that the most difficult challenge is knowing your own process well enough to write it down. As he puts it, that’s 80% of the work.
He, Brett, and Khe get into the shift from "single-player" AI (one person, one laptop) to "multiplayer" AI across a whole firm, the voice-memo trick Matt uses to pull his own process out of his head, and why he tells people to go try the one thing they're sure AI can't do yet.
A grounded, practical conversation on what it actually takes to get AI working inside an investment firm.
Timestamps:
[00:00] Intro
[00:59] — 20 Years of Software, From Data Infra to LLMs
[02:23] — Single-Player to Multiplayer: The Local-Machine Problem
[03:18] — Building a Company "Resource Brain"
[05:57] — Folder Structures and Markdown vs. Relational Databases
[07:48] — The Excel Problem: 1.2M-Token Financial Models
[10:46] — The Bitter Lesson of AI Engineering
[12:36] — How Time-Crunched CIOs Actually Stay Current
[15:42] — AI Psychosis and the Aha Moment
[17:27] — Turning a Research Doc Into a Shareable Website
[20:06] — Bucketing the Deployment Problem: Job to Be Done
[23:29] — Investors as Intuitive Pianists: The Articulation Problem
[24:07] — The Voice-Memo Hack for Extracting Your Own Process
[25:22] — Why It's Not a Tech Problem
[28:49] — The Last Mile: Getting From Prototype to Production
[29:42] — Assembling Existing Tools vs. Building Custom
[32:06] — Moving Beyond Basic Synthesis Skills
[34:20] — Skill Creation, Hill Climbing, and the Red Pen
[36:27] — Building Evals and LLM-as-Judge
[40:06] — Debugging the Model: The Goodwill Impairment Trap
[42:25] — The Tool Stack: Claude Code, Codex, and Mobile
[49:45] — Chinese Models, Open Weights, and Token Efficiency
[52:13] — Frontier Intelligence for Judgment, Cheap Models for the Rest
-----------------------------------------------
Want to actually build these workflows yourself?
The AI Accelerator is Fundamental Edge's 6-month cohort for investors who want repeatable AI workflows. Learn More below:
https://www.fundamentedge.com/ai-accelerator
Watch the full podcast series on our site:
https://www.fundamentedge.com/invest-with-ai
Follow Invest with AI on:
Trying something that you think is impossible for AI to do, and it actually does it. And basically, that's when you're building these tools or designing these systems, it's kind of figuring out the right mix of how you structure your data and how you kind of tie these tools or these pages into that data so that they can kind of find what they need.
SPEAKER_01You're working on a model on one laptop with you know the plug-in, you go home to your other laptop, you forgot to sync it, then you're like SOL.
SPEAKER_00All right, welcome to the next episode of Invest with AI, where we explore the intersection of fundamental investing and artificial intelligence. I'm really excited to have Matt Stockton here today. Uh, Twitter is a wonderful animal if you know how to use it well. There's a lot of noise. The noise to signal ratio is incredibly high, uh, but it's a wonderful place to find people smart people sharing smart things about AI. And Matt is one of those accounts that I've followed religiously. Uh, he's been working a lot on the deployment of AI into firms. And so we thought it'd be great to have him here today and talk about some of the things he's working on, some of the challenges he's facing and where we're at in the deployment of AI. So, sort of my mind, I view Matt as a little bit of a one-man FDE shop. FDE is the hottest, hottest thing going right now in in AI. So, Matt, thanks for thanks for uh being with us. Maybe you start with just a little bit of your background and uh and what you're working on.
SPEAKER_02Yeah, sure. No, I'm glad to be here. Thanks for inviting me on. Yeah, so I'm Matt Stockton. Um, I've been building software professionally for about 20 years now, um, and for the last decade, mostly focused in kind of classical data infrastructure, machine learning, um, you know, building trained models, things like that, operating them. And then um, you know, a couple of years back when large language models kind of started hitting the scene, and I got just kind of really deep into them and really wanted to understand how that world connected to uh to classical data stuff. And so just went deep. And um, you know, that kind of turned into me helping companies implement uh uh tools or capabilities using these large language models. Um in my background, I've spent um most of my time bouncing between hedge funds and kind of uh VC backed tech startups uh in the fintech space.
SPEAKER_01So thank you, Matt. And definitely Matt is one of those accounts where every time I read a tweet from his, my IQ goes up uh a little bit. So um, you know what I'm also noticing on this conversation is that the three of us have had the luxury of stepping away from a full-time job to explore this. And it's something that I definitely I know none of us take for granted because imagine trying to keep up with this fire hose of information if you're working 14 hours on a on a trading desk, right? That so uh, Matt, I wanted to start off, you and I have been chatting a bunch about this kind of the latest struggle that I've seen with my training and consulting clients, which I'm kind of loosely calling the challenge to transition from single player mode to multiplayer mode. And um, what I mean by that is it's funny, right? We spent the last decade, 15 years like getting everything into the cloud, everything is shared, everything is SaaS. And then now, you know, with the advent of cowork and code and open claw, everything moves back onto the local machine, right? And so you have two laptops, you're working on a model on one laptop with you know the plug-in, you go home to your other laptop, you forgot to sync it, then you're like SOL. And so, you know, this idea that you know everyone's kind of taking context that they need for their individual contributor task, you know, copying it onto their local machine, running their analysis, then pushing their information back into the share drive, like that doesn't feel like a scalable way of sharing knowledge, of using AI. And so you and I have been talking a bunch about, let's, for lack of a better term, a company resource brain, right? How do you take institutional knowledge that's both in struct, you know, structured data, unstructured data? There's a lot of it over 10, 15, 20 years. It's in people's email boxes. And people want to just talk to it from cohort or code or or codex. Um how should we be thinking about this exercise?
SPEAKER_02Yeah, yeah. No, it's an interesting question. And I think you're you're right in terms of like taking this from single player mode to multiplayer mode. Um, there's a lot of benefits, I think, even from teams kind of sharing their workflows. Like if someone discovers a workflow that's working with these tools, like having a way to share that. But you're right, ultimately it kind of all traces back to the data. Like where is the data living? How are you storing it? How are you capturing it in your workflows? And you know, a lot lots of the smaller firms, like they they're they don't really have rigid or robust processes to do this. Um, and so sometimes you're you're coming in and and you're looking at it and you kind of have to figure out, okay, we have this historical way of storing information. How do we how do we structure that in a way so that we can get utility or value out of these models? And I guess the way that I kind of think about it initially is like you definitely don't want to boil the ocean when you're you're coming in and trying to solve this problem. Like, I like to take a very much like a job-to-be-done approach. Okay, let's try to break down what you're trying to do as a firm or as a company. Let's name those workflows. Um, and then take take one or two of them and try to it try to systematize it, try to operationalize it, try to figure out hey, is the data not landing in a shared spot? How can we make that happen? Hey, is that data not connected to our tools? How can we make that happen? Um, so I I like the concept of this digital brain and talking to your data and all that stuff. Um, but you kind of have to tease apart, you know, what are the sources of information you care about? Are you actually capturing those in your workflows? Um and, you know, just try to make incremental progress, right? Um, and then I think another big thing here is you kind of have to have everyone on board, right? You have to have the team on board if if the way of doing things to get value out of these tools requires you to capture some new data or something like that. You gotta have to, you gotta get people aligned on that process change.
SPEAKER_01Do you think that um, you know, there's a big push, especially in like the power user community, AI community, of you know, really good um folder structures with markdown files, with metadata, with indices, kind of bypassing the very kind of heavily structured, you know, relational databases. Um, do you think that that that works for, you know, that works well for an individual contributor, but does the folder structure with you know good markdown files and tagging, does that can that expand beyond a small group of individuals?
SPEAKER_02Yeah, yeah. It's it's a it's an interesting topic and basically like just kind of drilling out a level, right? When we talk about these models, um you know, they obviously have to have access to data. And the the models, it turns out they're really good at exploring folder structures, like using the tools that are kind of baked into them to open folders and look through things and search files. Um and that's actually really it's really useful for kind of unstructured data. Um, you know, you think about um creating some taxonomy where you're organizing your your your data by company or by theme or something like that, and have have these tools be able to search that. That's super useful. Um, I think there's definitely still a spot for structured relational data, right? You still need databases and data warehouses, especially for um, you know, the numbers. Um, so it's kind of a mix, right? Um, and basically that's when you're building these tools or designing these systems, it's it's kind of figuring out the right mix of um how you how you structure your data and how you kind of tie these tools or these agents into that data so that they can kind of find what they need.
SPEAKER_00One of the things I I've been wondering is, you know, if I have PDFs, PowerPoint slides, et cetera, in a folder structure, I understand how that can connect. You know, most investors live their lives with their nose in Microsoft Excel. And I I had I did an assessment of one of my models and it was 1.2 million tokens. And so when I initially connected SharePoint to Chat GPT, I'm like, oh great, chat GPT is gonna be able to talk to all my financial models. This is amazing. And it it was nowhere close, right? Is that a is that a token issue? And how do you see how do you see uh firms handling that Microsoft Excel issue, like you know, extracting and speaking to you know 17 tab 1.2 million token financial models?
SPEAKER_02Yeah, yeah. No, it's interesting. I I admittedly I haven't spent a ton of time uh connecting Excel workbooks, right? Um a lot of the work that I've done is kind of taking more unstructured information, like transcripts and and filings and things like that. Um yeah, I mean, there's there's definitely, you know, with these models, right? Like you have to figure out um how to both pre-process the information you're giving to them and also kind of like segment your your tasks that you're giving to them. So that there's this concept of a context window with these models, right? Which is how many tokens the model can pay attention to. Um and that is is clearly growing, right? There's there's there's models out there with million uh token context windows. But the fact is, is like the quality degrades the more things these models have to look at. So when I ultimately, like if you if you kind of step back a level and say, hey, I have this data that's gigantic, right? Um my mental model is how do I split apart the task so that I can kind of segment the task to make it a smaller, uh a smaller set of tasks that feed into the end result, um, just kind of breaking apart the task to make it more um uh easier for the model to accomplish each task, right? So, like for example, uh one project that I I I've worked on is taking these thousand-page PDFs of unstructured information and having to build a process that goes into there and actually like distills the information, pulls out very specific aspects of that PDF. Um, you definitely don't want to just point clawed code at the PDF, right? Like you want to uh pre-process it, you want to pull out the text, you potentially want to put that text in a vector database and and and be have the model be able to search it. Um I don't think there's a like a silver bullet, but I think my mental model is if it's not working, if it if it seems like it's not working when you use the model, how can you break apart that that that workflow, that thing that you're trying to accomplish into smaller pieces?
SPEAKER_00That makes sense. Um and you've you've talked a lot about uh the bitter lesson of AI engineering on Twitter. Just walk through sort of your experience with the bitter lesson. Like to what degree have many of these fixes been obsoleted by the next iteration of the model? And where do you think we're we are on that obsolescence curve?
SPEAKER_02Yeah, yeah, it's interesting. So it's it's kind of like um you know, the the speed of iteration uh in this space is so fast that basically you you kind of have to be positioned to um kind of like throw away a lot of your work or or or or repurpose it or redo it. Um and I guess like a couple good examples of this uh in my experience is um you know way back when we're getting these these early models like GPT-4.0, you'd have to do a lot of things to instruct the model on how to get work done. You'd have to be very specific, you'd have to overly prescribe what the model does, right? And now um the newer models, GPT-5. Um you you don't actually want to constrain them as much, right? You actually want to let them breathe a little bit when you provide instructions. And uh, you know, ultimately you you keep having to reinvent things when you're building with these tools, which is which is hard because it's it's it's the um kind of the life cycle of building is shrinking in that you kind of have to reevaluate how you're using these tools. And I think that's just going to continue to happen. You kind of have to have your um, you have to be aware of the techniques that are useful for the time period in which you're building, right? And those change relatively quickly.
SPEAKER_01What um what do you how do you advise folks, like especially ones that don't have the in-house expertise, to you know, like you know, you're mentioning things like RAG and vector databases, and you know, I know enough to to be dangerous to ask one or two good like follow-up questions on that topic, but a full-time CIO, you know, small team of PMs, they they they can't um they can't invest that time. And so how do you like how does someone from the outside know kind of the unknown unknowns to be dangerous from an infrastructure, AI infrastructure perspective while not getting lost in the weeds, you know, besides hiring three of us?
SPEAKER_02Yeah, no, that's a really good question. Like, how do you if this is not the thing you're you everyone has a job and it's not to learn how to use AI, right? They have they have an actual job. And so how do you how do you how do you stay in tune to what's happening? Um my view, and and this is just it's kind of silly to say it, is like you just kind of have to be in it, like you have to use the tools, you have to spend time experimenting. Um there's no replacement for just going and doing the thing, right? Like going to try something. Um, even even for me who's in the weeds, like these things, these things change, right, over time. And so you just have to be in it. Um, so I think it's more, it's almost more like a a process or a team operations uh concern. It's like you kind of have to both have people uh at the ground level that are curious and willing to experiment and have the time to experiment. And I think also you kind of have to have the view as a firm of like, hey, we're gonna, we're gonna spend time uh using this and we're gonna try to to to integrate it into our processes. In terms of like where to look to learn, I mean, the anthropic, uh, openai, all of their all of their resources, all of their training is is really good. Like their documentation is great. So you can just kind of go there and you know do their do their tutorials, read their docs and and and get up to speed relatively quickly. But I think the most important thing is actually spending the time doing it, which it's it's hard, right? You're you're you're doing something else and it's not learning AI.
SPEAKER_01Brett, I'm curious in your world, you know, you've you see folks that, you know, CIOs and PMs that, you know, if they're talking to you, they have the vision that this stuff is important, but they probably are super time crunch that they're not, you know, firing up a cloud code instance on a Saturday night to you know test a rag implementation. How how are you finding that? And how are you seeing that kind of in the folks that you talk to, kind of balance the, you know, I've got to like I need to be the CIO that uses the tool, and at the same time, I'm the CIO. I've got 9,000 responsibilities, and playing with cloud code is probably not one of them that takes priority.
SPEAKER_00I think we we've probably all seen the same thing of usually at a firm of 20, you have two or three who had some sort of aha moment and they get a little bit of AI psychosis. I say that kindly because I think we probably all three have some degree of AI psychosis. And I think what stimulates that, at least for me, is like trying something that you think is impossible for AI to do, and it actually does it. And you're like, holy moly! Like, I can't believe that these tools can do these things. That takes, to your point, Matt, like a bunch of experimentation. But that is such a seductive, addictive feeling, and all of a sudden you're doing, you know, you're sort of trying other things late nights and weekends. So I'd say pretty much every firm has like one or two of those people um that are trying to bang the drum for the other individuals to sort of bring them along. I think the singer single player versus multiplayer gain like really, really resonate, really, really resonates. I think that's the tricky uh challenge of how do you diffuse that uh uh information to the uh to the broader to the broader firm. Because most of those people, like you ask them how many hours they've spent experimenting to get to that level of fluency with the tools, and it's usually like hundreds of hundreds of hours. So that's that's the that's the that's that's that's a trick. Um and then even me, like almost you know, every week, like the interface on codec changes or cloud cowork changes, you're like, wait, what's this button? What is codec's projects? Like, what is this? What's perplexity computer workflows? Like just staying up with the the architectural changes of the UI is is hard enough to do.
SPEAKER_02But uh Yeah, I I really love your example of of trying something that you don't think is possible for these models to do. Um, I I agree that when you kind of frame it for people to do that and then they go do it, that's kind of generally when they hit their aha moment because you know, lots of people are still still using these tools more like a Google search for for what it's worth. They don't really truly understand um how capable these tools are. I mean, a recent example for me is I've been really into trying to take uh written content that I've used the models to build, you using upstream data sources, and instead of just having that written content, basically turn that written content into an interactive website. So you think about you have uh you built a company deep dive or you you're you're doing some research on some topic, right? And you know, you get this 10-page document, it's like, okay, do you really want to share that? Is someone going to read it? Turn it into a website, have send them a self-contained HTML, JavaScript CSS, where they can just click around. And like that just kind of works now. Like, if you if you if you tell the model to turn this stuff into a website, you can get an HTML sent it to someone. And like, I don't know, if you were to ask me if that was possible like six months ago, I'd be like, no way, can't you can't just one-shot a file that you can share that someone can just click around in.
SPEAKER_00Yeah, you guys probably saw the Tim Ferris uh tweet where he talked about his books having declined 80% over the last five years, and he sort of attributed it the recent accelerating drop to to AI. And um makes sense to me, man. Like I used to read 20, 30 books a year, and I sort of look back at myself and like maybe I've read one or two this year, because exactly that. Like to go get all this information on a topic and learn it in a distilled way is a superpower of these tools. It's pretty crazy, actually.
SPEAKER_01Did did you see yesterday they announced uh clawed code can create artifacts that are then shareable within your team, so you don't even need to fire up like a hosting environment. So that same example, Matt, you described, you could create this like beautiful, I don't know, an initiating coverage report via HTML on Snowflake and then just put a button, push a button and automatically deploy it to your entire firm.
SPEAKER_02Yeah, yeah. I saw that announcement. I haven't dug in. I have an ever-growing backlog of things to to read about and these companies ship even faster and faster.
SPEAKER_00So what uh we talked about the data cleaning problem. Like when a client comes to you, Matt, and says we want to make these things work, what are how would you bucket that exercise? Like, what are the big buckets of workflows that you think firms need to focus on to really go from interested in concept to we actually want this, you know, productized and enterprise grade to to make decisions with agents?
SPEAKER_02Yeah, yeah. I mean, I I like to take um like a uh a job to be done type approach. It's like, okay, like what are your what are you doing now? Like what are the workflows that your analysts or your team are doing? What documentation are they producing? How are they how are they sharing that? What are the upstream things that they're they're gathering, right? And and you, it's kind of a discovery um process where you can talk to the people and say, like, what are you doing? Where where are you getting your data from? What are you looking at? How do you how do you store your notes? And so just understanding um the workflows, right? Acknowledging that part of what these tools might help you do is optimize that workflow. Maybe it will help you gather the upstream information faster, distill it in the ways you want to faster. But it might be some something new, like some new process that's kind of like goes across your your existing flows. Um, so I think it's like understanding the key data source. And you know, in this space, like there's there's a handful of them that that seem pretty important, right? Like transcripts, SEC filings, obviously your your own personal notes, your RMS. Um, and so it's figuring out how you can can get those um in a way that um can be accessible to to the models, right? Um you know, when you when you look at things like SEC data or or transcripts, like there's there's providers now where you can essentially pay money, get get access to that information and have that accessible to the models. You have to you have to wire it up. Um you know, I think the the more interesting pieces are um what are you what are you doing with this? Like what are what are you what are you planning to do with the data? How do you want to structure it? Um how do you want to distill it? And there's a lot of utility in helping um teams write that down, like talk through that. Because I think a lot of this stuff is kind of just, you know, it's in your head, right? It's this institutional knowledge that's just in people's brains, right? So I think a big benefit is extracting that kind of that specialized intent or knowledge and writing it down because it turns out if you can, if you can really specify what you want these models to do and you have access to the data that has that source information, they're really good. I think a lot of the um a lot of the work goes into just kind of writing down what you want to do. And I always like to tell people when they're learning AI is like, hey, are you a good communicator? Like, can you write things down? Okay, like that's actually 80% of the work is like, you know, is describing your intent clearly, right?
SPEAKER_00Which is challenging for many investors because so many investors are these sort of intuitive, almost like concert pianists, like they sort of can create a masterful uh trade, uh, but asking them to you know sort of turn around and write an 80-page operating manual of you know, sort of how they make decisions is a is a harder, is a much harder exercise. Like, how do you how do you think we we crack that? Um ideas that disambiguation because like you know, specificity is really critical to sort of overcome the the stochastic nature of the underlying models, right?
SPEAKER_02Yeah, yeah. So my my biggest uh hack or view for this is it turns out like the models are actually helpful to help you to help extract this information from you, right? So like one of the one of the biggest ways that I use these tools if I'm trying to build something or or describe what I want, is I will truly uh take a voice memo of me just like talking, right? Extensively, maybe go on a run and just record myself talking if I have to build a project. And like I'm not even um I'm I'm truly just rambling my thoughts, right? Because it turns out if you take that voice memo and turn it into a transcript and load it into a tool like Cloud Code and do something like, hey, here's where I'm at with this idea or this thing I need to do. I need I want you to ask me questions exhaustively until you're you're clear on what I'm trying to do, and that's a complete story. So you can actually use these tools, right, to kind of help you um help you build that. You like you don't have to start from a blank slate. So that's kind of my view on how I I um I get this information out on paper.
SPEAKER_01It's it's interesting because we've spoken about you you used a an analogy bread of you know, a pianist, an intuitive pianist. I saw this quote, I can't remember who said it, uh, but it's basic, oh Dan Daniel Meissler. He said, uh, all of knowledge work is just vibes and spreadsheets. Right. And um it's LLMs don't do well with vibes, right? Like they want instructions and they want intent. Um and it's funny because uh, you know, Matt and I have spoken to to different companies, just you know, uh discovery work and the the at the end, like they just think like you just like throw all this data in an LLM and then it's like like the magic wand waves, and like every answer you could ever want is there for you. But the reality is, as Matt and and you are both alluding to, is that you have to like understand what output you're trying to get. And that is actually not even a tech question. In fact, the tech makes the need for this process documentation or process mapping way more important than it ever has been. And uh, I'll be curious to see how financial services people do with that because especially investment on the investment side, notoriously don't really care much about that. They have their process, their process works. Let's call it a day. And so I don't know if either of you have thoughts on like, are people ready to be sit in a room with Matt for four hours and be like, so tell me when this piece of information comes out, what are the seven things you do and the three things that you think about?
SPEAKER_00It's sort of become the singular focus of almost singular focus of the consulting we're doing, just helping people map that. Like we've built a map of sort of what we we view as the style agnostic fundamental investing process into 800 sort of primitive, you know, subatomic steps to sort of show, you know, a salad bar, if you will, of the various steps to sort of inspire, help people address that blank page problem. Um, but it is hard. Like the there's such a combinatorial complexity of what investors do because the way a multi-manager utilities analyst approaches his or her day versus a family office long only banks analyst approaches his or her day, the data they're looking at, the decisions they make, what what good management means, what a good, you know, what a good bank it is versus a good tech company. It's a very specific exercise. And so that cloning of your own process is a is I think a real bottleneck. Um I like your processes, Matt. Like we've we've we've done similar, but it still does take time off the desk to really start to start to uh to do that.
SPEAKER_02Yeah, for sure. I mean, I I think it's a very um it's a similar story to the to to having to spend the time to learn AI. You actually have to spend the time to build the materials that will give you this benefit uh with these models, right? Um yeah, I mean, you know, pulling yourself out of the weeds to do this is hard, right? Because everyone has has a has a real job to do and it's not learning AI or building the tools to make the AI better later. Yeah.
SPEAKER_00What um one other bucket I've really been struggling with a little bit is um, you know, with the the open tools now, whether it's cowork or codex or perplexity computer, I can get to very good prototypes very quickly on pretty much every workflow of the investment process. I still struggle a little bit with consistency. I struggle a little bit with accuracy in those, and I struggle with the token economics of how that scales up to an investment process. So that sort of like last mile problem of productioning those demos still feels like a a hairy issue. What uh what have you seen? Like what are the sort of the the bags in your tool, you know, the tools in your toolkit or what you've seen clients do to actually production these these demos?
SPEAKER_02Yeah, so it it um you know my view is that if you can find a way, like you should if you can find a way to use to assemble the tools that exist in creative ways and and kind of limit how much custom stuff you're building, um, that's ideal, right? So example would be okay, if you can use one of the main harnesses like Codex or Cowork and you can build a refined set of skills for your company that are aligned to what you're trying to do. And then you can have the suite of um data sources, whether they're you know an MCP server, which you know allows you to connect to to an API via the via the models. Like if you can do that and then share the knowledge of like how you how that system that you built fits into your workflow, like what are the expectations of um of when people use the thing and and why, uh, that goes a long way, right? I think you know, and and that requires you to have access to the data sources, requires you to spend some time building out your intents and these skills. Um, I think ultimately, you know, and this is this is probably a good thing, but like once companies are getting value out of that, they oftentimes hit something where, oh, we want to we want to do something a little bit more custom, like we want to run like reports on a schedule, or we have a data source that we want to integrate that you know doesn't have an MCP um server. So like that's kind of when it's tricky because you if you're like a smaller firm, you can't expect to have like an engineering team to do this. Um, but I think you know sometimes you actually do have to kind of build something, right? Um and back to the bitter lesson idea too, is like you have to be cautious here because things are like continually changing, right? So like you don't want to um you don't want to build build something that might be thrown away immediately, or you want to build something in such a way that such that it can be like repurposed later, um, which is is why I actually really like this this concept of skills because when it when it comes down to it, it's kind of just like writing down what you want in English, right? And um it's a little bit bigger than that, but that's the simplification, and that's gonna be, I think, durable over time, right? And useful.
SPEAKER_01I was about to ask you about skills. You're a big proponent of them. And I've found that many of the folks I work with, they can kind of get their first couple skills, you know, like oh, uh, you know, earnings call, a transcript reviewer, kind of like the blocking and tackling. But once they kind of get those kind of core, and they tend to usually be summarization skills or synthesis skills, they struggle, and myself too, to kind of expand the map of what's possible. And I know like like OpenClaw, for example, a lot of the code is actually skills files. And so, how do you how would you guide folks who are kind of earlier in their skills journey to move beyond just synthesis skills? Like whether it's a mental model or an actual like tactic that you that you would recommend for them?
SPEAKER_02Yeah. Um, you know, I I I I still like to think about um in terms of like, okay, what are you trying to accomplish? Right. Like, not necessarily like what's the skills library I should build out, but you know, back to Brett's point earlier is like try something that you think the model will not do correctly, and and and hopefully you'll be pleasantly surprised. So I think it's more about expanding, first of all, expanding people's view on like what these things can do, and then experimenting with it. Like maybe you do something that's like more cross cross-cutting in nature than what you're generally using these tools for. Um, but it turns out too, like if you um if you if you go at it with an applied approach of just trying something and you get to a really good output, like you can turn that into a skill, right? Like you can ask the the model to be like, hey, like let's do that again later. Can you help me turn that into a skill that I can reuse? Um so I I like that applied approach versus you know, just like once you kind of do have a recipe that works and you're like, this is what I want, like I think that the objective there is then to kind of make that durable and and reusable, and then ideally, you know, share that knowledge back with your team, right? Like embed that new skill or that workflow in the in the shared repository where other folks have access to it, like share that knowledge.
SPEAKER_00Matt, have you found any best practices on skills creation? I mean, back to prompts, we had a speaker say, like, for beyond four to six pages, the the chat bot starts to you know lose attention. Are there any equivalents that you found in in skills in terms of you know depth versus you know, depth first uh versus efficiency or other other elements in creating skills that you found helpful for for people to think about?
SPEAKER_02Yeah, um, I mean, I think you know, the these models, it's funny, they have a skills creator skill, which you can go in and use to create skills. And um, the point of that is like these models obviously know what they expect to see within these skills. So I think that's useful to try. Um ultimately, I I think it's more about I like to this concept of like hill climbing, right? Where um you try a thing, you you look at its outputs and you use your brain to say, hey, is this good or bad and why? Right. And so, like ultimately when I'm building these things out, um you you kind of have to have this iterative loop. So maybe you build something that does something, produces an output, and then you say, okay, like let me record myself talking about what's good and bad about this specific thing. Let me take that feedback, the output of the skill, the skill itself, and do do work with the model to iterate the skill itself, right? I think kind of refining it is a pattern that I I like to use, um, just trying to continually improve it. And and ideally, you know, for some of these things too, it's really important to have kind of uh systematic evaluations, like if you're kind of producing something, you you you ideally have a way to test its outputs, which is which is tricky if it's text, right? Because it's it's it's sometimes very subjective. But how can you specify what correct is and and have a way to kind of continually test that your skills are doing the right thing over time?
SPEAKER_00Have you thought about building evals? Because that's sort of hard as well, too. Even like I'll eval things across three different LLMs and I'll get a little bit of a different scorecard on my on my evals.
SPEAKER_02Yeah, yeah. It's definitely it's it's tricky with with the written text, right? It's a lot easier if you're evaluating. Oh, is this number correct? Did I extract this specific sentence from this specific um section of the the filing? Um with with um with with text, I mean you can obviously do something the technique is LLM as judge, where you're using a LLM itself to review the outputs, but that that's a little tricky because then like, okay, what is what is the judge doing? How are you how are you evaluating that? Um I mean, I think uh oftentimes like when I'm building these these workflows for for the space and the output is like a written report, um, you get a lot of utility by just reviewing those with the person that's consuming it. Like you say, okay, let's step through this memo and you tell me what's good and what's bad about this, and we'll record that and transcribe that and use that as input to try to figure out, okay, how do I adjust this skill? So that's, I mean, that's not an uh automated evaluation, but it's kind of an evaluation loop, which I think is is is really valuable, right? Like because you're kind of going back to the person who's using the outputs and getting their feedback and trying to integrate that back into the upstream uh instructions.
SPEAKER_01Matt, has your view? I know early in the skills uh shift, you were a very uh big supporter of Skillcreator. I think I saw a tweet from you recently that is uh maybe more skeptical uh of Skill Creator. Uh I know personally that I've pushed people to use Skillcreator. I use it all the time. And now it's these like overbuilt. Someone even asked me, do you think they're overbuilding them to just like uh subconsciously have you use more tokens? I'm like, dude, I just need you to like print one number and you have like you wrote me a book.
SPEAKER_02This kind of ties to a more like general view I have here is that um like these models are amazing, right? Like you can type things in and get things back, and they're they're great, right? But there is this weird kind of like dopamine hit of like just getting things coming out of the machine, um, to the point of like sometimes like you know you're not reading things that you should be reading. So clearly you shouldn't say like go create me a skill that does a memo and then just say, okay, this skill is done, right? Like, there's a lot of value, I think, from like stepping away from these tools, like and just opening a notepad on your on your system and like writing down what you want, like typing words or or describing what you want. I think that is the failure mode, is that like you don't actually spend the time to read what these things are producing, whether it be the skills themselves or some some meta instructions that you're using. Um, but it's a balance, right? I think you know uh I'm I'm assuming that the the companies have spent a bunch of time on these skill creator skills such that they produce the shape of output that does best in their system. But yeah, like you got to spend the time to make sure that what you're using is like what you actually intend to use.
SPEAKER_00And the the hill climber mode is the sort sort sort of mindset is so valuable. I've sort of thought about it as like the red pen, like the red pen phase, because I I feel like so like the natural impulse of Wall Street users with these tools is you run an earnings preview, you get it back, and you're just like, oh, this is awful, right? Rather than saying, okay, this isn't great, but how can I red pen this and give feedback back into the model to go back in and change the skill and do that two or three or four times. I even find myself sort of having to nudge myself to hill climb with that red pen, red pen mode. Like you're just not gonna get there right out of the box. Like the first shot is almost never going to be ideally how you want. Ideally, how you want. I'll give one example I give is LLMs think that goodwill impairments are really, really scary things for companies. Like you can't buy the stock because there's gonna be a goodwill impairment coming, which is just an accounting, accounting thing. So you get that back. You have to be like, no, don't never in the future cite goodwill impairment as a key risk to to a company that's more of an accounting dynamic rather than a than a fundamental dynamic. So there's a lot of these different debugging things that you have to go through to to get these things to speak your language.
SPEAKER_02Yeah, yeah. Yeah, I think another thing that that's interesting too is um, you know, the these models are ultimately they're they're good at you know distilling text, they're good at writing code to produce things, um, but but but they're ultimately non-deterministic, right? Like they're they are non-deterministic machines, right? And so um if you're trying to solve a problem that requires determinism or is is is a calculation, like sometimes it's better to just write the code or do it somewhere else. So like you you definitely, I mean, LLMs, this is this is a tool, right? And so like you can't for some of the things you're trying to do, maybe it's not the right tool for the job. And just like oh always having that in mind too, is like it's it's just it's something that you can use in your broader workflow, but it's not always the the the thing you need to reach for for everything.
SPEAKER_00Matt, how do you see the the tool stack evolving? I mean, there's sort of been this debate on um, you know, using actual agents like OpenClaw and Ermies. Um, I've seen more and more of my friends sort of shift from building in Cloud Code to actually using Cloud Co-work to sort of lean on the consistency of the harness. So it feels like there's a little bit of a trend of abstraction into the super apps. Here's your perspective on that and where you see things going over where you think what what do you see as like the current trend and where do you see things going in that regard?
SPEAKER_02Yeah, yeah. Um, admittedly, I never got into the open claw or Hermes thing. Like my tools that I use on a day-to-day basis are basically Claude Code and Codex. Um I I think that tools like Codex um have made incredible progress in the last six months to be much more accessible to a wider and more general audience, um folks that aren't um doing code. I mean, it's it's unfortunate that Claude Code is named Claude Code because people are like, okay, I don't code, I don't need that. But for the longest time um before cohort came out, I was trying to get people that are technical enough to use it, but are not coders to use the thing, right? Because I think the big the big break. Through there is like you got to use the tools that have access to your file system and can use the operating system tools. Like that's clear to me. Like you shouldn't be using Chat GPT in the browser. Um, and I think Codecs and Cowork, they they they have they're they're going to continue to make that user experience work for a wide audience. Um, so I think they will kind of consume a lot of the workflows and kind of like that's where you operate, right? That's where you do your work. Um, it's it's amazing how many data sources and tools kind of plug into that ecosystem now. So my my gut is that will continue to happen. Um it is still feels very early though, because I mean, you know, we're obviously in the rabbit hole. Like, I don't know how many what percent of people doing uh work out there is actually using codecs. Probably less than 1%, maybe I don't know. Um, but I do think eventually, my view is that the that category of tool is going to kind of subsume a lot, a lot of work. People will be spending a lot of time in that type of tool.
SPEAKER_01I'll jump in as someone who probably burned like 30 hours on open claw. Um, I think the the and then I stopped once once they took away the subsidy, once I had to actually pay for my Opus calls, I was like, whoa, this is not worth $50 a day. Um but one thing that I've observed, particularly on Codec. So why did I like OpenClaw? It removed two key frictions for me. One was it's just nice to be able to text from your phone. Like that, I think that's really until you actually do it, you're like, you're like, well, I could just open the app and chat. Like, no, no, no. Like we live inside our messaging app. So there's just something so powerful about the lack of friction, which which I really miss. I use it a lot less uh uh because I got rid of the texting thing. The second is cron jobs, and for non non-nerds, which is just scheduled tasks, which this is for another podcast, but like for something that seems so simple is like deceptively hard to do. They break, you don't, there's no way to track them, and so on. And OpenClaw made cron jobs really easy. And uh, but now, especially with codecs, the code, I don't know if either of you have used codecs on your mobile phone. It is an amazing experience. It literally maps to what you would see on on code on uh your desktop. So it so you can run full codecs from your desktop, unlike Claude Cowork, which has uh dispatch, which is kind of just weird. It just I don't get it, it doesn't work for me. So that would be one, and then the automations are really, really good as well. So you again, you can have an interface to like schedule like show me all my scheduled tasks is actually quite easy to the point that I didn't buy a Mac mini, but I did set up an old Mac as like an always-on server for OpenClaw, and I'm actually in the process of converting that into an always-on codecs so that when I'm traveling, I can still talk to my codecs instance because my machine will be on 24-7. So it's effectively gonna become like a local server serving codecs that I can talk through through an iPhone gateway. And that like I'm this close. This weekend I should get it done, and it's gonna be awesome. Yeah, so a very simple one. Again, I I start with a lot of personal agent things. So uh I have an agent, it's still rough, that basically can schedule calendar meetings for me. And so that's it has delegated access to my calendar. And so I'll just uh go into codecs on my phone and say, schedule a meeting with Brett this Friday at 1:30 p.m. And then if I want, I can have it send you the invite as well. And so, like, but I need to be it needs to be on my codecs, right? Because the permissions are local to my machine, they're not like in the cloud. So that would be one. Another one I have, this is super cool. Uh I call it the CRM tickler file. And so I have a CSV file that has all the people in my network and a frequency in which I would like to just like stay connected with them. And so then I have a skill that actually, well, first I have that hooked up on codex to iMessage. Because like 70% of my work, like, I don't think I've ever emailed Matt. It's like always been through to NU as well. So it's like, what's the point of a CRM if it can't read your iMessage today in this day and age? It's kind of pointless. So I have it hooked up to my iMessage and my my Latour Gmail. And every Friday it runs, it basically looks through both sources and sees like who falls outside of the window of like, you know, who do you need to get in touch with? And then it it sends me an email of a list of the people that are like past their window. And so I could, but let's say it runs every week. But let's say I wanted to run that on like a Wednesday night and I'm on the road, I could just say run the run the CRM tickle skill.
SPEAKER_02That's cool. Yeah, I've never um I have not gotten into the use the AI from the mobile thing yet. And I'm kind of resisting it because I'm already I'm already spending so much time on this stuff as it is, right? And so it's it's interesting though. I mean, I think that's a big part. Uh you're not the first person that said the tech, the texting aspect was really important to them to be able to kind of manage and run run these things. Um, but yeah, it's it's interesting.
SPEAKER_00What um, Matt, what are your thoughts on the I've heard a lot about the Chinese models. There's been this sort of wall of like token, you know, token burn, token maxing, token efficiency. Uber ran through their tokens for the year and four months. But it seems like people are pointing to sort of the the fast follow of the Chinese models as a as a sort of antidote to that, um, with sort of routers on top of that. What's your what are your thoughts on that? That's that space.
SPEAKER_02Yeah, it's interesting. Um, you know, I I have dabbled with running local models on my machine, just just as a way to just see how it works, right? Like download some weights and run them. And I don't have a super beefy machine, right? Like, so the the models aren't great. Um, but the the smaller models um and some of the open source models, they're getting really good. I mean, if you look at the benchmarks, and you kind of have to take the benchmarks with a grain of salt sometimes, but the the open open weight models are are are very effective. And my belief is that most people's work right now, like that they're actually doing, probably doesn't require like the frontier model, anyways, right? Probably a lot of the tasks that people want to use is very serviceable by these smaller models. Um, so I think it has legs, and I think um, you know, these these smaller models will continue to get better. Uh I I think um, you know, companies if if they want to cost optimize, this is going to be a route that they go, right? Like and you can already see like uh Shopify put out an interesting post a couple months ago. They they took a workflow that they were running at scale on a frontier model and kind of uh um, I don't know if they fine-tuned a model, but they did something to run it much more cheaply, right? And like they got a hundred X, I think, cost improvement there. Um, so I I think um that will happen. I think it's it's it's tricky because it's it's much easier, I think, to just like use the frontier model and run after it. But I think as companies have things that they need to do at scale uh and it's it's repeatable and they've proven it it provides value, like that cost optimization um uh angle will become more important. What what are your thoughts there?
SPEAKER_00Uh I I don't have very detailed thoughts. I just I I just hear people talking about it more. So it seems to make sense to me. I've I've been a fan of you know uh tools like Perplexity Computer, which is sort of like an abstracted open claw that has access to 18 models, and I find it just to be more consistent than I do a Claude, which is really important for the for the workflows that I do. And I think people people are trying to struggle to figure out Excel as well, too. And this common response I get is that you know, frontier and you shouldn't you shouldn't try and update a model with frontier intelligence. A lot of that should be done in a in a coding environment. So, how do you start to think about actually deploying more deterministic use cases uh at potentially local, low-cost models, and use frontier intelligence for a stack of things when you really want to think through, you know, think through a debate or apply judgment and sort of think deeply with the with the the strongest intelligence, but to deploy frontier intelligence across a full stack of the sort of agentic investor process doesn't feel like one, it doesn't feel like the most effective methodology. I don't want a stochastic, non-deterministic tool in the in the you know quantitative realms of my investment process. Um so I think that's part of why I've been more excited with agents. Like you see a little bit of glimpse of that of like the LLM is the orchestrator of these various tools, calling subagents, spinning up coding environments, et cetera. Um but that's about as that's about as deep as my as my knowledge goes in that in that in that space. Um sort of feels like this spinning up of subagent. That's when I started to like I I I gave up on cloud code when someone's like, well, to actually do that, you have to spin up subagents and do all this. I'm like, I don't know what I'm doing. I'll just find a tool that can automatically spin up subagents, which perplexity computer does. And codecs is it's gotten much easier to spin up like many, many subagents in codecs as well.
SPEAKER_02Yeah, yeah.
SPEAKER_00Well, this has been a wonderful, um, wonderful Matt. Almost certainly this conversation will follow Sutton's uh bitter lesson curve. We'll probably be laughing at uh at the conversations we're having, you know, here in in 12 months. How anachronistic were those conversations we had about uh subagents and Chinese models, et cetera. Um, but thank you so much. So we'll you know love to have you back on and sort of stay, stay, stay in tune with uh the frontier here. Please keep tweeting because it's part of my educational stack uh as well, too.
SPEAKER_02So uh yeah, for sure. You're right on Twitter. It's like um if you want to be in the loop, you have to be on there, but you also have to ignore a lot of things that are on there.
SPEAKER_00Yeah, exactly.
SPEAKER_02And you got to put the phone down once in a while.
SPEAKER_00I tell people my rule is just like wait for I hope I hope Andre Karpathi can still can still tweet at anthropic now because I'm like just wait for Andre Karpathi to write about something and then like spend a couple of days thinking how that applies to finance and investing. That's sort of been my like ironclad rule of uh of uh you know staying staying ahead on AI and finance.
SPEAKER_02So yeah, yeah, you gotta find the people that that that are talking about the right things and just pay attention to them, I think. Yeah.
SPEAKER_00Great. Well, thank you so much. This was really educational and and entertaining. Yes, Matt.
SPEAKER_03Thanks for having me on, guys. Appreciate it.