Friendly Show
Yaro and Adrian are online and conference buddies that enjoy speaking about development, tech, SaaS, and product development.
Let's be friends!
Friendly Show
Meet SQLite Jesus, Stephen Margheim
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Steven is on a mission to make SQLite good enough for production.
We talked about
- SQLite and Litestack
- contributing to open source
- being "the guy" in a niche, creating your own luck
Exploring SQLite in Rails Development
Speaker 1Hello friends , and welcome to season two of Friendly Show . Yeah , so today we have an special guest . It is Steven Memenheim , or ?
Speaker 2Marguheim , marguheim .
Speaker 1Yeah , and basically I know Steven from Twitter as the like eskulite guy . So I like keep reading your tweets over the months and I'm just like getting more and more convinced that like how we should just dump Postgres on my scale and like eskulite is the future . There are so many new features . So , yeah , we're going to talk to Steven today and actually I had this wonderful chance to meet Steven . Well , we both did agent , so we met Steven live in . It was at Fern , yeah , at Halvedic .
Speaker 3In Bern , yep and at Rails world as well .
Speaker 1Yeah , you know the beer temple .
Speaker 3He was one of the beer temple guys , the crew .
Speaker 2Praying at the temple .
Speaker 3Yeah .
Speaker 1Yeah so , Steven , it's a real nice to have you here with us .
Speaker 3Cool . So you know , you're an American expat living in Berlin with his wife and his two dogs . And their two dogs , You're like a contributor to Rails and the eskulite three Ruby gem , as well as maintainer of a handful of gems aimed at making Ruby on Rails the absolute best you know platforms to in the world to run a eskulite project . Okay , so do you want to start a little bit of telling us where do you work and how did you end up in Berlin ?
Speaker 2Yeah , I work for a company here in Berlin called Testio . We offer crowd driven testing services to companies and I am the engineering manager there . Okay , so I'm not doing development work as my primary day to day responsibilities , but one of my responsibilities is being effectively one half of the research and development team at the company , so the CTO and I do nearly all of the sort of initial experimental projects to see is there something here ? Do we need to build like a small one off application for a larger customer ? Are we expanding into a new market and want to see if it has legs and is viable ?
Speaker 2And it was through that that I started using SQLite in production , because I needed to be able to build an application quickly , and Rails is quite good at that .
Speaker 2But what I found over time was that , even more importantly , I needed to be able to maintain that application with really small windows of time between calls .
Speaker 2So it's like if I have 30 minutes between the end of one call and the start of another call , I need to be able to read the bug report , go into the production application , figure out where the problem is happening , figure out how to solve the problem , implement the solution , deploy the solution , go into production , check that the solution actually solves the problem and then write the person that reported it hey , it's been fixed .
Speaker 2And I have like these 30 minute windows to do all of that . And I found that the more tools , the more in direction that I had in the applications , the more difficult it was to work with these applications in that kind of way , and I initially just started off like just running Rails defaults and pushing it to a Hatchbox server and seeing what it was like , and almost immediately it was a noticeable difference , like even just the simple thing of like SCPing the production database down to my laptop and being able to debug in my local instance of the applications the totally sandbox but with all of the production data and like step through the problems . And yeah , that's been maybe . I deployed the first Rails application into production here at work a little bit over a year ago and the more I used it , the more I was like this is incredibly productive for me in these constraints , and so I just started trying to polish the rough edges that I came across as I went deeper into building and maintaining these applications .
Speaker 3Nice . So basically , you're trying to stay as lean as possible and do as much as you can in the shortest amount of time . Gotcha , tell us a little bit about the rough edges . What's stopping people ? I mean what's not enough when running . You know on a regular basis . What's not enough when running . You know Rails new and using SQLite . What's missing there ?
Speaker 2Yeah , Well , luckily I can give you some different answers for like what was missing when I first started versus what is missing now , because there has been a lot of progress and that's exciting to see . Like early on it was sort of just running into these brick walls of knowing of these Rails features from working with Rails applications using Postgres and then going to use them . So whether it's like generated columns that was like one of the very first ones that I hit and looking at the SQLite documentation is saying SQLite supports this and Rails supports this , but the SQLite adapter in Rails doesn't tie these two wires together and there were a small handful of features like that that were honest , you know , just like just copy and paste the Postgres adapter code , almost . So I did a number of pull requests like that at first just to wire up these features that had been neglected from the SQLite adapter .
Speaker 2But today I think that the biggest rough edge is that SQLite has a database level write lock , so not row level or table level .
Speaker 2Only one query can be writing to the database at a time , and a lot of people presume that this means you can never have like production grade applications back to SQLite , which isn't really true . Just back of the napkin math , roughly like when you move from queries taking milliseconds to queries taking microseconds you can pack at least 100 of these queries in the same amount of time it took you to do another one . So linearizing them does not harm you a lot , but you do have to contend with getting locked or busy exceptions in your standard Rails application because you are going to spin up multiple threads by default in production and they're going to have multiple connections . But there are pull requests that are open and being evaluated and making their way into main and then eventually into new versions of Rails that will resolve that rough edge , but it's still there . That's probably the most frequent one and you're guaranteed to see those exceptions in your bug tracker when you put a SQLite application today into production .
Speaker 3So I'm not a super big database buff , I don't know a lot of the internals , but what this sounds to me is like what Postgres , I mean , and every database had when serverless came around , when you could have a service , something on Lambda that was spinning up like maybe 100 or 200 clients at once , because you have this surge of customers , of users , and then they would all connect to your same database , which usually it will take a few connections , like 10 , whatever 20 , but now they have hundreds all at once .
Speaker 3And there were these solutions like one of them is PG Bouncer , which is basically a manager of connections and you send all the connections to the PG Bouncer and PG Bouncer connects to the Postgres and whatever . I don't know a lot of the internals , but what it seems to me like this is something that we need Some kind of I think it's called a stopcock , some just one place where you can connect and it will figure out when to write , when to whatever . But maybe I don't have that much of deep understanding , but having something like this , like somebody said , okay , it's okay , just put on PG Bouncer and you're good , you're going to be good , you're not going to feel the difference this is something that I would like to hear for SQLite , like , hey , just use this and everything is covered , it will retry whatever it needs to retry or handle the connections or the rights or whatever . Is this something at the Rails level that should be added , or is that like to the SQLite Ruby Gem ? How do you see that ?
Speaker 2Yeah . So I think that the simplest place to have that like the buck stops here and this is where we're going to figure out the linear order of operations is SQLite itself , like it has everything to handle . That itself the problem . There's two key problems that exist today . One exists in the SQLite 3 Ruby Gem and one exists in Rails . So the SQLite 3 Ruby Gem is SQLite exposes a C interface , and so if you want to hook into it and work with its various hooks , you can only do that in C . So the core of the SQLite 3 Ruby Gem is writing a lot of C code to connect it into Ruby code and to allow you to interact with the SQLite interface through Ruby .
Speaker 2Sqlite has native support for retrying busy connections . So you have two concurrent connections that are both trying to write . The first one that gets in , starts to write , acquires the database lock . The second one goes , sees that there's a lock and you can set up how you want that retry mechanism to work . The default , most common use case , is to just set up a timeout , so it's going to not quite exponentially like logarithmically sleep and retry up to however many milliseconds you say and when you set the timeout parameter in your database YAML file in Rails . That's what it is wiring up . The problem is that the SQLite 3 Ruby Gem tells the C code in SQLite hey , there's a timeout for five seconds , so you have a busy connection , and it's like ah , let me run this timeout . Logic retry , sleep retry . That happens in SQLite's code in C and the SQLite 3 Ruby Gem does not instrument that code in such a way as to release the GVL in Ruby .
Speaker 2So , SQLite is sleeping and retrying but it is holding the Ruby's global VM lock and so no other code is able to run . So SQLite allows you to do concurrency , but the Ruby interface doesn't connect with that . So there is . I wrote a pull request to the SQLite 3 Ruby Gem to implement a non-blocking timeout . So you follow the same logic sleep , retry , sleep , retry up to a certain amount of time . But during each of those sleeps we release Ruby's VM lock and once a new version of the SQLite 3 Ruby Gem gets tagged , I'll open a pull request into Rails and wire up the database YAML timeout configuration option to this new non-blocking retry . So that's like one problem , but it's being solved . It will be solved .
Speaker 2The second problem in Rails is that it doesn't currently work with SQLite's transactions in the most intelligent way . So SQLite has a somewhat novel concept of transaction modes . So by default , transactions are what's called deferrable , which means that they don't acquire SQLite's database lock . They presume maybe you're just doing a pure read-only transaction . In that case no need to acquire the lock . So we're going to presume that you won't do a write until you start to do a write instead of that transaction , and at that point we'll try to acquire the lock , but because you're inside of a transaction and SQLite is very aggressive about its isolation guarantees , if you try to acquire the right lock inside of a deferrable transaction and another connection has the right lock , it won't go through the retry mechanism , it immediately errors . Sqlite will just instantaneously throw an exception because it can't guarantee if it retries the state of that , the initial reads or what happened before .
Speaker 2There is another mode for transactions called immediate mode , which says I want to immediately , when I'm starting the transaction , acquire the right lock and if another connection has the right lock , it'll enter into this retry mechanism and sleep retry , sleep retry .
Speaker 2So I have a pull request open right now for Rails to ensure that all of the transactions that Rails wraps write operations from ActiveRecordIn are immediate transactions , and so when these two features come together and are in Rails , what will be the case is that we will push the responsibility to SQLite to figure out the linear order of write operations as we can currently send them in . It will have however much time you set in your timeout for each operation to get executed , and the default is five seconds . That is very sane and you're very unlikely to see any exceptions unless you really have a problem with your database and I've monkey patched things in my own application to get these fixes in there and the exceptions have complete . I have not seen a database locked or database busy exception since implementing those changes , so I think we're going to get to that place by Rails 8 .
Speaker 3Nice , nice .
Speaker 1I'm kind of under the impression that , yeah , you can try using the SQLite introduction now , but it's better to wait for like a year or so until it is really ready .
Speaker 2Yes , yeah Well , it depends , it depends . So I specifically built and released a gem called the ActiveRecordEnhanced SQLite 3 adapter a really beautiful name .
Speaker 1I'm very creative .
Speaker 2To backport these features because you know Rails releases can be slow and sometimes you can't quickly upgrade to the new Rails version . But you still want these features . So , whether it's generated column support , whether it's a non-blocking SQLite retry mechanism , whether it is immediate transactions , right , if you install the gem it automatically enhances the SQLite 3 adapter , patches these changes in and you'll get there . So the goal is to make the default Rails experience for everyone . You don't have to know about this gem , you don't have to know to install it . Excellent , and probably it will take through the end of 2024 to like get everything in default Rails . But you can get all of these things today if you do Rails new and then bundle add , you know , activerecordenhanced SQLite 3 adapter .
Speaker 3You should . We'll talk after , but I'll show you how you can create a template for that and it'll be a little bit easier . But , yeah , definitely , definitely a good path forward . It's a little bit more . It's a little bit deeper than what I understand about . I mean , I understood what you said , but more . I'm not super passionate . I feel that you're super passionate about it and I think this is what we need . You know you're the SQLite guy . You're PRing improvements to the SQLite driver , to Rails . Okay , you can take care of that thing . And I'm personally like I'm super , you know , happy that there is such a person .
Emerging Ruby SQLite Ecosystem
Speaker 3I remember when we met the last time at Helvetic , I asked you like , hey , I would love to see how you do some of the you know , I would say common things that you do with Postgres or we are used to doing with Postgres , like you know backups , or you know restores or like logging or other things . How would you do those with SQLite ? And you told me about the light stack and , I think , extra light . This is something else . Tell us what ? What are these two ? What are the light stack and extra light ? How do they work into our current , you know , workflow ?
Speaker 1Yeah , I think Julian Rubish has been releasing like a series of blog posts about the lights . I think there are like five of them . I haven't read them yet , but maybe after Steven explains it'll be easier to digest them .
Speaker 2Yeah , yeah , those are . Those are on the app signal blog and they're they're excellent . I would highly recommend them . So , yeah , let's , let's start talking about the emerging larger Ruby and SQLite ecosystem .
Speaker 2So light stack was honestly the the light bulb moment for me . That like activated my passion because I had been running a couple of Rails applications in production at work with SQLite and was enjoying the developer experience . But I hadn't pushed very hard and it's like when I hit rough edges it's like , okay , just you know , avoid that corner , avoid that edge . But I came across the light stack gem on Twitter and I went to the GitHub repository . I read the , read me and it was genuinely like one of those lightning strikes light bulb switches on . I had a epiphany . I had simply lacked the imagination to consider the possibility that Rails has only a handful of IO bound components that need some sort of data persistence to some degree and SQLite , being an embedded database , could be architected in such a way to be the back end for all of them .
Speaker 2So you want a cash back end ? Here's a SQLite database . You want your pub subserver for action cable ? Here's a SQLite database . You want your database for active record ? Here's a SQLite database . You want your job queue ? Here's a SQLite database .
Speaker 3That sounds like a lot of SQLite databases .
Speaker 2Yeah , well , it's four , but they're just files . They for most of them , the data is not persisted for all time . They don't grow to infinity . But sure , it's a lot of SQLite databases , but SQLite databases are basically the simplest , cheapest kind of database you could possibly have .
Speaker 2So I saw that and it opened my eyes to the possibility and I had already been attracted to this core principle of simplifying the number of moving parts , simplifying the number of tools to make this maintainability as like , quick and seamless as possible . And so I just was immediately like , wow , this is the future For the kind of apps that I'm writing in the kinds of constraints that I have . This is 100% going to make my life much better . And so I got connected with Muhammad Eali , old Mo on Twitter and GitHub , who created that project , and just started chatting with him and asking him a bunch of questions as I was reading the code , and what I started to realize was that there was a lot of wisdom in his code that was generally applicable . And so I found myself basically just like becoming the John the Baptist to his Jesus Right , like , okay , you have this wisdom , let me learn it , understand it and find ways to apply it .
Speaker 2But that's the LightStack project . It's basically what is ? Sqlite files backed all of your data needs for a Rails application and the extra light project , just to really quickly cover it is also really exciting and I personally use it often and I'm working on putting just a little shim layer to allow Rails applications to use it as their active record driver . But it is an alternative SQLite driver , that is to say it writes C code to expose SQLite C interface into Ruby . The biggest key difference between Extra Light and the SQLite 3 Ruby gem is that the Extra Light driver was built from day one to ensure that Ruby's global VM lock is released whenever SQLite is doing work . That's gonna take more than a couple of microseconds and so it allows for concurrency in a way that the SQLite 3 Ruby gem doesn't Most .
Speaker 2By default , the SQLite 3 Ruby gem is going to hold the GVO and Extra Light is going to release it , so it just plays with concurrency better . And that's a project by Sharon Rosner , who is a pretty well-known Ruby dev . If you've poked into some of the not Rails sides of Ruby , he's amazing and there's been a lot of activity lately . He's been adding a lot of really cool features and that's another example where I just go and read his code and get inspired and try to just bring that back to the larger ecosystem . But yeah , both of those are really vibrant , awesome , exciting and inspiring projects in the emerging Ruby SQLite ecosystem .
Speaker 1Nice , but what's actually the point of using SQLite ? I imagined so that , like at some point of time , heroku removed the possibility of having a free Postgres database , and that's the approximate time when I started hearing about SQLite . So I guess it's cheaper and from what you are doing like you launch a lot of tiny projects to demo or something so I guess it's use SQLite for quickly prototyping and launching into production without having to pay for a Postgres database . So I have the impression that you would use SQLite to decrease costs , like not to pay for a Postgres , but just like pay for Astro storage for SQLite files .
Speaker 2Yeah , I think that that's one of the reasons , but there are others , so let me list some of them , and also not to like chastise you or anything , but just because I think it is an important piece of nuance , because another part of the myth , I think , is that , like , SQLite can handle tiny things , but if you've got a real size thing , you need a real database . And I have one of the applications , the very first one that I built . It's the oldest . I mean we're doing millions of dollars in revenue through that application . It is used by companies whose name everyone knows . I mean , it's real , you know .
Speaker 2But so just to say , like , there's the scale thing . There's a lot of myths there , but cost is certainly one , I think , related to cost . But for me far more important is reducing the number of moving pieces and reducing the number of actual servers that need to be up for the application to be running . So this is sort of like an inversion of the standard logic for high availability . It's like , hey , if you want high availability , you need to have three instances of everything that you're doing , because one is none and two is one , and yeah , there is some logic to that . But when you have an application and in order for it to be running smoothly , the application server needs to be up and running , the Postgres server needs to be up and running and the Redis server needs to be up and running . If any one of those three servers goes down , your whole application goes down and the service area is there is larger .
Speaker 2Yeah , but the other one that honestly drives a lot of my passion in the open source world is that I think that it is important for Rails to really be more aggressive about lowering the conceptual barrier to entry for new developers to get applications onto the internet . And I think it's specifically important right now because I think that that has been one of the major advantages for the front end JavaScript world for like the last decade . They did a really good job of marketing this idea that you could just write a little bit of code , sign up for the free tier of 15 different SASSs and get an application onto the internet , and there is no better hook for a new developer than having an actual URL for their application .
Speaker 2Like getting it running on your laptop is cool , but when you can send your mom the URL , like that is electric and I think that it's easy for Rails developers in particular because it's so mature and so many of us are mature developers , have been working in web development for over a decade to underappreciate like the actual conceptual complexity of like these guides like even just come , all right , it's like hey , it makes it super simple . All you need to do is understand containerization , all you need to do is understand what traffic is and what the first proxy is and how it's a load balancer , and there's like 18 different bits of jargon that if you don't understand them they're gonna take you months . And then even the simpler ones are like okay , let's build an application . Well , well , all that is gonna require is one server that's a load balancer , three servers for your application . You're gonna need one Postgres server , but you want it to be highly available , so you wanna have two different replicas . Here's how you're gonna set up your replica mesh . Now , obviously you're also gonna want Redis for your job queue , but you're gonna want a different retention policy for that Redis than the Redis for your cache . And here's how retention policies work . And you're gonna have those two servers , but obviously you want high availability , so you wanna replicate each of those servers . And then all of a sudden it's like you have a network topology of 10 servers and that is .
Speaker 2I'm not saying that that's a bad way to deploy applications . That's a great way to deploy applications when you have those needs . But if that's the story that we are presenting to people , for here's how you get an application on the internet that's like good and solid and like legit , not like hey , here's my little toy demo and it's just real scaffolding , I just don't think it is approachable , and React has started to become unapproachable . So let's take the opportunity to make Rails really approachable and to say like this isn't a toy , this isn't for tiny applications , this isn't for demos Like you can run millions of dollars of business on a single machine .
Speaker 2You can make it highly resilient , you can debug problems in five minutes and you only have to really learn two tools Like I barely know Docker , I know Rails and I know SQLite and I am incredibly productive and I know almost nothing else kind of a two trick pony , but it's possible . So that's . I think those are three interrelated reasons , but cost is not , in my mind , the biggest one , but it is a nice perk , I would say .
Speaker 3Yeah , I was just thinking when you were doing all that description about the map of servers and you ended up at , okay , you need 10 servers , and in my mind all it was , yeah , and I only have 10 customers right now . So , yeah , I think this whole thing , I think it's a thing that us engineers are doing that over engineering our expectations as well . Like , I need a real database because I'm going to build a real app , a real product that is going to have like millions of customers , right , and I cannot toy around with SQLite . You know , and that's why , like , that's what you're hearing from senior developers , people that are really building at scale , right , maybe they have different needs , who knows ?
Speaker 3And I think that for a junior developer or something that's starting out , you're kind of afraid of like going , like I'll say , down on the SQLite path or the things that are not real , even though they actually know that they're going to have 10 customers in the end , you know , for the first year or two or whatever . So , yeah , it's a thing of managing the expectations and how we talk to people and we should be a little bit more , a little bit less strict about , yes , this is the only way to do it . You know , like I said , like load balancers and multiple services , yeah , we should be a little bit less strict about that and , yeah , make it more approachable and make it fun a little bit Like what the hell .
SQLite Directory Project Discussion
Speaker 3We started with PHP and FTP , like I did that , and it was all good .
Speaker 1Yeah , so around 2015 , when I was just learning to build right also , it was common you would like have SQLite in the development group in the gem file and you would have like Heroku 12 factor and or something like that and the gem Postgres in the production group and like in the database for ML . You would have like , in the development you used SQLite and in production you would use Postgres . And I remember I really build my fingers at some point of time when I was like doing development in SQLite . Then I tried to deploy it to production and the app crashes because , like the I don't know the schema or the like queries that I was doing in the that were compatible with the SQLite database were not compatible with the Postgres database . So I kind of spent a while on that and since then all this , when I created like Rails new , I used type Rails new database equals Postgres .
Speaker 2Yeah , I hit those same problems myself . I remember the first time and this has actually been changed recently but I had a whole bunch of queries that we're using the double equals operator . So I had some string SQL and I was using double equals to check equality and I I mean this was years ago , I can't even remember why . I guess I was running the tests against SQLite it must have been that and everything was failing . I could not figure out what was going on and it's like at the time SQLite only supported single equal sign for equality checks . I don't know . Yeah , I remember and I was like that just looks ugly , I don't want to do it . It looks like a sign . It doesn't look like a Boolean check , yeah , and those kinds of gaps and even like just the gaps were like the Rails adapter doesn't implement a feature for one database when it could can get frustrating for sure .
Speaker 1Talking of light stack , so there was this repository Ruby video for the website rubyvideodev , maintained by Adrian Polly , and it's running on the light stack . So I was actually exploring the code base and it's quite interesting to see like light stack gem installed and being actually used . I will leave a link in the description .
Speaker 2I want to do a SQLite directory , similar to Gem file directory or Kamal directory , to have a place for us to , as a community of web developers using SQLite , to like build a repository of proof that , yes , it is production ready , because it's being run in production and has been for years . And then I'm also working to just like collect . There is a decent amount of smart thinking on how to use SQLite in the context of a web application , but there's not a ton of it and it's very dispersed . It's dispersed across time . You know you'll find like I have found some like really excellent stuff from like 2010 . And it's dispersed around the internet . So I'm starting a curation project to just like gather as much and have a centralized place to get as much of the good content . I saw a tweet from you that you started working on .
Speaker 3SQLite directory . Tell us what's that , what are you building there and how can people help ?
Speaker 2Yeah , so this is a project directly inspired by Josh Pickford's Gem file directory , which was him building an idea from Rob Zulkis , I think , to just have a place on the internet for people to submit . Here is an actual , real web application that is running in production and it's using SQLite . And as a community of web developers using SQLite , we can build up a lot of the web applications and build up this directory of demonstrable proof that SQLite is in fact production ready because it is being run in production , and I'm specifically wanting to allow people to provide a little bit more information , if they want , optionally , about how they're running it . So like how many CPUs are you running it on ? How much RAM do you have ? What's your disk size ? What host are you running it on ? Do you know information about your max request per second or your average request per second ? To just help build a clearer sense that a lot of these myths that like , okay , sure , if you only have 10 requests per second , sure I guess you could use SQLite ? And it's like no , I mean DHH was just tweeting about it they have a machine , one of their beefy machines from their CloudExit and they put their new once product campfire on it and low tested it and hit 50,000 concurrent requests , so slightly more than 10 .
Speaker 2So , yeah , that is just a project for the community to start to build out a collection of applications that are actually using SQLite in whatever ways they are , so that we can aggregate Like this is real , like this is . I worry sometimes that people start to think that these things that start to have a bit of hype on the internet are just being driven by the inertia of the hype and it's all smoke . So just to like ground it and say , hey , I'm not talking about this because I'm selling anything , because I'm trying to make money . I run multiple applications at work on SQLite and I've really enjoyed the different developer experience that it has unlocked for me . I think that other people might want or need that developer experience and I have the opportunity to take the time to help push this niche inside of the Rails community forward . So , yeah , that project is just this hope that we can build a large enough collection of real applications running SQLite with enough detail to just put to bed this myth that it's for toy projects , it's for tiny projects , it's all theoretical .
Speaker 1Actually , sqlite directory sounds not just contained to Ruby world , but it can be not Ruby applications but all kinds of applications running SQLite . And also you mentioned SQLite directory and you mentioned like GM file directory and drop Zolkus's native directory . I'm just in a trend here . There should be lots of directories popping up like this directory , that directory . I'm sure one of the next weeks I will be like HTMLX directory . Yeah , you should check that the domain is not taken .
Speaker 3I'm trying to take directory . Does directory right now ? I'm one line . It's unavailable , goddammit .
Speaker 2I genuinely hope that if you visit it , it just redirects you to yahoocom .
Speaker 3Yeah , let me do that right now . Cool , so like people know you as the SQLite guy . Why is that and why ? How can other people be the guy of anything ? It's an off topic question or whatever comes to mind . How can other people be the guy ?
Speaker 2Yeah , I actually really love that question because it both happened accidentally and on purpose . So let me unpack both halves of that . The accidental half is that I just sort of somewhat accidentally started . I just experimented to see what would it be like to run an application with SQLite . Would it help me out , given the constraints I had ? And I enjoyed it . And so I did the next one and the next one and I hit a few rough edges and wrote some monkey patches in my applications and continued on . But it became intentional after I watched the Lerakon talk by Aaron Francis on creating your own luck .
Speaker 2And he described sort of his journey and it started off similar to mine . It's like I have been doing Ruby on Rails development for almost I guess maybe 10 years now and have benefited so much from the community and from the people who are active and present and energetic in the community . But I've always been very much like a voyeur , very passive and just sort of leaching from the community . But I can confess , as he also confessed , like I always thought like but I'm good , I'm good at this , I feel confident that I'm at least just better than average and good enough that I could do something of value , I could theoretically give value back to the community . But like what and where and how ? Do I not get wrapped up in my own ego and just I don't know how to do it ? I just never really pursued it . But I had started to enjoy this different sort of experience running these applications at work with SQLite and I saw his talk and around the same time I came across the Lightstack project and I was like , okay , I want to really explore this and see if I can make this work . But I wanted to try to do it without installing a gem , because that's another part of the aggressive simplicity of these R&D projects . It's like as few gems as possible . And yeah , so I just started to like push into it and I just sort of told myself , you know what , like , let's just try some things out .
Speaker 2And there was a really critical moment I opened . So I just started writing , I resurrected my blog and I started publishing and I was in a really energetic mood . So I was like publishing every day and after like maybe five days or something , I had gotten some feedback on Twitter and I had written a post about how to configure a SQLite database to better fit the context of a web application and I got some tweets back on oh yeah , you can also tweak some compilation settings and that can help a few things . I was like , oh , that's cool . So I started experimenting with how could you configure the compilation flags for SQLite in your Rails application . And I couldn't quite figure it out .
Speaker 2And I opened an issue on the SQLite 3 Ruby repository and one of the maintainers there , mike DeLessio , was like hey , or , I sort of I framed it initially as here's a bug , I'm trying to do this and it's not working , and it's a problem with the SQLite 3 Ruby Jump , fix it . And he was like , hey , I don't quite understand the problem , but maybe we could hop on a call and I could , you know , pair with you and see what's going on . And I'm like , okay , cool , like I'm free . So he just hops on a call with me , we dig into it , we talk . Through talking , he helps me understand that I'm the thing I'm trying to do . I shouldn't try to do it in this way , which is hard . I should do it in this way , which is easy . And all he has to do is make one change to the SQLite 3 , ruby Jump and then it'll be easy and everything will work . And I'm like that's awesome . And he makes the change and pushes a new release and in that interaction , like how , like open and easy and encouraging he was , it just sort of like pushed me over the top . I had like this momentum . I was like you know what , like I need to give , I need to stop taking and I need to give . And so I started and I opened a couple pull request to Rails to just implement some of these missing features . And he paint some Shopify developers to have them actually review it . So it wasn't , you know , one of like 300 PRs that never even gets read no shade , just a description , and from there you know like you just get a little bit of confidence in it , allows you to keep going .
Speaker 2I want to pull out from like that story , like , I think , two key points . And the first one is that if you recognize that you have been receiving energy and value from a community , that's only possible because there are people who are active and energetic and positive in that community . And they're just people . I mean , you know , we're all just people . We've got lives and jobs and families , and like there's nothing particularly difficult or special about just being present and positive and energetic Doesn't , doesn't require any special skills or special interests , is just like be there and be positive , be energetic , and that's how we keep these communities alive and thriving .
Speaker 2And then the second key point is like inertia is the most powerful force in the universe . So if you can just start , it's amazing where you will end up , like I literally I tweeted about it because it kind of stunned me a bit to look back and be like I have now written the C code that is in , you know , the SQLite three Ruby gem , like one of the most popular gems downloaded by most Rails applications , and I can definitively say I don't know , see , like you can't . You know , give me a coding challenge and say solve this challenge and see , but I learned enough that I could do this and that's just , that's just inertia , right , like that's just , you do things and it gets easier to do the next thing , gets easier to do the next thing , and the next thing you're doing is bigger and it snowballs . So really just like , just start , find something
Building a New Web Framework
Speaker 2.
Speaker 2For me it started off with an interest , it turned into a passion through the energy that I received back and the kindness that I received back . Like old Moe , he was like , hey , you want to hop on . I was like , I want to understand your gem , can we hop on a call ? And you just walk me through the source . He's like Sure , hop on a call . You know , mike , hop on a call with me . These kinds of activities , this kind of presence and energy like drew me in and now I'm just trying to like give that back and I hope that more and more people enter into that kind of way of thinking . Say you know what , just let me be present , let me be positive , see where it goes .
Speaker 3That sounds great and it seems like he he made you a believer from one one phone call , one video call and that's just great .
Speaker 2Yeah , I saw him at Rails World two and I got to thank him and tell him like you know , probably for you that was just like a random 30 minutes , but it was actually like really meaningful and valuable to me and the way in which you sort of like didn't just treat it like oh here's a bug report or something you know , he was so present and positive and it really can have such a profound effect like these things that seem so small to us . That's why I really keep harping on it like presence and positivity are profoundly high . Leverage things Right . Like it's one of the things I I like about this show and I like about like the general perspective of it . Right . Like it's easy to under appreciate the value of friendliness , but it is like profoundly valuable and and can really change the trajectory of people and their lives and communities . So friendly show for the win .
Speaker 3You're making us blush man , thank you .
Speaker 1How can we make Steven blush ?
Speaker 3Yes . Well , I have one more question . This is something that I can't remember who , but somebody told me , like I think it was at Helvetic . I said Do you know ? There's this guy here ? You know , steven , he's . He's working with Joel Draper , our new framework , which is totally out of this world . Like I got to find him and talk to him and then , like I knew you and I said , okay , steven , tell me what , what , what are you working with ? Like we want to , we want to hear more .
Speaker 1The escalide guy working with the flag sky .
Speaker 2Yeah , maybe we should rename it to that . But yeah , it's , it's an exciting project . It's also a daunting project . Right now it has we've given it a code name because code names are cool . So it has code name , yippie .
Speaker 2But yeah , joel and I connected I don't know a while back and just have been chatting and and through chatting we came to recognize that we thought about web development really similarly and we were both interested in one particular question which is like , if you started a web application framework from first principles today , right , it's like , for example , some of those principles would be in VME drives are the default . Yeah , this is like the push of all of the solid gems . Is this recognition that you're not going to a spinning disk , like the difference between going to disk and going to RAM is not you know three orders of magnitude and performance . That's one big difference . Like containerization is something that is fairly well established and understood . I don't understand it as well as I'd like , but most of the other smart people understand it . And so that's like , just as an example , some of the first principles , like some of the things if you're just starting today you would take for granted and sort of not have backwards compatibility to like those are framework work well if you have a server with a spinning disk .
Speaker 2The other sort of main thought was OK , we know it in our bones that you could build an incredibly large , performant , valuable , resilient web application purely through vertical scaling . You could SQLite all the things . You could get massive , I mean even just on AWS . You can rent like these absolutely massive machines . But you know , no one else will believe us until we can demonstrate it . So that's like what is it going to take to be able to build like a truly world class web application on a single machine ? And just through chatting , you know we had some similar ideas around some principles and some technical decisions and so we just started hacking to . You know like what does the routing look like ? What does the model layer look like ? What have yous look like ? Hint their flex .
Speaker 1I was surprised .
Speaker 2But the vision and it's like constantly evolving from like what exactly does a , a beta version of this look like ? What gets in and what do you push off ? But the vision is to present an alternative to Rails and to Hanami that tries to strike a distinctive third track where it has the articulation of Rails . So both of us find that Hanami's a little bit too verbose . I don't like the pyramid of namespaces . You have to have five namespaces to do anything and visually I just can't stand it . I'm too OCD for that . So it's Rails-like in that way , but it's Hanami-like in that it is aggressively trying to not do dark magic , only light magic .
Speaker 2We want zero leaky abstractions . So we want something that is very expressive , very beautiful and very easy to write , but the abstractions mathematically can't leak . This is one of the things I really like about Flex . It's incredibly expressive , it's elegant , it's beautiful , but the abstractions can't leak . There's a direct , one-to-one mapping between elements , attributes and children and there's nothing else added . So that is the Yipi project . We took a break over the holidays and both took vacations and hung out with our families , but I'm actually getting on a call with him tomorrow . We're getting back into it . We have the core rudiments of routes and controllers and models and views , and we're just trying to figure out how much polish we want to add before we put something out there in beta form .
Speaker 3Nice Sounds great . I really want to dig in and check out the beta and I wish you guys luck , because I know this isn't easy building up a new framework and trying to think of everything yeah .
Speaker 2And it's tricky to find that balance between you want to do something world-class and not have any issues , but you also try to stop it . I have literally been meditating for months on how could you , even just theoretically , do zero downtime migrations with SQLite . I actually now have a plan , but it took me just thinking about it for months and then , at the same time , trying to experiment with a declarative approach to migrations is like , okay , what if you didn't write migrations ? What if you just described your schema in your models ? And if you make a change to the description of your schemas , we compute a delta and figure out how to get from state A to state B . And that led me into like I have been battling with myself to convince myself to not write a Ruby parser that generates a Ruby AST of SQLite's SQL syntax , because ultimately that's what I would need to do to do what I want to do perfectly . But I'm also like but that's going to take me a month and I don't want to spend a month doing that . So how can I find a higher leverage choice ? So , yeah , it's daunting on a number of fronts , but it's also . I think in general it's really valuable .
Speaker 2I would recommend to people . I would not recommend try to build a new web application framework , but I would recommend like to really challenge your assumptions and to challenge your priors and to really consider , like I wonder if I could do this thing that , on first blush , seems to me either stupid or impossible , and just give yourself 30 minutes to really think about it and see is it actually impossible ? Is it actually stupid ? I mean , this is sort of what ultimately landed me working with SQLite . Like everyone , I had read the internet , you know I have a sense . People say that's stupid . Don't do that . I spent a little bit of time and I thought about it and I was like well , it's not stupid for me , though , and I think that that's true of a lot of things , so take the opportunity to challenge convention .
Speaker 3That's just great advice , and I think it's a good , it's a good motto on which we can close . Tell me , tell us , steven , where can people find you online ?
Speaker 2Well , I am at Fractal Mind everywhere . I kind of got lucky and stumbled into good branding so you can find me on Twitter . I'm most active on Twitter , but I'm also on the Ruby Social Master Don . Fractal Mind is my GitHub handle . I write mostly about SQLite related Rails things , but also just other Rails things on my blog at fractalmindgithubio , and I say this every time I'm asked this question and I really mean it , so I try to take a little bit of time to flesh it out .
Community Support and Collaboration
Speaker 2I received so much value from people in the community who are willing to take time to talk with me and answer my questions and consider the things that I was trying to do and think through it with me , like , is this the right thing to do ? Is there a better way to do it ? And so I recognize that it is my responsibility to offer that back and I take that very seriously and I genuinely really enjoy it . So I know everyone says like please do reach out , but I really genuinely mean it . Please do reach out .
Speaker 2I have a karmic debt to pay . You would be really helping me out . And , plus , part of my passion is to find as many of the rough edges as I can and smooth them out , and I can only do so many different features in my SQLite apps . So if you have problems and you want to or you're considering , is this even possible ? Is this doable ? Like , write me on Twitter , dm me , write me on Mastodon what have you open an issue in a pull request or in a repository ? I am very much open and interested in having conversations with people and taking the opportunity to pay back my karmic debts .
Speaker 3Cool . Well , thank you so much for joining this evening , Stephen . We'll keep in touch on the channels on Twitter and hope to see you in the new events of 2024 and the conferences .
Speaker 2Yeah , thank you both for having me .
Speaker 3Okay , see you , bye .
