#47: Building an Instagram for DJs with FaunaDB with Keston Crandall

You can find Keston on Linkedin or by email at keston@squirb.com.

Check out Squirb at www.squirb.com and FaunaDB at fauna.com.

For more stories about real-world use of serverless technologies, please follow us on Twitter as @RealWorldSls and subscribe to this podcast.

To learn how to build production-ready serverless applications, check out my upcoming workshops.

Opening theme song:
Cheery Monday by Kevin MacLeod
Link: https://incompetech.filmmusic.io/song/3495-cheery-monday
License: http://creativecommons.org/licenses/by/4.0

Transcript

Yan Cui: 00:12  

Hi, welcome back to another episode of Real World Serverless, a podcast where I speak with real world practitioners and get their stories from the trenches. Today I'm joined by Keston Crandall who is going to tell us his story of how he builds Squirb all by himself, using serverless technologies. Hi, welcome to the show.

Keston Crandall: 00:31  

Hi, how are you?

Yan Cui: 00:32  

Yeah, I'm good. So we got in touch through the guys at FaunaDB and from what we discussed before the show it sounds like you've got some really interesting stories to tell here. And can you maybe give our listeners a quick introduction about yourself, what you've been doing and your experience with serverless so far?

Keston Crandall: 00:50  

Yeah, so my name is Keston Crandall. I recently graduated from the George Washington University with two masters degrees, one in Information Systems Technology and one in Business Analytics. I'm very passionate about music and electronic dance music. So, a long, a long time ago I used to, a couple years ago, when these contests, where you would be able to meet the DJ. And I thought that was really cool and how you could get these personalised interactions with musicians, so I kind of thought, how can you make that more ubiquitous across everyone and especially the more devoted fans because for some of them they were DJs that I just heard the first time and I just want some random contest so I probably wasn't the most deserving fan. So I thought of the idea of coming up with Squirb, which is sort of like Instagram, Pinterest, and some other features where you can subscribe to these artists somewhere to Pinterest and get rewards such as a phone call or maybe a meet and greet or going out to lunch with them or ask them a question or getting merchandise. So, and then there's also the social feature where they have a news feed you can follow their new releases. You can see their discographies. And you can also, there's another feature for, basically, it's crowd sourced promotion, so some people might not have as much money to be able to, you know, support their artists, but they might be able to promote on Instagram or Facebook, and they get points based on how well their posts do and then they can use that to spend on rewards that the artists, put up on their profile.

Yan Cui: 02:29  

Okay, so that's a pretty cool idea. I really like the fact that you have a way now to get in touch with your favourite artists through your platform. So how did you went about building your platform? What sort of technologies were you using to build this whole thing yourself?

Keston Crandall: 02:45  

So, we're using Azure Functions as our REST, REST layer that hosts our static landing page and hosts our entire REST API. Then we have Xamarin, which is for the mobile application. It's cross platform right now iOS and Android using Xamarin.Forms. We could also expand that to UWP and Mac if we wanted to, with a little bit of work. We also have FaunaDB as our database, and we have AZure Search API which powers the search bar.

Yan Cui: 03:19  

Okay, so you're using Azure Functions mainly as your REST layer, I guess, for the entire application, how long did that take you as a single person to put together? 

Keston Crandall: 03:29 

I would say it took me about maybe nine months working a little less than half time to develop the entire thing.

Yan Cui: 03:38  

And where would you say most of the time is spent? Was it spent on the UI side of things where you sort of customise the UI, the user experience and all of that?

Keston Crandall: 03:47  

Yeah, so the UI was, was a big portion crawling the relevant music APIs was somewhat difficult, because I was trying to link albums together. So when you get an album, you can open it up in Deezer or whatever. So crawling, I think I have Deezer Napster, Spotify, Apple Music and MusicBrainz are all crawled so that that was difficult to make sure that they all had good data. Yeah, I would definitely say that the user interface was the longest.

Yan Cui: 04:21  

So for your crawler. So these are Azure Functions that are being triggered by some kind of a cron job that pings those Spotify and other platforms and get the list of records and artists. Is that right?

Keston Crandall: 04:34  

Yes that's right, Azure Functions has something called timer triggers. And you basically set a cron interval and they'll be triggered every 10 minutes/hour, whatever you set. So I just keep going through that and every like 10 minutes I search for new stuff and it's just continuously crawling,

Yan Cui: 04:54  

And as a user that comes onto your platform and gets registered to the platform, how do you implement the user management side of things? Would you use something that came with Azure or you're using like a third party such as Auth0?

Keston Crandall: 05:07  

For user management, they just, whenever they change their account or make a post, they're just hitting a button on the app and then it sends data up to Azure Functions and then that gets stored in the database for the persistence in Fauna.

Yan Cui: 05:22  

Okay. So with Azure and Azure Functions, how do you, I guess decide to use Azure in the first place? Sounds like you came from a .Net background, because you're using Xamarin already. So was Azure the logical option when it comes to building the backend?

Keston Crandall: 05:40  

Yeah So Azure, if you're, if you're a .Net developer, I would say Azure has some of the best client libraries for C#, I mean that's I think what they develop first. Also, I was considering Lambda. The reason why I don't like Lambda as much as Azure Functions is because Azure Functions has the HTTP REST layer built in by default, and it takes almost no time to configure and the other nice thing is it's actually just an app service. So, when you, you can build your entire project and with one button in Visual Studio just ship it up so it's easy deployment, whereas Lambda, you have to manage each function that sort of own Lambda function you have to manage dependency for each individual function at service managing for the whole project. And then you have to do API Gateway for Lambda which is another kind of hassle so there's little more management with Lambda as opposed to Azure Functions for everything sort of just there for you and it takes a lot less time in my opinion,

Yan Cui: 06:37  

I think some of that you can mitigate a lot of those with the right tools. For example, when I'm working on with Lambda, I don't, you know, manage the dependency for individual functions. I developed them as a single project, and the whole thing is uploaded as one artifact. So there's a framework like the serverless framework where they kind of manage a lot of that for you. So it's not as much work as you may think. But I do think that having to have API Gateway is a bit of hassle in terms of configuration. If you ever do it by hand. Again, with the right tools like the serverless framework, it makes a lot of that easier, but you do have the additional cost upfront, well, the operational cost because API Gateway itself is not particularly cheap, but what it does give you, though, is a lot of control around authentication and all of that. So in this case, if I remember correctly with Azure Functions you to get the HTTP endpoints for free, but you don't have any way to authenticate requests with the HTTP endpoints. Is that right?

Keston Crandall: 07:38  

They do have a way for you to do. So I make mine anonymous functions so they're open up to the world and then I use my own custom authentication logic. But you are, you are able to have an authentication built in. I've not really explored that too much but you can set different authentication levels for each function.

Yan Cui: 08:00  

Okay. So with the database you pick the FaunaDB. And one of the, one of the things I thought Azure has got, has done really well is with the Cosmos DB. It’s got a lot of capabilities and from the outside looking in, Cosmos DB seems to be the de-facto sort of option we will go into. How come you chose FaunaDB what is the sort of main driver for that decision?

Keston Crandall: 08:25  

I actually really do like Cosmos. I think it's a wonderful database and I originally started using Cosmos for the first one or two weeks that I was developing this project. I think I was using the SQL API for Cosmos. The one thing that I was, as I was researching Cosmos more and more, I was thinking about transactions and doing payments and and moving funds around. And Cosmos, at the time when I was developing at least did not have cross collection, transactionality, so you couldn't have consistency across across collections. And that was a big issue. And the great thing about Fauna, is when you put anything inside of a do function in Fauna. Everything is either committed altogether or it's nothing's committed at all. So if you abort the transaction at any point in that transaction it fails and the whole thing fails, which is a very, very very useful thing. Another thing that is different now, but when I was developing Cosmos didn't have a consumption based truly serverless plan. They do now I think. But when I was doing it, you had to provision a certain number of, IOs, or RUs, or something. And I wanted to truly serverless where I could just put it up and it would automatically scale with the usage which Fauna had back then, and they do now. So yeah, I, I think some other reasons why Fauna is great is they allow you, their client libraries, especially in .Net have a bunch of functionalities. So the C# client library has an encoder and decoder so... Cosmos also has this but not everything has this, you can serialise all of your classes up very easily, and just create them in the database without having to property by property, write code to do that. So that comes out of the box. Another thing that Fauna is developing that I was really for is something that you probably would worry about and why you might choose Cosmos is they have the ability for you to host your database now in a specific region. So that can reduce your data egress charges and deal with compliance if you need it. So yeah, that that's some of the reasons why I picked Fauna.

Yan Cui: 10:55  

Okay, that's, that's such an interesting because, does that mean that the Fauna itself is runs on runs in Azure and that's how they're able to use the same regions and reduce the data egress charges?

Keston Crandall: 11:07  

Yeah, I don't want to like that but yeah, if you were. That's something that's very new and Fauna and they're working on. I think it should be available soon. If you request like an enterprise plan... But yeah, you can you can put it in a region, and it will auto scale and you'll still be paying in the serverless model, but you'll just get to choose where it runs.

Yan Cui: 11:30  

Okay that's such a quite interesting integration, model, pricing model for for Fauna. And before I think we, when we talked about this before the show, you also talked about potentially using some of the graph based queries you get with Fauna as well and you looked at Neo4j at the same time for some of those same queries.

Keston Crandall: 11:52  

Yeah, so I always think about my data like a graph and not like a traditional SQL data model. So in Fauna, the way you can do that is if you want to make for example an edge between two other collection types for example let's say with Facebook or MyApp you can like a post or you can like an album or something. That is an edge so it would be the like, the liker likes the likee and the likee can be a post or it can be different collection types whereas in an SQL database you wouldn't be able to do that because you would have to say, like, user like post and then you would have to enforce referential integrity from the many to many relationship to the post or the user but here you can have an artist likes the post, the user likes the post. And it's truly a graph model, you lose out on at least right now they don't have all the complex graph queries that something like Neo4j might have. But they said that they're very interested in that. And they're developing it and Fauna seems really on board with the graph train compared to the SQL train.

Yan Cui: 13:10  

Yeah, I've done quite a bit of work in the past with Neo4j and do love graph databases and do and, if possible, I would like to use a lot more. One of the problems I find with Neo4j just as, I guess, it's not easy to run it yourself. There's a couple of services out there like GrapheneDB that gives you hosted Neo4j, well I guess, the other problem with that with Neo4j is just that everything has to run in memory so you need pretty sizable machines to run them, which gets expensive quickly, and there's no horizontal scalability, I guess, well, you can have read replicas, but you can't have sharding. And is that something that FaunaDB addresses in terms of their scalability issue that you have with some of the solutions like Neo4j?

Keston Crandall: 13:55  

Yeah, so FaunaDB has pretty much, from what I can tell, an infinite scalability, and you can create as many indexes as you want and the indexes are the way you query data. So if you want to for example filter by, let's say, gender, you have to make an index for gender and then make an intersection or a difference, or make a compound index that has filters that has like terms for gender, and maybe age. Yeah, so, from my experience, you can just scale it pretty much as much as you want and Fauna takes care of all, all the scaling you don't really have to worry about that. In terms of querying, it is a, in reading, there's a lot more user setup than for example other databases. They require you to make indexes. So if you want to for example filter on gender and age you can make two index indexes one on gender one on age, and then do for example the intersection or the difference of the, or you can make a, you know, compound index that has terms of age and gender, and then put whatever you want to filter on. The great thing about indexes and Fauna, is they're very cheap, you can create as many of them as you want. You know, in an SQL database, the more indexes you create the worse your write throughput. In Fauna, that's just not the case, you can just have as many as you want, and it performs pretty much the same. So yeah that that's another reason why I like Fauna is because as a, as a user using the service, you really don't have to worry about scale or performance you can just figure out what data model you need, and make it and it should perform pretty well. You don't have to worry about for example running out of disk space because you have infinite disk space with Fauna. But yes, you will incur, I think more of a charge from your your disk usage in your bill from having extra indexes because it does. For example, for you index gender, you know you'll have male and female, for every single record in your user collection, and you have to, however much that costs in terms of if it's a boolean, if it is male or if it's a three gender for non-binary and you have a string there. Those bytes are going to cost you in storage for double storing them. So when I was developing it before I actually went live and did a lot of crawling, I actually had zero bill for the first, you know, nine months of development I didn't pay anything on Azure and I didn't pay anything on Fauna so I had $0 development cost. Once I went live my crawlers that are calling the music APIs which I was running, you know, 24/7. I accumulated in a month, I'm accumulating maybe like 50 million reads and somewhere between 10 and 20 million writes. So my bill ends up being somewhere between $40 and $60 a month for that, which is pretty manageable. It's still less than, if I would have went for example maybe a Mongo cluster or Neo4j cluster, I probably will be spending $200 a month just to make sure I have something that can handle a decent amount of traffic during a peak time or if something crazy were to happen.

Yan Cui: 17:11  

Yeah, so that definitely sounds familiar. I have had quite a few of those clusters in the past, even the dev ones you still have to have something just to be able to use it and they do rack up on your cost pretty quickly. And so you've been developing with Fauna, you've been developing with Azure for a while. How do you find that experience? Were there any sort of platform limits that you run into and make your life difficult?

Keston Crandall: 17:39  

Um, well with Fauna there were there were a few hiccups with the C# client library, but the development team was very receptive, they got those worked out. And now if anyone's developing with the Fauna client library, it should be pretty perfect because I've managed to work out a lot of those bugs and it seems it's a lot better now. In terms of Azure, there was a few pain points with, I was trying to do their durable orchestrators where you can scale out and scale in some of their, their functions. So for example, Let's say I want to bill users every month for their subscriptions. What you want to do is get all the users and then you want to scale out to have basically one function for each user and have Azure function sort of deal with that. I ran into some configuration problems where it would work on my laptop and on Azure, for some reason when I had the durable Nuget package, it would not work. So, and there were there were a couple other things with Azure where when you deploy it, it seemed like I had to work through a few problems. But after, you know, five or six hours I was able to work through that. It wasn't that big of a deal. I've heard that is sometimes one complaint with Azure Functions in the past. It seems like it's getting better at every year as they improve their, their service but people do say there are some of those weird little bugs you get with Azure Functions.

Yan Cui: 19:14  

That's pretty similar to what I've heard from other people as well. But in general, both the, both platforms like Azure and Lambda are improving all the time but there are always some niggling pains that you run into from time to time. And if you were to sort of build Squirb again, what would you have done different this time around, having done it again, having done it once already?

Keston Crandall: 19:38  

Um, to be honest, if I were to develop it in terms of the architecture and what technologies are used I think I would use it exactly the same way, I would have FaunadB, I would have Azure Functions. And I would use Xamarin. So I mean there's, there's some coding things that I've done that I might do a little differently but nothing too major. I think I would develop it pretty much the same way again.

Yan Cui: 20:02  

Okay, so I guess that's an indicator, that's a good indication that things didn't work out that badly then.

Keston Crandall: 20:09  

Yeah, they went pretty well. Almost all the bugs are UI bugs that I've run into on the mobile app so I haven't had almost any bugs with FaunaDB or Azure Functions and production.

Yan Cui: 20:21  

That’s really good to hear. So I would like to thank you for joining me today. Is there anything that you want to tell the listeners about Squirb and potential, maybe what you, what you're working on, are you doing any, any other new projects?

Keston Crandall: 20:37  

Yeah, so, I'm working on a sort of a Fintech project that deals with payments also and maintaining Squirb, while my partner is sort of promoting it. He owns a record label and he's kind of doing a small test with his artists. We haven't really tried to push it too hard. But in terms of Squirb but if you know any artists who want to come on board and want to join us, tell them to email us. They can email us at support@squirb.com. We'd love to have them on board. And if you want to try it out, just feel free to go to squirb.com, and you'll see the Android and iOS download links.

Yan Cui: 21:18  

Great. And how can people find you on the internet?

Keston Crandall: 21:22  

You can find me on LinkedIn by searching Keston Crandall. It's LinkedIn slash in slash Keston dash Crandall. It is probably the best way to connect with me, or you can email me at keston@squirb.com.

Yan Cui: 21:38  

Okay, great. I'll make sure those details are available in the show notes as well. And once again, thank you very much for joining me and sharing your experience with us. And yeah, take care and stay safe in this strange times.

Keston Crandall: 21:52  

Yeah, you too Yan. Stay safe.

Yan Cui: 21:55  

Okay. Take care, bye bye. 

Yan Cui: 22:08 

So that's it for another episode of Real World Serverless. To access the show notes, please go to realworldserverless.com. If you want to learn how to build production ready serverless applications, please check out my upcoming courses at productionreadyserverless.com. And I'll see you guys next time.