This is part 2 of my conversation with Joe Emison, who has been building serverless applications since 2015 with Firebase on GCP and then later with AWS Lambda. We talked Joe's recruitment philosophy for Branch Insurance, where he primarily hires junior frontend developers, for two very pragmatic reasons. We also talked about pain points that Joe sees with adopting serverless today and why Microsoft will be the main challenger to AWS in the cloud computing market.
In part 1, we discussed the value of serverless, especially to startups, and why he has built Branch Insurance to be fully serverless. Joe is a strong proponent for the monorepo approach, and we had a good chat about why he feels so strongly about this approach and how his team structures their application in a monorepo. And of course, we also covered the pain points of this approach and the tradeoffs his team has accepted.
You can find Joe on Twitter as @JoeEmison and check out his new venture gobranch.com.
For more stories about real-world use of serverless technologies, please follow us on Twitter as @RealWorldSls and subscribe to this podcast.
Opening theme song:
Cheery Monday by Kevin MacLeod
Link: https://incompetech.filmmusic.io/song/3495-cheery-monday/
License: http://creativecommons.org/licenses/by/4.0/
This is part 2 of my conversation with Joe Emison, who has been building serverless applications since 2015 with Firebase on GCP and then later with AWS Lambda. We talked Joe's recruitment philosophy for Branch Insurance, where he primarily hires junior frontend developers, for two very pragmatic reasons. We also talked about pain points that Joe sees with adopting serverless today and why Microsoft will be the main challenger to AWS in the cloud computing market.
In part 1, we discussed the value of serverless, especially to startups, and why he has built Branch Insurance to be fully serverless. Joe is a strong proponent for the monorepo approach, and we had a good chat about why he feels so strongly about this approach and how his team structures their application in a monorepo. And of course, we also covered the pain points of this approach and the tradeoffs his team has accepted.
You can find Joe on Twitter as @JoeEmison and check out his new venture gobranch.com.
For more stories about real-world use of serverless technologies, please follow us on Twitter as @RealWorldSls and subscribe to this podcast.
Opening theme song:
Cheery Monday by Kevin MacLeod
Link: https://incompetech.filmmusic.io/song/3495-cheery-monday/
License: http://creativecommons.org/licenses/by/4.0/
Yan Cui: 0:00
In this episode of Real World Serverless I finished my conversation with Joe Emison where we talked about his recruitment strategy and why he only hires junior front end developers for his fully service stops and the challenges that he sees with Serverless adoption today. And just also want to touch on something that you just mentioned there that unless you've got all these superstar developers, coding is probably not going to be perfect the first time they just finished writing it. One of your really interesting philosophies around recruitment, which I think is very different from pretty much everyone I've spoken with is that you primarily hire junior front end developers. So what is your rationale behind this approach and how well has it worked for you so far?
Joe Emison: 0:55
Well, so the rationale's several reasons. I think the primary philosophical reason for this is that, so I've been working with software developers for about 25 years now and certainly over the last 15 I have noticed that when you hire senior developers, or any developer you hire with as an experienced developer in the tools that you're using, you're getting experience. But the other thing you're getting is generally speaking a belief that important decisions about architecture and ... Well important decisions about application architecture and important decisions that will really affect the future of that application need to be in the domain of those senior hires. But if you combine that with two realities today, one of which is that we have no good continuing education practice in software development. There is simply nothing out there that will make it simple and easy for anyone to understand what the best way to do things is today versus five years ago or even 18 months ago. And if you combine it with, and this is certainly true in America and certainly true in larger cities, but a real arrogance in software developers. It's not all software developers, but it is, there is a real brogrammer culture in software development that has, I mean I think it's epitomized by the sort of stereotypical response on Hacker News, which is just brimming with self-confidence, arrogance and generally derision for whatever thing they're attacking. There aren't a lot of support of, "Wow, that's really interesting. I hadn't thought of that before. That goes against everything that I've done to date, but maybe I should look at it." Like that's never a response you see, that's just not any. But that's how you learn, right? That's what your continued education would do.
Yan Cui: 3:20
Absolutely. That's one of the best thing I find about being a programmer, constantly someone telling you something that you haven't thought about, your world shattering views coming in, which I think that's where you kind of learn the most in terms of what you don't know. But like you said, "You almost never get those kinds of responses whenever you try to challenge someone's pre-existing views on how things work."
Joe Emison: 3:43
Right, right. And so there is this just challenging recruitment then, especially in Serverless I think, which is that ... So I mean I'll give some examples. So I built these two Firebase applications and I mean I designed and then great software developers build these two Firebase applications that were Serverless. and we were essentially bought by a company and they were brought in-house at that company with their much larger development teams. And instead of, and there was great traction and people were really happy about them. And instead of continuing and putting resources behind feature developments, the immediate decision of the technology leadership, and I mean I was actually the CIO within the organization for a while. But the immediate decision amongst the kind of VPs of engineering and all of the teams who were responsible for all of the infrastructure within the organization was that the most important development work that could be done on these was ripping out Firebase and replacing it with SQL Server. And that's insane nonsense. That is obviously not the right decision. I mean there were problems with the application. I mean the application was working fine, but this was, I don't know if it was an existential threat to the SQL Server database. I mean I suspect it was, it certainly felt that way. The challenge then is how much time, and in that organization there was a lot that needed to be worked on. And so it would not have been a good first choice decision of mine to spend all my time fighting for no don't pursue this as the next sort of major development work. Because I had lots of other things I needed to deal with. But in starting Branch, a real question I had was, "So we've set up and built this great Serverless infrastructure and architecture." It has really great velocity and we've had these front end developers come in and they basically are full stack developers in Serverless, because it's some Node.js functions. It's understanding some cloud formation YAML. We use Stackery to manage a bunch of that stuff. So it's kind of like understand how to use a Stackery CLI tool, which is really simple. There's just a lot of these very easy things to do that you can take. You know, front end developer can easily learn everything else that's needed, especially if you're using Node.js for your functions. And so I was facing this situation of, well if I hire a senior developer then I have to go off or a senior VP of Engineering, and I need to go off and raise more money or deal with other aspects of the business, what's going to happen? Is it going to be, are we going to be running on containers when I turn around. How do we keep these business benefits that we have of Serverless? And then combining that with, so we're in Columbus, Ohio and a lot of the market for senior developers in Columbus, Ohio is very much .Net senior developers and Ruby on Rails senior developers. And so there aren't necessarily an enormous number of Node.js senior developers. I mean, it's relatively new and has been relatively derided by a lot of these same programmers as a bad backend language, which I think is thoroughly untrue. And so all of that made it quite easy to say, "Well, let's go get some front end developers who pass a skills test of showing that they kind of think organizationally the right way, they can take a design in Zeplin and render it properly in CSS and have a sort of a good mental model of how react components would work and where you would put certain styles and component code, how you would organize that. And then let's invest in them. And that's really what we've done. And it's worked out really, really well. Eventually you need managers and VP engineering and senior developers and that's, I mean certainly from an HR perspective and from a coaching perspective and all of the great things that senior developers and managers can do. And so it's still an open question for us as to is that something that will come out of the developers we've hired. Will we hire more senior developers and sort of struggle with this particular task and probably spent a really long time recruiting managers and a VP engineering when we need one. But for now it's working quite well. We have roughly 10 full-time developers now working, and I will say that a lot of the stack choices and the service choices have really changed, even the requirements that we have on how many developers per product manager, how many developers per kind of scrum master or person who's trying to ... Or business analyst who's trying to unblock developers. The whole thing is a lot smoother than it normally is in a non-Serverless environment in my experience.
Yan Cui: 9:09
That's great to hear. And I love the pragmatism that you apply to all the technical as well as business decisions at your company. So in your view, what do you think are the main challenges that we have when it comes to building a Serverless architecture or application. And what are some of the technical platform limitations and things that are making your life difficult?
Joe Emison: 9:32
So I think that debugging and local development, so we know how to do local development and we know how to debug, but there's still less simple and accessible for most developers than a more traditional non-Serverless development environment. One of the things that can be great, just to give an example, if you're an Amazon web services and you're writing lambdas. Being able to write a lambda in the console, save it and test it there is an amazingly fast and great development experience.
Joe Emison: 9:32
But as soon as that, and this happens to us for basically all of our functions is, as soon as they get too big to be able to do them in the console. And again, there are a variety of things you can do. You can take CloudWatch events and sort of inject them through node and run those. And so we do those. We write these sort of sample event files and we can run locally against them. But it's not 100% the same environment. We're pretty spoiled in Serverless I think. And in our ed Branch and our development environments where things are just the same.
Joe Emison: 0:00
So again, we just don't have these separations where people say, "Well, they work in my environment and not yours." But this is one of those situations where sometimes those local lambda invocations just end up being different from remote ones. And so one example of those is sometimes the local lambda invocations will run, and maybe this is largely maybe a node problem, but sometimes those local indications will run to completion. And sometimes they will kind of terminate early when they're running in Lambda, or run by AWS Lambda. And so that's something that's really hard to identify and really hard to debug locally. It's not that common, but you do get enough difference in those environments where that is a challenge. And figuring out why something isn't working is a challenge. And so sometimes you end up doing the development and then running the deploy within your local environment. Which can take, in most of these cases we're talking four or five minutes, but it's still, your dev test cycles are not great. This is the worst with a CloudFront, right?
Joe Emison: 0:00
You know, when we were building and setting up the server-side rendering and keep in mind we have multiple react apps. So we have server-side render ones, we have other ones. There's a lot of path prefixes in CloudFront and different routing Lambda@edge functions and testing those. And the dev test on that was awful. So those are not tiny problems, but they are, I think that's an easy trade off on those. And we still don't love the latency that you can get on CloudStart. Although again Lambda with its provision capacity is a great, at least somewhat solution for that.
Joe Emison: 0:00
There are those. And then I think, this is true about every piece of software, but getting better at observability, getting better at how we aggregate all of our events and being able to dive into specific historical records like what Honeycomb I think is focused on doing. That's something that, at least in my experience has been slightly more challenging on the Serverless side. But again, not insurmountable and I think is the ... And I don't think most organizations do that well at all anyhow today. And so maybe it's easier to when you've got a good solution to implement it in Serverless. But that also can be a challenge just trying to look back at what exactly happened based upon CloudWatch, in particular CloudWatch insights is kind of annoying and difficult to use. But overall I'll take those downsides to get all the nice benefits.
Yan Cui: 13:36
So given some of these tooling, or lack of tooling support for local development and for observability. Do you think those are the areas that a cloud provider like AWS should be focusing on improving? Or are there some other areas that you think should deserve some more attention as well?
Joe Emison: 13:54
Well, I think in general, so if I were going to take cloud providers in general, I would say that Microsoft really doesn't have a product like Google and Amazon have. And one day Microsoft is going to need to wake up and realize that what the Amplify and AppSync ecosystem in Amazon and what Firebase at Google provide is something entirely unique that they don't have. And so Microsoft Azure, I've often said they need to buy Netlify, which is kind of the last independent, although maybe Azure are. Which is a GraphQL service that can sit in front of relational databases that can give some of those benefits as well.
Joe Emison: 14:42
But Microsoft needs to do something in this space. Anyone who's I think doing Serverless on Microsoft Azure that isn't just simple backend functions, but is trying to build full platforms and ecosystems that are Serverless. Just know you're missing these key tools that exist on other clouds that really change, it make your applications a lot less complex. And then I think specifically within Google and Amazon, the things I like CloudForce to work on end up being very highly specific things. So neither of them has a good way to serverlessly index data to something like a Serverless Elasticsearch. And so there's a service called Algolia, which is phenomenal, which does this, but it's an independent company. And so both Google and Amazon will just have, you basically have to run your own Elasticsearch VMs. Managed Elasticsearch at Amazon isn't that much better than just running your own Elasticsearch and it's not Serverless.And so there's a service called Algolia, which is phenomenal, which does this, but it's an independent company. And so both Google and Amazon will just have, you basically have to run your own Elasticsearch VMs. Managed Elasticsearch at Amazon isn't that much better than just running your own Elasticsearch and it's not Serverless.
Joe Emison: 15:25
And so one other thing is that the authentication frameworks that live in it is if you want ... One of the biggest benefits you can get out of Serverless is by not ruling your own authentication. And so Amazon's Cognito, and Firebase's Firebase Off are great services, but they both lag behind the independent in the space Auth0 by such an inordinate gap. And especially Cognito has lots of pain points and hasn't had a great development velocity at least from the outside.
Joe Emison: 16:27
And so the main things I'd like those cloud providers to do is just to continue to build up these critical infrastructure services and get them in the ballpark of the features of what you can get from independent services out there. But beyond that, I think Amazon's doing a great job with all of the work they're doing with Amplify to make things easier for front end developers to just go and build these entire applications and backends. But from a front end development toolkit and mindset I think that that is great work. And I definitely think Amazon should continue working on that Amplify path. Because I think that is that that's ... The future is going to be, in my belief the future is going to be front end developers are going to obviously do more and more and more of the full stack in conjunction with the cloud provider. And so the more the cloud providers are helping front end developers get there faster, I think the better.
Yan Cui: 17:25
Yes. It's funny that you mentioned Cognito, I get so many questions from customers about Cognito because it's so complex. And like you said, there's so many different missing features and pain points that customer find it very difficult to navigate themselves. But at the same time I also find that Auth0's pricing model is ridiculously expensive-
Joe Emison: 17:25
Oh, it's insane.
Yan Cui: 0:00
... especially when you've got, this whole MAU-based pricing model, it's just the death penalty for a lot of companies that have no recurring users. I've got a few customers who are in the insurance world where the customers will come to your website, they ask for a quote and never come back. But they pay for every single one of them as a recurring monthly user. So when they look at a pricing for something like Auth0 it just becomes astronomical.
Joe Emison: 18:12
No, I completely agree that Auth0's pricing puts a lot of potential customers totally out. And I do think that you get so many benefits for Cognito being integrated into the Amazon ecosystem, especially within AppSync. I mean, it's phenomenal to be able to write these functions and not have to deal with any sort of authentication or verification of user. But just be passed information about this is the user, they've authenticated, here are the groups they're in and you do no work. I mean, it's fantastic. But I'll tell you the number one, if you asked our customer support team what the number one pain point that our customers have with all of our insurance products, all of our systems. they would all unanimously say logging in. Cognito's password reset and email verification setups is very rigid in terms of how it handles things.
Joe Emison: 18:45
And there are a lot of people who are somewhat afraid of their computers and who aren't that good at figuring out where is that email. How Cognito sends its verification codes and when it sends them have caused a very small, but non-zero portion of our user base to require a lot of our time. And we have built all these custom ways to reset passwords by making eight different Cognito calls and sending out custom emails based on them. So it's a challenge within Cognito. And so I would love for Amazon to fix those problems and get it closer to how great Auth0 is. But I agree that we could not afford Auth0.
Yan Cui: 20:11
And in terms of this, I really love the point you made about Microsoft not having that sort of front end developer centric tooling like you have with Firebase and with Amplify. Do you think a lot of that is because Microsoft has traditionally be very strong in the enterprise world and a lot of enterprise customers just haven't been asking for something like that and that's why they're sort of lagging behind in that particular space?
Joe Emison: 20:37
Yes, I think that's exactly it. I think Microsoft has a model of doing at least iterative development. And that model is that they go to the CIOs and VPs of engineering and senior developers who don't have a good way to do continuing education. And they ask them and they're very good at responding to them. But in a world where those people are going to ask you for faster horses instead of the car, you're just not going to get significant benefits there. And so what that means is that if you're Microsoft, you have to buy those innovations. And I think that's fine.
Joe Emison: 21:22
I mean, I'm fully on board with Microsoft buying Netlify, taking their leadership and letting them run the strategy. I don't think that that's a bad way to go, but I do think it means that Microsoft runs the risk of missing out on really important changes in how we do software development. And they are right now with Serverless. But again, if I were to bet on Google improving its development velocity and its customer centricness with Firebase or Microsoft figuring out the right acquisition, I'd bet on Microsoft. Just because I've watched Google, I feel like not properly treat Firebase post its acquisition of Firebase. And I really think Google had ... Google had basically a five year headstart and is now well behind Amazon in my view. And so that's why we, I started building on Google but switched.
Yan Cui: 22:19
Yeah, I think I agree with you as well in the sense of betting on who the main challenger to AWS is. It is going to be Microsoft. At least these guys understands enterprises and how to do business and support for enterprises. Where all the stories you hear about Google, just dumping customers, they're just not dealing with those kinds of support their customers needs. And also when you read about that announcement recently that, "Oh, unless they become the leader in the cloud, they're just going to pull out." I mean, how can any company have confidence in a partner like that. That you have to be number one in three years time or they're just going to pull out of the market altogether.
Joe Emison: 22:57
Right. And it plays into the entire reputation of Google that they just kill stuff off at will and you can't rely on them at all. And Amazon still has it simple DB that's been running for those customers. I completely agree with that. I also think that, I think Google recognizes its problems in serving enterprise customers and I think that they're ... I mean they're trying to hire a lot of salespeople so that they can at least feel enterprisy on the sales side.
Joe Emison: 23:14
I think the thing that I worry about as well though is that if you don't have a way of building product and of innovating on product, then you're in a lot of trouble. And Google has this huge problem that they're, at least as I see them have a lot of arrogance that they know what's best. A lot of arrogance that if you're not using the services the right way or in the way that they think you should, that you're the one who's the problem. And that's just in high contrast with how both Amazon and Microsoft view customers.
Joe Emison: 23:42
And so I think culturally Google, culturally and from a leadership perspective, I just don't see how Google gets on a track where they at all are leading on the product side. And so I think that yes, I don't think they'll make their numbers. I think that we'll see more stuff killed by Google based upon how they seem to think about things.
Yan Cui: 24:29
Yeah, That the cultural difference is startling. I interact with a lot of AWS people, they almost got this, I don't know, this sense you get where they tell you to beat them some more. Tell them what they're doing wrong.
Joe Emison: 24:59
Yeah, they do. That's very true yeah.
Yan Cui: 0:00
So Joe, thank you so much for taking the time to talk to me today. It's been an amazing conversation. Is there anything else that you want to tell the listeners, maybe Branch is hiring and how can the listeners find you on the internet? Joe, thank you so much for taking the time to talk to me today. It's been an amazing conversation. Is there anything else that you want to tell the listeners, maybe Branch is hiring and how can the listeners find you on the internet?
Joe Emison: 25:07
Yeah well, if you're in the U.S. and let's see, if you live in Arizona, Illinois, Missouri, Ohio or Texas, you should definitely try Branch, quote Branch for your home and or auto and or renters and or umbrella insurance at ourbranch.com. And if you are a senior developer of VP engineering and Branch sounds interesting to you and Columbus, Ohio sounds as a wonderful to you as it does to me, I'm always happy to get emails at joe@ourbranch.com. We're not presently hiring that position, but we will be, and love to start conversations with people as well. And just happy to talk with people about Serverless at all. But yeah, but thank you very much for having me. This was great.
Yan Cui: 26:00
So that's it for another episode of Real World Serverless. Thank you guys again for joining us for this wonderful conversation with Joe Emison. To find the show notes and the transcript for this episode, please go to realworldserverless.com. I will see you guys next time.