#57: Serverless at Purple Technology with Filip Pyrek

You can find Filip on Twitter as @FilipPyrek

Links:

For more stories about real-world use of serverless technologies, please follow us on Twitter as @RealWorldSls and subscribe to this podcast.

To learn how to build production-ready serverless applications, check out my upcoming workshops.

Opening theme song:
Cheery Monday by Kevin MacLeod
Link: https://incompetech.filmmusic.io/song/3495-cheery-monday
License: http://creativecommons.org/licenses/by/4.0

Yan Cui: 00:13

Hi, welcome back to another episode of Real World Serverless, a podcast where I speak with real world practitioners and get their stories from the trenches. Today, I'm joined by Filip from Purple Technology. Hey man.

Filip Pýrek: 00:25

Hey man. Thank you for having me.

Yan Cui: 00:28

Yeah, thanks for joining us. I guess that I’ve been kinda aware of you for a little while now. And some of the things that you've been talking about the work you're doing at Purple Technology, and we’re actually both speaking at the Serverless Summit this week, so I'm really looking forward to that. So for the audience who haven't heard about Purple Technology, can you maybe give us a quick introduction about what the company does? And what is your role at Purple Technology?

Filip Pýrek: 00:52

Sure. Yeah, Purple Technology, we are based in Brno in Czech Republic in Europe. We are developing software for global brokerage companies or in forex business. And we are developing for them, their web applications, we can say, inside which they can... It's something like internet banking, where people can deposit money, withdraw money, manage their accounts. And also a big portion of our work is also KYC AML processes and their automatisation. And that's essentially what got us to adopt serverless.

Yan Cui: 01:27

Yeah, so that's interesting, because as a FinTech company, I guess some of the things that are really important for you are around security, I guess how much does that play into the decision to use serverless because I've spoken with a few other I guess FinTech companies in the past and the security was always one of the top reasons why they decided to go with serverless.

Filip Pýrek: 01:48

Yeah, in our case, we are not that heavily regulated, even though we are so after all our research, we figured out that AWS has all the necessary certifications and that it's basically okay for us to go. So we decided to adopt Step Functions in the beginning together with Lambda in order to automate all the processes and because we needed to see inside the processes because they are very complex. And Step Functions enable it as that was a deal breaker for us, let's say four years ago, when we started with serverless.

Yan Cui: 02:25

Four years ago, I guess that's when Step Functions were still kind of new. I remember it first came out as a much better version of Amazon simple workflow, which I used to work with, and it was really difficult to use and Step Functions changed that a lot. So can you maybe give us some examples of workflows that, you know, they're implementing with Step Functions? And one of the questions I get a lot is, you know, Step Functions is kind of expensive. So you know, when do you use it versus using something else, like for example, should I be writing my workflows in the Lambda function, or use a Lambda destination so that I can trigger another function after one function finishes? At what point does it make sense for you to use Step Functions instead?

Filip Pýrek: 03:08

Yeah, since we are really more focused on the complexity of the processes, rather than on the traffic, in that case, the pricing of Step Functions really doesn't have any impact for us. For us, the biggest business value was that we can see inside those complex processes because they have like 20, 20 steps for example, and we… it's very important for us to see inside which step each client is and those steps can eventually fail because they're contacting external systems. And we need to be able to, like rerun those steps and then continue the execution. So for us the ability to have the nice tool where we can visualise the processes where we can track them was really the deal breaker and we since we don't have like billions of clients, it's not that important for us the pricing, let's say, because it's really small in our in our numbers. But in case somebody is using some automation, which is running, like I don't know, every 10 minutes and is having several steps and 10s of iterations. And I have to also try to do something like this, then it can get expensive. That's true, but for like the business processes it is worth it, I would say.

Yan Cui: 04:26

Okay, so guess it’s the fact that you can quite easily visualise transactions and where they failed and debug things. But where you mentioned, you want to be able to rerun the steps that failed, Step Function doesn't do that right now. So are you implementing some custom things to allow you to say, resume from a failed step?

Filip Pýrek: 04:47

Yeah, that's correct. We actually sort of created our own pattern for this. We call it the break-away pattern, which means that there is always a try catch for these risky steps. And if the if the step fails, we used to use Step Functions activities. But nowadays, we use the callback pattern in Step Functions, which basically saves the task token inside the database and notifies us that hey, look, this step failed. Take a look at that. And eventually we can rerun it, or we can continue with different input and stuff like that. So yes, we needed to create our own pattern for that. And also I will be talking about this in, in in the talk in Serverless Summit actually. So that's a good point.

Yan Cui: 05:37

Okay, so let me just make sure I understand what you're doing. So for all the risky steps that you want to be able to resume, you're using the callback pattern, you're storing the token in your database, and when the failure happens, you're not reporting back to Step Functions with the error instead you're calling something to let your developers know that, hey, something's gone wrong, please take a look and fix it before the task token expires, so that you fix a problem and then rerun the step and then reply to Step Function with the token, with a success as opposed to an error.

Filip Pýrek: 06:10

Yeah, exactly. That's correct.

Yan Cui: 06:11

Okay, all right. That's actually quite clever. That's quite a nice way to do that. I guess that the main downside I can think of is just that it adds a bit of complexity to your state machine definition. So there is a lot of those double dollar sign to get a token and then have to save it and then process it off band, out of band.

Filip Pýrek: 06:29

Yeah, but we were able to create sort of a system which processes these failures. As as the step itself, then with serverless framework, we are able to, like put the steps inside like variables, so it gets a bit better. But as you said, it's in the end, it's a bit messy. The chart the graph, the visualisation and stuff like that. Yeah, that's true.

Yan Cui: 06:53

Okay, so you started with Step Functions back in 2017. And I guess now it's four years later, almost five years later. What's changed? I imagine you're using more and more serverless components now, you know. I guess, Lambda, API gateway. How much has that grown in your company?

Filip Pýrek: 07:11

Yeah, basically, we started like four years ago with automating the process, one by one, and nowadays, I can say that we are like 90% serverless. And hopefully during next year, we will make it to 100%. Yeah, we started adopting API gateway, DynamoDB, Lambda, obviously, and Event Bus. That was a huge feature for us. Yeah, basically, like every service we find useful for any use case like Cognito, for example, AppSync, yeah, all that, all the stuff we can use for web application development, basically. Yeah, so it's a lot of them.

Yan Cui: 07:51

Yeah, that makes sense. And I guess the fact that you are adopting serverless more and more means that things are going well, that, you know, is bringing value. So what would you say are some of the biggest wins since the company started to adopt serverless?

Filip Pýrek: 08:06

Yeah, we are in contact with external consulting companies. And they told us they have not seen any team to be so efficient. Let's say that in so small group of people, like four years ago there were four of us. Nowadays we're like 15. And we are growing. They saw the complexity and the scale of the systems. And they said that well you're working really efficiently, and we said that, yeah, this is thanks to serverless actually, because we don't need to, we're just writing the code and we don't need to worry about all the operations and stuff around. Yeah, we like to give a perspective. We have, for example, more than 600 Lambda functions in production. We have 20 plus state machines, which all have like 20 or more steps. We have some DynamoDB tables and stuff like that. And we are growing still.

Yan Cui: 09:03

Yeah, it's funny that you mentioned the fact that you guys are working so much more efficiently because there's just less things for you to have to worry about. I guess it calls to mind something that Corey Quinn recently wrote about the unfulfilled promise of serverless so that I think that kicked off a lot of conversations in the community. I think some of the things that Corey mentioned makes sense that, you know, we're not just writing business logic, there's still a lot of other things we gotta write, you know, like the Step Function definitions and things like that, but compared to the amount of other, I guess, undifferentiated heavy lifting, that's the Amazon's favourite word, you know, things that we have to do before. This is still a big step forward, it's not maybe as far as I guess Warner promised all those years ago in terms of just writing business logic, but it's still a big step forward. And I think you guys are making a lot of good use of your developers time. And so you can do more with fewer developers. And I've seen the same thing with lots of small companies that are able to do so much because they don't have to worry about infrastructures and having to worry about patching the images, machine images and updating their OS dependencies and all of that stuff. And certainly I find myself taken on projects that before I would have had to hire a whole team to do and now I can just do it myself. I have to say the productivity gains are massive here. And I'm also quite curious as well in terms of your stack. You talked about the serverless framework, you talked about using the API gateway, DynamoDB, Lambda and so on. What are some of the things that you're using tooling wise in terms of, for say, monitoring your system? Step Functions gives you all that visibility in terms of that visualisation graph you get which is great. But how do you, you know, debug a problem, say in your Lambda functions, or just getting alerts when something goes wrong in your web layer? Are using anything so specific?

Filip Pýrek: 11:02

Yeah, in terms of the of the whole stack, we created two or three years ago, our own we call it purple stack, which was supposed to serve our needs. It's basically a mono repo which contains arbitrary number of serverless services, and also a front end hosted in S3 CloudFront. And the AppSync API, and then back end services, which are usually the state machines and stuff like that. And we manage it all together with Lerna. We are developing in in TypeScript. That's how we manage the infrastructure, let's say the code wise. And in terms of the operations, basically, as I mentioned, the most important thing for us are the failing steps inside the Step Functions. So we have a Slack bot for that which is notifying us in terms of alerts. Other than that, well, if something starts failing, we are looking into the CloudWatch logs, we are using the structured logging. So we are logging our logs in JSON. And thanks to that, we can search for those logs in CloudWatch Insights, which is also a very useful service from AWS. Yeah, that's the that's the main debugging. We are also looking at the X-Ray or Lumigo for example. But let's say it didn’t catch up. It didn't get the traction inside the team.

Yan Cui: 12:30

Okay, I actually got a question in that case about your use of mono repo. It’s an approach I've used quite heavily myself as well and I've seen quite a few different, different approaches to how you do deployments with a mono repo. I've seen a lot of people, they would have, say, a separate folder for, say, shared libraries. And so in their project if it's all JavaScript, and that they can use a symlink to reference those shared library and so when they bundle things together, both the front end and back end can share code, and you use a webpack to bring them in. And then I think you said you use Lerna. Is that so that you can detect which services have changed based on the files that was touched by the commit so that you know which one to deploy?

Filip Pýrek: 13:15

Yeah, basically, that's a very, very good point. We quickly stumbled upon this issue that the deployments started to take like 15 or 30 minutes. Yeah, Lerna is basically in, in JavaScript world, a tool for managing the mono repo in terms of installing packages, linking dependencies, with symlinks as you mentioned. Lerna also has this ability to like filter out packages or services, which changed until some commit in the past. So we created our own utility, let's say we call it Lerna smart journal. We have also a blog article on our blog site that this allowed us to, after every deploy, create a git tag, and since then git tag in the next deploy, we know we can detect what has changed actually, including the sub dependencies inside some packages and stuff like that. And then just deploy the services which really changed so that was a deal breaker deal breaker for us, I would say.

Yan Cui: 14:15

Yeah, sounds like that's quite a similar approach to what the Lego team is doing. They are also using Lerna to detect based on a commit which files have changed and based on the file that changed which services that depends on those shared files should be deployed again. So they don't deploy everything. So I got another I guess question around your sort of use of serverless technologies in general. So you've been using it for a while now, are there still some technical challenges? Or other I guess, maybe cultural challenges that you find, you have to face quite regularly? Because we see the tooling has improved a lot and also the services has become more capable over time, but there is still... I can still hear people complain about occasionally you couldn't do X or Y or Z. So in the work that you are doing are there anything that's particularly frustrating for you?

Filip Pýrek: 15:11

Yeah, the biggest thing for us right now is, is for example, the fact that we are not able to really hire serverless engineers here and maybe not that much, even across the world. Because there is not many of them. But I guess that's okay. Since this is such a new, new industry, and we can teach people. And the other thing is the tooling. Yeah, you are right. Just yesterday, I kicked off a discussion in AWS community there is like about serverless framework and whether other people think that it is slowing down its innovations or not. Yeah, for example, what I like is the is the SAM Accelerate feature, which I think first was introduced by Serverless Stack that allows you to just deploy your code changes and not wait for for your whole CloudFormation stack to deploy and stuff like that. So I would say that, that's, those are the places where we need to innovate more. If we say, All right, we want to debug inside the cloud. Well, then we need to be able to quickly deploy it into the cloud, which is not happening nowadays with huge CloudFormation stacks. Also, yeah, I would say that's the main thing. That's the main thing. Yeah.

Yan Cui: 16:30

So I use the serverless framework myself as well and it does have that deploy function command that you can deploy single function. Is that not enough for your use case, or is it because you're using Step Functions so you often have to deploy multiple functions in one go?

Filip Pýrek: 16:45

Yeah, well, it's, it's a good start, but it froze there in the in place. But the Accelerate, as far as I know, it automatically detects the changes you've made and deploys it immediately, like, for example, Webpack watch or stuff like that. So in case of serverless framework, you need to go and manually hit the deploy. But in case of Sam Accelerate, it just does it automatically.

Yan Cui: 17:14

Right. Okay. Haven't used it before. So that sounds like it could be quite a useful thing to do. Do you do any sort of testing locally, because one of the things that I've done so that I don't have to deploy to figure out that there's some problem. And when I run those tests locally, I can also catch, erm, put a break point and catch problems, especially when I'm working with JavaScript.

Filip Pýrek: 17:36

Yeah, what we develop locally is, for example API with AppSync Simulator from Ben, Benoît Bouré. That's that's very useful thing. But for example, in case of Step Functions, well, we can do the unit tests. But sometimes we need to deploy simply some services, like event bus and stuff like that. It's hard to emulate locally without using some local state. I'm not sure how was the status of that right now. So yeah, we're developing locally like, let's say 50% of time.

Yan Cui: 18:11

Okay, and I guess with Step functions, if you need to update the Step Function definition then you have to do a full deployment anyway. So I'm not sure how that's gonna... how (inaudible) going to do that.

Filip Pýrek: 18:24

You don't have to do a deployment or you do have to, I didn't hear that.

Yan Cui: 18:28

Yeah, if you change the Step Function definition itself, I think that needs to be a full deployment, right?

Filip Pýrek: 18:34

Yeah, that's yeah, that's like one of the one of the issues but it can be also it can be solved in the same way like, like the like the Lambda deployment that you just overcome the CloudFormation and you do a direct API call. But of course, this cannot be done for all the services. Yeah, that's, that's true.

Yan Cui: 18:55

Yeah, I guess another thing I've done in the past is the just I've just used the console to edit things because the, for Step Functions, the console also lets me visualise it, which is something that I find difficult to do when I'm working with the serverless.yml and using the Step Functions plugin, because I'm just writing YAML and I'm not really visualising this not for complex state machines. Have you started using the Step Functions visual editor yet?

Filip Pýrek: 19:25

Not really, as far as I'm concerned, but I think there is a plugin to VS code which can visualise the YAML for you, so maybe some of our developers are using that, but I'm not really sure right now.

Yan Cui: 19:40

So Step Functions, I think, a couple months ago introduced a new feature that lets you design the... yeah, so that's not the, not just visualising, I think designing visually is is much more powerful. You can still switch back to YAML, but I think a lot of the workflows... I like to design them visually. And I think this has been quite a useful thing to...

Filip Pýrek: 20:02

Yeah, I love the editor. I just used it. Yeah.

Yan Cui: 20:05

Okay, cool. And what about in terms of testing the state machine itself? Because I imagine not all of your steps are using Lambda. So have you found some approach to test the state machine themselves?

Filip Pýrek: 20:17

Yeah, basically, we deploy the whole application, let's say, we are able to deploy the whole feature. And in that case, we let QA test the application itself and this application is starting the state machines and they then they have access to Step Functions to check if the execution is being executed correctly, and then they can see the result inside the application. So yeah, that’s...this is basically how we tested like end to end I would say.

Yan Cui: 20:45

Okay, I guess re:Invent is just around the corner. Do you have any sort of wish list items for Step Functions or Lambdas that you like to share?

Filip Pýrek: 20:55

I don't really have any wish list, I would say. I mean, I love the pace that how AWS is developing everything around serverless and basically all their services. So I trust them, I would say in this this way. Yeah, that's... I don't have any wish list basically.

Yan Cui: 21:14

Okay, so in terms of hiring is there something that you're looking for specifically, I guess maybe someone who's listening to this podcast right now may be interested in. Is there somewhere they can go to see your job specs and what roles you have?

Filip Pýrek: 21:29

Yeah, sure. They can definitely go to purple-technology.com and they will see the open roles there or they can they can check out our blog, which is blog.purple-technology.com. If they have any questions, they can contact me on Twitter and we can chat about that. Or the next week, we are organising Serverless Brno meetup for the local community. So we can also meet there. Yeah.

Yan Cui: 21:54

Okay, I'll get those links included in the show notes. So is there anything else that you like to mention before we go, any sort of personal projects or other things that you're doing that you want to give a shout out to?

Filip Pýrek: 22:05

I’m not sure about that. Yeah. I love 3D printing. In my free time, I have created a collection of 3D models which are related to AWS, for example, little Lambdas or the icons of the services. For example, Jeff Barr really liked them on Twitter. So if you like 3D printing, definitely check it out. Check my Twitter and you will find the links to download the SDL there.

Yan Cui: 22:33

Okay, I'll put the link to your Twitter profile as well so that people can check out your 3D printed AWS logos. So Filip, thanks again for joining us today and sharing your story with Purple Technology and the serverless.

Filip Pýrek: 22:47

Thank you very much for giving me the giving me the space here.

Yan Cui: 22:50

No worries. I'll see you tomorrow at the Serverless Summit.

Filip Pýrek: 22:53

Yes, see you tomorrow.

Yan Cui: 22:55

Ciao. Bye bye.

Filip Pýrek: 22:56

Bye bye.

Yan Cui: 23:10

So that's it for another episode of Real World Serverless. To access the show notes, please go to realworldserverless.com. If you want to learn how to build production ready Serverless Applications, please check out my upcoming courses at productionreadyserverless.com. And I'll see you guys next time

Real World Serverless with theburningmonk

#57: Serverless at Purple Technology with Filip Pyrek

Listen to this podcast on