Containers, Containers, Containers - AWS Edition Artwork

Cloud Coffee Talk

Cloud Coffee Talk is a podcast by cloud professionals for cloud professionals. These are relatable, deep dive, unscripted discussions, where technical talk is mixed with the real world challenges of people, process and technology. Each episode features a different domain discussion with 1 or more guests who are passionate about cloud and technology.

All Episodes

Cloud Coffee Talk

Containers, Containers, Containers - AWS Edition

June 06, 2021 • Darren Weiner and Erik DeRoin • Season 1 • Episode 3

Darren and Erik talk about Containers: This episode discusses the AWS Elastic Container Service (ECS) in some depth. General containerization concepts and challenges are discussed:

Moving from the monolith
Developer challenges
Fargate vs. EC2
Democritization of DevOps
Cultural challenges
ECS

[0:00:15] Darren: Welcome to Cloud Coffee Talk AWS edition. Sponsored by CloudButton. These are real world problems, solutions, and thoughtful discussions about working in the cloud. This podcast is for cloud professionals at every level of the organization, from the executive team to those with their hands on the keyboard, putting out fires and making the world a better place.

Just to reiterate, this is an unscripted discussion for those that are passionate about cloud technologies. It's not meant to be beginner level or cloud 101. It's meant to be something hopefully a little bit different. I am Darren Weiner, I am the owner of CloudButton, an independent consulting company focused on AWS Cloud. This week's episode is Containers, Containers, Containers. And my co-presenter joining me on this fun discussion is Erik DeRoin. Erik why don't you tell everyone a little bit about yourself?

[0:01:04] Erik: Hello everybody. I'm Erik DeRoin. I'm a site reliability engineer for a company called TrainingPeaks here in Boulder. Well, Louisville Colorado near Boulder. We do training software, coaching software for endurance athletes.

[0:01:17] Darren: And you do a lot with containers. You actually are, aren't you, almost entirely on container platform at this point?

[0:01:25] Erik: No, we are mostly on EC2 instances.

[0:01:28] Darren: We're going edit that out, Erik.

[0:01:31] Erik: Well, you can leave it in. That's fine. I think that's actually a good point.

[0:01:36] Darren: So, talk about your experience with containerized platforms on AWS at this point.

[0:01:43] Erik: Sure. We have a few production systems running in AWS - All of our stuff is running ECS Fargate. We have an Nginx reverse proxy that runs a fair amount of traffic for our www and marketing sites that directs traffic there for various things. They're all under our WWW umbrella. We have a WordPress instance running in there. We have another backend service in there that does some search that integrates with our ElasticSearch and is an API endpoint for integrating with some of our training plan searches.

Our tech stack is a 20 year old application, most of it is monolithic-esque that is running on EC2 instances. We started to peel off leaf nodes as we break apart the monolith. We've actually been doing some interesting stuff in determining whether something should live in Lambda or whether something should live in a containerized service. We default to trying to put as many things in Lambda as possible. But we actually found that containerization is a good middle ground on the way to full serverless. Our goal and our ideal would be to full serverless.

[0:02:48] Darren: Talk about that a little more, because the container and serverless boundary is a really interesting one that a lot of people I think are dealing with. What are those criteria? And it's changing, literally, almost month by month in terms of the capabilities of serverless versus containers. So what are those boundaries that you're dealing with?

[0:03:12] Erik: Yeah, that's an interesting question. I think first of all, we are doing it, arguably both of them are serverless. Fargate in my mind is serverless. It is just containerization. You don't really need to manage any of the servers. That's the benefit of it. It's not really an AWS managed service. It's just running a container serverlessly as the same way that Lambda runs a function serverlessly.

[0:03:36] Darren: That's an interesting point that I want to get back to later actually.

[0:03:39] Erik: Yeah, we can talk lots about that, I think would be good for us. It's really a matter of how we want to tackle a project, what's the size and scope of it. So certain things are going to be easier to peel off into Lambda functions. Lambda services, actually. We really want to think about Lambda as more than just a function, but as a service layer, it's not just going to necessarily be one function, there's going to generally be a few that are working together.

What we've moved into Lambda so far has been around event driven processing. So 'pub-sub' stuff. Stuff that's maybe more complicated, that's bigger, that the ability to move it directly into Lambda is just going to be way more difficult, it makes more sense to peel that off into an independent decoupled chunk. If we're dealing with monolithic code...

[0:04:26] Darren: A lot of what you're describing is because of your starting point being a monolithic environment, it's just much easier to break that into a containerized environment versus the additional re-factoring effort that would be needed to move towards Lambda.

[0:04:43] Erik: 100%. Yeah. The first thing we need to do is decouple the actual code, the actual internal architecture of how dependencies are injected, how libraries are shared, how that kind of common code is distributed around. So building that and starting to break that apart. There're also things that - you can't talk about monolithic architecture without talking about a monolithic database. I think anything that interacts with out monolithic database makes a little more sense to leave in a container for now, until we can start to break apart the database as well. We were able to do that with some of our event driven stuff and move that into DynamoDB and that was a good start. But that's also much more limited in scope and what we're able to do. There's lots of interdependencies within our database that becomes much more complicated to break that stuff apart.

[0:05:31] Darren: First of all, I love containers, I love what you can do with them. I love the simplicity. I started out in data centers with pizza box servers and then of course moved over to EC2 and the amount of management - configuration management, patch management - The TCO associated with managing large fleets of EC2. It's so nice to have that all go away and not have to think about that and really focus more on things that are closer to the application. So I'm a big fan of containerized environments where there's not a good application...where you can't move it into more of the serverless, the Lambda and that's going to be another great podcast that is coming up soon.

[0:06:24] Erik: Yeah. And Lambda, don't get me wrong, Lambda adds a lot of complexity, in my opinion.

[0:06:31] Darren: Any time you start breaking things down into more and more microservices, you have all these - you need to have some really solid microservice patterns to be successful and where you don't things get complex. But let's park that for the next…

[0:06:45] Erik: Sure. Well, I do want to say though, that the benefit of containerization versus something like Lambda comes down to why I think containerization is so strong and why I love containerization so much is that your development environment and where you're writing the code and what's being done right on the local developers machine is essentially what's going to production. We talked last time a little bit about how you can write code without understanding how it runs in production, and containerization eliminated that problem. I can specify that on my machine, I'm going to run in this environment, and I know that when it goes out to production, it's going to run in the same environment. I might be able to throw some more hardware at it, but the configuration of the server of the internals of the guts is the same.

[0:07:35] Darren: as long as everyone's working locally in Docker, which...

[0:07:37] Erik: Sure

[0:07:38] Darren: ...is a real challenge. Sometimes...

[0:07:38] Erik: Yes

[0:07:40] Darren: ...because like a lot of things in the cloud, a lot of it comes down to culture - and that's going to be a theme of a lot of these podcasts - the container culture and how you get people moved over there. Some people just latch onto containers, other people don't. And then there are some significant challenges when you're working with these things locally.

[0:08:07] Erik: Absolutely.

[0:08:08] Darren: Some of those are technical. So for example, as of right now, you are the one, I think, that talked about the volume mounts, and the performance issues related to volume mounts when you're working locally. So a lot of developers when they're working locally and they want to iterate very quickly, you don't want to have to rebuild your container every time and redeploy it. So they will do a volume mount locally. Well. If you try to do that on a PC or Mac vs. a Linux OS, there's actually some significant problems with the drivers that create lots and lots of performance issues. Then there's always going to be certain little network issues that show up as well. So, there's still some friction associated with working locally that depending on the developers that you're dealing with those can be a little hard to sort out.

[0:08:52] Erik: Yeah, absolutely, I agree with that. I think there can be developers who want things to be a certain way and they might have some ideas and they might be different than, what my ideas would be or the Director of Engineering ideas might be or whatever it is, and that's when it comes down to: I think an engineering organization needs to come together and either come up with some level of working agreement, come to some level of principles and understanding of how we do our work and how we go about doing that and then we can make determinations from there.

I think there's lots of solutions to these types of problems, but I think we need to, within the group, within the organization, need to share those and need to understand that, and you need to have some people who are driving whatever your company's decision for best practices of those are, and make sure that people are following and adhering to those.

[0:09:38] Darren: Irrespective of that. I think there needs to be an acknowledgement that local container environments just have some technical challenges that you just need to be willing to accept as an organization and be patient with that.

[0:09:52] Erik: Yeah, but I'd say that's true of any local development. At our company we have the ability to run everything locally and that is its own challenge and its own bits and pieces of that. It used to be when I first started there that it would take you two weeks to set up your laptop because you have to build and run and do everything. Slowly but surely, we've automated and optimized and now you can do it in an afternoon or whatever, but it still takes a while to get it all set up and that it be standardized and configurable for everybody. Then people started using Macs, we were all Windows .NET stuff and then we started having Macs and people running VMWare and mixing between that. We were able to automate all that.

I think anytime you're doing local development, it's going to be tricky. Containerization, I think to your point though, adds another level of complexity on top of that in getting all of those things to work. Like you talked about volume mounts or whatever. File system, cross file systems, whatever it could be...versioning between services.

[0:10:47] Darren: I think that the local dev environment more closely aligning with the higher environments - Stage, Prod - is great for me. I think the huge benefits of, once you get it over into the AWS environment, of the immutable infrastructures and the replicatable environments and you know exactly what you're getting your running through the same pipelines, same images is so powerful.

[0:11:16] Erik: I will say though, as we're talking about this, that the local development to - especially Fargate - I've actually found to be pretty difficult. There is a gap there that would not be as significant if you're running it on your own hosted server.

[0:11:34] Darren: In terms of being able to troubleshoot and work in that environment?

[0:11:39] Erik: In that like running the Docker container locally is going to be run through - whether it's Docker compose or just cli or whatever you might need to run that - is not entirely but almost a 1-1 to what the task definition is. But I can't run ECS locally effectively. I have to go run it in AWS. So, to do local development and know how that will translate over - how to get my Docker compose file to translate to a task definition - is knowledge and cognitive load that people need to figure out and understand.

[0:12:15] Darren: There is a translation there, not everything is created equal even though they're they're trying to, for instance, right now with the ECS CLI you can actually convert a Docker compose to a task definition and drop it into the ECS AWS environment. But that local - that's a challenge with a lot of these serverless services. The local environment doesn't always mimic, and there's lots of little band aids and little additional tools. That challenge is likely going to persist for a while. I do think the gaps going to narrow over time. I certainly hope it does because that can that can obviously really slow things down. As soon as possible, I like to get things into the AWS environment and start doing as much testing there as possible.

[0:13:10] Erik: Agreed.

[0:13:11] Darren: In terms of using EC2 versus Fargate-based ECS, What's your experience there?

[0:13:15] Erik: I haven't done it personally on EC2, outside of like, you know, looking at the docs and just kind of really poking at it at a high level. The benefits of Fargate, in my opinion, are so strong and it's so established that I don't know why you would do it. I'm sure there's good compliance reasons for doing it in EC2, but I can't think of any other, from at least an operation or management point beyond some sort of security or compliance reason, for running it in EC2 in my mind at this point. You can tell me that I'm wrong and I'm super okay with that.

[0:13:49] Darren: I actually think Fargate ultimately ends up having a smaller blast radius if you design it correctly. I think it makes a lot of sense. I think that the biggest issue I have with Fargate is when things go wrong, so troubleshooting,

[0:14:05] Erik: Yeah

[0:14:06] Darren: You have these services that just keep re-spinning and could be a number of reasons for that. Maybe some of that's going to show up in the logs. Maybe it isn't. That ends up being an area that when you're initially deploying services, if you have a problem with it, Fargate can be a little frustrating. Although they just released in March - It's still limited. It's a limited release but ECS Exec, which allows you to Exec into EC2 or Fargate-based containers. I haven't tried it yet, but I like that because that could really help as a troubleshooting tool. Not to go in there and change your immutable infrastructure. That's not the point. The point from my perspective would be to really troubleshoot and figure out what's going on if there is an issue with the deployment.

[0:14:52] Erik: Yeah, we went through a little bit that when I deployed our first container, well deployed our second container service. The first one was an API for an experiment and then we migrated another service in there. And then...so third, sorry I did the third service, it was sort of unique in that we were replacing something - the Nginx reverse proxy - that sat on EC2 services that for whatever reason seemed to have some issues and would go down and required constantly weekly reboots and was just an operational nightmare. We moved it into ECS Fargate for basically cost and size and scalability benefits as well as adding in some of the health check opportunities and things like that, so that we could hopefully have it stop doing these issues - one day we can talk about what that actually was: a DNS problem with ALBs changing their ip addresses and how Nginx caches DNS - but figured that out way later....not important.

But I would get into that cycle where we would start spinning up new ones and then I go and look at the logs: Wait, why the heck is this spinning up a whole bunch of new ones over and over again? What the heck is going on?

[0:16:04] Darren: That's one of the things I wish they would dial in with ECS - and they do have circuit breakers now, which, I don't know if they work in every situation - they are a little funky. But to be able to quickly get that feedback. Right now, it's a little challenging to kind of get that feedback into some sort of monitoring channel. You can do it - EventBridge and whatnot - but definitely not ideal. So hopefully they'll dial it in because that can be a little frustrating.

[0:16:32] Erik: It's also hard to figure out what the threshold is there because sometimes it needs to be replaced. I don't want to know every time. It doesn't necessarily if that's the correct behavior that it should be doing it. But if it's done it 15 times in five minutes maybe I want to know - whatever that threshold - dialing some of that stuff and it's difficult. Good alarms are always hard...

[0:16:48] Darren: Alerting is all about the thresholds.

[0:16:51] Erik: Absolutely. What's normal for you? What when do you care?

[0:16:56] Darren: The EC2/Fargate thing. I do think that's another area also where the gap is narrowing. They've come a long way with Fargate. They're putting a lot of effort into it. You could do EFS on Fargate. You can do almost everything on Fargate right now. The reason I run EC2, there's sort of two dimensions to that. One is where cost may be a factor. Fargate is great for scheduled tasks and short running workloads. When you're running a lot of containers all the time, you can't binpack Fargate the way you can with EC2, so you end up just spending more for any particular workload. Depending on the particular situation, that's something to think about.

The other thing is sort of older - This goes back to sort of older code bases - where Fargate might not be fully ready, although there fewer and fewer places where I'm needing to worry about this. But if they are older systems that probably shouldn't even be running in terms of, you know what they're running on, that's where I stick to EC2 as well. But overall, hopefully the price will continue to drop on Fargate. And when you start doing smarter savings plans, you can, you know, do compute based savings plans, you can minimize some of those cost differentials as well.

[0:18:22] Erik: Yeah. And for us, we've definitely had some overpowered EC2 instances sitting out there for a while that moving into Fargate actually saved us a bunch of money. But these were traditional server roles versus ECS based.

[0:18:37] Darren: Right now, of course, if you're going straight from EC2. But even with the EC2-based ECS, before capacity providers, it wouldn't be unusual to over-provision ECS clusters because you just wanted to make sure to have some run room there.

[0:18:56] Erik: Yeah. And I do like the Fargate model, even though the price per compute is a little bit higher, but you only really pay for the compute you use - certainly enticing for somebody.

[0:19:06] Darren: It is nice, but of course, if you run into memory issues, the memory issues can be a little a little bit of a...

[0:19:13] Erik: I haven't run into any yet, so don't jinx me.

[0:19:15] Darren: I will not jinx you. Talk about setting up container pipelines, there's the sort of the worker loads, on ECS and then there's maybe more the web facing systems that might need a load balancer. So, talk a little about that.

[0:19:32] Erik: Yeah. Most of ours at this point are going to be web facing. So, we have load balancers attached to them. We need to level up ourselves in this way. Right now, we're running code pipelines - code build/code pipelines - for a lot of those. And we're just sort of letting the CloudFormation task definition updates, basically manage that. It spins in new ones for you, rotate out the old ones and it just kind of works. We have the load balancers that it builds attached in the same CloudFormation and then they set their own DNS after the fact so that if the load balancer is replaced it could go update its own ‘A’ record down the road. It's all sort of self-contained within that nice little package. We have spent a fair amount of time doing CodeDeploy but we spent it around some EC2 stuff that we were working around and we found that how CodeDeploy works for EC2 is very different from how it works for ECS - Fargate especially - and it's much more appealing on the ECS level. So that's our next task is to spend some time with CodeDeploy, getting some more blue green deploys running. Right now, we're just kind of using the Dev and UAT environment to verify those and then ship them out to Prod kind of full steam ahead which is not ideal, but it's actually worked fairly well for us for now.

[0:20:48] Darren: Ultimately, if it it's not breaking, you can always create more complexity in your deployment patterns later if needed.

[0:20:56] Erik: Yeah because these are leaf nodes for the time being, or supporting infrastructure, it's not as dramatic or impactful if something were to go sideways although I can't think of any instance where we've had anything really truly sideways outside of me doing something that I knew was going to go sideways, like cutting over some DNS in the middle of it or whatever. So we've been a little bit fortunate in that we've been slow in our growing pains, somewhat intentionally. We don't want to just move over a super important workload without running through some of those paces. So, this latest project that we're working on is going to put us through our paces. It's going to be a much more foundational piece of some of the stuff that we're building going forward, so we're going to spend the time to do that a little bit better and build in some, hopefully, smarter blue green style deploys.

[0:21:46] Darren: One of the things I really like about containers and this goes for serverless as well, is it’s one area where there's a real democratization of DevOps that happens in this space. In other words, the developers can really own this from start to finish and build this into their bitubucket pipelines, their code pipelines. And one of the interesting challenges that I seem to deal with a lot - I deal with a lot of obviously web facing applications as well - There's a shared load balancer and so here you are, you're ready to deploy a new service on Docker, but you need to actually make sure that it's registering into a target group under a particular load balancer that might have some listeners and some rules associated with it. What I find is that I define a lot of infrastructure at AWS as the somewhat static constructs that are sort of further removed from the application, and the constructs that are more closely tied to the application. Think about a load balancer, any of the networking components, that are these fairly static constructs and I'm going to deploy out a new target group and put it into this load balancer. I'm going to create a rule, this is the priority for that rule. I'm going to create that over here. But to deploy a service, the shell of a service, you need some sort of service definition, but the service definition, the task definition really should live with the code. It should live effectively as a json file in your code base as well. That way if you need to make a change to an environmental variable or something else, it becomes very easy to do. But it's a chicken and egg problem in that you need to deploy this infrastructure with this initial definition but that's not the definition that ultimately might be deployed or changing over time. It's interesting, because everything else gets built into the pipeline very easily. But I always find this...

[0:23:51] Erik: So, you talk about the case, if I spin up all the load balancers and target groups and DNS off in this pipeline over here and then you run your ECS and it spins up its own target group and rotates those in. And basically those target groups you created over there because you have to create target groups with the load balancer are now worthless. I'm not doing anything and they just kind of sit out there?

[0:24:09] Darren: Well, the question becomes, where do you create that, and what does that look like?

[0:24:15] Erik: Yeah.

[0:24:16] Darren: It's a really interesting - I deal with this a fair amount where these gray areas of what you can really push into the sort of application-driven pipelines versus the infrastructure pipelines. All as code, right? It's all as code either way. But in terms of sort of ownership and manageability of it - there's a lot of gray areas there.

[0:24:42] Erik: I guess to me it's going to come down to...we've talked a little bit about culture, some of the culture around that. We've done both at our company and depending on different cases, I personally, for new stuff that's been moved into containers or Lambda or whatever, I really like everything being bundled together. That code and infrastructure and all supporting pieces are in one repo, one definition, however you want to break that up and it's self-contained, and that it does everything that it needs to do and it's all there. I guess I like the safety of it, really what it comes down to, is that if the load balancer were to change then we know the next time it runs, it's going to update the load balancer. If somebody accidentally deletes something and who knows what that is, which pipeline do we have to run? We know we just have to run THE pipeline and it will do that. I like that.

Also, I like that where the code runs matters to the person who's writing the code and have those things tied together. Now that can also be a double-edged sword. You can have some people who can make some messes, you don't know what they're doing. As you started talking about this, I thought you were going to talk about potentially where developers, building their own Dockerfile, they're starting to pull in some questionable, some crazy version of something from who knows where and whatever repo and they're putting change mod (chmod) 755 on every directory and just like, who knows what? And trying to just hammer the stuff through and no best practices or no security and all those kinds of things that, you open yourself up to that kind of stuff. You need to have checks and balances and controls around those sorts of things.

[0:26:17] Darren: Well, that's where the image scans are a really nice feature.

[0:26:20] Erik: Yeah.

[0:26:21]Darren: …Built into the Elastic Container Repository. As long as it's a supporting base image, which unfortunately, I still have to deal with some that are not, that's another story. But those are the kinds of, that's the kind of direction you take that to make sure that if you are democratizing this, that you have some checks and balances in place.

[0:26:43] Erik: Yeah. And I think that everybody is on the same page and everybody is working from the same place. You don't just kind of drop this stuff in somebody's lap. We have had some discussions about how you want to attack this particular problem or this particular container, this particular image, how do we want to build that? How do you want to deploy that? Let's have some conversations and let's make sure that people are on the same page and people understand what's going on and why we want to do one thing one way or the other and again, with the democratization, I guess implies that everybody is informed and that's to me the hard part is disseminating that information, getting people to care about something that they may or may not care about or that they may or may not be interested in or want to spend their time with. It takes time to tune containers. It takes time to tune some of that stuff and get it right and get it solid. I certainly understand why people don't necessarily want to deal with that.

[0:27:35] Darren: Yeah. It slows you down, doesn't it?

[0:27:38] Erik: And again, we go back to additional cognitive load and supporting that. Does the business understand the cost of going to these containers, going into these various microservices? What are the impacts of that? That's one of the actually interesting topics we could probably talk about around containerization is performance and metrics and impact and how you go about measuring that? How do you go about understanding the cost benefit analysis for your customer of going to a service-oriented architecture and what that may or may not mean for them? How does that affect response times? How's that effect end user experience? How does that time to deploy, time to ship new stuff? Time to, you know, move over a service into a new - or break something apart into a new service before you can start adding or building new features, so that is delaying deployment. There's lots and lots of cost benefit analysis around all this stuff. That's something that the business organization, the Engineering organization needs to be able to have conversations about.

I'm assuming if you're going into containerization, you're going into some sort of service-oriented architecture of some form. You need to have an understanding of what that means, how to manage that. Because again, you're adding complexity to something. Say what you will about monoliths and the challenges that they provide, but they are a little bit more accessible than potentially a large microservice architecture. You look at AWS' service map that they love to show off. That's just like lines between tiny dots at this point, you can blow it out into the screen, you can't even understand what's going on. It's super complex. It took a long time and a lot of effort and a lot of very intelligent engineers to build something like that and it's much harder to fully grok that than it would be all in one very nice repo.

[0:29:32] Darren: That's probably more of a larger microservices discussion. Because the observability and keeping track of the moving parts and all the intercommunication is definitely a whole area in itself that's evolving and creating confusion.

[0:29:52] Erik: But I would say that's been our biggest, actually, our biggest struggle with containerization is that part of it. Getting the piece of code up and running into containers - it's not particularly hard. Getting it to run in a way with all the stuff that's needed to support it, has been the hard part.

[0:30:11] Darren: And one of the solutions to that is telemetry. XRay, OpenTelemetry, which is a whole other area of significant rework and learning that is probably simply going to become more and more necessary, as these systems get split up even more.

[0:30:36] Erik: And I think about CloudWatch in the past year and a half has leaps and bounds improved.

[0:30:39] Darren: Yeah

[0:30:40] Erik: I think part of it is because again, this is where it's so hard to answer some of these questions and it's so important and necessary in doing this stuff, that people are going to outside services, other third party providers are building their own solutions and things like that. AWS has done a nice job making CloudWatch more available, more accessible and more powerful. I don't think it's quite there yet personally, in my opinion, but it's really dang close.

[0:31:11] Darren: The funny thing about that, I agree the features they keep adding are really significant, very useful. Early on with CloudWatch, to try to get some observability on these things, there are all the third party vendors and what I liked about the third party vendors is that they were so simple and CloudWatch was so confusing and actually over time a lot of these third party vendor tools have gotten even more complex and more features filled and CloudWatch is actually cleaned up a little bit even as they've been adding some features. I'm pretty excited where it's going. Because really that native integration and not having to push all this data out to third parties is pretty convenient. Still could be a little quicker. It would be nice if it was a little quicker with those logs, you know, just show me the logs a little bit faster, please.

[0:32:04] Erik: Yeah, and for us, CloudWatch is not our main destination. We use it a lot. It definitely falls into the - we like to joke, one is greater than zero, something is better than nothing. We'll throw it into CloudWatch logs or use CloudWatch metrics. Ideally, we're shipping it off into - we use an ELK stack for that, so we're going to push off into ELK and hopefully have it in a single pane of glass where you can tie all these things together or whatever. Whether we stay in ELK or move to the CloudWatch, or move to Prometheus or who knows, whatever tool down the road. Certainly open to a lot of that kind of stuff, but I certainly like the idea of it being in one area, but right now that's the function that CloudWatch provides for us - write to those logs and then have we have a service that reads off the logs and ships them off into another ELK for us.

[0:32:55] Darren: Containers are a great opportunity to move the monolith and to start getting rid of some of that undifferentiated heavy lifting associated with things like EC2 and configuration management of systems and all the associated - whether it's security or compliance issues, it just makes it much easier.

[0:33:16] Erik: We could have moved our stuff off of EC2 into containers at this point. But again, we're a 20-year old product. We are through various versions of .NET and we're in the process of migrating fully to .NET Core. At this point, we're still on the .NET framework on some stuff. It's just taking time to move off. We have old libraries, old dependencies we've needed to move and we're in that process. But we kind of just decided we don't want to deal with Windows containers. We don't want to spend the time, energy and effort to spin all those up. But we could have moved it over in that time. We just didn't really see necessarily the full benefit and speed and growth that could happen from there.

[0:33:57] Darren: There's the expense of refactoring it to this new environment versus accepting the technical debt that you have and knowing that things over time at some point, things are going to get, not refactored, but potentially just rewritten or replaced.

[0:34:13] Erik: I guess in my mind, and discussions and conversations we had around where the containers being lightweight fast and much less overhead, we found that Windows containers, Window Docker images were not necessarily that. It made sense for us to stay in EC2 for the time being. We have multiple layers, various EC2 instances for those various layers of our various tiers. That works really well for us for now. That's not where I want to be. It's not where I want to go, but it makes sense.

[0:34:44] Darren: Well, you're making business decisions that are based on experimenting and testing and determining what works and what doesn't.

[0:34:54] Erik: Absolutely. Yeah. For every service that we've put out there, we spent the time to load test, to tune, to understand how to scale it well. How to do those things really, really well. I think that's a really important piece is understanding the containerization side of: it works on my machine. I got it working, it's contained, it's isolated. I can throw that over the fence and it will make its way to Prod. And I know that the environment, everything there is scalable and response times and throughput and opera...Again, another word. I'm blanking on. I can't think about it, but along those lines, to run that in production is a whole other thing. There're tons of great tooling out there and stuff to figure out what those thresholds and what those benchmarks are for individual containerizations. What I really like about containers is it allows you to really tune that stuff well because it's focused on a little piece of architecture.

[0:35:48] Darren: Really minimizing the

[0:35:49] Erik: …scope. Yeah.

[0:35:51] Darren: Excellent. Erik. Thanks for joining me once again. This was a lot of fun. It's always fun to talk through these ideas. I think about a lot of this stuff all day long, but I don't get to talk about it as much. It's great talking through it with you.

[0:36:07] Erik: Yeah, same. It's one of the things that I do the same and I look forward to what people have to say and reading more about it and learning more about it. I'm a sponge in this category. I've been trying to dive as deep as I can, and I feel I have so much more to learn. It's a whole another level of compute and a whole other level of systems. I feel that some people have a head start on me, so playing some catchup.

[0:36:33] Darren: The depth of the cloud. There's something ironic about that, but I completely agree. You can go very deep. All right, thanks for tuning in. You can find us on twitter at @cloudcoffeetalk. We welcome all your feedback. Be critical, but be kind. Until next time. Have fun in the cloud.