#5: Managing complex AWS organizations at Moneyou

This is part 2 of my conversation with Olaf Conijn and Tjerk Stroband at Moneyou, a Dutch bank based here in Amsterdam, Netherlands. We discussed the challenges Moneyou has faced with AWS Organizations and the difficulty of managing a complex AWS environment using the built-in tools. And why they built and open-sourced org-formation as a way for you to manage your entire AWS organization using infrastructure-as-code (IAC).

In part 1, we discussed Moneyou's journey towards serverless over the last 2 years and why it serverless makes sense to Moneyou. We talked a lot about risk and many of the misconceptions about risk when it comes to serverless. And of course, vendor lock-in was a big part of that discussion! It's refreshing to see a bank approach serverless and the vendor lock-in debate with such thoughtfulness rather than just following the mainstream narrative without understanding the underlying risks first.

You can find Olaf on Twitter as @OConijn and check out org-formation.

For more stories about real-world use of serverless technologies, please follow us on Twitter as @RealWorldSls and subscribe to this podcast.

Opening theme song:
Cheery Monday by Kevin MacLeod
Link: https://incompetech.filmmusic.io/song/3495-cheery-monday/
License: http://creativecommons.org/licenses/by/4.0/

Transcript

spk_0:   0:00
in this episode off Rheal World serve Alors, I finish my conversation with a laugh and Jurich at money you where we talked about the challenges. We're using eight of his organisation and gaining control off your execution environment across multiple accounts and regions and how he drove them to develop their own tools. That allows you to apply infrastructures coat to a managing your age. This account as was your head of this organisation and the challenger. Remember we talked about that you had early on Wass you have so many eight of its accounts and ease the pain and asked to manage, especially with a best organisations and honest Lennon's on stuff you guys got special solution to this is the chalice about that? Yeah

spk_1:   0:55
So we we I think we started off on the right thoughts and this is just also timing but But it only started to set up a raid away Ask platform A devious organisations wasjust out and then we started creating with Ada Yes, organisation are different organisational unit where where we are able to great different development accounts for different development teams. Different production accounts for a production were close and then We have a number of shared accounts to do things that are cross cutting, like user for share services or moving. And that really helps us a lot to stay in control. Right? Because you, you know you can give users broker missions on development accounts because there's no community production daytime and then on your production accounts, you know that's there is probably no need for anyone to look into it. And if there is, you know you need Teo look backwards on what has happened, perhaps work something back into infrastructure is gold and having bean fetched in the source code. But as a resource problem, we indeed we have a large number of fatal Yesica and we asked that from team We profession these accounts and we managed a baseline of resource is on a lengthy accounts within our TVs organisation. And that is I would expect a problem that more companies have and we've seen this rights with other companies that have bean adopting a ws on building a platform and deciding to go with a multi account set up. So we probably just like a numerical. When he started tow, we started to ultimate to process and we started the ultimate process off creating accounts but also the resource profession ng we had of Yvonne solution which was internally developed. Which charge and I we spend a lot of time on making making sure that all those guardrails were there making sure that we had all the resource is making sure that we had a route 50 tree and certificates and crossing confirmation set up properly for all the developers and the proof against who are work. At some point we I think so. The unit off scale here or deployment wass there was was the baskets at some point in our own frame reveal about 150 tops and we were looking for a new solution to formalise the way we mobile radio waves organisation and formalised waiting model These the shared resource is

spk_2:   3:25
and a challenge leg predominantly in cross region and cross account provisioning of resource is right. For instance, we we typically employer wildcard certificate in our default region, which is Frankfurt because we do a lot of stuff in Germany, makes lessons. But then if we want to deploy, maybe I gave way or cloud from distribution. We also need the Lockhart certificates and Ulysses one right then unfolded arrest region. And then if we want. But we want to deploy our workload in Frankfurt, so we need to actually know the air and the certificate into Lizzie's one. But we don't want it like manually put that everywhere in parameters or in the scripts around in temples. And so that is an example of across region issue that at some point started to really take take some time and then not to speak the cross account commissioning, allowing one account or service to access and service in Africa and the complexity that came with that started to take a take. A kind of a poll on our team trying to manage dad across all those all those accounts and that basically, like Tio two are almost 42

spk_1:   4:35
or information which is which is approaching now hosted on Get up. We've developed most lost all of it in our in our own time. We apply it within. We didn't go me here. We use it, we're on users. We have a great time using it, especially since we know the complexity where we gain from using our information too many charter a DBS organisation. But also the resource is a pro. So all of these accounts and some process automation comes But it is a pure joy. We now see other companies also adopting it, which is which is great to see and we don't get a lot of positive feedback. So or information is a infrastructure s kowtow for eight lbs organisations. So as opposed to having to go into the console and create accounts, set up o use or service control policies you have? Yeah, mofo, which looks a lot like the information it has all the same. Syntex side those exclamation mark ref for references etcetera, etcetera where you define your organisation, you're on the tool and just like transformation will detect any changes, right? So if if you change the name of organisational unit that makes sure that the actual resource is and all you have to reflect that change, it will create a count who create all use. It will melt accounts into our use. Same goes with surfing for policies. And then we had a couple of other things that were useful for us. Like I am alias boss for policies. You defined in a single base. You can apply to your organisation, support levels, all these things that we used to do either manually or true different scripts. We we put into this model where we can just checking into source control. We can have the whole bonnets on DH. Our resource is eight of us. We'll always match that model. And that helped us tremendously. So, interestingly enough, the aim for or from nation was never just to be a infrastructures goto for aid away his organisation, the ambition and the and the reason we started this wants to solve the problem off. Staying in control of resource is gross income permissions that we have to manage within each other. And we told about this quite some time with just experience. We have been doing this with separate confirmation crypts and tools that booty scripts together. And if you really fundamentally well want to solve this problem, glad information and it's in taxes, not efficient. Right. Confirmation runs into context off they account in a region, and this interacts doesn't allow you to create any link to references across difference account two regions, so having the interest of purest gold fell for organisations more anything was a stepping stone in being ableto include this organisation foul into transformation templates and then reference parts of your organisation from within confirmation. So what we did there is a bit like Ah, well, it's a bit like expectorant programme. All of a sudden you'll have a confirmation fowl, which then described your resource is and get to specify which resource is Go where. Any problem as simple as a friend, since a guard should be set up, will require you tow execute differently information templates to different accounts, but it's still part of the same problem U F one master account with member resources for each member, and any member of God gets master resource That needs to point that the monster so using using this infection confirmation and there's two you can create, references to resource is in your organisation, Yellow found. So I can't blame the county from any account biological name or I can specify a number of guns that resource needs to goto. Even the links between these accounts and these resources get translated toe para metres and tow output. That thing could melt into each other, and this is I. I think that with our previous solution and 150 tasks, we were really hitting the limit off that solution, being scalable and useful for for our purpose. And at this point, we are. We're still migrating into the solution, but it's ah definitely gets us a lot better feeling of being in control. All these resource is allowing to reason about these being able to share these best practises. But dollars

spk_2:   9:16
and from a complaints perspective, there's there's benefits as well, right, because basically, we have to dissolve her coat that can be audited way have the definition of our organisation in a single almost file, and so it becomes very easy to see you. Kay. This is our organisation, and he could be. It could be verified through Consul, but also at a whole process to talk about guard duty, for instance, right, which is he's a pain to set up through the console. Basically, if our policy would be every production accounts is a member within guard duty? We have a single template that defines that every accounting. The Production organisation unit is a guard to remember. So when we add a new account or we create new account, Um, and added to the production organisational unit, it will automatically become a guard to remember that mechanism is very we. When we create a new account, we have to decide in which unit it goes. And based on that, the whole account baseline is provisions that includes everything like Cloud Trail, aggregated cloud trail monitoring, alert, configure rules that we have guard duty. And we really don't have to think about this. And we have ah, updates the law. Basically, that describes what the changes were made so we can review that. And from a compliance perspective, that's it's very easy, because we can always say, Look, these are the policies and we can verify that they are applied correctly and we put it in

spk_1:   10:52
stores control because overdressed so we could refuse these requests. And I don't think it was just like any other Silver development likes level and and as a result, and this is then probably again, a very exciting part of what we're doing. You see, all these patterns arrives like people used to compose silver out of different functions, and at some point you moved right of us for instance, and you start to Basel these different services together. But but now something we haven't told to reason about our software in the context of an entire ADA. Where's organisation with all the benefits you get from there older than the heart security boundaries, all the account permissions that you might have for mai mult half limiting the blast radius If something goes wrong. One account you effectively don't have to worry too much about the others because it's probably gonna be

spk_2:   11:48
coming there. And this is I find it very exciting

spk_0:   11:53
on Frank if compared to things like the landing zone wishes, okay, creating a new landing zone and you account for that. But then it becomes quite hard to grow our update to the definition of the landing zone, where something like this I'm using it myself for your client project. And it's because so easy to say, Okay, what, as part of our land is all, we should not have another component. Just be, I think, to my stack my definition for my landing zone and then just run the update house on. Off you go you provisions to all the accounts or the region's according to my calculations cell up, which is something that's that's really going to do without such as coach, because I've had the same thing with previous companies whereby it's okay I've got is a template for a new account. But then, once they're guests brought out, he just becomes impossible to update existing accounts. We want add a new thing to your landing zone just well, good luck. Sometimes we have to create new account because he's just too difficult to update them. It's

spk_1:   12:52
like he's cogeneration tools. You had only go one way. Yes, on this very only 11 time off

spk_0:   12:58
on DH. Yeah, So if anyone want to try out the information linked to the report is gonna be available in the show Notes on. There's also gonna be a document that shows you over the 50 plus features and some example set up. Then you can just copping to your new accounts well, and also I don't remember all of mention if you've got existing organisation, you can also just want one command to import into a yam afar, which you condense start to use that she update your organisation and start to manage that using infrastructures coat. So you guys are basically sold His one massive problem that a lot of customs have. We ve the b s man comes to managing organisations and accounts. Are there any other areas where aid was come to a much better job where you make your life a lot easier as a developer Ondas architect

spk_2:   13:49
One thing in particular that we've experienced a couple of times and known last year or so is that our approaches Infrastructures code, period where there's nothing we're gonna do manual because where calls lazy, it just doesn't work and it doesn't scale. You keep forgetting and stuff breaks The CLA information support for some 80 vast features is somewhat lacking behind and are doing a great job. And I mean, it's very liable. They have awesome features like stack drift which we which we use, for instance as well. But it is sometimes lacking behind and that there's something that did we ask. You definitely improve on

spk_1:   14:31
anything, Kate. We have his strength is sometimes also its sweetness there. They have a really false turn around new feature if they come up with Logan Services. But sometimes there's like little things that just don't work as smooth as she would like to friends. I don't know. It's probably sold by now, but things like using a universal second factor on the C L I. And it's it's just these little things that you could expect to be there that might not be there, that some things take away a little bit from the joy. But then again, they probably wouldn't be able to put out the services in the same basis they were doing at the moment. If they would have to worry about all the specific use case.

spk_0:   15:14
Yeah, and they want that along. Similar lines ahs, intense off edoema strength also become his. His own weakness is that there's so much inconsistency between different teams. AP eyes contracts documentations you name it. There's just so many just weird little inconsistencies that have been named differently. Different casing face existing one AP Carbonneau, another distracting nuts, especially when I try to be a lot of his tools on top of the head of the services. Another thing you mentioned about the stretch of code and the calf mission not being supported. The documentation on a beer's just also drives me nuts this. Well, they talk about in distress is calling all the time. But then tutorials is always go to the consul. Go to the screen. Stick of butter. Weather help. So,

spk_1:   16:02
yeah, this is one of the things if you if you go to one of these tutorials on us and it starts with Well, in this example, we use one on someone asked you count this and to do unless the other account the tree to tree as they entered a guard and the bits and pieces off the information that you just have to piece together. Yeah, I love tto. Make all those examples into into our information information. Yeah, and and And you, you get a best director that you can. Beckett. In a single template, you can share with someone else. You have logical account things. Yet there's probably lots of ground to cover, lots

spk_2:   16:35
of room for improvement. But on the whole, they've been making our lives very, very easy and exciting. So and especially also looking at the operational port we've been using their purity features I am is one of the key features that nobody talks about that much, but is kind of the glue that makes it possible to use this in a responsible way, at least, privilege is one of our core principles, and we've been able to, especially with our new tooling, be able to set this up and really little kid, said his principal and features like a missus. Well and customer managed, he's

spk_1:   17:12
told our services from Amazon that are really work to look at if you work within, say, financial services or in your highly regulated industries. Uh, we ask on thick, which is there. TIO define policies and rules as to what's combined. Three sources are continues in monetary teas and get reports off these and, of course, ploughed through. These are very important. Services are set up. These are These are the cornerstones in in how we build our platform in a blind way.

spk_0:   17:44
Yes, certainly, Finger I am is really underappreciate service, especially look it. But then, if you guys have that much time, if you're so assets management on a joy, especially for your functions, he's just crazy. I used

spk_1:   17:57
to work for Microsoft. I love Microsoft and I love my only ex colleagues Microsoft, and

spk_0:   18:04
I'll be sure to end it out but I do think I am. And Crouch Oh is also cultures is so powerful the seven events you can do with it just not just in terms of security. More thing also just another them off information you can do with Cloud Trail that I was bent and Nam There's just so much stuff I used to do myself and I just vital function bank Every time something has created something,

spk_1:   18:29
yeah, event bridge nowadays is actually just ah layer on top ofthe blood which events, right? But anything you would like Tio Ultimatum hole. There's love defence that you already have on DH something she just great your own. It's a great mechanism to extend processes and due process automation. Then one of the key features that we also that that allows us to really tie everything down and plug all

spk_2:   18:58
the all the holes is deep permission balance, which is also a very

spk_1:   19:03
powerful feature aside, I am that really, like closes the loophole, so to speak, anything we can do to sleep better we get we re embrace its permission. Boundaries is one of the guests Every people that I have not heard of permission, boundaries please look it up Because if you want to stay in control of your eight of us accounts and then this is definitely one off the waste.

spk_0:   19:28
So for those of you who had never heard of it, his basically a way for you to say I am Rose cannot create. I am Rose are more committed than itself, which again is a common look. Hold people worry about that once the budget has compromised and even potentially just get abin assess the by creating new I am most anything you can assume gets another thing to work. Looking at where I am is they're looking at really understanding the conditions you can apply, which is another against we demand really advanced feature than you get with I am but again, not that often understood on the EU's properly. So we're coming after the hour mark and we want to thank you guys for spending time talking today. You say anything else that you'd like to share with the audience? Maybe How to define you on Internet? Is his money used to hiring? Maybe.

spk_1:   20:18
Yeah, definitely. I think we're always open to new, brilliant and talented developers that wantto join us in circles.

spk_0:   20:26
So, yes. In that case, I think guys again, I hope to see you guys around.

spk_1:   20:30
Yeah, you to keep up the good work and we'll be listening to her post. Got guts.

spk_0:   20:34
Thank you. Answer. Yeah. Take it. Thank you. So that's it for another episode ofthe real World service. Thank you guys again for joining us for this wonderful conversation with Old off and Derek at money you to find a show, notes and the transcript for this episode piss Go to real world, serve a list or come. I'll see you guys next time.