Announcer:

Welcome to HSDF THE PODCAST, a collection of timely and informative policy discussions brought to you by the Homeland Security and Defense Forum. In this first part of a two-part episode, Charlie Armstrong, former CIO at Customs and Border Protection, leads a discussion on tactical edge computing at DHS with Sonny Bhagowalia, Assistant Commissioner and current CIO at Customs and Border Protection, and Scott Bowman, Deputy CIO, FEMA. This program was originally broadcast on September 23rd, 2022.

Charlie Armstrong:

Well, good afternoon everyone, and thank you for everybody who's participating here in-person and out in the meta universe, we really appreciate it. Especially big thank you to our participants, the Assistant Commissioner and CIO from Customs and Border Protection, and the Deputy CIO for Operations from FEMA. Obviously, now we're getting ready to move down into the down and dirty part of cloud, and what it takes to get there, and what things are like along the way, and what bodies are left afterwards. I'm going to start off with kind of an open-ended question just for the Assistant Commissioner down there. Tell us a little bit about your journey so far. You've obviously been doing this cloud journey for a number of years now, so tell us a little bit about your experience so far, and what have you encountered along the way, what lessons have you learned?

Sonny Bhagowalia:

Well, good afternoon everyone. Well, I think the first thing I learned is I started off with dark hair, and now I've got gray hair, so there we go.

Charlie Armstrong:

It's one more stage.

Sonny Bhagowalia:

That's one more stage of [inaudible 00:01:47] Well, good to see you again, Charlie. I think the first thing is, I remember I was Interior CIO. So at that time, Vivec was sort of asking me, he's got some ideas on something called cloud, Federal Cloud. It was a really interesting journey. I'm talking about 2009 timeframe. So this whole thing started off with a policy that sort of, in our day job we were CIOs, in the night job sort of helping out corollary duty. So I think this whole thing that he started was a really good concept, and it's taken all this time to really get that adoption up to the level that we are today.

Having served as a five time CIO, I've just seen from the other side of it what it takes to really make things happen. It takes time. It's supposedly by 2025, this will be a 1.3 trillion dollar market. I'd say the government adoption, which let's say government is about what, 80 to 100 billion that we spend on federal government alone? I'd say about 10, 20% is cloud now. It's not really still to the level it needs to be, but it has really taken off in the last couple of years. So first things first, obviously I'm an engineer and a CIO by training and experience, so I get to say what I got to say, but this is what I've been approved to say. So sometimes I'll say, in case you're wondering why I'm looking at a piece of paper, it's not because I can speak ad nauseum about the subject, but really there's some points that I think the team has prepared and what's been approved that I want to really talk about.

I think the first thing I'll just say, we operate at the speed of mission. Matt talked about some mission and what you've talked about. You've been there, Charlie, so you know that at the speed of mission, our operations, we are transacting 10 billion transactions a day with 40 billion data exchanges. We are processing trillions of dollars. One of our systems processes 4 trillion dollars. We're the second largest collection agency at 98 billion dollars. We're processing 1.2 million passengers, pedestrians every day. So what we got to do in this journey is make sure that it's secure and reliable, and operates at the speed of mission. We're encountering 2.3 million encounters at the border. All of this stuff needs to be there. So we've got about 282 legacy apps, and I'll talk about that. We've made tremendous progress in terms of getting at least 60% of the cloud. So all of this takes time, but it takes cultural change, and I think we are all about moving from an application level portfolio of those 282 apps, to getting to an operations portfolio so that stuff just works.

So the app team under my Deputy CIO and Chief Software Officer works with the infrastructure team under the Infrastructure Officer, and then with the CTO bringing in innovation, and then working with our agents and officers, how do we innovate at the speed of mission? So there are four things that I think this journey that I focus on, which throughout all these years, I think we can only do 104, and now I think we can do about 304 faster, meaning we can deploy things faster where. For example, in Operation Allies Welcome, when we had the evacuation of Afghanistan, we deployed in seven countries around the world, and at the forward end we were deploying things... Usually I have like 40, 50 teams doing concurrent agile software development, giving two week deliveries. I was able to do that in some areas next day, on a mobile platform with facial comparison.

How is that possible? There is stuff on the cloud at the back end that allows you to do that. Facial recognition of the technologies. Better, more resilient, and making sure that a lot of the stuff that we do stays up longer, is reliable. In some instances, for example, in one of the systems, we used to have potentially a five minute outage. I think Charlie, you know what I'm talking about. We're down to five minutes. We've not even had an outage actually, touch wood, although I am getting a little something over here, but it's not cloud, it's something else. My point is, why is that? Because we have done the work, and to me, this is the other part that I want to talk about, is this is not a buzzword compliant kind of thing. This is a lot of people doing solid work. You got to do the work to get there.

I think that's also part of the journey. You got to do the engineering, you got to have the processes. So it's people, policy, process, technology, governance. You got to have all of that stuff working together. I think we're certainly doing that. I will say, the other thing is agents need this stuff on these phones anywhere, anytime, for any mission, securely and reliably. So how do we make that app work so that it can be getting that information on the cloud, and making sure the data is secure? We got a lot of things like Records Act, FSMA Act. I mean, there's so many things that we have to comply with, as you know Charlie, and I'm sure my colleagues will talk about that as well, is that to me that agility is there, but that performance is really, really important. You also got to... So in addition to the faster, better, it's got to be more secure.

When I came in 2018, we were having 40 million cyber attempts on us a day. Today, it's 100 million attempts a day. 4,000. 10 billion transactions, 40 billion data exchanges, 100 million attempts. I don't know if it goes any grayer than this, but that's what it takes. People are working really hard. I'm really proud of the team as to what we are doing on that. I think that's another message in government. There's a lot of good work being done in government, by industry and government all working together to make that happen. We also have a strategy, we have a cloud strategy. We have that 60% already done, we need to get out of that data center to save money and then reinvest. We're also doing a lot of stuff on our data center optimization, and also cyber strategy, and an overall strategy as to what we can do for the mission.

But at the end of the day, Charlie, bottom line is our journey to the cloud, I'm happy to report that from sort of an idea and a vision, it's now a reality. It's still happening, I still think there's still a little ways away, and I'll leave with you with this thought. Out of those 282 apps, we also have something called high-value assets. I don't know if you have looked at that executive order, that these are the most important systems in the country that run the mission of the country. So I've got some systems that run all trade in the country, all border, all national security, all travel. These are massive systems, and as you know these, you can't just move to the cloud just like that. That's why they're sitting in the data center now, we have a tier four data center. That stuff is moving last, but we want industry to help, and we'll be putting some things out there to get you to work with us because it's going to take that level of engineering to make sure those work.

So I'm really excited about this question. I think it's very positive to see where that journey started in 2009 to where we are today. I think adoption is increasing, and I think everyone's bought into this. For example, all this collaboration that we did during Covid was seamless. So even though 70% of our... We are the largest law enforcement agency in the United States at 66,000 people, everything was done seamless. 70% of that was working in the front lines at the 328 ports of entry, and also our border patrol stations, and so on and so forth, along with my staff that was there. But 30% was teleworking. All of that remote work, telework, seamless through what collaborative tools were in the cloud.

So all of that has been working, and now with Microsoft Teams and Webex and all this stuff, we're able to not only communicate from anywhere, also process that information. I think Matt talked about some of the things on TBS, all of that stuff is done with two second adjudication in the cloud. So all of that stuff is faster, better, more secure. Lastly, I'll say more affordable. I will not say cheaper. There's a reason I say more affordable. More affordable is a pay you go utility, that even though you outlay maybe a little bit less over time, it may be the same or a little bit over, but look at all the other benefits that you get. Faster, better, and more secure.

Charlie Armstrong:

Mr. Bowman, obviously you've got a big mission at FEMA, a very important mission out there providing not just disaster relief, but grants and flood mapping and all kinds of things. Tell us a little bit about what's going on at FEMA in terms of cloud, and what you've experienced so far.

Scott Bowman:

So definitely a lot going on. I feel like I'm the person in the middle here. Back when we started the cloud journey, my hair was darker and I had more of it. I'm losing it and it's turning gray, so I'm transitioning to that final stage here. It's good that I'm sitting in the middle. We've learned a lot. I would say we're about almost halfway through our journey, not quite as far as long as CBP is. Going back to those early days when the vet issued the cloud first instruction and information, getting federal government to start accepting the cloud. We had some systems that went out, they kind of predated FedRAMP and got out there early. That was the exception, not the rule, so it has been slow going, slower than we would've liked. What we've realized here recently over the past few years is that we needed more governance and structure around our migration to the cloud.

So within FEMA, we issued a cloud smart computing directive a few years ago, and that's to ensure that FEMA is in compliance with the OMB Cloud Smart strategy, and going forward, ensuring that we go to cloud first. Cloud is the first solution that we look at. If it is impossible, then that's fine, we'll go to other solutions, but we need to evaluate the cloud as we go forward. What we've been working on over the past few years is actually standing up what we call FEMA enterprise cloud environment within the top three cloud service providers. So we have now established a footprint there. Our customers have different requirements. As you mentioned, flood insurance is very different than our grants management, which is different than our individual assistance and our direct assistance to disaster survivors. So they have different needs, and we didn't want to limit ourselves to one cloud service provider or another, so we've actually established footprints in multiple environments there.

There's a lot that goes into that. It's not just going and procuring the cloud service provider and standing that up. It's the security, it's the network connectivity, it's doing all of that appropriately. It's implementing zero trust as we go forward. So it doesn't happen nearly as fast as we would like. I wish we were done at this point, and we are not. There's a lot of work that goes into doing it and doing it right. I would say the other thing at FEMA is we're not limiting ourselves to one solution or another, so we're looking at software as a service, platform as a service, and infrastructure as a service, and leveraging all three platforms. So as much as possible, we want to go with software as a service, low code, no code, but we are definitely, especially for some of the legacy stuff, looking at infrastructure as a service, standing that up, and also leveraging platform as a service.

A few things that we've learned along the way. Obviously, the need for good contracts. You've got to be able to not only set up the cloud service providers and the connectivity to them, but you've got to have the expertise in house in order to ensure that it's done correctly. Definitely contracting is a big thing, secure connectivity, monitoring the cloud, setting up the full logging, everything that's needed in the cloud to ensure that you've got a secure environment. The other thing that we've done is establish a lot of inheritable controls for our authority to operate to streamline that process. As we all know, the governance and the security seem to be what take the longest. They take the longest. The technology is easy. You could go out today and stand up a cloud environment and be up and running by the end of the day, but you don't have that governance and that security that's required by the federal government.

Another thing that we learned along the way is the cost management and cost estimation, ensuring that we stay within the bounds... As most of us know, the government operates as a debit card, not a credit card, so we need to have the funding in place before we consume the resources. So ensuring that we don't over-burn, over-consume was critical. That has been one of our main focus areas. Also understanding what our customers and what we are getting into as we migrate to the cloud. What we don't want to do is migrate to the cloud and find out that it costs a lot more. So far we haven't found that, but that's been part of that implementation of our cloud group that does the analysis and the cost estimating, to ensure that we know exactly what resources we need, what we're going to be consuming, and then ensuring we stay within those bounds. If something is all of a sudden going crazy and charging a thousand dollars an hour instead of a dollar an hour, you need to know that immediately and work to find out what's going on and stop that. So those are some of the things that we've gone through, and like I said, we're about halfway through with our cloud journey to this point.

Charlie Armstrong:

Thank you. I really thought when I retired from federal service, I was going to stop getting taskings from Luke McCormack, but it appears that we have a question that got passed over from his group. It also kind of dovetails into what I was hoping to discuss in it. So let me ask, what has been the impact on your workforce... I know AC you talked a little bit about what your structure is within your organization. How has that evolved over time? I know Mr. Bowman, you talked about the need for a different way of doing contracting and the ramifications around that, but overall, what have you all done in terms of looking at the workforce changing things, and what's your strategy there?

Sonny Bhagowalia:

We saw that ball being passed to us faster than the Washington Commanders, I guess when they did the toss sweep, right? Sorry. Okay, no more jokes on Washington Commanders. All right. I think first of all, there's a lot of things here on the workforce side of it, because as we are trying to deploy faster, better, more secure, more affordable, there are different skill sets that go with each of those words. I'll add two more words to that, transparency and accountability. Scott talked about accountability. The CFO is always like, "Hey, the CIOs always come and ask for money." There's always a sob story. "The dog ate my homework, I didn't really do all the stuff, there's more data coming in, all kinds of security challenges." Absolutely legit reasons, but the first thing is with the workforce, I first try to tell them what the cloud program is.

So a lot more training, communication, outreach and awareness, not only for the stakeholders first. So I've got now first with the stakeholder workforce. So the stakeholders need to know what does cloud do for them? They have bought in through dashboards, we have done communications, I've got something called a trusted partnership initiative where I sit down monthly to quarterly with each of the 20 stakeholders within CBP. I also sit down with all the countries. We operate in 98 countries. We're also operating with the Five Eye countries. In fact, we're going over to meet with them. So those CIOs, we discuss what those are. Anyway, all the stakeholders need, first of all, a transparency. So show them where they are in their cloud journey, and what benefits they're getting. Then the accountability, which is sort of like TBM concept. We were lucky enough to be the, I think second federal agency to get an award for TBM, but we have open dashboards that I share with everyone so that they can understand what the Amazon cost is, what this cost is, what the operational move cost is, all of that stuff.

I think what that's done, Charlie, is take that aspect of it and educate that workforce to be with us. On the other side of the provider... That's the consumer side of it. On the provider side of it, everyone's concerned a little bit about their job, literally, right? It's like, "Hey, I used to do server huggers and all that kind of stuff." They did a great job if you think about it. Also, we always talk about legacy. It's the car that brung you. I drive a 1994 SUV. It is reliable as hell, it's better than the ones that I got now and it works. So some of the stuff works. We got to just be a little bit appreciative of the fact that legacy technology works, it's survived this far, but cloud is better.Now, how do I make that example to make sure it's not only about the technology, it's about the data? We in the government do a really poor job of the classification of the data and how to bring that and migrate that data correctly, and then do the business continuity of all that. So I think training, giving them other things to do. For example, if you're not doing technical... Most of the [inaudible 00:19:04] are doing technical supervision, but we've got a lot of folks who are also doing technical work in conjunction with our contractor workforce, so make it a win-win situation. Give them training in terms of what cloud is, the management of the application so that they can be embedded [inaudible 00:19:21] the agents and officers as to what the mission is, and how they can deploy faster by understanding the business process, what this cloud can do for the mission.

I've got people who are embedded, for example, when they were doing Operation Allies Welcome, we were also doing the Uniting for Ukraine. I can't talk too much about it, but I can tell you there is substantial, substantial progress that has been done in those area. I'm talking 100,000 people in [inaudible 00:19:45] in Afghanistan, all done very quickly. The other stuff done days with full accounting of what's coming in and out, who's coming in, what level. All that stuff is all cloud-based. It's all cloud-based with app. So I think app delivery, the training, making sure that the mission is there so they understand what the mission is, and then see how the solutions can be made to make that happen, the security of it, data, right? It's all about the data. Where is that data? So I think training in that area, defining the key roles, responsibility, authority, I think that's been key.

We're also looking at sort of stuff in terms of governance in the area of cost. This has been the biggest challenge in cloud in my view, cost. It's very hard to look at the cost looking at that. So I think that training of that, and then I think just making sure that people understand that their job is no longer the technology, in some cases development, but more of the management, and then working together in these cross-functional teams they can get these things done. It's very energizing. Uniting for Ukraine, massive, massive program. For example, the Operation Allies Welcome, the largest airlift in US history. We were doing this in days, fully up operational. United for Ukraine, fully operational. Largest border search, we're doing it right now.

Anyway, I can go on, but my point is all of this is only possible because people are learning, but at the end the day, you got to give them confidence that they're here with us, learn some new skills. It's a chance for them to advance. I'll just stop with this, we have got something like 58 AI ML projects going on 158, robotic process automation projects going on. These are all new things that can learn data, a new thing on data. So I think all the [inaudible 00:21:29] are very excited to learn some of these newer projects, and we need to tell them that that's how it is, and learn a little bit about cloud as well. I think I'm really impressed with what the industry's doing in this area. I think we need to team up with them to offer sort of, don't be afraid of the cloud. Automation will not take your job away. You can use this because there's plenty of jobs to do. I look at it as, there's plenty of work for everyone, and there's not enough hours in the today to do that mission. I don't know if I've answered your question, but that to me is the workforce answer.

Charlie Armstrong:

Thank you. Thank you. Mr. Bowman.

Scott Bowman:

Yeah. I would say for the workforce fear concern, "What are you doing? Why are you moving my cheese? Why are you taking my job? What am I going to do?" All of the emotions I think are the initial reaction to that, but at the end of the day, once we sit down with the team and understand... The big thing with them was communicating with them, making sure they understand what we're doing, when we're doing it, how we're doing it, why we're doing it, and also ensuring that they have the training that they need to be successful as we go forward. We're still working on that, we're about halfway through. I think one of the things that they've really started to realize as we're going through the process is a lot of what we have on prem, the legacy environment, as Sonny said, it's the car that brung ya.

I mean, a lot of that's virtualized today. They do that remotely. They're not coming into the office every day and touching the server or touching the storage area network, so a lot of those things they weren't doing day to day anyway. Really, the cloud, it's someone else's servers, it's someone else's data center. So the things that we're doing virtually in our data center are very similar to what we're doing in the cloud and the different cloud service providers. We're doing it remotely. So as we've gotten into it and learned more, it isn't as big of a change as they thought it was going to be, so there's less fear the more that they've learned.

Sonny Bhagowalia:

Could I just add one thing on that? I think in our culture in America, too often we are like, "Yeah, it's old, it's legacy," shaming. "Oh, you know that?" Actually, it's more genius. When I went to PCI [inaudible 00:23:38] I found that they had a VAX computer working for 40 years. I called the guy and gave him an award, but then I told him we got to go to the cloud. But, I embraced the guys, like "You are a genius for keeping this thing how long?" A PDP, whatever. You know what I'm saying? He was buying stuff on eBay and making the workshop work. He can easily learn the cloud. It's nothing. So we got to change this culture of "Yeah, it's old and everything new is shiny and great." It's not, there's a lot of other challenges. You got to do that. I think to Scott's great point of also about the fact that once they learn all this stuff, I tell you what's exciting is when you give them your part of the mission, the Operation Allies Welcome, when we recognized the team, a lot of them were in tears because it represents the greatest values of America.

We honored our commitment and we helped somebody. I mean, you could see that mission happening. So when they're involved in that mission, it's not about servers and cloud, it becomes something else. I think that is where this is going. So it becomes a service that works. Now, I will say with industry, we need to make sure the stuff stays up, but that's what so far, we've only had one or two things here and there, but overall it's gone much, much better. There are two or three more sigmas of reliability that we've gained. I think people are gaining more... Instead of previously it was like "My jobs," I think they're learning more like, "Hey, how can this be used for this mission or this project that you're doing?" They go like, "Oh my goodness, I can serve that mission faster and better, so let me go there."

Scott Bowman:

One of the things I've communicated to my team and the various folks throughout FEMA is we didn't become IT people and technology people to do the same thing every single day, year after year after year, [inaudible 00:25:16] We did this to learn new stuff.

Sonny Bhagowalia:

Yeah, exactly.

Scott Bowman:

Technology is evolving continually, so rise to that challenge, learn new stuff. It's not scary. Once they get into it, they are accepting it and willing to learn new things and move off of some of the legacy solutions that we have implemented.

Announcer:

Thank you for listening. Subscribe today so you never miss an episode of HSDF THE PODCAST, and visit HSDF.org for more information about the Homeland Security and Defense Forum.