How to build an AI Center Of Excellence (COE)

In this third episode of The Macro AI Podcast, hosts Gary Sloper and Scott Bryan explore the strategic implementation of AI governance through Centers of Excellence (CoE), providing a roadmap for organizations of all sizes to effectively harness AI capabilities. 

What is an AI Center of Excellence? 

The hosts define an AI Center of Excellence as a centralized hub that drives all artificial intelligence initiatives within an organization. This nerve center connects technical implementation with business objectives, ensuring AI projects align with company goals whether they focus on revenue growth, efficiency improvements, or enhanced customer experiences. 

A successful CoE brings together diverse talent across the business.

Models for Different Organizations 

The podcast outlines various CoE structures to fit different organizational needs: 

• Formal CoE: Dedicated departments with full-time staff, independent budgets, and leadership - typical in Fortune 500 companies and tech giants 
• Virtual CoE: Cross-functional teams drawing expertise from existing departments without creating new structural units - ideal for mid-sized organizations 
• Hybrid CoE: Small core teams of AI specialists supported by part-time contributors from across the organization - balancing focus with flexibility 
• Lean AI Approach: Targeted implementation focusing on high-impact use cases, often leveraging external partners or cloud-based tools - suitable for smaller businesses and startups 

Building an Effective AI CoE 

The hosts provide a step-by-step approach to establishing an AI Center of Excellence: 

1. Define Vision and Objectives: Connect AI initiatives directly to measurable business outcomes and align with company mission 
2. Assemble the Right Team: Combine AI technical expertise with business acumen, scaling appropriately for organization size 
3. Build Secure Infrastructure: Establish cloud platforms, networks, and security tools to protect AI systems from threats 
4. Set Governance Guidelines: Create clear policies for AI development and usage that ensure fairness, transparency, and regulatory compliance 
5. Start Small, Prioritize Security: Test concepts with low-risk projects before scaling to enterprise-wide implementation 
6. Cultivate Collaboration: Train the broader organization on AI fundamentals and secure executive support 
7. Measure and Adapt: Track meaningful metrics and continuously evolve the approach as technology advances 

Leadership Through AI Transformation 

The episode addresses the human challenges of AI adoption: 

• Communication: 
• Leading by Example: 
• Addressing Fears: 
• Fostering Innovation: 
• Se

About your AI Guides

Gary Sloper

https://www.linkedin.com/in/gsloper/

Scott Bryan

https://www.linkedin.com/in/scottjbryan/

Macro AI Website: 

https://www.macroaipodcast.com/

Macro AI LinkedIn Page:  

https://www.linkedin.com/company/macro-ai-podcast/

Gary's Free AI Readiness Assessment:

https://macronetservices.com/events/the-comprehensive-guide-to-ai-readiness

Scott's Content & Blog

https://www.macronomics.ai/blog

Transcript

00:00
Welcome to the Macro AI Podcast,  where your expert guides Gary Sloper and Scott Bryan navigate the ever-evolving world of artificial intelligence.  Step into the future with us  as we uncover how AI is revolutionizing the global business landscape  from nimble startups to Fortune 500 giants.  Whether you're a seasoned executive,  an ambitious entrepreneur,

00:27
or simply eager to harness AI's potential,  we've got you covered.  Expect actionable insights,  conversations with industry trailblazers and service providers,  and proven strategies to keep you ahead in a world being shaped rapidly by innovation.  Gary and Scott are here to decode the complexities of AI  and to bring forward ideas that can transform cutting-edge technology  into real-world business success.

00:57
So join us,  let's explore, learn  and lead together.

01:07
And today we're exploring how business leaders can build and manage an AI center of excellence or COE to steer their organizations into the AI driven future.

01:25
Let's start with the basics.  What is an AI center of excellence  and what does it look like in practice? So we want our listeners to have a good macro level picture before we dive into the details. Cause I'm sure some of you  are probably scratching your head saying, what does this mean for AI? Yes, absolutely, Gary. So an AI center of excellence is a  centralized hub within an organization that takes the lead on driving everything related to artificial intelligence. So it's a place where

01:53
AI strategy is shaped, governance is enforced, and innovation is sparked. So the COE ensures that all AI initiatives align with the company's overarching business goals. So whether that's boosting revenue or trying to grab market share from competitors, ah improving internal efficiency or enhancing customer experiences,  or even just defending your moat from competitors. So it's...

02:18
It's also the go-to resource for expertise tools and best practices, making sure that AI is deployed responsibly and effectively across the board. Yeah, absolutely. So it almost reminds me of early stage, you know, JAD facilitated workshops. I'll probably reference that anybody that was around 20 plus years ago will know what I'm talking about, but

02:40
at the end of the day, at its core, it's like the nerve center for AI. So connecting the tech side with the business side and having all the stakeholders and workflows together. Exactly. Exactly. So it's not just about coding algorithms. It's about tying AI to real world objectives and outcomes. Very important to make sure that you're focused on objectives and outcomes. So the COE brings together a mix of talent. So think of data scientists if you have them, if not, can outsource them.

03:06
uh AI engineers, business strategists, security experts, even privacy and ethics advisors you want to consider  to make everything happen. Yeah. So I think this is very powerful. So in the conversations that I have, especially at the C level, so if you think CIO, CEO, even CFO, because  they all have to be part of this strategy,  when I'm having these conversations, you know, talking about designing a COE for them,

03:34
But what I always try to stress, and I'm guessing you concur with this, it doesn't always look the same at every organization. So if you agree with that, what are the different forms of an AI COE can morph into? Yeah, good point. So it's definitely not a one size fits all model. It's different across the board based on the size of the organization. So the structure of COE really depends on the size.

04:01
resources inside of the organization and what are the ambitions of the organization. So I'll break it down a bit. a formal CIO, COE in larger companies like a Fortune 500 or tech giant, you might see a dedicated department. So dedicated department, specifically there for AI with a full-time staff. So this team is solely focused on that with its own budget, leadership, and they define the roadmap. So think of it as a standalone unit.

04:28
like an R &D lab pushing AI projects  from concept to execution. So just changing size a little bit, you might want to downsize a little bit and say, for a slightly smaller company like a midsize organization, you might want to go for a virtual center of excellence, which would be more of a formal department is not something that they need. Instead, they might create a virtual COE,  cross functional team where experts from existing departments like, you know, IT, marketing operations,

04:58
They work together, they collaborate on AI efforts. So it's less rigid,  leverages talent that's already in-house without needing a big new structure. uh Then there's a hybrid version of that. So some companies blended to. They might have a small core team dedicated to directing the AI initiatives,  but it's supplemented by part-time contributors from other areas. So this hybrid approach  is more of a balance with a balance that focuses flexibility  adapting to the organization's needs.

05:28
And then there's the lean AI approach, we get a lot of questions from smaller businesses  all the time. You  know, a lot of questions from small businesses. So for smaller businesses or startups, a full blown CoE is just most likely not feasible. So instead, they'll adopt a lean model focusing their energy and resources on a handful of high impact AI use cases one at a time,  like automating customer support or predicting inventory needs. So they often rely on external partners, consultants  or

05:58
you know, cloud-based uh AI tools to fill the gaps. So with small teams, know, discipline is key to keep that regular cadence going and to follow through. That's where a lot of things drop off. They just don't stick to it. So the follow through is key. Yeah, that makes sense. That's a really good rundown, Scott. so, you know, in essence, if you're a multinational organization, you might have a formal COE with dozens of staff compared to a smaller retailer who could

06:28
you know, still benefit from AI with a lean setup and some outside help, right? Right. Yep.  And so the form isn't really what matters most. It's the function. So whether it's a big team or a  solo champion with a laptop, the COE's job is to centralize the AI leadership, bring them together, drive the strategy and make sure the organization doesn't miss the AI boat. Yeah. I like that flexibility too, because  you're not trying to

06:58
slot an organization that's smaller to then go add staff that's not required or you know whether it's technical business etc. It really means a company anywhere can find a way to make it make this work and make AI work for them.  You probably agree to that right? Yeah so yeah certainly. Yeah so uh so with that background that foundation let's  talk about how to build and manage uh whatever form it takes.

07:25
Okay. So let's get practical and tag team through the steps to build an AI COE. So what's the first step to building a center of excellence, Scott? Sure. Yeah. So it starts with, uh, defining your vision and the objectives. So obviously you need a clear why for your center of excellence. Maybe it's cutting costs by 10%. Maybe it's improving customer satisfaction scores or speeding up product launches or product development. So you tie those goals to your company's mission. So everyone sees the bigger picture.

07:55
Step one. Yeah, it makes sense. So if I think about step two, want to assemble the right team. So this could be a formal group or a virtual squad. You'll need AI experts,  security professionals,  and business minded individuals  as part of that group. For smaller teams and environments, I usually will suggest, you know, that might mean a one AI savvy leader plus some external support. So some consulting groups have access to a bench.

08:24
all forms of AI talent. So that's a good way to leverage that. There's some gaps and that talent, you know, can align specifically to your needs without having to go higher or find the right person or people. Yep. Yeah. So from there, uh, next you want to build a secure infrastructure for AI. That's obviously right in our wheelhouse for 20 plus years. So you need a cloud platforms, you know, network interconnectivity. Maybe it's multi-cloud connectivity. So you want to then

08:53
secure, make sure you've got the security tools. So to protect against any kind of risk, like, you know, hackers tampering with your AI models, obviously, security is not optional. So we'll consider that foundational. And we'll talk a lot more about that in future episodes, because obviously security and security and AI models is a little different, but obviously critical. Yeah, I completely agree. And I think if you were to fall on there as the next step, it really comes down to governance, similar to security. I view

09:23
Governance, privacy, ethical guidelines, really important. You want to make sure that you're creating policies for how AI is developed and used. So you want to make sure it's ensuring it's fair, transparent ah programs and meeting those regulations like GDPR, CCPA, for example. uh And then you also really want to make sure that you have regular check-ins to keep everything honest, especially, it's especially important since

09:53
regulations are still on the drawing board and they're going to continue to change. Just look at what's in the news, geopolitically, anything could happen. So making sure you have those regular check-ins is important. Yep. Yep. And then we're onto step five, which is kind of getting started really on the project. So you want to start small and again, prioritize security. I just said that, but I'll say it again. So you want to test the waters with low risk projects.

10:20
So like something simple, like automating the reports before scaling it up  and improve that concept  and make sure again, pardon your defenses, focus on security. Yeah. And if you think of step six, it's part of this,  which is really important from a culture standpoint, you want to cultivate collaboration. So train everyone, not just the center of excellence  on the AI basics and security to your point, Scott, get buy-in from the C-suite.

10:48
to the shop floor. if management and leadership is not fully bought in, how can you expect the individual contributors in the organization to follow through with this? So it's a team effort. Just a little bit of focus will drive a lot of ideas and hopefully generate some excitement and dispel any myths that people have. No, I agree. Yeah, training is definitely a good way to build some excitement. It gets everybody kind of motivated to try these tools and get on board with AI. So I agree with that. ah

11:17
Yeah, so finally, you want to measure and adapt. So you want to be able to track results. So your team, your  AI center of excellence wants to think about how can you track results?  So cost savings,  error rates,  whatever matters. So tweak your approach as the AI technology evolves.  It's  center of excellence is it's not static. So it's going to evolve with you, with your team, with the business. Yeah, I completely agree. we just put together a very solid playbook.

11:46
But a playbook is only good unless everybody follows through. So really part of this is, you know, outside of just building this environment, this center of excellence, it's leading the people through the change. And Scott, you and I have both led many organizations through change. let's  why don't we dig into that a little bit, because that's part of the soft skills that I think a lot of people really don't want to overlook when it comes to AI transformation. Yeah. No, thanks, Gary. Yeah. So leadership is key and we get a lot of, um,

12:16
outreach from executives because you know, this is something that they're thinking about, they're concerned about, and they oftentimes don't really know what they're talking about yet. So the biggest hurdle is, you helping them and thinking about how do you communicate the vision? So people aren't going to embrace AI if they don't get it, or worse, they'll even fear that it's going to replace their jobs. So leaders need to be able to explain how AI fits and what it means for everyone's future in the organization. how do you win them over in that situation?

12:46
because you've come across this. Good question. So leading by example. So executives do need to roll up their sleeves a bit. They need to dig in. They need to start using some AI tools. And we oftentimes guide them as to some that are a good starting point. But they want to join in on some demos. They want to share success stories. They want it to be not just talk. So they want to listen to great AI podcasts.

13:16
like this one  and stay current so that they can get those gears moving around, you know, what is AI and how can I lead my organization forward with some excitement?  Totally agree.  And really tackle fears directly, right? Be upfront. Yes,  some tasks might shift, right? Your job function may have some tasks that shift, but here's how we will re-skill you. You know, so training to oversee an AI system instead of doing the grunt work prior.

13:44
I mean, that's how you build a great culture and getting people across that finish line, right? So you're showing them the path of their future versus what they're doing today. And it can be exciting. Yeah, confident that you're going to help re-skill them. good point. um So another key is just sparking innovation. So there's going to be a lot of competition out there. You've got to be ready to think a little bit different. So give your teams room to play with artificial intelligence. So maybe it's a group challenge to.

14:14
up with a bunch of ideas. uh Nothing's off the table or it's a hackathon to solve a pain point. So maybe think of ways to reward the best ideas to get some excitement going in across the organization and inside of each individual business unit.  Yeah. mean, I think the worst thing that you can do is,  is over hype it, right? Keep it real. Start with small wins.  Try shaving an hour or so off of a weekly process that will help you build trust.

14:42
without setting some crazy expectation that AI is going to do a bunch of things that it won't do. So we, we,  in one of our last episodes, we talked about, um, you know, the rise of  agentic AI. Um, there's some off the shelf things that you and I have demoed for companies, um, you know, recently, and those are easy wins. So it's not like you had to go up in the organization. You're just improving and giving some cycle time back to the company. Yep. Yeah. So, yeah, let's think about.

15:12
you know, once you get started, so maybe some thoughts, Gary, how, how, do you rally a team to help pace the competitors that might already be using artificial intelligence in the business? Yeah. Yeah. I mean, we talked about, so you have the playbook, right? So how do you get the team to follow the plays and, sometimes call audibles, right? So I always say, paint the picture. they're, they're using AI to cut costs and steal our customers. We can sit still and let's compete, right? So share examples.

15:40
For example, Amazon's recommendation engine, and they say, we can do this too, and we can do it better. I think people really love to compete. It's energizing. Totally agree. Yep. So, uh, you know, maybe tap into that ambition, you know, that urge to compete. So  position your company as the leader, not the follower. So you host, like I mentioned before, some brainstorming sessions, let your team come up with AI solutions and have them  own the fight. That'll, that'll definitely get them energized.

16:10
Yeah, yeah, totally agree. mean, just being able to explain that again, not with a machine with you as a human to another human. That's, that's where people do get lost that  that's still going to be there. Um, so if I think about AI strategies for smaller businesses, right, there's a lot of smaller companies out there that probably are on the sidelines,  maybe nervous, unsure, untrustworthy of what this all means. So

16:36
about those smaller players without a formal center of excellence, how do they keep up in your opinion, Scott? Yep. Good question. And like I mentioned, we get this one a lot from a lot of, uh, lot of small businesses. So my recommendation is  start small. So, but think big. You want to think about what can your business do? Think big and that's going to help them push the envelope.  Uh, but focus on that one pain point, like using artificial intelligence to  predict sales trends. You know, nail that.

17:05
and then expand to next thing. Right. Yeah, I definitely feel tapping outside help is important. So, you know, some organizations use consultants when needed. You use it, might use a consultant like a CPA to file your taxes as a small business. This is no different.  Uh, if the, if the tax codes changed on a daily basis, would you have a CPA helping you on a daily basis? And the answer is probably yes. You want to be

17:32
within the rules of the IRS. This is no different, right? So it's okay to make sure you have the right consulting team to offset some of those gaps and  some of those areas. So you're not having to build this from scratch. Yeah. And I'd say that, you know, there are some platforms out there too, that have some pre-built, you know, AI solutions too. And that I consider that outside help too. So that's one thing that we...

17:58
point a lot of small businesses to where some of these pre-built agentic platformers, et cetera.  But they're definitely going to be smaller businesses are going to be learning as they go, obviously.  But the leaders want to push their teams to take online AI courses that get some energized.  Many of them are free or not that expensive. And that obviously helps to start build that knowledge in-house more than just using chat GPT to help their kid with homework. Right, right, right.

18:27
You can still pick smart tools, right? You can go for the plug and play options like a chat, GBT to build some of your content for blog posts, those types of things. Obviously we talked about security and privacy. You don't want to put intellectual property  that you wouldn't want the outside world to have access to. But even things like, like I mentioned, you know, the CPA example,  QuickBooks with AI insights. Those are easy wins  without having a tech team full of PhDs, right? So if you could help.

18:56
couple small areas within your business and use some of the AI, could even be a weak AI environment that some of these off the shelf platforms have. That's, that's a first step.

19:08
Yeah, and I'd say speed is critical. iterate fast, measure what's working. For example, how much time do we save on this process that we do over and over again? And then ditch the things that don't, clear them off the plate. It's lean and mean. that's how you do it with a smaller budget. Yeah, I totally agree. So we're going to wrap up to kind of summarize. So an AI center of excellence, formal, virtual, could be hybrid or lean.

19:38
Uh, it's really in your launch pad from mastering AI tailored to your organization and use cases,  right? Yep. Yep. And, um, like we talked about throughout this podcast, you know, leadership,  uh, leading through it takes, takes vision. takes action,  uh, discipline. So, so inspire your team, equip your team adapt. So whether you're, whether you're a big company and you're leading a large business unit or multiple business units, or you're the CEO,  uh, or if you're a startup.

20:07
It's the same thing. It's leadership takes vision. Yeah. Well, great, great episode.  Loved talking with you on this, Scott, as always. uh Can't wait to continue diving into future episodes  on these areas. Yeah, thanks, Gary. Good luck. Thanks for tuning in to the Macro AI Podcast. Share your AI journey with us.  For all ears, until next time, lead the charge.

20:39
you