What's Up with Tech?

Private AI Revolution: Broadcom's VMware Explore Highlights

• Evan Kirstel

Share episode

Interested in being a guest? Email us at admin@evankirstel.com

The explosion of generative AI has created an urgent need for enterprises to run AI workloads on-premises to protect sensitive data, maintain governance, and safeguard intellectual property. In this illuminating conversation with Tasha Drew, who leads the engineering team for private AI services within VMware Cloud Foundation, we explore how Broadcom is transforming enterprise AI infrastructure following their landmark announcements at VMware Explorer.

Tasha reveals how VMware's AI platform, now baked directly into VCF, enables organizations to implement comprehensive model governance, scale models efficiently as a service, and programmatically prepare data for RAG applications. What's particularly fascinating is VMware's commitment to "dogfooding" their own platform by using it to deliver VCF Intelligent Assist, ensuring the technology is constantly tested and improved under real-world conditions.

One surprising trend Tasha highlights is how organizations are increasingly migrating bare metal AI workloads to VCF for superior enterprise manageability. While data science teams may start with bare metal for experimentation, production AI workloads demand the scheduled maintenance, workload mobility, and scalability that VCF has perfected over decades. This shift underscores a fundamental truth: as AI matures within enterprises, infrastructure management becomes just as critical as model performance.

The conversation explores VMware's partnerships with AMD, NVIDIA, Intel, and Canonical, emphasizing their commitment to an open ecosystem approach that lets customers choose the hardware and software stack that best suits their specific AI use cases. Looking toward the future, Tasha shares exciting developments around Model-Context-Protocol (MCP) integration, agent builder capabilities, and tools for enterprises to safely implement agentic AI workflows with proper authentication and governance.

With real-world examples from customers like Walmart and manufacturing environments that require air-gapped AI infrastructure, this episode provides a comprehensive look at how VMware is making enterprise-grade AI infrastructure accessible, manageable, and secure. Don't miss Tasha's insights on what's coming next, including the technical preview of Intelligent Assist expected early next year.

Support the show

More at https://linktr.ee/EvanKirstel

Speaker 1:

Hey everybody, on the heels of Broadcom's VMware Explorer, we're taking a look back at the few days event and some of the amazing announcements and innovation from the show with Tasha Tasha how are you?

Speaker 2:

I'm great, super happy to chat today.

Speaker 1:

Well, thanks for being here. You had an amazing onstage presence, as well as so many announcements. Have you caught your breath? Are you back on Terra Firma after that amazing week?

Speaker 2:

Not yet. Yeah, we're just moving a million miles an hour over here, but you know, at least things never get boring.

Speaker 1:

Fantastic. Well, that's one thing. You and your team are not. So maybe introduce yourself and your role and team within Broadcom.

Speaker 2:

Yeah, so I'm Tasha Drew. I run the engineering team that is responsible for delivering the private AI services capabilities within VMware Cloud Foundation. So, really looking at model gallery, model runtime and ML API gateway, data indexing and retrieval agent builder, as well as the VMware Intelligent Assist.

Speaker 2:

Yeah, so much innovation announced in Las Vegas, maybe at a high level? How are you weaving AI into VMware Cloud Foundation? What's the bigger, let's say, vision behind the move? Yeah, so we identified really.

Speaker 2:

Chris Wolfe sort of made this big announcement at VMware Explorer 2023 about private AI and he really looked at where AI was going and the incredible excitement around generative AI that was taking off and he said, you know, he kind of put a flag up and said you know, there's going to be a huge driver for running these workloads on-prem in the data center with private and sensitive data that has governance and laws around it. How are we going to protect that data? How are we going to make sure that we have strong role-based access control across how we're sharing models, especially when those models have been exposed and fine-tuned on sensitive data and now need to be appropriately handled? And how are we going to protect our IP? How are we going to make sure there's no IP loss in various scenarios or potential injection of unsafe IP into our really sensitive code bases? So he laid out this sort of roadmap for how enterprises could achieve private AI and VMware and VCF has really been looking at how do we help all of our customers achieve the goals of private AI with a lot of different product capabilities that we've been putting out there for the last couple of years.

Speaker 2:

So now, kind of transitioning to where we are right now, we have a nice AI platform, now baked into VCF, for customers to be able to manage and have model governance, scale their models as a service for all of their users, for very efficient use of those expensive GPU compute infrastructure, and then start to get their data programmatically into vector databases from original document locations to start building it, being able to easily build RAG applications. And now we're going to use that same platform and capability to also deliver VCF intelligent assist. So use the exact same platform to deliver in-product capabilities. Meaning we're constantly dogfooding our own platform, scaling our own platform, testing it, living it, breathing it, which is really the best way to deliver a rock-solid platform if you have to live and die on the hill that you're giving other people.

Speaker 1:

Fantastic, and, of course, vmware, but about simplifying infrastructure. Now, how are you going about making AI-ready infrastructure easier to consume?

Speaker 2:

Yes, yeah. So basically, what we've seen is and this was a surprise to us initially was there was a huge movement to have all of your AI workloads in bare metal configuration. So initially we said, look, we see that right, like for training, you're going to beat those GPUs to death anyway. There's not a great virtualization use case. So what we're gonna go after is let's make it easy to serve inference, scale it, do have all of the sort of enterprise grade capabilities that you need when you're delivering production services and let's make it easy to do fine tuning. Those are gonna be our major use cases and let's make it easy to do fine tuning. Those are going to be our major use cases and we've seen a lot of uptick there.

Speaker 2:

But we've also started to see that people are actually starting to migrate bare metal AI workloads on to VCF Because they really need the manageability the enterprise grade manageability of being able to do scheduled downtime, scheduled maintenance. How are we going to move these workloads around? How are we going to scale right, like, let's just have a platform that's been doing that for 20, 25 years now, addressing the needs of the AI workloads? So what we see now is the bare metal use cases are more being left to the data science team. Where they're experimenting, sure, go for it. But as soon as something starts moving into production in the data center, people really want it running on VCF.

Speaker 1:

Fantastic and you announced so many new partnerships and integrations that expanded the ecosystem at VMware Explorer. What role does that play, do you think, in accelerating AI adoption with companies like AMD and Canonical and Booktube and so many others?

Speaker 2:

Yeah, so we really see our role to play is open ecosystem. So how are we helping people adopt best-of-breed technologies and technologies that fit their use cases in a really seamless, predictable, manageable way? And so, as far as a lot of the announcements you saw from us this conference you saw, you know AMD, we are continuing to invest heavily within NVIDIA. We have support for Intel, gaudi, you know. So really kind of just showing like you can run all of your workloads on VCF and have access to all of these different accelerators, depending on what your business case is. We're going to be there with you With Canonical just really having a nice strong enterprise Linux story right for people. A lot of folks have heavily invested in Canonical and Ubuntu already. They have that in their tool chain and they love having it fully supported in the product as well.

Speaker 1:

Fantastic opportunity. So the industry is abuzz with autonomous operations and agentic AI. How close are we to fully self-managed self-healing infrastructure, data centers and more?

Speaker 2:

Yeah, that's a great question. So you know, right now the industry is abuzz with MCP model, context protocol, advanced agentic workflow. These are all things that we are adding to the agent builder capability in VCF. So over the next few releases from us, you're going to start to see MCP, you're going to start to see us helping enterprises operationalize how they manage MCP. So MCP itself is just a protocol. There's a lot of capabilities that the ecosystem is only starting to realize. It needs Standard authentication mechanisms, oauth integration.

Speaker 2:

How do you make sure that you as an IT team have control over what MCP servers you're choosing to recognize and then what tools in those servers you really want your teams using to start to kind of try to bulletproof the workflows that the agents can take and make sure you kind of limit your liability while still being able to take advantage of the technology. So we're going to be adding support for that. For our customers building their agents in VCF. We're going to be adding a tools gallery and also building in a distributed tracing so you can see exactly what that model to agent to tool trace looks like, who's doing what, where you know, checking authentication at every step of that interaction to make sure you don't have any privilege escalation, baking that all in for the customer but also baking it all in for the intelligent assist. So now you can start to have MCP servers per function in VCF.

Speaker 2:

You can start to say what do you really? You know, what do you? So one of the things I'm doing with customers is I know everyone's really excited about MCP because there's been so much buzz. But if you take a step back you're kind of like what are the workflows people want to have? And I hear a lot of ones that just are wonderful. Like I just want my users to be able to be in a Slack channel and say, list VMs and get a list of all their VMs. And I don't have to go to this weird spreadsheet I've been maintaining and I don't have to go to this weird spreadsheet I've been maintaining and so really kind of starting to understand what are these workflows that just reduce toil on the VI admins that we can start to completely automate with a lot of these tools.

Speaker 1:

Fantastic. One of the things I love about Explore is always the real world examples where customers talk about their tools, Use cases and AI workloads and Walmart was one example Really doubling down and standardizing on VCF. That must be pretty rewarding. Maybe talk a little bit about that or any other customer stories anecdotes that you recall from the big week.

Speaker 2:

Yeah, yeah, I've been involved with Walmart off and on just from a number of different projects back to my Octo days, and so seeing that big announcement with them was really awesome. We've had such a great partnership with them for so long. I think for me, some of the stories that really stuck out were from a lot of our partners in the AI space who are looking at how do we operationalize delivering models as a service and then how do we start to bring workloads that have been really cloud native because the cloud is where we're starting to. The cloud is kind of where everyone starts out building AI, because you have the ease of access to GPU-based infrastructure, you don't have a long procurement cycle and you have every tool on earth. But then, as people scale to production use, what you really start to see is that the cost of maintaining those services can become very, very great, and now that you have a predictable workload, people want to bring those back on-prem and start taking advantage of the cost savings of a data center workload.

Speaker 2:

And so I myself ended up in a lot of customer and partner briefings where it was just walking through the different use cases, customer by customer, partner by partner, for how they want to deploy these. One of the ones that I found the most interesting was definitely vehicle manufacturing lines right, like completely air-gapped has to be on, you know, like has no room for error, but just running a rack per production line, basically right. So all of those like really kind of unique configurations for how people want to deploy the services and run the services, but then also seeing the standard patterns between them at the platform level.

Speaker 1:

Oh, super exciting to see that unfold Well, thanks. So much for just a little flashback to the busy week, and I can't wait for what's next. What are you excited about over the next weeks and months? Probably has down delivering all of this technology to customers.

Speaker 2:

Yeah, we're getting ready for our next big release here and so really just focused on, you know, getting private AI into everyone's hands. That was our big announce at VMware Explorers that private AI is now included with VMware Cloud Foundation. So really focused on the enablement there and know that there's a ton of excitement for folks. In addition to that, you know, the big features that my team's really focused on is that initial MCP capabilities within Agent Builder and Private AI those initial integrations and then getting our tech preview of the Intelligent Assist out early next year.

Speaker 1:

Can't wait to see it. Thanks so much for the update. I'll let you get back to work, but congratulations on an amazing week.

Speaker 2:

Thanks, evan, great to chat.

Speaker 1:

Thank you and thanks everyone for listening and watching and sharing this episode. Take care.