NatChat - The Chip Shortage Explained: What Rising Memory Prices Mean for Your Infrastructure Strategy

In this episode of NatChat, Nick Diesel, Multi-Cloud Solutions Architect at Natilik, is joined by Matyas Prokop, Technology Director for Multi-Cloud, to dig into one of the biggest challenges facing IT leaders right now - chip shortages, soaring memory prices, and what it all means for organisations trying to plan their infrastructure.

This episode covers:

- Why memory and chip prices are rising - and what's really driving the shortage

- The link between GPU memory demand and the rising cost of standard server memory (DIMMs)

- What's really creating memory demand - and why prices are unlikely to drop back to where they were

- Whether to buy now or wait it out — and what clients are doing in practice

If this episode got you thinking about your own infrastructure strategy, now is the time to act. Natilik's Multi-Cloud Discovery Workshop is designed to help you understand your current environment, identify the right balance across on-prem, cloud and hybrid, and build a roadmap that works for your business

Book your session today here.

Transcript

SPEAKER_00 0:15

Hi and welcome to another edition of NatChat. And uh today uh I'm joined by Matthias uh Prokop. Uh and did you want to go ahead and introduce yourself?

SPEAKER_01 0:24

Sure. Uh thanks for having me, Nate. Uh I'm Matthias Prokop, I'm the technology director in Natalic looking after multi-cloud. So everything from data centers all the way to cloud.

SPEAKER_00 0:31

Fantastic. And I'm Nicholas Diesel. I'm one of the uh multi-cloud solution architects here. And today we're going to be discussing everything microchips, chip shortages, and RAM price increases. Yeah, we love it. Yeah. Matthias, do you want to set the scene around some of the latest changes and updates?

SPEAKER_01 0:46

Sure. So uh it's obviously very exciting to be in the data centers these days. Uh it felt like you know nobody cared about us for uh like decades, and now like you know, for the last few years, uh it we're we're in the midst of it, uh, which is a good fun. Um I think what we're seeing, me and you with our clients, uh, is lots of like you know, heated meetings, uh fighting for the compute. Um we're seeing you know massive sort of e delays and uh you know increasing the lead time. So from what it used to be weeks, uh, we're seeing it changing into months, sometimes you know, even over a year, um which is challenging, but um it sort of like you know sometimes forces you into like thinking out of the box and thinking about the different solutions. Um we see obviously the increase in price. So um if the supply is uh smaller than the demand, obviously the price is going up. Uh so mainly driven by the memory prices, um, which I'm sure we will get into a little bit more of a detail uh today. And um I guess like you know, what what what what cause it um is always the big question. I think the the big driver behind this is what public cloud providers are doing. So uh they've deployed like you know massive amount of money into the market. They're spending you know hundreds of billions, I think it's over 700 billion, which they're deploying this year, uh buying everything from compute to networking to power generators. So it's a huge amount of money, which is obviously you know creating huge demand, uh, and just supply can't can't keep up with that. So uh that's what's happening. You know, some vendors are better in it, some some vendors are are worse. Um, and uh we're just trying to navigate and help our clients with it.

SPEAKER_00 2:36

Yeah, we've particularly seen it across the board, whether it be Cisco, Dell, HPE, Supermicro, we've seen across the board sort of price increases and memory, CPU translates through to full full orders. We've seen cancellations, we've seen um lead times pushed out massively, and you know multiple triple figure uh sort of increases in terms of prices. And uh it's causing a bit of uh unrest in the market. We've seen everything from you know clients panic buying ahead of time. You know, what was 12 months out is now being purchased now because it's because of the long lead times and you know the fear of further price increases. Um but I mean, yeah, this is all driven from shortage of you know CPU and memory, particularly memory. Uh HMB memory is now being heavily manufactured for the sort of AI industry, and like you say, the hyperscalers and cloud providers are taking up everything they can find. Um, there's a few memory companies that have uh pulled out from you know commercial markets altogether, they've uh retooled their environments to provide HMB specifically for the AI data centers and things like that. And it did it doesn't seem to be slowing down anytime soon.

SPEAKER_01 3:44

Uh yeah, yeah, it doesn't. It's always like interesting. Like, okay, so so what what caused this, right? Like, you know, why the memory is suddenly in demand? Like, you know, there was always a demand for the memory, but why so suddenly there's like you know, such a big demand for the memory?

SPEAKER_00 4:02

Explosion of AI, really. Um, you know, I mean, obviously, more complex models that you want to run within sort of AI factories and things like that require more memory capacity, um, more speed memory. And you know, the faster you can generate those tokens, the more they can be useful. Um, you've had an instance a while ago where you were generating, you know, hundreds of thousands of tokens. Now it's you know in the multiple millions, if not billions of tokens for the world to consume. Um it seems every every day I I hear someone talking about you know, we're generating tokens for this or for that. Um and AI seems to be playing a bigger and bigger role in everyone's lives, whether it be you know just Chat GPT or Copilot or something like that, or more to the more niche requirements of AI. Everyone is generating tokens.

SPEAKER_01 4:47

Yeah, so but but this that that's GPU memory, right? So, like, you know, how how the GPU memory is related to DIM memory, where we are seeing such a huge increase in price.

SPEAKER_00 4:57

It's coming from the same manufacturers. Um you must remember that whether it's GPU or uh DIMS in terms of just RAM, it is the same little silicon chips that are going on in it, whether it's the 32, 64 or the 96 gig um little chips that are going on it, you can only provide so many of them. And a lot of the manufacturers are realizing that if they are producing HMB specifically for GPUs, um, they can retool and sell it at 10 times the price because HMB memory is much more expensive for GPUs. Um and that obviously gets plugged onto a GPU and it goes into a particular board and it gets gets used for AI. But um, if they start pulling resource away from the sort of general lines and put it towards HMB that's going specifically for GPUs or for data centers, it makes a bit of a constraint on the normal DIM processing um that just is standard memory, which again drives the price up.

SPEAKER_01 5:55

Yeah, it's it's interesting. Like we're talking in video specifically, right? Uh they they are using the HBM. Like, yeah, you know, there are obviously vendors coming up to the market, and I don't want to like, you know, like sidewalk from the topic we have today, but there are vendors already there who are exploring how to build GPUs mainly for the inferencing, which are not gonna need the HBM, right?

SPEAKER_00 6:20

Oh, definitely. Um, you know, I mean there's uh quite a few interesting GPU options that are coming out of China at the moment that's competing against Nvidia. Uh AMD's got their own stuff, Intel's always never want to be forgotten as well. Uh but yeah, you must remember that there's only two or three really big players in terms of you know creating uh memory chips, as it were. Samsung SK, INEX, and Micron. Uh and like I say, Micron recently pulled out of the commercial market and now are only enterprise driven. So it's literally only those three companies that are providing most of the world's supply in terms of memory chips. Um, you know, TSMC is obviously still never to be ignored, but that's more around CPUs.

SPEAKER_01 7:00

Okay, so so like you know, if if I, you know, let's say I'm the memory uh fab and I'm like, you know, making like and I know like you know there's a massive business in there. So like first thing I would do is like, you know, I'm gonna build up like you know extra fab. And then next year, like, you know, obviously it will take some time, but like, you know, next year in 12 months, I'm gonna have a fab and I can generate more memory and I can make more money. Like, if that would work this way, we would probably already see like a decrease in the price. So why is why is why is it not happening?

SPEAKER_00 7:34

Oh, definitely. But spinning up something like a memory fab or even a CPU fab is you're talking about hundreds of billions of dollars. And again, the kind of companies that make the machines that actually do the lithography onto the actual chips themselves, um, because the way they produce is a process called lithography when they create those chips. And there's only one company globally that provides those machines. Um, and they are stressed as well to provide as many as they can. Intel are spinning up their fab as well, and it's the same supply of machine. Um, and trying to get that particular kind of production in terms of a fab uh running, the clean room and everything like that, that takes years and yeah, millions, if not billions of dollars uh to get it up and running. And I do know that TSMC is spinning up another fab. I do know Micron is looking at putting up another fab, and they're in production and they have been for about a year or so because we knew this constraint was coming. Um but we're still looking at lead at least two or three years out until they are fully up to production.

SPEAKER_01 8:36

Okay, so so if I'm if I'm on a market, like you know, let's say I'm the customer and I'm on a market and I want to purchase uh compute, and it's like all over the place. Like, you know, like it's increasing constantly. Sometimes there's a little bit of a drop, but then it's gonna go like you know, it's going up again. Like, what would be your suggestions? Like, you know, should I wait it out? Like, you know, like is this gonna end you know next year? Because you said like you know, three years. So like let's say they start building like last year, it will probably, I guess, like get better next year. Uh or like, you know, should I expect like, no, actually, this is gonna get worse before it's gonna get better, uh, and maybe the price is not gonna go even like you know, below where we are.

SPEAKER_00 9:18

It might do. I mean, you know, if we listen into some of the analysts, and again, I don't have a crystal ball to look into, but some of the analysts are saying, listen, it's gonna get a lot worse before it gets better. Um, you know, the if we eventually look at the end, we get we might have a you know oversupply of memory, but at the moment it's massively short. And you know, companies like Nvidia aren't slowing down anytime soon with the release of the new N1X chip and things like that. They're dipping their toes into CPU desktops and things like that. So the increased requirement for memory is just going to go up, as far as I'm concerned. And we've seen that from a lot of our clients, and that's why it's a very difficult time to say yes, buy or hold out. Um, where we've seen clients that have actually purchased early, they've avoided additional price increases that have now been put uh put out across the board. So, you know, they bought at the right time, but arguably should they have waited. Um it's really depending on the business and you know how the business wants to do business. You know, can you afford to wait another two or three years in the hope that it's gonna get better? Do you buy now and potentially make a loss? It's uh a challenging time for everyone, as you know, as far as I'm seeing. And like I say, we've got it across the clients. I've I've seen it day in and day out. Some clients saying, Yes, we need to buy now before the price increases come, and some clients saying, Listen, I'm gonna sweat it out. So a lot of clients as well moving to moving to cloud and cloud-based options, um, basically to avoid it.

SPEAKER_01 10:48

Yeah, I'm I'm always like obviously my role in this company is always like looking at what's gonna be in a year or two, which is always like you know, great fun, because like you know, who knows what's gonna be in 12 months. So I'm I'm just guessing here, right? But when you think about it, like the assumption of the supply will increase and therefore it will sort of like balance it out with the demand is on the is based on and then therefore the prices will go down. It's based on the assumption that the demand will stay the same. But now the question is like you know, is it because like it feels like the demand is going up and up for the last you know four or five years, and it doesn't feel like you know there's any signs of that demand increase is you know slowing down, and then on top of it, you can you you can like you know, sort of like thinking about it, like you know, especially for the last six months, which is like gonna be probably another like accelerator in terms of like what the demand is going to be, is the agentic AI, right? So like you know, that's a whole new world, and the agentic AI like it changes the whole architecture, like you know, we've seen what Nvidia is doing, like you know, what they're when they're introducing Vera, etc. Like that it changed the whole architecture, how they like building their products and solutions. And one of the first things we've seen with Vera was much more focused on a CPU. So, like you know, like CPUs are coming back after like you know, like so much focusing on GPUs lately, and the memory requirements are definitely going up as well because suddenly what you have with the agentic AI, you we see like you know, using much more tools, uh like you know, that's the whole point of the agentic AI, like you know, leveraging the tools, and using the tools that means you know much heavier on the CPU and much heavier on the memory. So, my question would be like, you know, is this going to have the impact on uh you know this like you know memory prices? And are we about are we heading into another bottleneck? So like now we have the GPU memory bottleneck, so are we heading into the another bottleneck, which is going to be CPUs?

SPEAKER_00 12:56

Correct. Well, I mean, you know, as you rightfully said, you know, the what Nvidia has released with Vera and things like that is moving towards a CPU-focused um processing of the actual agentic AI agents. Now, as far as the way the processing works, you're just shifting your focus. So you're shifting from a GPU-based compute with GPU-based HMP memory to CPU compute with system memory. So then we're looking at things in terms of upgrading system memory. So then large-scale system memory is gonna be needed, higher speed system memory. We're gonna be moving from DDR5 to DDR6 and looking potentially further past that. Uh, far more dense uh chips are gonna be available and things like that to try and meet that demand. So you're not really getting rid of the demand, you're shifting it from one focus to the other. Um, still putting massive strain in terms of the um the way that the memory uh shortages are currently being applied. Um it may require a different tooling or retooling from some of those many uh memory manufacturers, but it's still gonna be a uh constraint on that supply.

SPEAKER_01 14:02

And so, like, you know, then there is this like another angle in question. It's like how how is this different from like what we've seen in the 90s and like you know, mid-2000s? Like, you know, obviously there's been always these like you know, spikes that went down. Like this feel like slightly different, like is it or is it not? It is slightly different, but it's at a much larger scale.

SPEAKER_00 14:24

Um, I think when we looked at stuff in the 90s when it was exploding, it was quite um quite price sensitive in those sort of times when it was available to a select few. It wasn't as widely adopted as it is today. I mean, you've got uh you know kids running agentic agents on cell phones and things like that, using cloud services and things and generating hundreds of tokens and things. So it it's much more available to the public these days and to businesses, and the sort of market for uh AI and agentic AI within the business to generate profits is off the scale. And I think that's gonna generate so much demand for compute, whether it be CPU, GPU, but the memory is obviously still gonna be there. Yeah. Um, because you now, like you say, with agentic AI, you've got agents spinning up other agents, generating more tokens, and you know, go you've got inter-agent connectivity and things like that, and it's just going to get much more involved.

SPEAKER_01 15:19

Yeah, it also feels like when you look, uh, there's been like this graph of how much like you know, constraints it is now in the memory field when it comes to technology. So, like, you know, we were able to like you know, sort of like follow the loss of like, you know, we're gonna be exponentially growing like every year, you know, more loss and and stuff like that. But it feels like last few years we slow it down in the memory space, and we're starting to see like you know, technology constraints as well. So, like, you know, jump from DDR5 to DDR6, it it will be possible, but like you know, how much we are able to actually compact in one chip, it's like you know, it's starting to slow down. So we're starting to like hitting these like physical limit limitations as well, right?

SPEAKER_00 16:07

Moore's law is starting to break down. Yeah, um, the smaller we go in the memory space, yeah. In terms of the CPU and the GPU market, it's it's blown past what Moore's Law puts into place on the memory space. Yeah, it's now about density, um getting as much as we can out of those chips and things. But um, yeah, I I don't think the supply for that will ever wane. And I I I I don't think personally, from what I'm seeing, the prices will come down. I think that will just be the new norm going forward.

SPEAKER_01 16:38

Okay, so you're you're saying like it's not gonna go down, that like you're saying. That's my let me rephrase it. Like, you know, yeah, exactly. You're predicting, like, you know, it's not gonna go down.

SPEAKER_00 16:48

Well, you spend a lot of time talking to analysts and um, you know, having behind closed doors meetings. Anything you can share that you've heard?

SPEAKER_01 16:55

Well, obviously I can't, right? Like, you know, they would have to they would have to shoot me like if I would say something. But I'm predicting, yeah, I'm just guessing. Uh I I don't think it's gonna go down, like it will eventually go down, but it won't go down below what we have today. Like, there's literally no signs of slowing down. Like, you know, when you see like how much money all these big players are deploying into the market, there's no sign of like you know slowing it down at all. So we're probably gonna see like you know, another like you know, 800 billion uh next year. Uh the businesses are doing well. Like, you know, when you look at the Google, AWS, like especially AWS, they had a huge spike in like how much profitable they are. When you're considering how much investing, it's crazy how well they're doing. So like, you know, all these businesses are like well-oiled, uh, well-oiled machines uh who like generating loads of loads of revenues and loads of profits. So they will be using it to sort of like you know not staying behind. Um, you see, like, you know, I think Entropic is a perfect example. If you are like underestimate your projections, how you need to grow in terms of the compute, then you are really struggling with like accommodating everything what your customers are needing. And so Entropic is like, you know, like limiting it here and there, trying to do their best, but you know, their supply of the like resources is very limited to when you compare it to Google and and and AWS, and that's probably where you don't want to be. So these guys, they will like you know keep spending money uh on a compute as well, and then you obviously have that enterprise side, and they're like, you know, there are no signs of like slowing down as well. Like, you know, you have these like you know, quant traders and who are like investing heavily into their computers into their data centers, um, enterprise like outside of these like big you know finance houses, like you know, it's it's doing pretty much the same thing. Um I I really do think that in the long term we will see clients if if the if the pricing, which I do believe in a public cloud will stay at least the same, but it will you know definitely go up. You know, there's been already like you know a couple of increases in the public cloud rate. You know, I do believe that more clients will move their AI uh on-prem. Uh it's just like you know, we're seeing first few clients, we're having conversations with the first few clients who are testing the AI in a public cloud because that's you know brilliant environment for that. But once they start rolling it out into the production, they start into figuring out like you know, there's actually lots of limitations in the throughput. Uh there's limitations in terms of the latency and availability of the services, so they're moving it back on-prem. So I think we will see you know increase in buildup in the data center space uh in in in that area as well. So um, yeah, I I don't I don't see like you know why this should change in the next two or three years. Uh it's gonna it's gonna be pretty much the same. So like, you know, supply will be still behind the demand. Um, and it's gonna be 2030, and then in 2040, we're going into the space. So uh data centers in the space. Yeah, data centers in space.

SPEAKER_00 20:02

I mean, you know, given that, like you said, you know, clients bringing stuff back on-prem and you know, the the the need for AI growing and things like that, um, you know, from uh your position as multi-cloud director, what what have you seen with clients? What is Natalic gearing up for in terms of offering to clients? Because AI seems to be across the stack, whether it be networking, collaboration, contact center, uh, AI is ingrained in everything. So how's Natalic making it available to clients?

SPEAKER_01 20:28

Yeah, um, so obviously, like, you know, we have services like you know, like the the the AI, um AI services where we can help clients to find the use cases for the AI. Um so I think you know that's the that's the best approach where to start, uh, sort of identifying uh these use cases, then help them to spend some time with their teeth with their teams and implementing the AI uh into their processes. Uh, because it's not necessarily with AI about building products and building you know features, it's more about like how you can implement the AI into their business processes. So we're doing that, and then when you have these and you have it you know tested, uh then you can start rolling it out. So that's where we are providing all these pipes for AI. So you know, either it is data center, uh it is your public cloud environment. Uh, we have obviously not select cloud offering uh where we are uh providing like data server in cloud uh or AI server in cloud uh because it started becoming quite obvious that AI is the accelerator for these kind of conversations where some of our clients wanted to bring AI a little bit more under control because you are providing loads of your internal data to AI, not just for the training, but like you know, in the even the inferencing, like you're sending loads of data out, uh, and uh that can be very sensitive uh data. So you don't want that. So um, you know, bringing the AI a little bit more under control with things like Natalic Cloud or on prem data centers, it will make sense. Uh, and it already is starting to make sense to lots of. For our clients.

SPEAKER_00 22:00

Yeah, you see, that sort of brings the conversation full circle when we look at that to say, listen, a lot of clients are bringing it back on-prem, not investing in the hyperscalers and that due to costs. And, you know, those costs would only come down when the memory prices come down and the availability is more. So, you know, it that sort of plays back into that message to say, listen, it's not slowing down anytime soon and it's going to continue to be an ongoing thing.

SPEAKER_01 22:24

Exactly. And like, you know, we you remember, like, you know, we were talking to a couple of our clients who thought that if they move into the public cloud, they will have the access to this like unlimited resources. But what we're seeing, you know, uh with some of the public cloud providers, they are requesting like a business case to increase the quotas and available resources for a client. And so there's obviously like resource constraint uh in the public cloud space as well. So uh it's gonna be just interesting to see like you know uh how how this is gonna pan out for the for the public cloud providers because then clients will be like questioning this whole like you know, everything into the public cloud, like if we are concentrated there, like why why we're not having our own environment uh where we would have the constraints anyway, but at least we could like predict and we would have like a little bit more of a control about like you know our roadmaps and how we're gonna be building up our infrastructure.

SPEAKER_00 23:16

No, guaranteed. That's definitely you know something to watch out for. I mean, if if public cloud providers with the amount of money and you know sort of business force they have behind them are having constraints where you have to create a business case in order to get a little bit of GPU, um, you know, that's obviously indication that the market is struggling. There is a constraint somewhere. And you know, as you said, you they're looking at investing data centers in everywhere. I've heard you know, Microsoft are looking at putting data centers back into the Arctic, the Chinese are looking at putting them in the sea again, and Elon Musk is looking towards the stars. Yes, exactly. So yeah, it's always an exciting time to look at where it's gonna go. I think that could be the next discussion is you know, are we running out of power? Are we running out of space?

SPEAKER_01 23:56

Yeah, yeah, it's gonna be interesting. Like, you know, like when you think about it, okay, in 2030, we will get out of these all bottlenecks around, like hypothetically, out of like, you know, CPU, GPU, memory bottleneck. So, what's the next bottleneck? Like, you know, I I suspect it's going to be how quickly we can actually build those data centers. Uh, it's not gonna be probably like you know, how much space we have, especially in the US, it's probably gonna be an issue in the UK, but in in the US, I I don't feel like you know that's an issue. Uh, and when you think about it, like you know, most of the compute right now is being built up in the US. So US, like, you know, I think in the long term will have the most of the compute. Uh, but then like what's the next constraint? Like, you know, like are they gonna be able to keep up building these data centers? Like, you know, like maybe the power generators, uh, and then maybe like you know, that like sending the data centers into the space will be like you know, next next logical step, how to sort of like you know avoid these these these constraints. Um but that's for another one.

SPEAKER_00 24:57

Yeah, I mean with uh sort of uh data center providers, Link Dell, HPE, things like that, reporting you know triple figure growths year on year and things like that. We've got to find a place to put it somewhere. Yeah, but yeah, as you say, that would be a conversation for another another day, another topic. Not till it will be building the rockets in a ten in a ten years. Not till it can space.

SPEAKER_01 25:17

Not till it can space, yeah.

SPEAKER_00 25:18

Not till it space. That's the one. That's the one. Well bro. You heard it here first, folks. Anyway, thank you again for joining us. Uh thank you, Matthews. I appreciate you. Thanks for having me today. And yeah, tune in again for another Nat Chat. And uh thank you and have a fantastic day. Thank you.