Micron's Record Profits, Apple's CXMT Plea: AI is Eating All the Memory

Austin and Vik break down the current memory crisis, where AI demand is driving unprecedented profits for memory makers like Micron, while simultaneously causing inflation and price hikes for consumer electronics. They discuss how even premium brands like Apple are struggling to secure supply, leading them to controversial suppliers. The hosts also explore the long-term implications of AI's insatiable memory appetite and its impact on both data center and consumer markets.

Chapters:
0:00 Memory crisis hits
1:20 AI impacting consumers
3:00 AI causing inflation
6:48 Consumer demand drop?
8:49 AI demand inelastic
10:56 Long-term memory outlook
11:02 GoPro's memory woes
12:24 Apple's pricing power
18:20 Apple seeks CXMT DRAM
21:38 Shrinkflation for phones
23:16 Korea's memory investment
26:10 Micron's killing profits
33:00 Why AI needs so much DRAM
40:27 Future of AI training
44:34 Cost-optimizing inference

Follow Chipstrat:
Newsletter: https://www.chipstrat.com
X: https://x.com/chipstrat

Follow Vik:
Newsletter: https://www.viksnewsletter.com/
X: https://x.com/vikramskr

Follow Semi Doped:
Get more of Austin and Vik daily, free!
Sign up: https://www.semidoped.com/

Transcript

SPEAKER_01 0:00

So check this out. Micron booked more revenue in a single quarter than in any full year of its 50-year history. So they booked basically $41.5 billion, and its gross margin hit 85%. That's more than Nvidia's gross margin. And for a brief moment there, Micron as a memory company was worth more than Meta. And to add up to all this, the South Korean government is investing over half a trillion dollars in SK Heinix and Samsung just to increase memory capacity so that they can own the memory business entirely. And in the midst of all this, Apple is now increasing prices and they go into China CXMT to get memory capacity.

SPEAKER_00 1:10

What's up, man?

SPEAKER_01 1:13

Not that much. We've just been uh dealing with the memory crisis. I don't know. I'm not dealing with it. Like, what do I have to do with it? I'm not buying any devices. My Mac M3 Pro is doing amazing. So I don't have to buy a laptop or anything. But I am also being more careful with it. I'm not gonna drop it anytime because if I have to replace that thing, it's terrible. Uh it's it's the prices have gone up. Apple has raised prices and all that. Um yeah, this was uh basically the entire discussion in uh like the bar yesterday. I met a bunch of friends, um, and then we were having a few beers, and uh the this was the whole discussion is like memory prices. Like, what does the world come to? Like everybody's talking like LLMs, uh, and I mind you, I'm not in the Bay Area, okay? This is of course Bangalore is a tech city, so it's not unheard of, I suppose, but it was very fascinating. Everybody's like, uh, oh man, I can't buy a laptop. This is Apple has increased prices. Another friend of mine was like, uh, I can't even buy musical instruments anymore because you can't buy these synths and stuff because all of them have um you know memory in it. So even musical instruments have gone up, you know. This is serious memory crisis, it's hit the streets.

SPEAKER_00 2:30

Totally, totally. Man, yeah, if it's a conversation at the bar, then it's truly impacting the consumer.

SPEAKER_01 2:37

Exactly. Uh when people start talking about the fact that memory prices from AI are affecting their day-to-day purchases, uh, we've uh we are we are in trouble now. Because I don't think the general population understood how much memory AI has been sucking up. We've been talking a lot about it on the podcast on and off, right? But uh that's the when it hits the streets like this, kind of gives you a visible uh reaction to like you know, people asking me to like, hey, you do the podcast, you know, what is what what's happening with memory? I'm like, yeah, yeah, let me tell you about HBM, let me tell you about DRAM, you know, and uh they're all like, oh wow, like is it that much, huh? And I'm like, wow, back to you. Like to me, like, you know, I've been thinking, oh, so much DRAM. We've always been talking, it's like second nature, like memory is short, yeah, yeah, we know this. Memory companies are making money, sure, we get it. You know, we it's in the news all the time for what we do. But for everybody else who is not uh on this train entirely, like we are, uh, it's news actually. It's like uh AI is killing my consumer devices.

SPEAKER_00 3:49

Yes, yes, yes. So uh, you know, the most simple definition of inflation is just like sustained increased prices for a period of time, a year, two, something. And traditionally, when the consumer thinks about like inflation, core inflation, it's usually like the price of food, but it's also like the price of oil, essentially, like gas and whatnot. Um, and so people, normal consumers have to tend to pay attention to various random industries based on the things that they buy that cause inflation for them. Um, so for example, in the United States, people, if you like to eat beef, you might know that like there's a shortage in the US beef herd or cattle herd. And it's a very random thing, but you're like, yeah, because when I go to buy steaks, now it costs way more than it used to. And what's interesting is because of AI, now the normal consumers are gonna have to start paying attention, or at least right now they are, to the memory market. Like normal people are gonna have to learn like, what is memory? Why is it so expensive? Because to your point, it's just like, yo, I'm just a person that produces music with a synth and suddenly it's expensive. And you just start pulling on that thread to find out why. And what's super interesting is as much as I love AI, which I totally do, it's interesting to say it as plainly as like AI is causing inflation. Like AI is gonna make it so that you know your grandparents who are in the United States that are on a fixed income and social using social security, for example, their social security dollars don't go as far because the price of certain things have gone up because AI. So it's kind of crazy to think like how it's truly impacting just everyday people around the world, too. It's not uh a lot of the AI is produced in the United States, but it's impacting people around the whole world, which is also kind of crazy to think.

SPEAKER_01 5:35

I think the whole sentiment in the market now, because of these increase in memory prices, is that when it hits the consumer and when the consumer starts to feel the pinch, the sentiment seems to be that uh people are gonna stop buying devices. And as much as we talk about AI and the deployment in data centers and how many GPUs people are buying and how much these data centers are spending, the consumer market is actually very significant. It is a it has always been a very big driver of semiconductor business. So it's not to be discounted or trifled with, right? So when the prices become this high, and when everybody sees their electronics prices jumping up, the first tendency is like what I said in the beginning, right? Like, look, I better not drop my laptop because I'm not going to buy anything in these prices. And that's the thing that everybody is going to think about as well, at least in those lines. Most people, I would think. So that what that means is that if people stop buying consumer devices, now you're going to have uh uh demand drop, and then the supply will like equalize better. Um and you know, once the supply once people stop buying consumer devices, some of the the market's gonna pull back or something. So you know people anticipate this because they've seen this happen in the past. And so the market kind of softened last week, I think, around this news. That's my interpretation, anyway.

SPEAKER_00 7:02

Yeah, so definitely consumers are going to delay purchases, or potentially you might, depending on which kind of consumer you are, you might say, like, well, I have to buy a new laptop, like uh, like my it's at work, it's time to refresh. I have to. They're probably gonna say, let's just delay, delay a quarter or whatever. Or they might say, Well, we'll get you that new MacBook still because it's been three years. We promise you that, but we're not gonna go all out on memory, of course. Um, but yes, it's definitely gonna impact consumers. And then the the interesting part is it's not only gonna impact the low-end consumers, but even high-end consumers. I think that's where it's really interesting, is when you have at peep, you know, Apple always has like the best customers who are willing to pay the most for the premium experience of having an iPhone. But it it's pretty crazy when even like the premium customers who are willing to spend three, four, five thousand dollars on a laptop, even they are saying, like, well, I yeah, I guess I better wait or I'd delay it. Let's see. The the question I have is so if we have a drop in demand from all the consumer devices, of which there are many, and it and it does consume a lot of memory, is that actually going to help even out the price, memory price? I don't want to say gouging, but like, you know, uh increases, because the question is, is that gonna improve supply? Um, or is that supply, that extra supply, if you will, that's not coming from consumers, is just gonna be sucked up by the AI and the hyperscalers, you know, and all the accelerators who can't get enough, right?

SPEAKER_01 8:40

Yeah, the answer to that comes from looking at the supply commitments already that are awaiting these memory companies. It looks like people are like booked out through 2027, at least. Like that's what Micron said in their quarter, at least, that uh this is not going to let up in 2027 because they have these uh long-term agreements, uh, and then they also have it as like a somehow like a prepayment clause, which is basically that you pay for it whether you take it or not, kind of thing, right? So they're going to continue to make money, and AI book orders are in, they're all booked up, so they have been pre-paid for. Um so even if this softens, I don't I don't see why uh it will let up in the near term. In the long term, I think supply capacity is being built up, and when that comes online, uh maybe then uh we will see some uh uh something letting up here. But up until that point in time, I don't see anything changing in the near term. But let's let's see how right I am in a year from now. We should revisit this episode.

SPEAKER_00 9:48

Right. But but so you're saying if it's not gonna let up in the near term because all the uh AI accelerator companies are just they're already pre-paying for it and they want to suck up any extra supply that's possibly out there, whether the supplies from a new fab or the supplies from the lack of consumers buying it, um, that's gonna be really hard on consumer device companies because it means that they're gonna continue to have this pain. Um, and so we, you know, I know you were you and I were talking earlier about um there are some companies that are having pain and trying to work around it. So, like GoPro, for example, tell us about GoPro.

SPEAKER_01 10:24

GoPro was such an awesome company. It was such a lifestyle statement to have that GoPro stuff. When it came out, you know, you could people were attaching it on their guitar headstocks, and then you could see them playing a live show with the guitar moving around. It was so amazing. Obviously, there's all these adventure sports that came out. I even considered it for some time uh mounting it on my surfboard when I lived in San Diego so that I could like see you know how it looks as I surf. You know, you see these surf videos of people going down the line and all that. So it was such an amazing concept, but today what has happened is they don't have money for memory. They know, I mean, how much will you pay for a GoPro anyway? It's already something that you don't really need. It's it's a fun thing to have, like to do all these crazy things. But nobody really needs a GoPro, right? And now the company uh is become a penny stock, basically. There's a good chance that they'll actually go under, which is a sad thing because it's a it's a device that had a lifestyle statement at some point, and now the memory prices is driving companies like this out of business, you know. It's it's sad actually.

SPEAKER_00 11:33

Yeah, it is sad. But yeah, you make a good point. Like, especially any discretionary spending um where people don't definitely need it, those companies are gonna definitely be hurting. Um, what about Apple? How are they trying to deal with not passing on such crazy price hikes to customers?

SPEAKER_01 11:52

So Apple is actually the biggest giant of our times, right? Before Nvidia, they were the first trillion dollar company, if I remember right. This is a company that has been the uh the gorilla in the room, so to speak, for decades because they had pricing power. They could always come in and say, hey, we're gonna sell a billion units. You know, we need this price or we need this performance, you know, we need this capacity. And there were companies before whose sole existence relied on Apple orders. I worked for some of them actually. Uh so their whole existence is whether they make it into the next Apple design cycle or not. And Apple was, in a sense, brutal with their own business practices. Um, it's kind of well known in the industry that they're really, really tough to deal with. And nobody would, if they can avoid it, if they didn't have to make money, they'd rather avoid Apple. But Apple are a tough customer to work with. And this is the first time that Apple has increased product prices mid-cycle. This is not, you know, uh a next refresh, like next generation that comes in or whatever. They refresh prices in the middle of nowhere, like saying now the same product is now more expensive. They've never really done that, uh, as far as I know.

SPEAKER_00 13:11

Yeah, yeah, it's pretty wild. Um now tell me, is shouldn't I feel like there's a uh a paradox here, which Apple is the you know, big big company, they make the premium products. They could probably afford to pass on the pricing to their customers the most because they've got the most premium customers, right? So if if you let's say you were making uh an Android phone and I was making an iPhone and like memory is gonna get more expensive, like it should be easier for me to pass it on, like, hey, it's things are gonna be this phone's gonna be an extra hundred bucks, then for you to pass it on as an Android phone maker because you're like 100 bucks. Like I can't, I can't charge my customers a hundred bucks. But then paradoxically, isn't it Apple who's probably getting the best memory prices because they have such scale? So like is me as Apple, wouldn't my cost for memory actually be less than yours, Android maker, because because I have such scale? Or what do you think there? I mean, I know there's I can't I capture Android or I Apple captures a lot of the margin, but I do know that there obviously there's uh you know Samsung and other um Android phone makers that still sell tons of units.

SPEAKER_01 14:23

Yeah, so I think the distinction that we have to make here is premium versus non-premium device. The whole argument that I've been reading in recent times is that premium tier devices are actually okay because they have a high enough selling price that they can absorb the price increase uh in memory and they're they'll be fine because of that. Um although their gross margin will reduce. But in general, yes, uh it's it's okay. Now, with the Apple price increases, I don't know if Apple wants to hold on to their gross margins and therefore they increase the price of the device in lockstep with what memory prices are increasing uh so that they can retain gross margins and look good in the next uh earnings cycle. So it's a gamble, right? Like because people may not buy as many if you increase it. So now you lose on revenue anyway. So that's that's I'm not sure which way they're going. But there was news, I don't know, we've mentioned this on this podcast too, because we've spoken about memory so much. Apple was supposedly buying up DRAM like crazy. They were buying it up at premium prices. If at all, they could only lock out their competitors from DRAM, they were gonna do it. And if there is a company that could do it, it's Apple with their big you know cash reserves, right?

SPEAKER_00 15:44

Right, right, right. Yeah. So interesting to think about like, yes, there's price elasticity here. So the the more they increase the price, the fewer people that want to buy it. Um, so of course, if they could absorb it all, then it wouldn't affect the demand. But then of course it's gonna affect their margins. And so, you know, always as public companies, you have to play that game of how much of a margin hit do we want to take versus how much of a uh like fewer units do we want to sell? But then on top of it, it's kind of like, well, are investors gonna compare them to the previous quarter and say, hey, you sold less units than expected? Or let's say the previous year, you know, year over year comparisons are probably best in a seasonal um thing like smartphone sales, or are they just gonna compare them to everyone else and say, yeah, your margins went down, but they didn't go down as far as everyone else. So actually, we will still reward you for that.

SPEAKER_01 16:36

Yeah, uh the expectation these days has become very AI-centric, right? Like everybody wants a massive increase in uh earnings and uh massive boosts. Because the idea is like, why would I invest in Apple when I can invest in Micron? Because Micron is making money. How about I invest in Micron? Uh like you you guys are losing gross margin, so why should I invest in Apple? Like, so it doesn't usually investors are not really tied to Apple as a company or anything, they're just there for the money, right? So I mean you're here to make money. So there's a logical thing is to go to the company that makes money. Uh but so I don't know, like ultimately it seems like uh even the premium TF was not spared. And recently over the weekend, there was also this discussion that now Apple is asking CXMT, which is a Chinese DRAM manufacturer, for uh DRAM supply. And DRAM uh this CXMT is actually on the entity list, which is a list of you know no-no companies to do for US companies to do business with in China because those companies are also suppliers to the Chinese military. So you don't want to, as a US company, give money to a company that then supplies the Chinese military because that's going to bite the US back in the future, right? They'll have a stronger military or whatever. Anyway, so they put CXMT on the entity list, and now Apple is asking the US government, can we please get CXMT to give us DRAM because we need the supply? This is like the another for another first, actually, because I don't think there's any company who has gone to the US government and said, like, hey, you blacklisted this company, but now we actually need to buy from them. Right. Imagine Apple actually doing this. Actually, that's a big deal. Um, yeah.

SPEAKER_00 18:30

Yes, the things not on my 2026 bingo card. Yes, Apple asking if CXMT could come or any Chinese company could come off the entity list. Um it just goes to show the lengths that they're willing to go to to prevent the inflation that we talked about, um, and obviously try to get more supply. Of course, the interesting thing is it's a temporary problem in that, like you said, we know more supply is coming online. Um, Micron, Samsung, XK Heinex, these guys are building fabs. They're gonna come on in 2028, 2029, and so on. If you ask for the uh CXMT to come off the entity list, that's bringing supply on indefinitely, uh, not just to solve this temporary problem, but indefinitely. Um, which of course, when you start to game theory this out, you wonder if the big three have on had it on their game theory bingo card uh that CXMT supply could get brought online. But it it I'm sure it does feel like a slippery slope in that it's like, yes, you would like more supply right now, but do you want C CXMT DRAM in you know a year from now or two years from now? I mean, maybe at that point they just stop buying from them. They just say, hey, thank you for your service and for helping us in 2026. Even though you're not on the entity list, we don't want to buy from you anymore. But it just yeah, it's just so interesting to think through all the implications.

SPEAKER_01 19:54

Yeah, so now we are the the long and short of it is that the premium companies, premium tier companies like the Samsung, Samsung is a bit different, right? Because they have they can they have DRAM supply by themselves uh and they make phones. Think about that. Like Apple doesn't have Apple doesn't make DRAM, Samsung does, and they also make phones, which is a unique advantage if you come to think of it. But you know, apart from these premium tier companies, uh all what about all the lower, mid and low tier? They cannot absorb the costs, you know. So the one thing that will happen in this scenario is that those phones will stay at the same price, but it will get de-spected. So a phone that was selling for four GB of RAM or whatever, some low tier, mid-tier phone, will only go with two GB of RAM now. And so they get de-spected, and that is one way to handle the lower tier. But nobody likes that. So now you're like, okay, fine, I've spent the same amount of money, but I get a crappier phone, right? Right.

SPEAKER_00 20:55

Totally, totally. Um yeah, that that's uh what do they call that when uh in like consumer foods where you're you you pay the same, but your bag of chips is like half as big. Yeah, isn't it a form of inflation? Yeah, yeah, yeah. There's like a funny term for it, but this is the same thing. Um shrink shrinkflation, that's what it's called. Shrinkflation. Shrinkflation hits low-end consumer phones. So you mentioned something interesting. You were it was kind of an aside, but you said, oh, Samsung, they make phones, they make memory, they also have a foundry and can make logic chips, right? Yeah. So in theory, yes, exactly. So in theory, Samsung could presumably make quite a lot of uh a phone, like you know, vertically integrated to some extent. And it would be interesting to think that through. Probably in if you're like kind of doing like the Taiwan China war gamings of like what happens if TSMC goes down, you could ask like which smartphone makers are best poised to continue to be able to supply new smartphones and maybe. Samsung.

SPEAKER_01 22:01

I think Samsung plays this game too, because for profitability, they also have to make an internal decision as to where to direct their DRAM supply. Would you sell it for a nice hefty profit to AI companies, or would you sell it into the consumer market and lose money? Like they will even refuse their own handset division if it comes to it, because it's all about getting the most out of the DRAM supply you have. So it's very funny actually. Talking about Samsung and SKINX, maybe this is a good time to mention this. The government itself, right, Korean government, is investing north of half a trillion dollars in DRAM capacity expansion in just these two companies. Because they realized that these two companies are now hold the key to the AI semiconductor super cycle. Because yes, there's TSMC and there's Taiwan who has the key to manufacturing. Yes, there's Nvidia in the US who have the dominance and training chips, and also all the other hyperscalers, they're all US-based. All this is good. But what do all of them need in common? What is the one breaking point? Is memory. And who has all the memory? I mean, two-thirds it's Korea, right? Like Micron is US, but two-thirds is a Korean-based company, like SK Heinex and Samsung. So the Korean government is like all in. They basically pushed all their chips in and said, here you go, we are we are going to own this thing, and they're spending half a trillion dollars on it. And then interestingly, in other Korean uh memory company news, it's like SK Heinix actually surpassed Samsung for the first time to become the most valuable company in Korea, ending a reign that Samsung has held for 25 years. So that's the power of memory. So the other thing I wanted to tell you was uh about Keoksia, because Keoksia is not really a DRAM memory company, it's more of a NAND flash maker, and they spun out of Toshiba. And they recently passed Toyota as the most valuable company in the country. So you can see what AI is doing. Like Toyota, it's a big deal, like it's in every country, but now a NAND flash company that Toshiba actually spun off. What a bad idea! I mean, talk about a bad bet. So funny.

SPEAKER_00 24:26

Yeah, pretty wild, pretty wild. Things are changing, that's for sure.

SPEAKER_01 24:30

Yeah. Finally, we have to talk about uh, you know, Micron. Uh we mentioned it in the cold open, obviously. But uh Micron is let's just say, for all the people who are suffering in the AI memory crunch, you know, the GoPros going out of business. Uh these I also read about a small uh company that was making some routers or something. They were selling their routers for like, I don't know, $500, let's say. And uh now they have to sell it for $2,000 to even stay profitable. And they're like, the the maker's like, no, nobody's gonna buy this at $2,000. This is not a product like that. You know, uh sometimes electronics has a certain intrinsic value. Like nobody buys a laptop for $20,000. You buy a laptop for like, I don't know, $2,000, right? There's like a logical limit to certain class of devices that you can't just break. So this is like one of them. So for all the suffering that is going on in the memory industry, uh, Micron uh is uh among other memory makers, uh, is making a killing, right? They are at uh 80, 84% or something, 85% profit margin, which means that you basically are selling it um for four times more than you made it for. So if you make some make this uh widget for like 10 bucks, you can sell it for like 50, I think. Yeah. So anyway, that's a lot of money to be made.

unknown 25:56

Yeah.

SPEAKER_00 25:57

Yes, totally. Yes. Microns, gross margins. I think in the last, which I wrote about this on Chipstrat, you can check it out. They're in the last um four quarters, it went from 45 to 56 to 75 to now almost 85%. And they're actually projecting like another 85, 86 percent next quarter.

SPEAKER_01 26:17

That's insane. Earlier, you mentioned maybe this isn't price gouging, but you know, I I'm gonna say it is. It is absolutely price gouging, and I think that a lot of people now have this inherent hate for memory companies because everybody's thinking because of these three. And so basically, it's just like I think when the down cycle comes, nobody's gonna like it, but everybody's maybe secretly, you know, saying it had to happen. I'm glad it did, you know. Right, right. These people back and where they belong, you know. Yes, yes.

SPEAKER_00 26:54

I mean, too, you know, to Micron's credit, like when I think price gouging, I think of someone trying to take advantage of people. And I think to their credit and other memory makers, it's just that literally their customers are like, no, I'll pay more than that dude because I need it. And that dude's like, no, I'll pay more than them because I need it, you know. And it's just, I mean, there's probably literally almost to an effect like a bidding war going on, you know. So it is obviously very opportunistic. But if you have a customer saying, like, hey, you you'll sell that to Vic for $3, I'll give you $5. You know, you can't, it's how do you say no to that?

SPEAKER_01 27:26

But maybe that's one way of looking at it that is like, yeah, maybe they're offering more to get the supply locked in, and Micron's just like, sure, why not? You you pay more, you get it. And the next person comes by, like, sure, you can. Or maybe they're just going, like, no, that's that's not our price. This is our price. You want to take it or you want to leave it? Like, you can leave it because you don't take it, somebody else will. You know, so there's this whole FOMO thing. Uh, we'll put up this on another chart I have from like the Wall Street Journal that shows basically uh microns adjusted uh operating earnings every fiscal quarter from like 2023. 2023 to 2024. If people are not aware of the history of this company, they had negative earnings. Okay, they were losing money, it was pretty miserable. Um but then you can see what happened to this chart as you go from basically uh the beginning of 2024, they start making a turnaround, and now they're making, I don't know, 41.5 billion in a single quarter. It's it's insane. Like if you look at the period between 2024 and uh 2025, 26, the last two years, they haven't made 41 billion dollars in a quarter in put together in all those years. Now they do it in a single quarter, you know? Yeah, that's it's fine, it's interesting, I think.

SPEAKER_00 28:49

It's wild, yes. It is it is so wild. I mean, we see this across AI, uh, these companies where they're spending more or buy or making more in a quarter than they did in an entire year. So for the CapEx guys, you know, for the hyperscalers, it's like all of a sudden now they're spending in a quarter in capex what they used to spend in an entire year. And of course, on the other end, the people who the hyperscalers are buying from, they're making more in a quarter than they used to make in an entire year. And it's just pretty wild to have all these companies suddenly making more per quarter than they did an entire year. And for some of them, in an entire, you know, several years added up. And for Micron, I mean, you know, 2022 wasn't that long ago. 2023 wasn't that long ago. So pretty crazy how their fortunes have turned so quickly.

SPEAKER_01 29:34

Yeah, I like the way you uh put it in your Substack. Basically, you didn't show the most recent earnings data in your Substack. It's like you see, um, memory is a cyclical industry. You can see it goes up, it's like a sine wave. It's almost like a sine wave if you draw a line joining all the peaks of these bar charts you have in your Substack article. Um, by the way, whoever's listening should go read the whole article. But then later you're like, ah, but I sneakily did not put the most recent one. Here it is with that recent one in, and then there's like this huge bar like goes up. Like that's totally yes. You tell me, is this cycle, is this a cycle? Does it look like a cycle to you? That's a peak. That's a right, right.

SPEAKER_00 30:15

Well, well, and what's interesting, and and I I linked to this nice chart from Doug O'Laughlin where he he tries to show where different industries or like submarkets are in terms of a cycle. And in the and I I I think of it as like a merry-go-round. Like there's these four quadrants of in is your inventory increasing or decreasing, and are your sales increasing or decreasing? And if you're in a quadrant where your inventory is decreasing and your sales are increasing, that's obviously very good. Prices are gonna go up. But eventually it like tips over and you start to build back up inventory. So even if prices are high and you're still selling a lot, you're building up inventory. And then eventually there's gonna be a point when you have so much inventory that you actually tip over and you don't sell as much. And then you know, you start to get in this place where it's like, oh, now we've got more inventory than we need and prices are going down. And it's kind of like a merry-go-round, you just go all the way around. And so a lot of people are are, you know, trying to make the argument that, oh, we we have all these new demand drivers, which is totally true, but they're kind of arguing that that like merry-go-round doesn't exist. And I think that the merry-go-round still exists, but now I think there's other drivers that are going to keep you know AI in a particular quadrant for a lot longer because, and we can talk about this, but like actually, let's definitely get into this really quick. When you know, when you buy HBM, you know, you're buying DRAM wafers essentially for stacked HBM. You you can unpack this. Um, and then also there's people needing to buy DRAM. So, for example, I need lots of CPUs for Gentec AI, and those need DRAM. Um, and so like there's these different drivers that want to take the same wafers. And it's kind of like no matter where you sliced it, it feels like the demand is going to continue to be high. And we know that supply is not gonna increase until 2027, 2028, 2029. I've got it in the Substack as well. And so it it there's I I'm a believer that the the the merry-go-round is still there, but there's just so many drivers that things aren't gonna tip over, the supply is not gonna catch up, the inventory is not gonna decrease for a lot longer.

SPEAKER_01 32:18

Yeah, so basically, to uh to the point of unpacking what you said, it's that we are really this short of memory because of AI. Okay, that's one broad way of putting it, but essentially because we need high bandwidth memory, which requires stacking these DRAM chips on one on top of each other, um, to make those DRAM chips that you have to actually stack actually takes three times as many wafers to make the equivalent number of bits. You know, so I don't know, like to simplify that statement, let's say you have to make a one GB DRAM chip. Now, if I were just making a one single chip that I didn't have to stack up into HPM, I would maybe do make it in one wafer, you know, and you can get one wafer's worth of DRAM chips or whatever, uh one GB chips. Uh and let's say you, I don't know, maybe you get uh a thousand chips out of a wafer, each of which is like one GB. Okay. Now you want to make the same one GB chip, but that one GB DRAM uh is going to go into HBM. Uh you're going to need three times as many wafers to make that same bit capacity if it were going to be used for HBM. Okay, there's a there's a technical reason for this. Uh we won't get into too much of it, but it's basically that you have to drill holes through this thing and have through silicon viers, which is how you stack them. You actually stack them by connecting them through each DRAM die. Whenever you drill through it, you your bit density drops. You can't have as many bits now because you can't put memory cells around these drilled holes for safety reasons. It won't work. So, because of that, basically, for every one GB of HBM and for every one GB of DRAM, the HBM takes three times as many wafers than uh the regular DRAM does. So it's a big suck up, right? It's sucking up all the DRAM supply big time. Uh that's one big, big reason. Uh the other reason is that we right now even uh need DRAM chips just as is. Forget about HBM. We need DRAM because apparently agentic AI needs CPUs, and now CPUs need a lot of DRAM. Great. So now you have to you know get a whole lot of DRAM chips. Then it turns out that we want to run long agentic workloads, which means the AI has to remember stuff literally forever and then you know, uh keep the context in its brain of what it is doing in a large code base or a very long context conversation. You want to keep asking it questions. You don't want it to have amnesia like ever, and you want instant answers, right? You don't want to wait or anything. So now all of that context that it holds in its brain is also stored on DRAM. It may not be stored in HBM. It could be stored in HBM, but it's very expensive. So mostly you can store it in DRAM, which is the next fastest memory. So all of that memory thing that AI has to remember is also stored in DRAM. And now people are looking at pooling DRAM. So what you do is you take all these DRAM sticks and put them together in an appliance and uh say, like, okay, look, this is my my KV cache server, you know. So it could be CXL pooled DRAM or it could be CXL pooled NAND flash. So everything is going into AI. It doesn't matter whether you're looking at uh like agentic uh CPUs, HBM, or just like KV cache storage, yeah, all over. It's all DRAM, it's all memory. So this is this is why we are in this pickle, so in a long story short.

SPEAKER_00 35:56

Totally. And that's just talking about essentially like the data center demand for DRAM, or like the enterprise demand, if you will. There's obviously still consumer demand for DRAM. The interesting thing is that the data center um buyers basically are inelastic in their demand. So like if the price goes up, they still want as much. And how much do they want? Every wafer you can make, right? Everything. And it's almost in like the curve, you know, the elasticity curve almost goes straight up. It's like the price goes up, I don't care, I'm all the same, you know. Um but but on the other side, the consumers, it's very elastic. And so there's this interesting sort of bifurcation into data center demand and consumer demand, and data center demand kind of bifurcates into HBM and DRAM, and DRAM kind of bifurcates into CPUs need it, and you know, maybe some storage appliances to supply like your KV cash needs it, right? And so you just have like all these demand drivers. Um supplies coming online. Meanwhile, of course, the the consumers are hung out to dry and it's impacting how far grandma's dollar can go, you know, and she can't get you the new GoPro for Christmas because they're going out of business, right? And so it's like consumers are getting beat up. Um, but it it's very difficult to see demand ever waning. So I don't think people should think that data center demand wanes. I think consumer demand will wane, right? Because it's like phones, like we talked about my laptop's expensive, I'm gonna put it off. But the data center, they're just gonna keep buying. Um, and so that's why you know we kind of think it's this AI super cycle in memory, if you will, because like like you said, there's a little sine wave, and then all of a sudden, boom, it's like it is accelerating, it's going a lot higher. Uh so it's good, it's gonna be there for a while.

SPEAKER_01 37:42

Yeah, yeah. I have uh I think we should like wrap this up. I have one question, I think, that will uh maybe we can think on it because this is like a big of a bit of a macro question, and uh I I don't know if anybody really knows the answer, but I'd just like to know what you think of it. So, what do you see happening from now to 2030? Okay, but that's like a broad question. So let me pin that down a little bit more. So, given that now uh leading frontier labs uh are not really being allowed to release leading edge models like fable slash mythos was cut off by the US government, then uh OpenAI Chat GPT 5.6, uh what they code name Seoul, uh was also not released for general public use. There is a lot of fear of distillation of these models from you know Chinese competition, since that is one front of it. So maybe the frontier won't advance as much due to other reasons. It's not actually scaling or uh technical reasons, it's more like political reasons or uh competitive reasons. Because now do you think that the next generation of leading edge models are going to be trained with the same gusto, knowing that they will not be released, right? So that is my first question. Um, what do you think will happen between now? Because the training seems to be having an inflection point of sorts. So, what are your thoughts on that? And then I'll ask you something else because I have more.

SPEAKER_00 39:15

So, if you assume that the government will always just prevent the leading edge thing from going out, then it does raise the interesting question, which is like, well, how much money do we want to spend on advancing the leading edge if we can never let it go out? If if that was all true, of course, there is an argument that the leading edge labs would continue to train even better models and just use them internally and just get continuously better and more efficient at doing what they do to improve their business. Um, however, I'm not so sure that the government is going to prevent this forever. I think interestingly, this could be like um shooting oneself in the foot. And that lately with the agentic AI stuff, it felt like the American labs had really run ahead of the Chinese labs. Now that it's not just training the model, but it's also like training the harness. And so it's like this bigger system that you're training and delivering. And Fable is so awesome, you know, that the government's like, whoa, wait a minute, let's not do that. Um and but but interestingly, that actually gives pause and gives a chance for Chinese labs to start like figuring out how to train harnesses and sort of catch up again, if you will. So I won't be surprised if there's like the pendulum swinging back and forth where it's like, oh, never mind, Fable's back on. We figured out how to feel comfortable about its safety because we feel like the rest of the world is catching up and we need to run ahead. Which by the way, this is gonna impact anthropic and open AI's um IPOs. It's gonna impact their, there's gonna be revenue left on the table because they weren't allowed to share the best technology with the consumers who would pay the most for it, right? So I think there'll be all sorts of incentives for the government and for these companies to get these models back out there. But I I definitely see your point, which is it for as long as the government prevents them from doing it, it actually puts it kind of disincentivizes maybe training the next model at the even on those scaling laws of like, yeah, now we need a million accelerators, now we need five million, right? And so then, you know, to your point, maybe that could actually flatten the growth of the demand curve a little bit for memory or for accelerators or whatnot.

SPEAKER_01 41:19

Yeah, yeah. That's the that's that's the kind of point I wanted to make. That if there is this giant rush towards AGI is being uh limited or governed by other competitive or political uh interferences, then it's questionable like how much more we will need. But that's only the training side, right? The demand for inference is still enormous. Like we have just big started with agents. We have not even scratched the surface of what is possible. I truly believe that, right? And uh it seems like it's very, very helpful. Like the people I've spoken to have said, I can't believe this thing. Like all companies now need to have an agentic AI approach to doing their job. Or their product, if it's a software company, needs to have AI in it somewhere. Otherwise, as a startup, even a software startup, let's say, they don't have any valuation. You can't go anywhere if there isn't a component of AI. So it's a strong driver everywhere. And it's it's it is uh helping productivity gains. Uh, if you see that OpenAI jalapeno uh chip release with Broadcom, uh you know, they say in the press release that it is accelerated by using Chat GPT. You know, so I'm I'm sure it wrote some code to verify the chip or whatever. So and there are like a whole lot of EDA startups doing AI-enabled this and that. And all of these are like productivity gains. So we have not even scratched it, the surface of what's possible. And I truly believe that we are going to only need more tokens from here. It's not like we're going to need less tokens. This leads me to my second question. I think now companies are becoming a little sensitive to what the token costs are. It's not so much as to the number of tokens, it's about the cost of tokens, which is where things like the GLM 5.2 model comes in, because it is a very capable model, but you can serve it at a fraction of the cost of what like OpenAI can, right? Or you can talk about the DC Deep Seek V4 Pro, and maybe that's enough intelligence. You don't have to have always like the cutting edge intelligence for everything. You know, so so basically my question is like, do you think that people will stop using uh these frontier models that are so expensive in an attempt to cost optimize inference going forward?

SPEAKER_00 43:45

My hot take is both are going to grow. I think companies are gonna continue to just be thoughtful and say, guys, we don't need Opus to do these little uh I'm summarizing. Uh these 50 articles and giving you a daily thing on it. Like you don't need Opus to do that. Just use Sonnet or whatever. So that kind of stuff is gonna grow. And which, by the way, someone else is gonna say, don't use Sonnet, let's buy a token generator and let's just use an open source thing and just generate these tokens for free, if you will, for the price of the capex of buying the on-premises token generator. And let's just use open source. So I think that's definitely gonna grow big time because lots of CFOs are gonna go, wait a minute, we just went crazy. We have to. But there's people are gonna always want the frontier, and the frontier always is gonna unlock more. And every time I touch the new frontier, I can do way more like productive software building and automating and stuff. Um it was amazing with Opus. It was amazing with Fable when it lasted. We had a great like 24-hour session where so much happened, and I want Fable back. And so I definitely believe that Frontier will always unlock new things. It's gonna be Frontier video models, it's gonna just be like continued frontier things. So I think both will grow. And of course, I do think that companies are gonna start to like fine-tune their, and I think they're doing this already, fine-tune their own models for their own proprietary use cases because we can't expect that Fable's gonna be amazing, but it's not gonna like have the best like biology data for your company as you're like doing RNA sequencing or whatever stuff. I don't know anything about that, but right, you can tell that that's not just gonna be baked into Fable. You're gonna have data, you're gonna fine-tune that fine-tune stuff may continue to be done on open source models, American or not, which we don't seem to have a ton of American ones right now. Um, but I so I see both growing. That's my reaction. What about you?

SPEAKER_01 45:35

Yeah, I think there's one more thing that'll big uh become uh a standard going forward, is basically model routing. Because uh you essentially have to put the right requests to the right model. That is going to be a very big cost optimizer for companies uh to make sure that the wrong request isn't burning frontier model tokens, you know. So that's uh that's a piece of uh engineering as somehow that's going to become very important within a company.

SPEAKER_00 46:04

Yes, yes. I even want to do this on my own. I've got lots of agents that are doing different tasks, and I want to get better at saying, like, oh, these tasks should only use these models, which right now how I solve it is I just use open router and I'm like, yeah, use Gemini flashlight or whatever for these tasks. But it's still hard to like, there's not like a good user interface for visualizing all this stuff, but I'd like a better way to know here's all the different agents, here's what they're doing, here's the models they're using, and make sure that it is sort of cost optimized, which uh, you know, I'm sure that there are startups probably out there that are solving this problem already, and I just haven't learned about it. So if you're listening and you know how to help me here, feel free to send me an email or comment on it.

SPEAKER_01 46:45

Let us know. We we really only do hardware stuff. We don't know much about this stuff otherwise. We we play around with models, right? We don't know what do we know about uh yeah. Uh but then the other thing is like somebody mentioned that like, no, no, no, I don't think companies can deploy on-premise um you know models like at scale or whatever, which to which I actually replied that no, actually, currently if you see how big companies do simulation low workloads, because they do run a lot of workloads uh on-premises, and they don't want all your simulation data to leave the company. So basically, these uh load share facilities or LSF farms uh are basically data centers run by companies and they are only within the company network. You can't reach them otherwise, right? So that's how I ran simulations, for example. All the big simulations were always dispatched to a server, uh server somewhere in the world, and I got my results back. Um that's how it always works because all the Linux terminals I was given in in many companies in the past have always been a virtual virtual machine. It's just like a 2GB RAM virtual machine. That's not nearly enough to do the engineering work I had to do. So it was always sent out on LSF. The same way, like companies can develop AI farms, you know, they can put in hardware locally and they can deploy local models, um, even even pretty big models uh that does exactly what they have to do. So they don't actually have to do it. Companies, I just read this piece of news too. Like companies are now like Meta are now worrying that using Cloud on the cloud or whatever is somehow leaking data to the model and then is it will be used to distill future models. So they're like issuing warnings to their people, like saying, like, be careful, like don't give away all the information or whatever. So, yeah, there are all kinds of concerns coming in as AI becomes more mature in its use in the industry. So it's interesting to see how all this will have an impact. So it's not only about look, look, we need more AI, more AI, but it's it's about noticing that frontier uh models seem to face some kind of headwind at the moment. Like you say, it may not be forever. And companies are getting smarter about what they actually use because they need to show revenue. You can use all the tokens you want, but at the end of the day, your CFO is going to come knocking like, okay, you blew, I don't know, $100 million worth of tokens this year. What revenue, but my company revenue hasn't proportionally skyrocketed. I want to see multiples of that hundred million spent on tokens because we we even laid off people uh for these tokens, remember? Like, you know, so I need to see the revenue. So this is this is going to become a big uh big driving factor, I think.

SPEAKER_00 49:35

Yeah, totally. It's always going to be at what cost. So obviously at what monetary cost, but also at what security cost, at what latency cost, and so on. So all right, folks. We hope you like our AI's eating memory deep dive. Uh thanks for listening. Thanks for uh our YouTube commenters. We love you. There's like a core group of you. Thank you for that. Thank you for our podcast listeners. Um, thank you to the person out there that was like, yo, where's the podcast? It's late this week. Uh we don't really stick to a schedule per se, but we try to get it out weekly. But we had some travel. But I love that some of you love the podcast so much that you're telling us, guys, get it out there. I'm ready for it. So that's awesome. Um, keep listening and we'll keep bringing this to you. Um, feel free also to check out our substacks, vixnewsletter.com, chipstrat.com, and share this with a friend.

SPEAKER_01 50:24

Thank you. And check out semidope.com where you get all these uh, you know, daily news sticks but with our with our little words in it. Think of it as a little readable version of this podcast brought to you every day. So yes, yes.

SPEAKER_00 50:38

If you like our podcasts in our we can't bring our podcast to you fast enough, check out semidope.com to get Vic in Austin every single day.

SPEAKER_01 50:46

Yes, and also leave us a five-star review on Apple. I've been told it's very important.

SPEAKER_00 50:50

There you go. All right, that's that's a wrap.