📈 The Poetry of Computational Power: What Scaling AI Teaches Us About Human Collaboration

Please see the corresponding Substack episode

How the engineering principles behind massive AI systems reveal timeless truths about cooperation, specialization, and the delicate art of working together

There's something almost mystical about watching a thousand chips work in perfect harmony. Each one a specialized genius, none capable of greatness alone, but together creating something that feels like magic. This is the reality behind today's large language models – not the breathless hype about artificial consciousness, but something perhaps more profound: a masterclass in the art of collaboration itself.

How to Scale Your Model: A Systems View of LLMs on TPUs ( by DeepMind )

This is Heliox: Where Evidence Meets Empathy

Independent, moderated, timely, deep, gentle, clinical, global, and community conversations about things that matter.  Breathe Easy, we go deep and lightly surface the big ideas.

Thanks for listening today!

Four recurring narratives underlie every episode: boundary dissolution, adaptive complexity, embodied knowledge, and quantum-like uncertainty. These aren’t just philosophical musings but frameworks for understanding our modern world. 

We hope you continue exploring our other podcasts, responding to the content, and checking out our related articles on the Heliox Podcast on Substack. 

Transcript

0:25

Have you ever just, you know, stopped and marveled at the sheer computational power behind today's AI? Especially those enormous large language models. It really feels like magic sometimes, right? How do these colossal models get built, trained, and deployed at scales that are frankly, well, mind-boggling? It's incredible stuff. Welcome to the Deep Dive. Today, we're pulling back the curtain on some of that magic. We're going to try and understand the fundamental principles behind scaling these immense AI models. You've shared some incredibly rich source material with us, and our mission today is really to cut through the complexity, extracting the most crucial insights. We'll zero in on the core hardware, TPUs and GPUs and the clever software strategies that bring it all to life. Think of this as your essential guide to understanding the backbone of modern AI, hopefully giving you those aha moments without getting too bogged down in every single technical detail. It's a fantastic challenge, really pushing the boundaries of engineering. And when we talk about scaling in this context, what we're really aiming for is something called strong scaling. Strong scaling. Okay. Yeah. That basically means if you, say, double the number of chips you're using for training or inference, you ideally want to see a proportional linear increase in your processing speed. Right. More hardware equals more speed. Simple. Exactly. More hardware equals more speed. That's the ideal. That's the dream, isn't it? Just keep adding more machines and your problem gets solved faster. Yeah. But I'm guessing it's not quite that simple. What's the first big hurdle? You've nailed it. The simplicity ends pretty quickly. On a single chip, right, performance is this balancing act between how fast you can do calculations measured in FLOPs, floating point operations per second. Right. And how fast you can feed data to those calculations, which is your memory bandwidth. with. Okay. But once you move to a cluster of chips, like hundreds or thousands, the game changes entirely. Now it's all about hiding the inevitable communication between the chips. Hiding the communication. Yeah. You want to overlap that data transfer time with actual useful computation. So the chips aren't just sitting idle waiting for data to arrive from their neighbors. And I imagine that gets harder as you add more chips because, well, more chips mean more conversations, right? Dave. More data flying around. Exactly. More chips mean a higher overall communication load. But here's the kicker. With more chips, each individual chip has less computation to do for its slice of the problem. Oh. So this leaves less useful work to hide that communication overhead. You're in this constant battle to avoid becoming bottlenecked by how fast your chips can talk to each other. It's often where the real art of optimization lies. I've heard people describe optimizing these models as almost black magic. Is it really just trial and error? Or are there underlying principles we can grasp? It can certainly feel like black magic when you're deep in the weeds. Absolutely. But what our sources reveal is that relatively simple universal principles are at play, you know? Whether you're working with one accelerator or tens of thousands, understanding these fundamentals lets you do really powerful things like accurately predict how long a matrix multiplication should take, identify exactly where bandwidth is becoming a bottleneck, and design efficient ways to redistribute data. It's less magic, more applied computer science, really. Okay, that's reassuring. So let's unpack this with some real hardware. We have two sort of titans in this space. Let's start with Google's tensor processing units or TPUs. What's their core philosophy? Right. TPUs are designed with extreme specialization in mind. They're primarily built to excel at one thing, matrix multiplication. It's crushing matrix math. Workhorse operation. Totally. The core of a TPU is its tensor core, and within that, the matrix multiply unit or MXU. This unit just crunches through BFLOAT16 matrix multiplies at astonishing speeds. For instance, a TPUv5e can perform, I think it's tens of trillions of BFLOAT16 operations per second per MXU. Wow. It achieves this with a systolic array architecture that's incredibly efficient for this kind of work. And crucially, these operations are heavily pipeline. Pipeline meaning? Correct, meaning the chip is always trying to stay busy. It's copying the next chunk of data into place while the MXU is calculating the current one. This helps prevent memory access from ever becoming the bottleneck. Alongside the MXU, there's also a VPU, a vector processing unit that handles more general mathematical tasks like activations, rail u, things like that. that so mxu for the heavy matrix lifting vpu handles the rest what about memory because that's always critical absolutely critical and tpus employ a pretty clever memory hierarchy first there's vm or vector memory this is a small on-chip scratch pad memory but its bandwidth to the MXU is incredibly high, like orders of magnitude faster than the main memory. Like an L1 cache almost? Sort of, yeah. Think of it like a super fast, tiny workbench right next to the calculator, but what's really unique is that it's programmer-controlled. Ah, okay. Not automatic like a cache. Exactly. Engineers can precisely manage what data sits in VM. which is a huge advantage for certain algorithms where you know exactly what needs to be close by. Then further out, you have HBM, high bandwidth memory. That's the main larger memory storing the bulk of the model weights, activations, and so on. Yeah. That programmer-controlled VM sounds like a real differentiator. Now, how do these individual TPUs connect to form those massive clusters we hear about? Right. They're arranged in structures often called trays or pods, A TPU v4 tree, for example, might have, say, four chips with eight cores, all connected to a CPU host via PCIe. Okay. But for massive scaling, these chips are connected directly to their nearest neighbors using the ICI or interchip interconnect. Think of it like a high-speed local network just for the TPUs. Communication between chips that aren't direct neighbors requires multiple hops across these ICI links. Gotcha. So distance matters. Definitely. Modern TPUs like the V5P offer substantial bidirectional bandwidth per ICI link, which allows for pretty efficient data exchange across large slices or groups of TPUs. And the big takeaway for TPUs then was... What's the summary? I'd say their simpler, specialized architecture combined with a really highly optimizing compiler often allows TPUs to reach very close to their theoretical peak performance, their roofline, with maybe less manual tuning compared to other platforms. The compiler handles a lot of the tricky low-level scheduling, abstracting away some complexity. Fascinating. Okay, now let's pivot to the other titan in this space. GPUs. Yeah. NVIDIA GPUs. They're often seen as more modular. Right. That's a good way to put it. Yeah. GPUs take a different architectural philosophy. Instead of one or two massive compute units like a TPU's tensor core, GPUs have hundreds of smaller streaming multiprocessors. processors or SMs. Hundreds. Yeah, an H100 chip, for example, boasts over 100 SMs. Each SM has its own local memory, often called SMEM, shared memory, which is kind of like a super fast L1 cache, plus its own set of registers. Okay. This modularity makes them very flexible. They use a SIMT model single instruction multiple threads where each core within an SM can operate quite independently, unlike the more lockstep SIMD approach on the TP.

MARK MIRCHANDANI: 7:43

So that SIM-T flexibility on GPUs, what does that actually enable for programmers that maybe the TPU approach doesn't? And what's the trade-off?

KATHLEEN MURPHY: 7:51

That's a great question. The SIM-T model gives programmers much finer-grained control, really. They can launch many different kinds of parallel operations across these hundreds of SMs, making GPUs incredibly versatile for a wider range of workloads, maybe beyond just pure matrix math. But that flexibility comes with a trade-off. like you said, managing all that parallelism can be more complex for the programmer. And there's maybe a higher risk of performance issues like cache thrashing if the memory access patterns aren't perfectly tuned by the programmer's code. Makes sense. And for their memory story, how does it compare? Like TPUs, GPUs rely heavily on HBM as their main memory storing weights, activations, gradients. Recent GPUs, like the H100 or the upcoming B200, offer massive HBM capacity and bandwidth, really pushing the envelope there. They also typically have a larger L2 cache compared to SMEM, offering a significant bandwidth boost for frequently accessed data that doesn't fit in the L1 SMEM. And as the tensor cores within the SMEMs have gotten bigger, newer generations like the B200 are even introducing new memory spaces. like T-MEM just to keep those cores fed. Always chasing bandwidth. And for connecting these powerhouses together, GPUs have NVIDIA's Envilink, right? Yes, exactly. Envilink is the high bandwidth, low latency interconnect that ties GPUs together within a single server or node, typically connecting, say, eight GPUs. It provides impressive full duplex bandwidth per link, allowing very fast communication within that node. Okay, within the box. Right. And what's really pushing boundaries now is things like the NVL72 system, which drastically scales up the size of that node to 72 GPUs and offers just an enormous jump in GPU to GPU bandwidth within that larger unit. Wow, 72 GPUs acting almost like one giant one. Kind of, yeah, from a connectivity standpoint. Beyond these local NVLink domains, GPUs connect using InfiniBand, typically. Exactly. NVIDIA's massive DGX SuperPod architectures, which can involve thousands of GPUs, use sophisticated network topologies like fat trees to ensure high, predictable bandwidth between nodes. They have that sharp thing too, right? In-network reductions. Ah yes, sharp. NVIDIA's protocol that theoretically allows network switches themselves to perform part of the reduction operations, like some ingredients. In theory, this could have communication costs for things like all reduce. Sounds great. It does. Though in practice, the empirical results mentioned in the sources show a more modest improvement, maybe around 30% benefit rather than the full 50%. Still useful, but not quite the theoretical magic bullet. Okay, so if we put TPUs and GPUs side by side now, what are the key differences for the listener to keep in mind? Not just raw specs, but maybe the feel or philosophy? Yeah, I think it fundamentally boils down to that difference in philosophy we touched on. specialization versus modularity. TPUs with their one or two large dedicated 10-ster cores are really matrix multiplication specialists, highly optimized by their compiler. I got it. GPUs with their hundreds of smaller, more flexible streaming multiprocessors, offer broader programmability, maybe more versatility, but often require more manual tuning by the programmer to really extract that peak performance. And memory-wise? Memory-wise, the TPU's VM, though smaller, offers that incredibly high programmer-controlled bandwidth right next to the compute, which is killer for certain algorithms. GPUs generally counter with larger overall HBM capacities and are constantly increasing their HBM bandwidths with each generation. But maybe the critical takeaway for you, something that often surprises people, is this. TPUs, especially the inference-optimized versions like the V5e, can often offer better performance per dollar. Really? Even if specs look lower sometimes? Yeah, even if their raw specs sometimes appear worse on paper. This is because their specialized design means they're just incredibly efficient at the specific tasks they were built for, primarily that dense matrix math that dominates large models. That's a great practical insight. Okay, so we have all this amazing distributed hardware, TPUs or GPUs. But how do we actually use it effectively? How do engineers speak the language of parallelism to coordinate thousands of these chips working together? That's where frameworks like JAX come in, especially prominent in the TPU world. But the concepts apply elsewhere, too. JAX lets you write, you know, pretty standard looking linear algebra code in Python, maybe NumPy-like. And its compiler handles the complexities of running that code efficiently across potentially thousands of devices. A core concept here is sharding. Sharding, like breaking things up. Exactly. You define a device mesh, think of it as just a logical grid of your devices, maybe 2D or 3D, with named axes like XYZ. And then using a special sharding notation, you tell JAX how to partition a large array, like your model weights or your input data, across this mesh. So an array might logically be huge, say 4 by 10, 24, but each device only holds a smaller shard of it, maybe 2 by 128. Right. And JAX figures out how to make it work. Pretty much, yeah. JAX then magically handles all the necessary communication behind the scenes whenever operations require data from different shards. It makes these sharded arrays behave, from the programmer's perspective, almost as if they were a single, un-sharded entity on one giant device. That does sound powerful, almost like magic again. So what are the fundamental rules then? Like for something common like matrix multiplication with these sharded arrays, when does that hidden communication actually become necessary and potentially costly? Yeah, that's the key question, right? Predicting when data needs to move across the network because that's where you can lose time. For sharded matrix multiplication, there are basically four fundamental scenarios that dictate if and what kind of communication is needed. Okay, let's hear them. All right. Case one. If neither of your input matrices is split or sharded along the dimension you're multiplying over the contracting dimension, then no communication is needed. It's all local work on each chip. Easy. Nice.

Case two: 13:50

if only one input is sharded along that key contracting dimension, you typically need to perform an all-gather of that sharded input first. Basically each device needs to collect all the pieces of that one array before it can do the full multiplication. locally okay so communication cost there yep case three if both inputs are sharded along that contracting dimension you first multiply the local shards you have and then you need to perform an all reduce on the results that means summing up the partial results from all devices to get the final correct answer everywhere another communication step exactly and finally case four is a bit more subtle If both inputs have a non-contracting dimension sharded along the same mesh axis, you often need an all-gather of one of the inputs first before doing the local multiply. Got it. So these rules are like the cheat sheet for predicting communication. Pretty much. They help you understand when the communication costs will hit and hopefully guide how you shard your tensors to minimize those costs. costs. So that all gather and all reduce you mentioned, those sound like critical building blocks, the basic operations. Can you break down these core communication primitives a bit more? Absolutely. Think of them as the basic vocabulary for interchip communication in these distributed systems. And all gather, like we just discussed, gathers all the shards of an array along a specific dimension, effectively replicating the full array on every participating device. Its cost primarily depends on the total amount of data being moved and your network bandwidth, not necessarily just the number of devices involved. Once you're in a regime where bandwidth is the limit. Okay. A reduced scatter is kind of the inverse. It takes an array that's already distributed where each device has a partial sum, maybe, sums up those partial results across devices, and then reshards the final summed result along another dimension. costs roughly the same as an all-gather in terms of data moved. Right. Sum, then split. Exactly. And all-reduced basically combines both. It first performs the reduction like a sum across all devices, and then it distributes that final complete result back to every device. So it's effectively a reduced scatter followed by an all-gather, meaning it costs about twice as much as a single all-gather. It's very common for things like summing gradients in data parallels. The most expensive one, generally. Typically, yes, for the same amount of data. And finally, there's an alt hole. This is more of a general rearrangement tool. It lets you reshuffle data between devices, often used to change how an array is sharded, like moving data associated with one dimension, to be sharded along a different mesh axis. It can be surprisingly efficient, sometimes costing only around a quarter of an all-gathered, depending on the network and the exact pattern. Okay. The crucial point across all of these is that their runtime, their cost, is largely determined by the total bytes being moved over the network and the available bandwidth of that network. This just underscores why those powerful interconnects like ICI and NVLink are so incredibly vital. This all builds towards the most dominant architecture in AI today, the Transformer. How does this deep understanding of hardware sharding and communication apply to making transformer math fly, especially during training? Right. For large transformer models, the vast majority of the computation, the FLOPs, happens in two main places. Yeah. The MLP blocks those feedforward layers and the attention mechanisms. The heavy hitters. Exactly. And a good rule of thumb to keep in mind is that training a transformer involves roughly six times more FLOPs than just running inference with it. That's because training requires both the forward pass to calculate predictions and the backward pass to calculate gradients, which is computationally about twice as expensive as the forward pass. So 1x forward plus 2x backward for gradients is 3x per step times roughly 2 for optimizer updates and other bits, 6x total FLOPs. Wow, okay, 6x more work for training. That definitely impacts how you scale things. Immensely. Understanding this FLOP breakdown helps us choose the most effective parallelism strategies to keep those expensive transactions. training runs efficient.- Let's dive into those strategies then. How do engineers actually distribute a giant transformer model across thousands of chips for training? What are the main approaches?- Okay, there are four primary approaches, and often the best results come from cleverly combining them. One, first step is data parallelism. This is usually the simplest and most common starting point. You shard your input data, your batch across all the devices, but, and this is key, you replicate the entire model's parameters and the optimizer state on every single chip. The whole model on every chip. Yeah. The only communication needed is an all reduce operation to sum up the gradients calculated on each device during the backward pass so everyone gets the same update. The main challenge here, as you can guess, is memory. Replicating the model, and especially the optimizer state, which can be large, on every chip eats up memory fast. This limits the size of models you can train or forces you to use very small batch sizes per chip, which might not be efficient. Right. Memory becomes the bottleneck. Exactly. So, to address that memory issue, we have strategy number two. Fully sharded data parallelism, FSDP, which you might also hear called 03. Okay, fully sharded. With FSDP, you still shard your activations by batch, like in DP, but now, critically, your model parameters and your optimizer state are also sharded across the devices. Each device only holds a slice of the full model. Ah, that saves a ton of memory. A massive amount. Parameters are then all gathered just in time, right before they're needed for a forward pass computation on a specific layer, and then discarded. Similarly, gradients are reduced-scattered during the backward pass instead of all reduced. This massively reduces the memory footprint on each device, letting you train much, much larger models. And the communication cost, does it go up a lot? Interestingly, no. The total communication volume ends up being roughly similar to standard data parallelism. That all gather for parameters plus the reduced scatter for gradients is roughly equivalent in communication cost to the single all reduce in DP. So FSDP is often seen as a very efficient memory upgrade over DP. Very clever. Okay, what's next? Next is tensor parallelism, TP. often associated with the term megatron sharding. Here, instead of just sharding the data batch, we shard the actual tensors within the model layers themselves. So activations might be sharded along the hidden dimension D and parameters within, say, an MLP layer might be sharded along the feedforward dimension F. Splitting the math itself. Exactly. This requires more intricate communication within the layer computation. Typically an all-gather of activations before the first matrix multiplication in a block, and then a reduced scatter of the output after the second matrix multiplication. So, more comms, but what's the benefit? The big benefit is that tensor parallelism allows for much smaller, efficient batch sizes per pod or per chip compared to just DP or FSDP alone. It helps keep the chip busy with less data, which can be crucial for fitting massive models or large pods. And it's highly compatible with FSTP. You can use both together. Okay. Unlock smaller batches. Yeah. And the fourth strategy. Finally, there's pipeline parallelism. This strategy splits the model layers across different groups of devices called stages. Stage 1 computes the first few layers, passes the activations to stage 2, which computes the next few layers, and so on, like an assembly line. Like a production line for the model. Precisely. It boasts very low communication costs in terms of raw bandwidth needed because you're typically just passing activations between stages, which might be smaller than weights or gradients. It seems almost free from a network bandwidth perspective sometimes. But there's always a catch, right? Always a catch. Pipeline parallelism introduces significant challenges. The code becomes much more complex to manage. It doesn't always interact cleanly with FSDP or DP. And the biggest issue is pipeline bubbles. Yeah, idle time. The first stage finishes its micro batch and has to wait for the last stage to finish before it can start the next one. Similarly, the last stage is idle, waiting for the first stage to send at work. These bubbles represent wasted compute time, reducing overall efficiency, especially with fewer stages or smaller batches. Plus, there's latency added because activations have to physically move between chips on the critical path. Wow. A complex landscape of strategies with different trade-offs. How does this apply to a real-world scenario, say, training a massive model like LAMA-370B? Great example. Let's imagine training LAMA-370B, a 70 billion parameter model, with a large batch size, maybe 4 million tokens, on a huge 8-960 chip TPU V5P pod. Right, that's serious scale. Definitely. Now, if you tried to train that only using pure FSDP or maybe FSDP combined with some basic sequence parallelism, which is like sharding the sequence length, our sources show you'd quickly become bottlenecked by communication. The chips would spend too much time waiting for all gathers or reduced scatters. Not enough compute to hide the comms. Exactly. The analysis shows that an optimal strategy involves a careful blend of these techniques. For instance, you might use a very large 2-24 way data parallelism, FAC, SDP style, combined with two-way sequence parallelism and four-way tensor parallelism. A three-dimensional approach almost. Precisely. This intricate combination across different parallelism dimensions is what's needed to distribute the work effectively, reduce the communication load per chip, and keep all those thousands of TPUs busy doing math instead of waiting. This example perfectly illustrates the core principle as you increase the degree of parallelism More ways to split the work or reduce the per chip batch size Your models tend to become more communication bound because there's less local compute per chip So effective scaring is this delicate balancing act strategically combining DP, FSDP, TP, and sometimes pipeline to keep compute high and communication hidden. Okay, this is where it gets really fascinating for me. We've talked about the hardware, the strategies, but how do the engineers actually figure all this out? What tools do they use to peek under the hood, diagnose these bottlenecks, and validate these complex combined strategies? Right, because it's definitely not trivial. That's where the software stack, particularly for TPUs using JAX, and its powerful profiling tools become absolutely indispensable. So imagine you write your high-level model code in Python using JAX. That code doesn't just run directly on the TPU hardware. It goes through several layers of compilation and optimization. Like a translation process. Exactly. First, it might go to something called stable HLO. That's a platform agnostic intermediate representation. Then it hits the XLA compiler. XLA stands for Accelerated Linear Algebra. XLA does a ton of heavy lifting, optimizing the math, fusing operations together where possible. Fusing, like combining small steps. Yeah, like merging a matrix multiplied directly with the activation function that follows it so you don't have to write the intermediate result back to memory. Saves bandwidth. XLA then lowers this to HLO, high-level optimizer IR, and eventually to LO, low-level optimizer IR, which programs the TPU more directly, scheduling memory copies between HBM and VM, scheduling the MXU operations, and so on. Finally, that turns into the actual machine code the TPU executes. That's quite a journey from Python to hardware instructions. It is. And for situations where you need the absolute bleeding edge peak performance, especially in really critical sections like, say, the self-attention mechanism, engineers can even use something called palace kernels. Palace. Yeah. It's kind of an escape hatch within JAX that lets you write lower level, almost hardware specific code to hand optimize a particular kernel. It gives you maximum control, but it's a tradeoff, right? You lose some portability and it takes much more development effort. Right. More power, more responsibility. So it's a deep rabbit hole. And to understand what's actually happening at runtime, when the model is training on thousands of TPUs, there's the JAX Profiler. Precisely. The JAX Profiler is this incredible multipurpose toolkit that allows engineers to truly understand and debug the behavior of their program running on the TPUs. It has several key views. What are they? Well, first there's the trace viewer. This gives you a super detailed chronological timeline of every single operation happening on each TPU core. You can see the XLA operations, you can see traces back to your Python code. So you can see what ran when. Exactly. It's invaluable for seeing the overall flow of computation, identifying repeated blocks of work like layers. and pinpointing exactly where time is being spent. Is it the feedforward networks? Is it the attention calculation? Is it waiting for communication? Okay, that sounds crucial. What else? Then there's the Graph Viewer. This visualizes the High-Level Optimizer, or HLO, graph. It shows you the dependencies between different operations, and importantly, how the XLA compiler might have fused several small operations into one more efficient chunk. It helps you understand the compiler's logic. See the optimizations. Right. And finally, especially important for debugging those memory issues we talked about, there's the memory profile viewer. This helps you analyze memory usage over time, see how much memory each operation allocates, and track down those dreaded out-of-memory, or OOM errors. Ah, the bane of every ML engineer's existence. Okay, so when you actually look at an XLA operation inside the profiler, say in the trace view, what are the key things you're trying to understand? What information is there? When you drill down into an HLO operation in the profiler, you're primarily looking for a few key things. First, obviously, what the operation is, is a matrix multiply data. a convolution, an element-wise operation, a communication primitive, like all reduce. Then you look at it as output-shaped data type, like BF16 or F3D2, but critically, you examine its memory layout. The layout, how the data is arranged. Exactly. The layout tells you how the multidimensional tensor data is actually organized in linear memory, the order of the axes, any tiling the compiler might have introduced for efficiency, any padding added. This is huge because a non-optimal layout or a mismatch in layout between producer and consumer operations can force the compiler to insert extra copy or transpose operations which waste precious time and memory bandwidth. Sneaky performance killers. Totally. You also look at the memory space indicated, like is the input coming from HBM, usually S0, or the super fast VM S1? Is the output going back to HBM when maybe it could have stayed in VM for the next operation? These details help you understand if the compiler is making optimal use of the memory hierarchy. So how does all this translate into real world optimization? Can you give an example of how using the profiler actually helped improve a model's performance based on the source material? Absolutely. Profilers are invaluable for diagnosing bottlenecks day to day. For example, let's say you're looking at an optimized transformer benchmark layer. An initial version might take, say, 90 milliseconds to execute. You run the profiler and maybe it pinpoints a specific reduced scatter communication operation that consistently takes 1.1 milliseconds within that 90 meters. Now, using your knowledge of the array size being scattered and the known ICI bandwidth of the TPU, you can do a quick roofline calculation, a theoretical estimate of the minimum time that data transfers should take. Kind of a sanity check. Exactly. If your calculation predicts, say, 1.05 milliseconds based on bandwidth limit, and the profiler shows 1.1 limits, you can say, "Okay, this operation is performing"very close to its theoretical limit."The bottleneck probably isn't this specific operation."The hardware is doing its job here." Right, look elsewhere. Right, but conversely, you might find an operation taking way longer than the roofline predicted. or you might spot an unexpected memory copy operation that the HLO view reveals is happening because of a suboptimal memory layout chosen by the compiler for an intermediate tensor or you discover that your chosen sharding strategy isn't distributing the computational load as evenly as you thought finding those hidden inefficiencies precisely one concrete example mentioned involved optimizing a transformer layer where careful profiling and subsequent code adjustments perhaps tweaking sharding annotations or fusing patterns managed to reduce the per-layer runtime from that initial 90 meters down to a much sleeker 80 meters. That's over a 10% speed up, which is significant when you're running trillions of operations. This iterative process profile, identify, analyze, refine, is how deep technical understanding allows practitioners to really push the hardware to its limits. It's constantly adapting as compilers get smarter and hardware evolves. What an incredible journey we've taken today. Seriously. We've peeled back so many layers moving from the specialized guts of TPUs and GPUs through that intricate dance of sharding notations and communication primitives. All gather, all reduce. Right. All the way to the strategic choices engineers make when parallelizing massive transformer models like LAM at 370B across thousands of chips. We even peeked behind the curtain with those profiling tools that reveal what the hardware is actually doing, kind of demystifying some of that black magic feeling. It really is a testament to the continuous innovation in this field, both in hardware and software. And, you know, as we wrap up, maybe consider this, this relentless pursuit of scaling these models, particularly the transformer architecture. is pushing the very frontier of hardware design and software optimization. Given that models like the Transformer have become so dominant, and optimizing their performance increasingly relies on understanding these really low-level hardware and compiler details, what new architectural innovations might emerge in the next few years? Will we see something challenge or complement this GPU-TPU duopoly? Maybe something that manages to abstract away some of this intense complexity once again for the next generation of AI engineers. That's a truly fascinating, thought-provoking question. What comes after this intense optimization focus? Something definitely to mull over. Well, this deep dive, as always, is just a starting point. The principles we've discussed today, the hardware differences, the communication costs, the parallelism strategies, they form a foundational map for understanding where AI infrastructure is heading next. Keep exploring, keep questioning, and you'll keep learning. Until next time on the deep dive.