🧠 Welcome to the curved space of everything

Please see our corresponding Substack episode.

🧠💥 Just discovered how your brain might be hiding explosive secrets in curved spaces. New research reveals why AI suddenly "gets it" - and it's not what you think. The math that's reshaping memory itself. #NeuralNetworks #AI #BrainScience

Interactions in Curved Statistical Manifolds

Source: Aguilera, M., Morales, P. A., Rosas, F. E., & Shimazaki, H. (2025). "Explosive neural networks via higher-order interactions in curved statistical manifolds." Nature Communications, 16, 61475. https://doi.org/10.1038/s41467-025-61475-w

This is Heliox: Where Evidence Meets Empathy

Independent, moderated, timely, deep, gentle, clinical, global, and community conversations about things that matter.  Breathe Easy, we go deep and lightly surface the big ideas.

Thanks for listening today!

Four recurring narratives underlie every episode: boundary dissolution, adaptive complexity, embodied knowledge, and quantum-like uncertainty. These aren’t just philosophical musings but frameworks for understanding our modern world. 

We hope you continue exploring our other podcasts, responding to the content, and checking out our related articles on the Heliox Podcast on Substack. 

Transcript

0:25

You know how sometimes the most complex things, like the human brain or the global economy, seem to operate on these hidden rules? Yeah, connections that are way deeper than they first appear. Exactly. Well, today we're doing a deep dive into a really fascinating new discovery that might just, you know, unlock some of those secrets. We're going to be talking about something scientists are calling explosive neural networks. Right, and the crucial role of what they term robots. Higher order interactions or HOIs. Think of HOIs not just as one-on-one chats, but more like group dynamics within a system. So like three or more things influencing each other all at once. Precisely. And our mission today is really to unpack this groundbreaking new framework they've developed called curved neural network. Okay, curved neural networks. Sounds intriguing. It is. And it's not just some, you know, abstract theory. It actually offers a kind of shortcut to understanding some incredibly complex stuff. Like what kind of stuff? Well, anything from how your own brain retrieves a memory you thought was lost to the mechanics behind the most advanced AI models out there today. Wow. Okay. So we're going to look at how these models implement something called a self-regulating annealing process. That's right. And why that leads to these really dramatic explosive changes. Explosive phase transition. Yeah. And enhanced memory capacity, too. Cool. And all this comes from a new article, right? Yep. It's titled Explosive Neural Networks Via Higher Order Interactions in Curved Statistical Manifolds. Got it. So let's start with those higher order interactions, the HOIs. What are we really talking about here? Okay, so in pretty much any complex system, physical, biological, social, you name it, many of the ways things depend on each other aren't just simple pairs, they're higher order. Meaning three or more components interacting at the same time. Exactly. And what's really interesting is that recent studies suggest this isn't like an exception. It might actually be the norm for how these systems are organized. Whoa, OK. The norm changes things. It really does. It means understanding these HOIs is critical. Why critical? What do they do? Well, they're behind a lot of collective behaviors, things like bistability. Where a system has two stable states, like on or off. Right. And hysteresis, where the system's current state depends on its past inputs, it's history. Right. Okay. And importantly for this research, they drive those really dramatic, explosive phase transitions we mentioned. These sudden, abrupt changes in behavior. Exactly. Like a switch flipping, but for the whole system's state. Very sudden, very sharp shifts. So not gradual changes. Not at all. Think abrupt. And these HOIs seem particularly important for how neural networks function, both the ones in our heads and the artificial ones. Okay, let's break that down. How do they affect biological brains? So in biological brains, they seem to shape how groups of neurons act together. They might contribute to sparsity. Sparsity, meaning only a few neurons are active at any given time. Yeah, which is thought to be really important for energy efficiency. Makes sense. And there's also a suggestion they could be behind something called critical dynamics. Ah, the idea that the brain operates at this sort of sweet spot between total chaos and rigid order. That's the one operating right at the edge, which many think is optimal for computation. And what about in AI? You mentioned artificial networks too. Right. So in AI, HOIs have been shown to actually boost the computational capacity of certain networks, especially recurrent neural networks. How so? Well, take things like dense associative memories. These are networks specifically designed to store and retrieve lots of patterns. Okay. It turns out that nonlinear HOIs are key to extending their memory capacity way beyond what simpler models can do. So more complex interactions mean more memory power. Essentially, yes. And there's a really strong feeling, a conjecture, that HOIs are actually fundamental to why the current cutting-edge deep learning models are so powerful. You mean like transformers in large language models? Exactly. The attention mechanisms in transformers and also the energy landscapes in diffusion models. The ones used for generating images. That's right. The idea is that these sophisticated HOIs might be the secret sauce behind their success. Okay, so if these HOIs are so important, I mean, clearly vital, why haven't we been modeling them properly all along? What's been the holdup? Ah, well, that's the big challenge. It really comes down to computation. Too complex. Massively. studying HOIs directly is computationally really, really difficult. Most models that we can actually solve, you know, analytically, Massively. they usually have to restrict interactions to just one specific higher order, like only groups of three or only groups of four. But reality isn't like that. Exactly. If you try to represent all possible HOIs, pairs, triplets, quadruplets, everything all interacting, you hit what's called a combinatorial explosion. Meaning just too many possibilities to calculate. Way too many. The number of terms just blows up. So research has mostly been stuck looking at really simplified homogenous situations or only very low order interactions. Right. Which isn't capturing the full picture. Not even close. So what was the breakthrough then? How did they get around this combinatorial explosion? This is where the curved neural networks come in. It's a genuinely new family of models. And they basically extend the classical ideas of neural networks in a really clever way. Right. How do they work? What's the core idea? They use a generalization of a fundamental concept called the Maximum Entropy Principle, or MEP. I've heard of that. It's about finding the most unbiased distribution given some constraints, right? Usually using Shannon's entropy. Precisely. But this new approach uses a different type of entropy. entropy called Rennie's entropy. Rennie's entropy. Oh. Okay. How does that help? The key is that Rennie's entropy introduces this extra parameter, a defamation parameter, usually called gamma. A defamation parameter. What does that do? Well, think of it as... It's warping the mathematical space the model lives in. It changes the geometry of the statistical manifold. Warping the space. Yeah, like curving a flat sheet of paper. And by doing this, by introducing this curvature controlled by gamma, the network can effectively capture HOIs of all orders simultaneously, but in a really concise, parsimonious way. All orders. Without listing them all out. Exactly. It's like building this nested onion-like structure in the model space, as they show in one of their figures. All orders. It neatly sidesteps that whole combinatorial explosion problem. Wow. That's really elegant, capturing all that complexity with just one parameter, gamma. It is quite elegant, yeah. Okay. But then you mentioned something else really fascinating, the self-regulating annealing process. What's going on there? Right. This is one of the coolest findings. That curvature parameter, gamma, it doesn't just sit there. It actively influences the network's dynamics through something called an effective inverse temperature. Okay. Think of it like an internal thermostat. Okay, an internal thermostat. Yeah. How does it self-regulate? So when gamma is negative, something remarkable happens during memory retrieval. The effective temperature of the network rapidly increases. Increases. Wouldn't that mean more randomness? Usually annealing means cooling down. Ah, but it's the inverse temperature that increases. So the effective temperature itself rapidly decreases. The network cools down fast. Ah, okay. Got it. So it gets less random, more stable. Exactly. It reduces the randomness of the network's transitions, making it converge much faster to a stable, low energy state, like finding the correct memory. And this creates a feedback loop. A powerful, positive feedback loop. The network's energy state influences its effective temperature, which in turn accelerates its convergence. It's like simulated annealing But the network controls the cooling schedule itself Based on its own state It accelerates its own process That's amazing It really is And conversely, if gamma is positive It does the opposite It decelerates the dynamics through negative feedback So negative gamma is like hitting the accelerator For finding memories A self-regulating accelerator, yes And this self-regulation, this acceleration This is what leads to those explosive phase transitions. Precisely. When you have large negative values of gamma, the network doesn't just gradually slide into a state. Oh It jumps abruptly, discontinuously. Okay. So you see things like multi-stability. Yes. The network can suddenly access multiple stable states. And you see strong hysteresis effects. Meaning its final state really depends on the path it took to get there. Absolutely. Its history matters immensely. Just like... Flipping a switch might depend on which direction you pushed it from. And these dynamics aren't just theoretical. They resemble things we see elsewhere. Very much so. They're strikingly similar to explosive transitions seen in models of, say, how rumors or diseases spread through higher order contact. Okay, contagion models. Right. And also in how groups of oscillators suddenly synchronize, like in Kuramoto models. Can you give a specific example from the network model? Sure. So in a simple case where the network is trying to retrieve just one stored pattern with negative gamma... the retrieval process can literally explode into the correct ordered state, converging way faster than a traditional network would. Much faster. And if you encode, say, two patterns, negative gamma can create these really complex hysteresis loops with potentially like seven different stable points, It compresses the range where transitions happen but expands the region where memories can actually be retrieved successfully. Seven points. That's complex behavior from that one parameter. It really unlocks a lot of dynamic richness. Okay, let's shift to the practical side then. Enhanced memory capacity and robustness. What does all this mean for actually using these networks? Right, so... Classical associative memories have this well-known problem, saturation. They just get overwhelmed if you try to score too much. Exactly. They fall into this useless, disordered state, often called a spin glass state. But these curved neural networks, both the theory and the experiments show, have significantly enhanced memory capacity. How does the gamma parameter play into that? Negative gamma values actually cause an expansion of the ferromagnetic phase. That's the phase where memories are stable and retrievable. That's the good one, yes. And it also expands the mixed phase. So basically, the network can reliably store more patterns before hitting that saturation point. More patterns stored reliably? That's a huge deal.

It is. But there's another side to it: 11:08

robustness. Ah, right. Is there a trade-off with capacity? There seems to be a fascinating one. While negative gamma boosts capacity, positive gamma does something different but also useful. What's that? It actually reduces the size of that mixed phase. And why is reducing the mixed phase good? Because in the mixed phase, the network is more likely to retrieve junk spurious patterns or distorted memories. I see. Things that look sort of like memories but aren't quite right. Exactly. So by shrinking that mixed phase... Positive gamma makes the memory retrieval more robust. It's less likely to get confused by these spurious states. So you have a knob, gamma, that lets you tune between maximizing capacity with negative values or maximizing robustness, precision with positive values. That seems to be the implication. It offers a design choice depending on the specific application. And you mentioned experiments confirmed this. They did, yeah. They ran simulations using the CIFAR-100 image data set. Okay, real-world image data. Well, process image data. They binarized the images, turned pixels into simple plus-one or omegas-one values, and encoded those as patterns in the network. And the results. into models of disordered systems called spin glasses. The curvature can turn their usually smooth phase transitions into abrupt, even explosive ones with hysteresis. That's quite a general effect then. It seems to be, yes. Zooming out, what does this framework tell us about, say, the big picture of modern AI? Could this help explain why models like transformers are so good? That's definitely one of the exciting implications. The framework provides a new lens. The idea that complex HOIs, captured effectively by this kind of curvature, might be a fundamental reason for the success of transformers, with their attention mechanisms and diffusion models, it's a compelling thought. Yes. And the accelerated memory retrieval part. That mechanism helps clarify how these advanced associative networks can operate so efficiently. It provides a potential theoretical underpinning. And what about insights into biology? Back to the brain Yeah, it offers potentially crucial insights there too We know there's evidence for both positive and negative HOIs between biological neurons A mix Seems like it And this mix could naturally lead to that sparse neuronal activity we talked about earlier Which is vital for energy efficiency Absolutely. So the study suggests that getting enhanced memory and sparse activity together, which this framework allows, is a really promising direction for understanding how biological brains code information so efficiently. It connects efficiency and capacity. In a very direct way. So overall, this work really ties together the maximum entropy principle, how these higher order interactions emerge, and the resulting nonlinear network dynamics. It sounds like a big step towards a more general theory. I think it is. A general theory of higher order interactions, which could help us understand complexity in all sorts of networks. way beyond just AI. Okay, so to recap, we've explored how these curved neural networks provide this incredibly elegant, concise way to model the really complex world of higher-order interactions. Right, and doing so lets us understand these explosive dynamics, makes memory retrieval faster, and even significantly boosts memory capacity. Things that are impossible or at least very difficult with older models. Exactly. And this deep dive, I think, really highlights how concepts that might seem abstract, you know, mathematical ideas like rainy entropy and curved manifolds, are having a direct impact on developing cutting-edge AI while also giving us these profound new insights into how our own brains might work. It's a great example of theory leading to really tangible, sometimes surprising results. Definitely. It reshapes how we can think about these complex systems. So it leaves you wondering if these hidden curvatures and higher order interactions are driving so much in AI and neuroscience, what other complex systems out there, maybe social networks, ecological systems, might be secretly governed by these same kinds of powerful group dynamics just waiting for us to figure out their underlying geometry?