Most Neurons Do Nothing and That's the Point!

This episode explores why biological neural networks are inherently sparse, with only 1 to 5 percent of cortical neurons active at any moment, and why this silence is a feature rather than a limitation. We trace the evolutionary pressures that drove the brain toward sparse coding, from the metabolic cost of each spike to the fixed energy budget per neuron, and examine the computational advantages that follow: greater memory capacity, more efficient representations, and robust generalisation. The discussion then turns to what this means for artificial intelligence, covering the Lottery Ticket Hypothesis, dynamic sparse training, Mixture of Experts architectures, and spiking neural networks. For engineers building at the deep edge, the conclusion is clear: strategic sparsity is not a constraint to work around but a design principle to build on.