The Sweetness in AI Bitter Lesson

This podcast episode explores why the most powerful intelligence isn't designed from the top down—it’s grown from the bottom up. We dive into Richard Sutton’s "Bitter Lesson" in AI, the parallel logic of Darwinian evolution, and how these massive, iterative loops provide a blueprint for excellence in business and design. From the "Deep Time" of biology to the parallel compute of modern GPUs, we discuss how the most extraordinary achievements are simply the result of an ordinary loop scaled to the extreme.

The New Frontier Post. Venturing forth into new realms of knowledge. 

Transcript

SPEAKER_00 0:00

AI's bitter lesson applies to life and work. How life's innovations all seem to work the same way. By Timer Ali Ahmed. Richard Sutton, a pioneering AI researcher, famously described the history of his field as a bitter lesson. Researchers kept trying to build intelligence with handcrafted algorithms, even as the right approach, let blind computation learn for itself, stared them in the face. When a brute force search-based program defeated world chess champion Gary Kasparov in 1997, many experts were, as Sutton notes, not good losers. Dismissing the victory because brute force, a simple algorithm that uses iteration instead of brains, wasn't how humans actually played chess. The pattern repeated in speech recognition, computer vision, and Go. The dumb, bottom-up algorithm of scaled computation beat the carefully engineered, top-down systems by a long shot. Why were AI researchers reluctant to adopt the right approach? Sutton provides a few answers. First, hard-coding human knowledge works well in the short term, giving researchers an intoxicating but temporary illusion of progress. Second, it is a matter of ego. We want to believe that our specific expertise, our deep understanding of chess strategy or linguistic syntax is the magic ingredient. We want machines to think the way we think. Admitting that a blind algorithm can completely bypass our hard-won human knowledge feels like a demotion of human specialness. When Darwin first published his theory of biological evolution, humanity had a hard time accepting it too. Life's complexity, its intelligence, couldn't have arisen by evolution's dumb, partly random process, many argued. A well-designed system needs a sophisticated designer. Creationists continue to believe in this kind of logic. Similar to AI researchers, their ego likely balked. The hand of God created us, not a dumb evolutionary process. Thanks to Darwin, we now know that life, in all its marvelous, intricate complexity, is shaped by a simple, brute force algorithm known as natural selection. It has only three steps variation, selection, repeat. This evolutionary algorithm can be depicted visually as an expansion and convergence of design space. When an organism's DNA varies through reproduction and random mutations, it expands the design space. When less fit organisms die, the design space converges on a solution. The cycle repeats. In one iteration, the improvement may be minuscule, but over time, these tiny improvements accumulate into something as beautiful as a butterfly. The chart below captures the first two steps of the evolutionary algorithm well, but it doesn't do justice to the third step. The unimaginable scale of time, the billions of years and trillions upon trillions of generations it took to create a butterfly. The parallels between biological evolution and how AI is built are striking. An AI model starts with random weights throughout the network, similar to how evolution expands the design space. The loss function acts as the environment, exerting selection pressure on the model's weights. Importantly, both AI and evolution derive their power from scaling. A massive amount of computing power enables parallel iterations on a GPU, akin to evolution's deep time, the unfathomable amount of time evolution has been hard at work, parallel processing organisms. This truth, that complexity and intelligence emerge by scaling simple, bottom-up processes, operates in multiple domains. Consider the cosmos. Some physicists propose that our universe is simply the survivor of a blind loop in which unstable, unfit universes collapse into nothingness, leaving behind only those in which the laws of physics allow a stable system like ours. The evolutionary processes play an important role in business. Consider continuous improvement, a closed loop system for iteratively refining processes, services, and products that took off in the 1980s. But my favorite example is the Double Diamond DD framework, which is widely used in innovation and design. The DD is an iterative process of expanding and collapsing design space, of divergence, variation, and convergence, selection. In the diverge phase, you explore wildly. Participants are asked to adopt a beginner's mindset and to remain open to all possibilities. This mirrors the randomization phase of AI and evolution. In the converge phase, you ruthlessly test and kill ideas if they don't meet the objectives established in the defined stage. It seems to me that intelligence in multiple domains emerges bottom-up from a relatively simple recipe of variation and selection. In the moment, it can feel mundane. That's why we don't notice nature's innovation engine, evolution, in our day-to-day lives. That's why another brainstorming session with sticky notes doesn't make us tingle with excitement. But when we stand under a T-Rex skeleton in a museum, we marvel at nature's creativity. When we see the curtain lift on a product we worked on for years, we get weepy. In these moments, whether we realize it or not, we are stepping back and looking at the grand sweep of time, how the slow accumulation of small improvements made something out of nothing. It can be a spiritual experience. The magic of life, which includes AI and innovation, is in the scaling, in the endless, diligent repetition. It's how the Grand Canyon emerged without a designer, only water rushing against stone for millions of years. Whether you believe in God or not, I hope you agree that there is, as Darwin said, grandeur in this view of life.