How Does a Diffusion Model Work, Part 1: Introduction

Diffusion models have become the foundation of modern generative AI, powering state-of-the-art systems for image generation, video synthesis, protein design, and more. But behind the impressive demos lies a surprisingly elegant mathematical framework.

In this first episode of a multi-part series, we begin working through the excellent MIT lecture notes An Introduction to Flow Matching and Diffusion Models by Peter Holderrieth and Ezra Erives. Rather than jumping straight into denoising or neural networks, we focus on the fundamental question: What does it actually mean to generate data? 

Topics covered include:

•  Why generative modeling is fundamentally a sampling problem
•  Data distributions and probability densities 
•  Representing images, videos, and other data as vectors 
•  Conditional generation and prompts 
•  Why diffusion models are framed as learning distributions rather than memorizing examples 
•  An overview of where the mathematics is heading in the remainder of the course 

This episode is intended for anyone who wants to understand diffusion models from first principles, building the mathematical intuition needed for later discussions on ODEs, SDEs, flow matching, score matching, and modern diffusion architectures. 

Lecture Notes:
https://diffusion.csail.mit.edu/docs/lecture-notes.pdf

Website:
https://arkitekt-ai.com

Contact:
support@arkitekt-ai.com

Transcript

SPEAKER_00 0:12

Hello, welcome to another episode of AI Research Today. So I'm your host, Aaron McClendon, founder of Architect AI, company that helps develop automated SaaS uh solutions. Uh, for specifically, we tend to target target small businesses. Um custom software builds AI integrations um automatically built using our custom AI platform. Um and yeah, been a minute since we did an episode. Apologies, just have gotten wrapped up with a lot of competing interests. Um, but we're here today, and what I was planning to do today was a little bit different than most episodes have been historically. So, historically, for most episodes, we've covered a specific research paper or development. However, I found on Archive a really interesting document, I guess I'll say, uh, out of MIT, which are lecture notes to their MIT class regarding diffusion models and flow models. Um, a lot of the stuff we've covered on this podcast has been LLM centric um and reinforcement learning centric. And so I thought it could be cool to switch up a little bit and do a multi-part series on how diffusion models work. Um, and it will be based off of a walkthrough I do of this this lecture material, which I found to be quite helpful. Um, so I'll post the the the lecture notes um in the chat. I encourage you to look at them. Um the MIT page itself also has uh video recordings of the lectures, which no doubt do a better job uh than me of explaining their material. Um but I just found it to be pretty interesting and uh kind of wanted to change it up. I don't know how many parts of the series we'll do, probably um two or three parts. Uh, there's about 80 pages in the lecture notes, um, but I won't go through all of them. So the series is goes, it's there's first an introduction which goes through um the basics of generative modeling, um, and then a rigorous definition of flow versus diffusion models. And then there are some algorithms for training uh flow matching, score matching. Um, then they have some sections on uh prompt conditioning, uh building uh networks uh from scratch. Um so that's one of the projects in the in the series is um building your own diffusion model. Uh so again, a really cool uh course, they have some reference material in the back, also probability primers on uh uh random variables and uh Markov chains and all that sort of stuff. Um, so we'll do a refresher uh as we go through the uh material as needed. Um I'll keep I'll try and keep the episodes around 30 minutes um so they're not uh too difficult to make it through the whole thing. And yeah, we'll probably do uh two or three episodes and try to get through uh at least one or two of the learning algorithms for actually how uh to train one of these models. So let's get started. So um they point out that in the last few years uh we've seen this kind of revolution in generative AI. Obviously, things like Chat GBT and LLM-based transformer models um have played a large role in that uh revolution. However, there's another parallel track that's been developed um image uh image generation models like Nano Banana Stable Diffusion 3, Stable Diffusion, uh, which I guess is still a company. I don't even remember. I remember there's a news article last I heard the CEO was walking around uh in his underwear downtown. Um, but anyways, they definitely released some pretty good open source models. Uh and um there was also the DALI model, which was a uh hidden from the public architecture, I believe, but was uh presumably a diffusion model. Um and there's also been interesting integrations between diffusion modeling and transformers. There's a paper out of NVIDIA um using diffusion model type things for um uh token generation and selection. Um and I guess sort of like the JEPA model is in the same vein of a diffusion model where you're sort of thinking in latent space sort of things. Uh VO3 uh video models. Um so yeah, a lot of stuff has been happening uh in that space, and they uh branch and the goal of the lecture series is to talk through um diffusion models and flow matching models. So again, these are uh the the the most popular architecture is Alpha Fold 3, is another one they call out. Um and uh they they all kind of started. Um, I believe the uh original idea for a diffusion model was from someone who was a physicist studying Brownian motion, which is sort of like stochastic movement of molecules through a medium. Um and uh the idea is to convert noise into data. So, like if you think of starting out with a TV screen that's all fuzzy, just the black and gray dots. Uh, I don't some people may not even uh remember when TVs did things like that uh before streaming services, but uh uh you hopefully know what I'm talking about. Um, and if you kind of wiggle the antenna on the TV, you can eventually get a signal start to come through. And maybe it's a little staticky at first, but the more you wiggle the antenna, the better the signal becomes and the clearer the picture is. And that sort of iterative um building of the picture from a noisy space is like what a diffusion model is doing. And the the evolution process, the guiding process from noise to data is facilitated by simulating differential equations, ordinary or stochastic differential equations. Ordinary corresponds to um flow models, um, and uh the normal diffusion models correspond to the stochastic STEs. So sorry, stochastic differential equations or STEs. Um so the actual implementation uh isn't too difficult. Um there is some theory behind it, uh and so we'll kind of cover that iteratively and build up on these things. So um let's start off just talking about uh what an image is. So an image consider an image of height by width, where we're we're thinking of pixels. So like if you think of like the MNIST data, you have an image of a number three or something like that. But uh really what an image is is just um encoded pixels. Uh and so you can think of that as like a matrix where there's a height and a width, or like columns and rows, and each index uh or cell is just a pixel, and the pixel has some color value. Uh so we think of an image H by W, height by width, and in the image we think of three color channels, RGB. Um each you you can you can specify any color by by specifying a tuple of uh intensity values for each of these color channels, RGB, and um for each pixel uh you have three values associated with it the r value, the g value, and the b value. So you can represent an image uh as uh height by width by three. So that's like your tensor, really. Um so you have uh H by W, which is just your uh pixels, and then for each pixel you have three dimensions, which are the three color channel intensity values. Uh, and so that's how we'll describe um an image going forward. And a video is simply a series of images in time, so uh, you know, just like a frame rate, right? Like everyone's always complaining about frame rates and video games and um the the 60 fps and stuff. That's just the number of frames you see per second for a video. Um, and each frame, it's like if you've ever drawn a flip book as a kid, I used to do that all the time, where you'd get like a stack of papers, draw a bunch of pictures, and flip it to make it look like the character was moving. Um that's like a video, it's the same thing, it's just a bunch of images that you cycle through and creates the illusion of movement. Um, and so if you now add a dimension time, uh which is just the image associated with a certain time step, for each time step you have an image which has h by w by three channels. Um, and so you can add an extra dimension to this tensor to get a four-dimensional tensor, where you have time by height by width by RGB three. Um they also in their initial examples um give a way to think about um molecular molecular structure, um, where you just have a tuple of uh vectors, and each vector describes a location of an atom in the molecule. Um again, they're they're just trying to describe how you can mathematically represent things that seem like images or more visual things can be encoded uh as an actual just vector of numbers. Um and since you can potentially flatten any of these tensors into a single uh very long uh string of numbers, uh you can just treat every object as a vector in some d-dimensional space. So, like our image, for example, is height by width by three channels. Um, let's just stack everything on top of each other to get a D-dimensional uh vector. Um and uh that's kind of becomes the input to the models um that we'll be talking through. So next they go into what does it even mean to generate something? Um so like say you want to generate an image of a dog, uh, there are a lot of images of dogs that you could be happy with, um, and that makes sense. Like there are a lot of different types of dogs, bulldogs and golden retrievers. Uh, there's no best image of a dog uh in general. Um and so the image of the all an image of a dog kind of you can think of it as a probability distribution over the space of images of dogs. So, like if I give you an image of a dog, a golden retriever, you can think of it uh as a probabilistic return from sampling over the distribution of all dog images. Um that distribution of dog images uh we call a data distribution. They call that p data. Um, and really it's a probability density uh where we're just assigning a value uh to this vector. Again, this dog image is just a vector of numbers. So our probability density is just a function that maps our D-dimensional uh vector image into a probabilistic value. Um and so we're going to get higher probability to images that look more like dogs. Um how good an image fits is uh replaced by how likely it is under the data distribution. Um so uh, and one thing I'll clarify that the P data isn't representing only dog images, it's just um images in general, and some of those images fit the description better than others. Um, and the probability is like what we would want uh to be high if the image is answering the request. I say, give me a dog image, you pull out from the bank of all images ever made a picture of a dog, and that gives you a high score. And so you say, okay, this is a good image to return to you. Um and then that's the uh generation sample process. So the uh key idea they outline here is that uh generating an object Z is model the sampling from the data distribution. So P data, we pull something out of the stack of images, um, that is generation. We've now generated an image. Um, a generative model is a machine learning model that allows us to generate samples from P data. So mathematically, I just said draw a sample from a distribution, uh, but then how do you do that technically? Um, and that's the generative model. So um the other thing they define here, uh, key idea number three, they call it, is a data set. Um, that's just a finite number of samples from the data distribution. Uh, this would just be like the training data that goes into our diffusion model, like uh billions and billions of images um pulled from various points on the internet uh that we've you know heard about in the news people getting litigated for ripping images they shouldn't have rights to and training models, etc. So you just get some giant dump of images, and you just assume that this data set is sampled from the space of all possible images that could ever exist. It's sort of like just a representative, hopefully, representative sample of all images uh that we as humans could see. Um so that becomes our data set. They also uh add the um conditioning ish uh conditioning rule. So so like we we might not just be generating um uh a random image, we might want to generate an image conditioned on a statement, meaning like let's say we have a statement, the example they give in the in the text, I'll use a dog running down a hill covered with snow with mountains in the background. This is just a conditional probability distribution. So given the space of all images, if the statement is y, um, we sample from the distribution that has been conditioned on y. So uh the statement y has occurred, uh, and so we sample from the p data uh that's been conditioned on that statement. And this is what they call a guided uh data distribution. So uh we want to be able, uh we want our model to be able to condition on an arbitrary choice of y. Um that's what allows us to put crazy statements into image generation models. Give me a picture of a monkey eating a popsicle. Um, that's because we trained it on an arbitrary conditioning parameter. Um, and so we want a single model which can be conditioned on any choice of this conditioning statement Y. Um and uh they point out um unconditional generation can be uh generalized to the conditioned case. Um and so initially we focus only on the unconditioned case. Um so the conditioned case is sort of like a natural consequence of the unconditioned case, an extension that they get to later in the uh lecture notes. Okay, so I'll summarize again uh the introduction. Um so uh we we have the task a generation, which is just sampling a vector of dimension d um from a probability distribution of images. Um and this this d dimension represents you know some arbitrary construction of an image that could be height by width by channels, or if we're training over molecular structures or whatever, space of molecules. Um and uh the p data, the true distribution we don't have access to, but we do have access to a training data set, which we presume as being sampled from the space of all possible images. Um and so given data set pairs where we have a sample from p data and a condition statement, y, um, we want to learn the distribution p data conditioned on y. Um and so uh the our model should our our model when we sample from an afterwards should be a function that allows us to sample from this distribution we're approximating p data um after training. Okay. Um that brings us to the end of the first chapter, like kind of the introduction. Um I think what I will do is cut it off here. Um, the next portion of the text gets into flow modeling and then the algorithm behind flow modeling. Um and then there's a discussion over extension into stochastic processes for diffusion models um and the algorithm behind those. Uh so to avoid going too long, because I feel like if I get in chapter two, it'd be weird to cut off halfway through the flow modeling. So I'll just keep that as its own self-contained episode. So I'll just stop this episode here, and then in the next part, we'll pick up um with a description, uh, starting with a flow model, which is kind of like the easier uh of the diffusion types, uh, and then we'll extend that using some stochastic noise into a stochastic differential equation, which we use to model our uh true diffusion model, um, unconditioned generation, unconditional generation model. So uh yeah, that's why I'll leave things for now. I'll put a link to all this course material in the description attached to the episode. Um, and I'll try to be better about posting these more regularly. Uh, if you have any questions, feel free to email me, aaron.mcclindon at architect ai.com, also in the description. Um, and other than that, hope everyone has a nice Fourth of July weekend.