🧠 Generalist Reward Modeling: Inference, Generation, and Scaling

See related Substack to go deeper. 

 The episode unpacks the paper "Inference Time Scaling for Generalist Reward Modeling" from Deep Seek AI and Tsinghua University, revealing a critical innovation in AI development that's flying under most people's radar.

Beyond the jargon lies a revolutionary concept: rather than just making AI models bigger, researchers have discovered more efficient ways to improve AI performance by enhancing how models evaluate their own outputs in real-time. The hosts expertly translate complex technical concepts into digestible explanations, comparing the process to getting multiple medical opinions or teaching a child with consistent feedback.

The research introduces "Generative Reward Modeling" (GRM) and "Self-Principled Critique Tuning" (SPCT) - approaches that enable AI to provide detailed textual evaluations of responses rather than simple numerical scores. More impressively, the DeepSeq GRM model outperformed much larger systems while using computational resources more efficiently.

What makes this episode particularly valuable is how it connects technical AI research to broader questions about evaluation, judgment, and learning - both for machines and humans. As AI continues revolutionizing industries and daily life, understanding these fundamental improvements in AI reasoning capabilities gives listeners crucial context for navigating our increasingly AI-augmented world.

Inference-Time Scaling for Generalist Reward Modeling: Deep Seek

This is Heliox: Where Evidence Meets Empathy

Independent, moderated, timely, deep, gentle, clinical, global, and community conversations about things that matter.  Breathe Easy, we go deep and lightly surface the big ideas.

Thanks for listening today!

Four recurring narratives underlie every episode: boundary dissolution, adaptive complexity, embodied knowledge, and quantum-like uncertainty. These aren’t just philosophical musings but frameworks for understanding our modern world. 

We hope you continue exploring our other podcasts, responding to the content, and checking out our related articles on the Heliox Podcast on Substack. 

Transcript

0:25

Ever get that feeling like you're drowning in information, articles, studies, reports, all saying different things. Oh, absolutely. Like trying to drink from a fire hose. Right. You just weren't the good stuff, the core insights. The stuff that matters. That's where we come in. Welcome to the deep dive. We take those mountains of data and we distill them down into those precious drops of understanding. Your shortcut to being seriously in the know. Today. We're going deep on artificial intelligence. Specifically, how to make those AI brains, those language models, even smarter. And you've brought some cutting edge research to this. the table. Straight from the labs. A paper called Inference Time Scaling for Generalist Reward Modeling. From the folks at BeepSeek AI and Tsinghua University. Now, the title might sound a little intimidating. A bit jargony. Stick with us. The idea at its heart. Pretty mind-blowing. It is. Really gets you thinking about the future of AI. Our mission, if you choose to accept it, is to crack this research wide open. Understand how they're making AI reward systems more accurate. more adaptable. And the really cool part, they're focused on improving the AI while it's working, not just during its initial training. On the job learning for AI. Love it. Okay, so first things first. These AI language models, the ones that can hold conversations, write stories, all that, how do we even improve them once they've learned the basics? Well, think of it like teaching a kid, right? They try something new, you give them feedback. A little praise, a little guidance. Exactly. And in the AI world, we call this reinforcement learning, or RL for short. So it's all about creating this feedback loop. So where does reward modeling come in? What even is that? Reward modeling. That's how we define what good looks like for the AI. It's the system that tells the language model, hey, that was a good response. You get a reward. Guiding it towards better answers. Yeah. Got it. But the paper you brought mentions it's tricky to create these reward signals for general everyday questions, unlike... say, a math problem where there's a clear right answer. Oh, it's a huge challenge. You see, for most real-world questions, what makes a good answer is pretty subjective. Depends on who's asking, what they need, all sorts of things. It's nuanced. Very much so. Unlike a simple equation, evaluating a complex explanation or a creative piece of writing, it's not black and white. So if we want our AIs to be truly helpful, able to handle all sorts of topics and conversations. We need to up our game in generalist reward modeling, which is exactly what this paper tackles. They make a point about not just training the AI more, but also using more inference compute. What is that exactly? Inference compute. That's the computing power the AI uses when it's actually generating responses for you. The resources it has on tap to think and answer your questions. So instead of just making the AI bigger, they're suggesting we use more computing power while it's answering questions to get better feedback. Precisely. And the key here is finding that sweet spot, making reward models that are general. They can handle any input and scalable. They get better with more computation without breaking the bank. Of course, nobody wants a million dollar AI assistant. That makes sense. So how does this research propose to do that? Enter generative reward modeling or GRM. What's the gist of it? Right. Well, they specifically focus on point-wise generative reward modeling, which sounds complex, but the magic is that it can assess different types of responses using language. Language is the evaluation tool. That's fascinating. How does that actually work? So imagine instead of just giving a number score, like a 7 out of 10, the AI writes an actual critique. scalar models, which just spit out a number, and pairwise models, which only compare two answers. Those are limited. They can't really get more nuanced, even with more processing power. With GRM, we get this richness, this depth of understanding. Like getting feedback from a teacher who can actually explain why you did well or not so well. Makes sense. The paper also talks about the power of principles. What's that all about? How do principles factor into making these reward models better? Ah, good catch. Now, this is where things get really interesting. Because in many real-world situations, there isn't a simple rulebook for what makes a perfect answer. So they explored using principles to guide the AI. Like guidelines for how to evaluate its own work. Exactly. And you might have heard of constitutional AI. It's a related idea where humans set predefined principles to guide the AI's behavior. To keep it on the straight and narrow, so to speak. Right. But in this research, they're exploring how these principles can be used specifically within reward modeling to improve those AI critiques. And they ran some experiments to see how much these principles actually help. What did they find? Well, they first tried using principles that the AI generated itself. No human intervention. Didn't really move the needle. The AI self-generated principles were a bit all over the place. Not very helpful. So like a student grading their own homework? Yeah. Probably not the most reliable system. Haha, exactly. But then they tried something different. They curated, very carefully, a set of principles, aligned them with what we humans would consider good answers. And that made a huge difference. The right principles, the right criteria are key. Absolutely. And this realization led them to what they call self-principled critique techniques. tuning or SPCT for short, their secret sauce for making GRMs even more powerful. SPCT, it sounds very technical, but the concept behind it is actually quite clever. It is. It's like teaching the AI to be a better self-critic, a more reliable judge. And they achieve this through a two phase process. They phase it. Yep. The first is rejective fine tuning. Kind of a mouthful, but think of it as boot camp for the GRM, getting it ready to generate those critiques and principles. in the right format for all sorts of different inputs. So getting the basics down, like teaching someone the proper form for throwing a dart before they even aim at the board. Perfect analogy. They start by generating tons of responses from pre-trained GRMs, but then comes the rejection part. They toss out any examples where the AI's predicted reward doesn't match up with the actual best answer. Making sure it learns from its mistakes. Precisely. And they also discard the really easy question. the ones the AI consistently gets right. No point in wasting time on those. Focus on the areas where the AI struggles. Makes sense. The paper also mentioned something called hinted sampling during this first phase. What's that all about? Right. Sometimes to give the AI a nudge in the right direction, they would provide a hint. Like a good response would include X, Y, and Z just to make sure it's on the right track. Subtle guidance. Okay, so that's phase one. What's the second part of SPCT? Okay. Phase two is where the real magic happens. It's called rule-based online RL. This is where the AI really hones its evaluation skills by continuously comparing its judgments to the actual correct answers. So it's like the AI is constantly practicing judging and getting feedback on its performance, trying to become a better critic. Exactly. And what's really interesting is they don't reward the AI for getting the format of those critiques and principles exactly right. Instead, they penalize it heavily if it deviates too much from its previous behavior. Keeping it consistent. So all this SPCT training is great, but how does it actually help the AI perform better when we give it more... inference compute, that computing power we talked about earlier, that was a primary goal of the research. Absolutely. And this is where it all comes together. SBCT essentially trains the GRM to take advantage of that extra computing power. One of the key techniques they use is voting. Voting, like in an election. Kind of. They essentially have the AI evaluate the same response multiple times. Each time, it might generate a slightly different set of principles and critiques, leading to slightly different reward scores. So instead of just one opinion, we get multiple perspectives. Exactly. And then by aggregating these different scores, taking a vote, they create a more robust and nuanced evaluation. Like getting a second or even third opinion from a doctor. You feel more confident in the diagnosis. Exactly. And to make things even better, they introduce a meta RM. Meta RM. Another layer of complexity. Don't worry. It's not as complicated as it sounds. This meta RM is like a supervisor. Another AI specifically trained to judge the quality of the principles and critiques generated by the main GRM. So it's like an AI judging the reasoning of another AI. Precisely. Precisely. During that voting process, the meta-RM helps decide which critiques to give more weight to, filtering out the less reliable ones. This ensures that using more compute leads to truly better evaluations, not just more of them. So did all these fancy techniques actually work? Could SPCT and this inference time scaling deliver? The results are very encouraging.

They tested their DeepSeq GRM model on various reward modeling benchmarks: 9:33

reward bench, PPE, RMB, and real mistake. These benchmarks evaluate different aspects of how well the AI can judge responses. Drumroll, please. Their model, DeepSeq GRM-27B, outperformed existing methods on most of these benchmarks. Marks. Impressive. And get this, it even achieved comparable results to much bigger models like Nemotron 4, 340B Reward, and GPT-40, despite being significantly smaller. So quality over quantity, in a way. You could say that. And remember that inference time scaling, they saw big gains there, too. As they increased the number of samples used for voting, the performance of DeepSeq GRM just kept getting better, and the MetARM played a key role in that improvement. It's like the more it thinks, the better its judgments become. Yeah. The paper also mentioned some ablation studies. Could you explain what those are and what they revealed in this case? Sure. Imagine you have a complex machine and you want to know which parts are essential. So you systematically remove one component at a time and see how the machine performs without it. That's what an ablation study is. Makes sense. In this case, they found that both the online RL phase and the principal generation were absolutely crucial for achieving those strong results. Yes. Removing either component led to a significant drop in performance. So every piece of the puzzle matters. Now, this is a really cool finding. The paper suggests that using more inference time scaling with DeepSeat GRM could be just as effective or even more effective than simply using a much larger reward model. That's a game changer. It suggests that maybe instead of just throwing more data and bigger models at the problem, we can achieve better results by being smarter about how we use computation during the evaluation process itself. Using our resources more efficiently. So this research seems incredibly promising. Yeah. But every scientific breakthrough has its limitations. And it sounds like they might not be quite as good as those simpler models on very specific, easily verifiable tasks, like fact checking, maybe? That's a good point. They do acknowledge a slight underperformance there, but they also highlight the known biases and limitations of those scalar models. They suggest that GRMs have more potential for improvement and could catch up, especially if we incorporate things like reference information into the reward generation process.

They also mentioned a very intriguing possibility: 12:08

using DeepSeq GRM as a process RM. Instead of just evaluating the final answer, we could actually look at how the AI arrived at that answer. Oh yeah, that's exciting. Since GLMs generate those textual critiques, we can analyze each step in the AI's reasoning not just the end result so we can see how it thinks not just what it thinks potentially really helpful in complex tasks where the thought process is just as important as the answer exactly and they also highlighted a bunch of other promising future research directions like integrating external tools with GRM so they can access information from the real world while evaluating like giving the AI a calculator or dictionary to use during its evaluation exactly And they're also looking at potentially separating the principal generation from the critique generation, which could make the whole process more efficient. Plus, they see potential in using DeepSeq GRM for offline evaluation to better understand the AI's weaknesses and how to address them. So much potential. So to wrap things up, what's the key insight you think our listeners should take away from this deep dive into cutting edge AI research? Well, first and foremost, this research is a big leap forward in creating more sophisticated, adaptable AI reward systems. The SPCT method, combined with that clever use of inference time scaling, opens up a whole new way to think about how we train and improve these powerful language models. And the biggest aha moment for me is the idea that maybe bigger isn't always better. We can potentially achieve better results by being smarter about how we use computing power during the AI's thinking process, not just by throwing more data and bigger models at the problem. Absolutely. It's about working smarter, not harder. And this understanding of how reward modeling is evolving is key to navigating the ever-changing landscape of AI. It gives you that deeper understanding, that shortcut to being informed about where this technology is heading. So here's a final thought to ponder as you go about your day. If we can teach AI to generate these dynamic, adaptive principles to judge its own work, what can that tell us about how we humans evaluate complex situations? About how our own internal reward systems work? Now that's a question worth pondering. And if this deep dive has sparked your curiosity, definitely check out the full research paper. There's a whole world of fascinating details waiting to be explored. And who knows what new insights await us in the future. Keep those minds hungry, folks, and keep on learning.