LessWrong (Curated & Popular)
Audio narrations of LessWrong posts. Includes all curated posts and all posts with 125+ karma.
If you'd like more, subscribe to the “Lesswrong (30+ karma)” feed.
LessWrong (Curated & Popular)
"We need 3rd party Training-Run Assessments" by Alex Meinke
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Training-run assessments conducted by a 3rd party should become a standard part of frontier AI safety.
By a Training-Run Assessment, or TRA, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRA can look at intermediate checkpoints, training rollouts, RL environments, reward signals, SFT datasets, and the process by which the developer responded to warning signs.[1]
In this post I will argue that:
Detecting Scheming may require Training-Run Assessments
By scheming I mean an AI covertly pursuing misaligned goals while deliberately concealing its intentions or capabilities from its developers. I restrict attention to “coherent” forms of scheming where the model pursues somewhat stable misaligned goals across context windows, rather than misalignment that surfaces only as isolated, context-dependent defections. [...]
---
Outline:
(01:23) Detecting Scheming may require Training-Run Assessments
(03:55) Why 3rd parties should perform Training-Run Assessments
(04:12) Developers may lack incentives to adequately assess scheming
(04:49) Developers' safety assessments lack credibility
(05:31) External evaluators can bundle expertise for assessing scheming
(06:17) 3rd party TRAs can be developed gradually
(08:50) Checkpoint evals
(08:54) What?
(10:30) How?
(11:15) Data inspections
(11:19) What?
(12:03) Why?
[... 16 more sections]
---
First published:
July 5th, 2026
Source:
https://www.lesswrong.com/posts/3HvvjffA65mHLwaWm/we-need-3rd-party-training-run-assessments
---
Narrated by TYPE III AUDIO.
---
By a Training-Run Assessment, or TRA, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRA can look at intermediate checkpoints, training rollouts, RL environments, reward signals, SFT datasets, and the process by which the developer responded to warning signs.[1]
In this post I will argue that:
- Final-checkpoint evaluations will be insufficient to assess scheming risks.
- TRAs can be more effective at detecting scheming.
- Frontier developers should involve third parties to do TRAs or verify safety claims by the developers.
Detecting Scheming may require Training-Run Assessments
By scheming I mean an AI covertly pursuing misaligned goals while deliberately concealing its intentions or capabilities from its developers. I restrict attention to “coherent” forms of scheming where the model pursues somewhat stable misaligned goals across context windows, rather than misalignment that surfaces only as isolated, context-dependent defections. [...]
---
Outline:
(01:23) Detecting Scheming may require Training-Run Assessments
(03:55) Why 3rd parties should perform Training-Run Assessments
(04:12) Developers may lack incentives to adequately assess scheming
(04:49) Developers' safety assessments lack credibility
(05:31) External evaluators can bundle expertise for assessing scheming
(06:17) 3rd party TRAs can be developed gradually
(08:50) Checkpoint evals
(08:54) What?
(10:30) How?
(11:15) Data inspections
(11:19) What?
(12:03) Why?
[... 16 more sections]
---
First published:
July 5th, 2026
Source:
https://www.lesswrong.com/posts/3HvvjffA65mHLwaWm/we-need-3rd-party-training-run-assessments
---
Narrated by TYPE III AUDIO.
---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.