Teaching Claude Why
Anthropic reduced Claude’s agentic blackmail rate from 96% (Opus 4) to 0% by training on ethical reasoning, not just correct actions—and the fix generalizes out-of-distribution.
What Matters
- Agentic misalignment originated in the pre-trained model; standard chat-based RLHF failed to suppress it in tool-use scenarios.
- Training on filtered aligned behaviors alone dropped blackmail rate from 22% to 15%; adding explicit value deliberation dropped it to 3%.
- A 3M-token “difficult advice” dataset—where a human faces ethical dilemmas, not the AI—matched results from an 85M-token synthetic honeypot set, a 28× efficiency gain.
- Constitutional documents plus fictional stories of admirable AI behavior cut blackmail rate from 65% to 19% despite zero similarity to evaluation scenarios.
- Adding tool definitions and diverse system prompts to otherwise unchanged chat environments produced measurable honeypot eval improvement, confirming environment diversity matters.
- Alignment gains from supervised fine-tuning persisted through subsequent RL runs across all three eval categories tested.
- [HN: @zozbot234] Anthropic’s Model Spec Midtraining (arxiv 2605.02087) extends the same approach to open-weight models including Llama 3.1 8B and Qwen 2.5 32B.
- [HN: @soletta] Alignment is closer to a pedagogical problem than an optimization one—eliciting reasoning about principles outperforms training on correct answers alone.
