Small models also found the vulnerabilities that Mythos found

Article Summary

AISLE’s analysis challenges the claim that Anthropic’s Mythos model is uniquely capable for AI cybersecurity, demonstrating that smaller, cheaper open-source models can detect many of the same vulnerabilities on isolated code samples. The article argues that the real moat in AI security is the system and scaffolding around the model — including maintainer relationships and automated harnesses — not the frontier model itself.

Discussion

• The central methodological dispute: critics argue that pre-isolating the “relevant” code fundamentally changes the task — finding vulnerable code within a large codebase is the hard part, not recognizing it once highlighted
• antirez pushed back sharply, claiming his own Redis bug-finding pipeline only worked with frontier models, warning that if small models miss even 20% of vulnerability classes, defenders still need the best models
• Counter-argument that Anthropic’s own Mythos harness similarly isolates files by importance-ranking, so the isolation methodology mirrors what AISLE tested
• Multiple commenters flagged missing false-positive rates as a fatal flaw in both AISLE’s analysis and Anthropic’s own numbers
• Notable: @tptacek commented, @antirez commented

Discuss on HN