Cloudflare ran Anthropic’s Mythos Preview against 50+ internal repos and details what security-focused LLMs can and cannot do at scale.
Key Takeaways
Mythos Preview’s standout capability is exploit chain construction: chaining low-severity bugs into a single working, proven exploit via iterative proof generation.
Generic coding agents fail at vulnerability research due to context window limits and single-stream throughput; purpose-built harnesses are required for meaningful coverage.
A four-stage harness (Recon, Hunt, adversarial review, deduplication) with narrow-scoped parallel agents substantially reduces signal-to-noise versus a single exhaustive agent.
Mythos Preview’s organic refusals are inconsistent: the same task framed differently or run again can produce opposite outcomes, making them unreliable as a safety boundary.
C/C++ codebases produce more false positives; model bias toward hedged findings compounds triage cost unless a PoC is attached.
Hacker News Comment Review
Commenters broadly criticized the post for vagueness: no raw numbers on findings, no severity breakdown, no false-positive rates, making the piece feel like a promotional recap of the Mythos announcement rather than independent analysis.
The “four lessons” section drew skepticism; three of the four were seen as restatements of the obvious narrow-scope principle, with only adversarial review noted as operationally distinct.
Several commenters flagged AI-written prose style and questioned whether the post itself was LLM-generated, which undercut trust in the findings it reported.
Notable Comments
@wslh: Points to XBOW’s independent Mythos evaluation as a more data-rich companion read.
@krupan: Notes the irony that Cloudflare’s proposed fix for AI underperformance is adding more AI tooling on top.
