DreamHost silently added a default agents.txt to existing hosted sites on May 7, using a proposed spec that was already renamed agent-manifest.txt.
Key Takeaways
The file sets Allow-Training: no, Allow-RAG: yes, Allow-Actions: no plus standard directory disallow rules for admin, config, and wp-admin paths.
DreamHost deployed it retroactively to sites without notice, which is the core complaint – existing site owners had no say in the opt-in to RAG access.
The spec DreamHost implemented had already been renamed from agents.txt to agent-manifest.txt shortly after its March proposal, making this a stale implementation on day one.
Defaults are reasonable for a blank new site; the problem is applying them to established sites with unknown existing policies.
Hacker News Comment Review
Commenters are skeptical agents.txt provides real protection: major labs (Anthropic, OpenAI) respect robots.txt for named crawlers like ClaudeBot and GPTBot, but Common Crawl and unnamed scrapers are unaffected.
The Allow-Training vs Allow-RAG distinction in the default file is seen as the genuinely useful structural contribution, even if enforcement is near zero today.
No consensus on whether hosts pushing default AI policy files helps or creates false confidence for site owners who assume compliance follows.
