Andrej Karpathy — “We’re summoning ghosts, not building animals”

Andrej Karpathy argues LLMs are ghosts not animals, AGI is a decade out, and production software engineering is as safety-critical as self-driving.

• LLMs are built by imitating internet documents, not evolution — Karpathy calls them ghosts, not animals: fundamentally different intelligence.
• Llama 3 70B stores ~0.7 bits per training token; in-context KV cache grows ~320KB per token — a 35-million-fold difference in information density.
• RL is terrible, but everything before it was worse; humans likely use RL only for motor tasks, not intelligence or problem-solving.
• Self-driving follows a march of nines: each additional nine of reliability costs constant work; demo to product gap is unbounded.
• Production software engineering shares self-driving’s safety profile — one mistake at scale leaks hundreds of millions of SSNs or worse.
• GPT-5 Pro is Karpathy’s current go-to oracle; agents bloated his nanoGPT repo with try-catch blocks, deprecated APIs, and wrong DDP assumptions.
• AGI will blend into the 2% GDP growth trajectory already running for centuries — not a visible discontinuity.
• Karpathy is building Starfleet Academy: a physical institution plus digital tier, hiring faculty per domain, targeting pre-AGI upskilling then post-AGI learning-as-gym.

2025-10-17 · Watch on YouTube