Francois Chollet — Why the biggest AI models can't solve simple puzzles

Francois Chollet argues LLMs are sophisticated memorization engines, not intelligence, and launches a $1M ARC-AGI Prize to prove it.

• Chollet’s core claim: scaling LLMs increases skill, not intelligence — intelligence requires on-the-fly program synthesis from novel inputs, not pattern retrieval.
• LLMs can solve Caesar cipher for transposition n=3 or n=5 (common in training data) but fail at n=9, proving memorization of specific cases, not generalized algorithms.
• Jack Cole’s 240M-parameter model reached 35% on ARC only by adding test-time fine-tuning; without it, performance drops to 1-2%.
• Amazon Mechanical Turk workers scored ~85% on ARC — that 85% is the prize threshold; $500K goes to first team reaching it.
• $1M+ prize pool: $500K for hitting 85%, $100K progress prize split between top Kaggle leaderboard scores ($50K) and best explanatory paper ($50K).
• Chollet blames OpenAI for setting back AGI progress 5-10 years by closing frontier research publishing and flooding resources into LLM monoculture.
• Best path to ARC solution is a hybrid: deep-search discrete program synthesis (small DSL, ~100-200 primitives) combined with LLM-style learned building blocks — neither extreme alone is sufficient.
• Prize requires winners to release solutions in public domain; contest repeats annually until 85% threshold is hit with a reproducible open-source method.

2024-06-11 · Watch on YouTube