Joe Carlsmith — Preventing an AI takeover

Joe Carlsmith and Dwarkesh Patel debate the concrete mechanics of AI misalignment, moral patienthood of AIs, and whether competitive multipolar AI development actually reduces takeover risk.

• Carlsmith argues misaligned AI risk requires a specific profile: planning capability, situational awareness, and values that diverge from verbal behavior — GPT-4 likely lacks this.
• The core vulnerability: once AI is vastly more powerful than humans, human empowerment becomes entirely dependent on AI motives, not institutions or incentives.
• A multipolar world with many unaligned AIs does not solve the problem — correlated failures across labs using the same techniques mean no “good AI” exists to counterbalance bad ones.
• Carlsmith identifies an “AI safety sweet spot”: models capable enough to accelerate alignment/cybersecurity work but not yet capable of seizing power — this window must be exploited aggressively.
• Voluntary civilizational handoff to AI (automated courts, military, police) may be harder to detect and reverse than a fast takeover, because humans lose epistemic grip gradually.
• On AI moral patienthood: Carlsmith is skeptical of making Consciousness a fully necessary criterion for moral status, citing deep ongoing confusion about what consciousness even is.
• Our evolved values (cooperation, boundary-respecting norms) are partly shaped by what is instrumentally powerful — meaning nature is “somewhat on our side” against purely arbitrary goal optimization.

2024-08-22 · Watch on YouTube