OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil

OpenAI CPO Kevin Weil argues writing evals is the must-have skill for AI product builders, and today’s models are the worst you’ll ever use.

• OpenAI has 3 million API developers; Weil argues vertical-specific data moats are where startups can safely build without being squashed.
• Writing evals — model test suites — is becoming a core PM skill; model accuracy at 60% vs 99.5% requires completely different product architectures.
• GPT-3.5 API cost was ~100x what GPT-4o mini costs today; capability has risen while cost dropped two orders of magnitude in roughly two years.
• OpenAI uses model ensembles internally: o-series for reasoning-heavy tasks, 4o mini for fast cheap checks — framed as a company of specialized humans.
• Libra (Facebook crypto) is Weil’s biggest career disappointment; remittance fees of ~20% still exist while WhatsApp reaches 3 billion users who could transact for free.
• Deep Research can do a week of human research work in 25–30 minutes; evals were co-designed alongside the product to hill-climb on hero use cases.
• Weil sees personalized AI tutoring as one of the highest-impact unsolved problems — studies show multiple standard-deviation learning gains, ChatGPT is free, yet no at-scale product exists.
• Example-based prompting (include sample inputs and ideal outputs in the prompt) is Weil’s top practical tip for getting better model output without full fine-tuning.

2025-04-10 · Watch on YouTube