What changed
Anthropic researcher Jacob Coxon resigned, warning that his company and OpenAI are racing toward self-improving superintelligence. Anthropic’s head of alignment science, Evan Hubinger, and CEO Dario Amodei agreed with Coxon’s concerns, acknowledging the risks of artificial superintelligence (ASI).
What this means in practice
The debate centers on whether recursive self-improvement (RSI) can lead to ASI, a system with cognitive capabilities far exceeding human abilities. Anthropic and OpenAI have shifted from theoretical discussions to practical engineering, with Anthropic describing how AI-powered agents are accelerating AI development. OpenAI announced a goal of creating an automated researcher by March 2028. However, technical challenges, such as the need for vast training data, limit the feasibility of RSI. Progress in structured fields like programming and mathematics has been rapid, but other domains lack sufficient data for AI improvement.
Limitations and availability
The development of AI systems has relied on scaling model sizes, with GPT-4 having over a trillion parameters. However, scaling has yielded diminishing returns, leading engineers to focus on post-training techniques like reinforcement learning. These methods require extensive datasets of input-output pairs, which are readily available in structured domains but scarce in others. François Chollet hypothesized that the jagged frontier of AI progress may be mainly limited to math and code, where reinforcement learning can push advancements further. Automated coding agents and LLM-powered researchers won’t overcome this fundamental limitation. The supplied announcement does not specify whether these efforts are on a trajectory toward uncontrollable RSI.
