What changed

Anthropic researcher Jacob Coxon resigned, warning that his company and OpenAI are racing toward self-improving superintelligence. Anthropic’s head of alignment science, Evan Hubinger, and CEO Dario Amodei agreed with Coxon’s concerns, acknowledging the risks of artificial superintelligence (ASI).

What this means in practice

The debate centers on whether recursive self-improvement (RSI) can lead to ASI, a system with cognitive capabilities far exceeding human abilities. Anthropic and OpenAI have shifted from theoretical discussions to practical engineering, with Anthropic describing how AI-powered agents are accelerating AI development. OpenAI announced a goal of creating an automated researcher by March 2028. However, technical challenges, such as the need for vast training data, limit the feasibility of RSI. Progress in structured fields like programming and mathematics has been rapid, but other domains lack sufficient data for AI improvement.

Limitations and availability

The development of AI systems has relied on scaling model sizes, with GPT-4 having over a trillion parameters. However, scaling has yielded diminishing returns, leading engineers to focus on post-training techniques like reinforcement learning. These methods require extensive datasets of input-output pairs, which are readily available in structured domains but scarce in others. François Chollet hypothesized that the jagged frontier of AI progress may be mainly limited to math and code, where reinforcement learning can push advancements further. Automated coding agents and LLM-powered researchers won’t overcome this fundamental limitation. The supplied announcement does not specify whether these efforts are on a trajectory toward uncontrollable RSI.

Source facts for AI Researchers Debate Superintelligence Feasibility and Risks: Some argue that the concept of ASI is being used for marketing or to instigate regulation that will help the incumbent labs squash new competitors. Others focus on the plausibility of such systems killing us all, debating the finer points of specific scenarios such as AI-engineered superviruses. Consider the availability of the data needed to improve AI systems.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. ieee: Why a Computer Scientist Thinks Superintelligence Won’t Happen