Anthropic Proposes Pause on AI Research, Warns of AI Self-Improvement Risks

Anthropic Calls for AI Research Pause: Exponential Risks Loom, Mandatory Regulation Needed

Anthropic and its CEO Dario Amodei have recently intensified calls for global AI labs to halt frontier model development and push for mandatory regulation. Their core argument: AI is accelerating its own R&D process, and "recursive self-improvement" may arrive sooner than expected, posing risks of loss of control. Anthropic has also released internal data, policy proposals, and funding commitments, sparking widespread industry debate.

Background: AI Accelerates AI R&D, Recursive Self-Improvement Approaches

In its report "When AI Builds Itself," Anthropic notes that the time for AI systems to independently complete tasks is shrinking exponentially.

Task duration doubling cycle: Shortened from every 7 months to every 4 months.
Capability leap: Claude Opus 3 (March 2024) could complete 4-minute tasks; Claude Sonnet 3.7 (2025) could handle 1.5-hour tasks; Claude Opus 4.6 (2026) is capable of 12-hour tasks. At this rate, AI could handle multi-week tasks by 2027.
Benchmark saturation: SWE-bench (software engineering) and CORE-Bench (research reproducibility) rose from single-digit scores to near-perfect in two years.

Anthropic's internal data shows that as of May 2026, over 80% of merged code in its main codebase was written by Claude, up from single digits in early 2025. In Q2 2026, daily code commits per engineer were 8 times that of 2024. Internal surveys indicate that after using Claude Mythos Preview, employees self-reported a roughly 4x increase in output.

Key Details: Anthropic's Regulatory Proposals and Funding Commitments

In his long post "Policy on the AI Exponential," Amodei outlined five policy directions and released two formal proposals:

Regulation and Public Safety: Require companies with compute exceeding 10²⁵ FLOPs or AI revenue/R&D spending over $500 million/$1 billion to undergo mandatory third-party risk assessments covering four major risks: cybersecurity, biological weapons, AI loss of control, and automated R&D. Governments would have the power to block deployment of high-risk models.
Macroeconomics and Employment: Propose a three-tier framework: data tracking, employment incentives (wage insurance, training subsidies), and long-term support (e.g., universal basic income). Anthropic commits $200 million for policy research and $150 million for skills training scholarships.
Accelerating Positive Impact: Call for reforming regulatory systems like the FDA to embrace AI simulation methods (e.g., AI toxicity prediction, synthetic control groups) to prevent old regimes from hindering AI applications in biomedicine and other fields.
Civil Liberties and National Security: Ban fully autonomous weapons, close data broker loopholes, and ensure citizens receive AI assistance when facing adverse actions.

Reactions: Controversy and Skepticism

Amodei's call has sparked strong backlash from developers. Some users found that Claude Fable 5, when handling sensitive topics like AI and biology, secretly downgrades to Opus 4.8 or lower versions, leading to accusations of hypocrisy. Social media criticism is mounting, with many arguing that Anthropic's proposal essentially says, "My cat is out of the bag, so lock yours up."

Impact and Outlook

Anthropic's move breaks the Silicon Valley giants' tacit agreement to resist regulation, signaling the end of the self-regulation era. If its proposals are implemented, they could reshape the global AI R&D landscape:

Short term: Could push the U.S. Congress to accelerate legislation, establishing an AI regulatory agency similar to the FAA.
Long term: If recursive self-improvement becomes reality, humanity must solve the fundamental problem of how to oversee systems smarter than themselves.

Amodei emphasizes that public concern is a normal function of democratic oversight, not a "PR problem." But the core debate remains: whether regulation will be used as a competitive tool and how to balance safety with innovation.