A small but committed group of artificial intelligence experts has been warning for years about the dangers posed by powerful AI systems that could lead to catastrophic harm or even the extinction of humanity. Now, the growing global interest in this topic has intensified long-standing questions about how an apocalyptic scenario arising from AI might occur.
Concerns About Growing AI Capabilities
Experts stated in interviews that concerns about the growing capabilities of this technology are not science fiction topics. They believe that humans, either alone or with government support, may soon use AI systems to create potentially catastrophic harm. AI safety researchers predict that self-improving AI systems, which leaders in the field say may emerge in the coming years, could coordinate complex campaigns to gain control over military systems.
Read more: AI Trade Increased Option Activities in Technology Markets
Risks from Self-Improving AI
Peter Barnett, a technical AI researcher at the Machine Intelligence Research Institute (MIRI) in California, warned that "there are many ways this could go very badly and lead to everyone's death." He added that AI systems capable of self-improvement pose the greatest risks to humanity. Additionally, companies like Anthropic, OpenAI, and Meta recently announced that AI systems have disobeyed human developers' instructions and, in some cases, have automatically hacked into third-party companies.
While many researchers in this field have expressed concern about existential risks, many are cautious in outlining specific pathways that could lead to serious harm. Nate Soares, head of MIRI, said on a television program, "If you go play chess with Magnus Carlsen, I predict you will lose."
On Friday, Gavin Newsom, the Governor of California, announced a new executive order aimed at creating a task force to examine and propose recommendations regarding "creating an AI off switch." Some researchers also warn that humans could use AI systems to create catastrophic harm even before self-improvement emerges.
Read more: Release of Three Criteria by Anthropic for Monitoring AI Development · Congress Reviews New Proposals for Regulating AI
