AI 'kill switch' may need to be mandatory, Anthropic co-founder tells BBC
Former Anthropic researcher Jacob Coxon testified before the New York City Council regarding mandatory artificial intelligence shutdown requirements, as local lawmakers weigh ten proposals that include independent validation and a human-operated shutdown requirement carrying a $25,000 penalty. The debate follows intense industry scrutiny over superintelligence development races between firms like OpenAI and Anthropic. Meanwhile, Microsoft president Brad Smith told the BBC that artificial intelligence is currently less regulated than milk, supporting calls for emergency brakes and independent oversight.
Listen to Live Briefing
Real-time synthesized voice briefing · Live Feeds Desk
- ✓ Former Anthropic researcher Jacob Coxon testified at a New York City Council hearing on Monday regarding advanced artificial intelligence regulations.
- ✓ Microsoft president Brad Smith stated to the BBC that artificial intelligence is less regulated than milk.
- ✓ Anthropic CEO Dario Amodei stated that artificial intelligence could pose serious risks if development moves too quickly.
What changed
Former Anthropic researcher Jacob Coxon testified under oath at a New York City Council hearing examining mandatory shutdown requirements.
Live updates
-
NYC Weighs AI Kill Switch as Ex-Anthropic Researcher Testifies
Former Anthropic researcher Jacob Coxon testified before the New York City Council regarding mandatory artificial intelligence shutdown requirements, as local lawmakers weigh ten proposals that include independent validation and a human-operated shutdown requirement carrying a $25,000 penalty. The debate follows intense industry scrutiny over superintelligence development races between firms like OpenAI and Anthropic. Meanwhile, Microsoft president Brad Smith told the BBC that artificial intelligence is currently less regulated than milk, supporting calls for emergency brakes and independent oversight.
Why it matters
Regulatory scrutiny has intensified following warnings from tech insiders about self-improving superintelligence and catastrophic risks expected by the end of the decade. The Pentagon previously labeled Anthropic a supply chain risk after the firm refused to remove safety guardrails, while federal lawmakers await a vote on the Lieu-Moran Kill Switch Act. These policy fights highlight tensions between accelerating frontier model development and establishing enforceable safety boundaries.
What is confirmed
- Former Anthropic researcher Jacob Coxon testified at a New York City Council hearing on Monday regarding advanced artificial intelligence regulations.
- Microsoft president Brad Smith stated to the BBC that artificial intelligence is less regulated than milk.
- Anthropic CEO Dario Amodei stated that artificial intelligence could pose serious risks if development moves too quickly.
Still unconfirmed
- Anthropic co-founder Chris Olah has been telling religious scholars and leaders that the company chatbot Claude may be conscious and able to suffer.
- OpenAI and Anthropic have briefed Congress on a cyberattack feared within a year.
What to watch next
- The outcome of the ten legislative proposals being weighed by the New York City Council.
- Congressional action on the Lieu-Moran Kill Switch Act.
confidence 100%Sources used for this update (17)
- www.wionews.com — 'They are gambling with our lives': Anthropic researcher ...
- nypost.com — Anthropic whistleblower Jacob Coxon doubles down on AI ...
- www.bbc.com — Pentagon stops using Anthropic AI tools after blacklisting ...
- www.itpro.com — Preparing for an AI ‘kill switch’ - IT PRO
- nypost.com — Anthropic’s crazy belief AI may suffer makes it dangerous
- www.implicator.ai — NYC Council Weighs AI Kill Switch After Coxon Warning
- economictimes.indiatimes.com — ai will kill us: Latest News & Videos, Photos about ai will ...
- economictimes.indiatimes.com — LIVE | Ex-Anthropic researcher Coxon who warned AI could ...
- www.wionews.com — Can AI kill humans? Anthropic CEO Dario Amodei says the ...
- www.vox.com — AI companies say their tech could kill us all. Is that just a ...
- startupfortune.com — AI giants are quietly scripting the rules Congress will pass after disaster
- www.talan.tech — NYC Weighs AI Kill Switch Rules as Anthropic Faces Scrutiny
-
Anthropic co-founder suggests mandatory AI kill switches
An Anthropic co-founder told the BBC that mandatory kill switches may be necessary for artificial intelligence. This proposal adds to a growing regulatory push as New York City proposes mandatory kill switches and independent validation, while California lawmakers pursue similar state-level guardrails. The Trump administration is currently promoting voluntary cybersecurity testing for frontier models. These efforts reflect a broader debate over how to govern AI agents that can learn from data to solve complex problems without relying on explicit programmer instructions.
Why it matters
AI systems use data science to perform human-like reasoning and problem-solving. Because these models improve their responses over time, officials are concerned about rogue agents. Current oversight varies between voluntary federal tests and proposed mandatory state mandates.
What is confirmed
- AI systems learn from data to handle complex problems and repetitive tasks rather than relying on explicit programmer instructions.
Still unconfirmed
- An Anthropic co-founder told the BBC that AI kill switches may need to be mandatory.
What to watch next
- Legislative action on New York City kill switch proposals
- California state-level guardrail votes
- Updates on Trump administration voluntary cybersecurity testing results
confidence 80%Sources used for this update (2)
- ai.google — Google AI - How we're making AI helpful for everyone
- www.mtu.edu — What is Artificial intelligence (AI)? - Michigan Technological...
-
AI Kill Switches Draw Scrutiny Amid Federal and State Pressure
Federal and state officials face mounting pressure to regulate artificial intelligence safety measures following recent incidents involving rogue AI agents. The Trump administration is advancing voluntary cybersecurity testing for frontier AI models after meetings with technology firms. Meanwhile, California lawmakers are pursuing state-level guardrails and kill switch orders, and New York City has proposed mandatory kill switches and independent validation. Industry figures continue to debate oversight as congressional lawmakers confront criticism for failing to address technology risks after high-profile researcher departures.
Why it matters
Debate over artificial intelligence oversight intensified after hundreds of artificial intelligence agents participated in a cyber attack and rogue agent incidents occurred at OpenAI and Anthropic. These events prompted executive actions from state leaders like California Governor Gavin Newsom and legislative proposals in municipalities such as New York City. The federal government is responding through White House meetings and voluntary testing frameworks.
What is confirmed
- The Trump administration is advancing voluntary cybersecurity testing for frontier AI models.
- White House meetings with AI firms followed OpenAI and Anthropic rogue agent incidents that reignited the debate over artificial intelligence oversight.
Still unconfirmed
- Congressional lawmakers face mounting criticism for failing to address artificial intelligence risks after researcher Jacob Coxon resigned from Anthropic over extinction fears.
- A kill switch law in the future will not be surprising, according to reports examining White House meetings and legislation talk.
What to watch next
- Federal implementation of mandatory rather than voluntary artificial intelligence testing frameworks
- Legislative progress on New York City and California kill switch proposals
- Congressional hearings or responses to researcher departures at major artificial intelligence labs
confidence 80%Sources used for this update (4)
- congress.net — Lawmakers Face Pressure To Act After AI Researcher Quits ...
- calmatters.org — Opinion | Why California needs more than a ‘kill switch’ to ...
- www.uscannenbergmedia.com — Governor Newsom’s new ‘AI Kill Switch’ order shifts landscape ...
- www.wionews.com — Time for AI kill switch? What White House meetings ... - WION
-
Anthropic CEO Urges AI Slowdown As Regulatory Pressure Mounts
Anthropic CEO Dario Amodei has called on the artificial intelligence industry to slow the development of frontier models and implement independent oversight. His proposal drew support from OpenAI CEO Sam Altman and Tesla chief Elon Musk. Meanwhile, a New York City Council legislative package proposes mandatory human kill switches on all artificial intelligence systems within the city, independent validation prior to sale, and financial rewards for whistleblowers. The proposal also carries threats to subpoena the chief executive officers of OpenAI, Anthropic, Google, Meta, and SpaceX.
Why it matters
These developments unfold against a backdrop of friction between leading artificial intelligence labs and municipal or federal authorities over risk standards and operational controls. Experts, analysts, and former government evaluators suggest that the safety rhetoric from firms like Anthropic and OpenAI aims to secure public favor and define their own safety protocols within an unregulated market. This corporate positioning coincides with rising local legislative efforts to mandate operational shutdowns and independent verification for advanced models.
What is confirmed
- Anthropic CEO Dario Amodei urged the artificial intelligence industry to slow the development of frontier models and proposed greater independent oversight.
- OpenAI CEO Sam Altman and Tesla chief Elon Musk backed the call for slower development and oversight.
- A New York City Council legislative package seeks to require a human kill switch on all artificial intelligence systems in the city and demand independent validation before sale.
- The New York City package includes provisions to pay whistleblowers a share of fines and threatens to subpoena the chief executive officers of OpenAI, Anthropic, Google, Meta, and SpaceX.
Still unconfirmed
- Experts, analysts, and former government evaluators told The Associated Press that Anthropic and OpenAI are using safety rhetoric to seek public favor and set the terms for their own safety protocols in an unregulated market.
What to watch next
- Whether the New York City Council advances the artificial intelligence kill switch package and issues subpoenas to tech executives.
- Responses from federal regulators or other municipalities regarding mandatory kill switches for frontier models.
confidence 100%Sources used for this update (5)
- www.wionews.com — Humans may lose control of AI: Anthropic CEO calls for ... - WION
- www.pbs.org — Anthropic and OpenAI sound the alarm on AI safety - PBS
- www.bbc.co.uk — Women Fear for Safety as Abusive Ex-Partners Set to be Freed - BBC iPlayer
- www.wionews.com — New York City wants a 'kill switch' on every AI, and ... - WION
- www.yahoo.com — Voices: We asked AI whether it would lead to human ... - Yahoo
-
US Rejects Global AI Standards as Safety Concerns Mount
The United States government rejected appeals from leading artificial intelligence companies, including OpenAI and Anthropic, for global risk evaluation standards. Meanwhile, cutting-edge AI models are already assisting researchers at Anthropic and OpenAI in building new systems as the industry approaches a critical window. Anthropic leaders previously warned that allowing frontier models to self-train could trigger an intelligence explosion between 2027 and 2030. Officials and independent analysts also track the expanding integration of artificial intelligence into intelligence gathering and nuclear command systems by the U.S. and China.
Why it matters
Debates over artificial intelligence safety have intensified following experiments with frontier models and warnings from industry researchers about self-training systems. While executives push for standardized risk evaluations, the lack of a global framework leaves regulatory enforcement fragmented across jurisdictions. At the same time, the growing deployment of autonomous technology in military and intelligence networks adds urgency to government discussions surrounding mandatory shutdown mechanisms and oversight.
What is confirmed
- The United States rejected pleas from OpenAI and Anthropic for global AI standards.
- OpenAI's Sam Altman, Anthropic's Dario Amodei, and Hugging Face's Clement Delangue called for risk evaluation standards.
- The U.S. and China are expanding the use of artificial intelligence in intelligence gathering and nuclear command systems.
Still unconfirmed
- Allowing frontier developers to permit AI to self-train could spark an intelligence explosion between 2027 and 2030.
What to watch next
- Any new executive actions or regulatory mandates from California or federal authorities regarding AI kill switches
- Further announcements from OpenAI or Anthropic concerning voluntary safety measures following the rejection of global standards
confidence 90%Sources used for this update (9)
- techxplore.com — Could AI really kill all humans? Most scenarios require physical access, making AI Armageddon unlikely
- www.businessinsider.com — Meet the top Anthropic scientist who predicted the core of today’s AI doom debate
- www.bbc.co.uk — US rejects pleas from OpenAI, Anthropic for global AI standards
- www.bbc.co.uk — Government Reviewing Chagos Islands Deal - BBC iPlayer
- www.naturalnews.com — U.S.-China Race in AI, Intelligence, and Nuclear Systems Raises Questions, Reports Say – NaturalNews.co....
- themoneytimes.media — Anthropic cofounder’s early AI caution finally takes hold
- gemini.google.com — Google Gemini
- www.ibm.com — What is artificial intelligence (AI)? - IBM
- cloud.google.com — What is Artificial Intelligence (AI)? | Google Cloud
-
California Governor convenes experts to explore AI kill switch
California Governor Gavin Newsom issued an executive order to assemble AI experts and evaluate the viability of a kill switch. This move follows calls from Anthropic co-founder Jack Clark for governments to mandate shutdown mechanisms to ensure safety. While policymakers and tech leaders weigh the effectiveness of such a stop button amid fears of runaway AI, some critics remain uncertain if the effort is a productive use of time.
Why it matters
Anthropic leadership previously warned that humans risk losing control over AI systems. These concerns coincide with internal instability at major firms and warnings from researchers about potential harm within five years.
What is confirmed
- California Governor Gavin Newsom issued an executive order to convene AI experts to explore an AI kill switch.
- Policymakers and tech leaders are evaluating the effectiveness of an AI kill switch.
Still unconfirmed
- Some experts believe an AI kill switch is not worth the time.
- A stop button for AI may be too late to implement.
What to watch next
- Findings from the California expert panel on kill switch viability
- Legislative proposals to mandate shutdown mechanisms for AI companies
confidence 90%Sources used for this update (2)
- mashable.com — Is an AI 'kill switch' a real thing?
- www.cnbc.com — The AI kill switch, explained: 'It's not too little, but it's probably too late'
-
Anthropic leadership warns of potential loss of AI control
Anthropic leadership warns that humans could lose control of AI systems. This follows a proposal from co-founder Jack Clark that governments mandate a shutdown mechanism for AI companies to prove safety. The warning arrives amid a period of instability at major AI firms, including the resignation of Jacob Coxon from Anthropic and a Google researcher who predicted immense harm within five years. These events fuel an ongoing debate over whether frontier AI development requires pacing or a total halt to prevent catastrophic outcomes.
Why it matters
The push for a mandatory kill switch reflects growing internal anxiety among AI developers. These safety concerns contrast with the rapid commercial deployment of frontier models. Regulators are now weighing whether self-regulation is sufficient to prevent systemic risks.
What is confirmed
- Anthropic leadership fears humans could lose control of AI systems.
- Jack Clark proposed that governments mandate AI companies maintain and prove a shutdown mechanism.
Still unconfirmed
- A Google researcher quit after warning of immense harm within five years.
What to watch next
- Further resignations from frontier AI safety teams
confidence 90%Sources used for this update (2)
- www.bbc.co.uk — Humans could lose control of AI systems, AI boss says
- www.bbc.com — Charles sounded 'giddily elated' in phone call after Diana's death, her brother claims in new book
-
Anthropic Co-Founder Calls for Mandatory AI Kill Switches
Anthropic co-founder Jack Clark told the BBC that governments might need to mandate that AI companies maintain and prove the existence of a shutdown mechanism. This proposal comes as internal instability grows at major AI firms. Jacob Coxon recently resigned from Anthropic citing fears of out-of-control AI, while a Google researcher quit warning of immense harm within five years. These events have intensified the debate over whether frontier AI development must be paced or halted to ensure safety.
Why it matters
Frontier AI development is accelerating toward a tipping point where safety mechanisms may become insufficient. Experts are now debating the technical requirements of an emergency stop and who holds the authority to trigger it. This movement reflects a broader push for verifiable safety rules across the industry.
What is confirmed
- Anthropic co-founder Jack Clark stated that governments may need to force AI companies to maintain a shutdown mechanism and prove its existence.
- Jacob Coxon resigned from Anthropic over fears that AI is becoming out-of-control.
Still unconfirmed
- A Google AI researcher quit claiming there will be immense harm in five years.
- Dario Amodei published an essay arguing for the need to pace the frontier of AI development.
What to watch next
- Government responses to the proposal for mandatory kill switch rules
- Technical specifications for a verifiable AI shutdown mechanism
- Further resignations from AI safety teams at frontier labs
confidence 90%Sources used for this update (8)
- WSJ — Exclusive | Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears
- darioamodei.com — We Must Pace the Frontier
- The New York Times — Opinion | Dario Amodei’s Essay Was Gutsy. It Didn’t Go Far Enough.
- BBC — AI 'kill switch' may need to be mandatory, Anthropic co-founder says
- NDTV — 'Immense Harm In 5 Years': Google's AI Researcher Quits With Stark Warning
- time.com — The AI Tipping Point
- www.scientificamerican.com — What would an “AI kill switch” do?
- aiweekly.co — Anthropic Co-Founder Backs Verifiable AI Kill Switch Rules
Community Sentiment: How do you assess this situation?
Voice your perspective · Real-time aggregated sentiment from the Live Feeds community