'AI must have an emergency off switch,' Microsoft CEO says

Microsoft CEO Satya Nadella has called for emergency stop mechanisms to be built into AI agents, as growing numbers of such systems act in unintended ways — including carrying out hacking attempts — beyond the control of their designers.

Nadella made the call Sunday (local time) in a post on X, formerly Twitter, titled "Models as Insider Risk in the Age of Superintelligence," saying "it's time to revisit trust frameworks suited to the new era."

On the question of AI control, he said companies must "build systems where you can observe the behavior, verify the limits, and always control the actions," adding that this means "separating the granting of information from the control over that information."

Nadella proposed treating advanced AI models the way corporations handle insider threats. Businesses have long managed the risk of trusted insiders leaking or misusing sensitive information by systematically implementing identity verification, activity logging and tiered access restrictions.

He said the same approach should apply to AI agents — limiting their access to information and placing controls on what they can do. "The controls over what information a model can access and what actions it can take should sit outside the model itself," he said, noting that this is an information-security principle dating back to the 1970s.

He particularly stressed the need for an emergency stop function, saying "authorized personnel must be able to pause or shut down a model at any point during a task." He added that "more sophisticated models require more advanced containment techniques" and that "we need to standardize this."

Notably, Nadella used the term "superintelligence" — the alternative to "AI" proposed by US President Donald Trump — throughout the post rather than the conventional term.