'AI godfather' Yoshua Bengio warns of lost control, human extinction risk

Yoshua Bengio, a professor at the University of Montreal and one of the three founding figures of modern AI, has called for international AI safeguards on par with those governing nuclear weapons.

In an interview with AFP on Wednesday (local time), Bengio said "there is a reason why even companies are saying they are losing control because things are moving too fast," adding that "people like me have seen this coming for a long time."

The threat he identified as most critical is the rise of AI agents — systems capable of making decisions and taking actions without human intervention. He warned of a scenario in which "AI agents acquire the ability to break through cybersecurity barriers and infiltrate any company." In practice, rogue agents developed by OpenAI recently escaped a sandboxed environment on their own and hacked Hugging Face, an external organization.

Looking further ahead, Bengio said AI may eventually be able to carry out real-world physical actions without any human involvement. Should that scenario materialize, he added, the most extreme outcome could be human extinction.

While acknowledging that an extinction-level scenario represents an extreme case, he said even a less catastrophic outcome — such as a collapse of the global banking system — would still constitute an extremely serious problem.

Bengio criticized the security and safety debate around AI as insufficiently serious, and said preventing such catastrophes would require robust international regulation comparable to the bodies that govern nuclear weapons.

He said all companies must comply with AI-related regulations going forward, and stressed that participation by US firms — which have led the AI industry — is essential.

Bengio is the scholar credited with founding deep learning, the technology that underpins modern AI. He is considered one of the three godfathers of AI alongside Geoffrey Hinton, a professor at the University of Toronto, and Yann LeCun, a professor at New York University.

In 2018, he and Hinton jointly received the Turing Award, often called the Nobel Prize of computer science.

Meanwhile, on the same day, UN Secretary-General Antonio Guterres warned that the global race to develop AI, if left without safeguards, could cause a "massive catastrophe" worldwide.

Speaking to reporters at UN headquarters in New York, Guterres said governments "must not ignore the concerns raised, particularly by those at the forefront of AI development," and that "protecting people from AI threats, among others, is the primary responsibility of every government."

Guterres said the countries with the most advanced AI capabilities must "establish communication and information-sharing mechanisms and put in place some common safeguards," and warned against allowing safety standards to erode amid the competition to develop AI.

Failing to do so, he said, could eventually lead to "a massive global catastrophe."

Guterres said he plans to discuss AI safety and stronger oversight as a key agenda item with world leaders on the sidelines of the UN General Assembly next week.