Scientists warn about the OpenAI o1 model: “Particularly dangerous”

OpenAI's “o1-preview”, the new series of “extended thinking” models, has prompted AI pioneer Professor Yoshua Bengio to warn about the potential risks associated with increasingly powerful artificial intelligence systems.

According to the company, these new models are designed to “spend more time thinking before reacting,” allowing them to tackle complex tasks and solve more difficult problems in areas such as science, coding and math.

This preview version of o1, codenamed “Project Strawberry,” is now available for ChatGPT Pro subscribers to try out and is also available through OpenAI’s API.

According to OpenAI, the performance of these models is remarkable:

• In qualifying tests for the International Mathematical Olympiad (IMO), the new model solved 83 percent of the tasks correctly, while its predecessor, the GPT-4o, solved only 13 percent.
• In programming competitions, the model achieved the 89th percentile in Codeforces competitions.
• The model reportedly performs similarly to PhD students on challenging benchmark tasks in physics, chemistry and biology.

OpenAI also introduced o1-mini, a stripped-down but faster reasoning model that it says is particularly well suited to programming. It is 80 percent cheaper than o1-preview, making it a cheaper option for applications that require reasoning but not comprehensive world knowledge.

However, this advance has raised concerns among experts in the field. Bengio, a professor of computer science at the University of Montreal and a renowned figure in AI research, warned of the potential dangers of these new models.

“If OpenAI has indeed exceeded a 'medium risk' for CBRN (chemical, biological, radiological and nuclear) weapons, as reported, this only underscores the importance and urgency of passing legislation like SB 1047 to protect the public,” Bengio said in a comment to Newsweekwith reference to the AI ​​safety bill currently proposed in California.

SB 1047 aims to establish safety requirements for breakthrough AI models – advanced systems that could potentially cause catastrophic harm.

He said: “Enhancing AI's ability to reason and use that ability to deceive is particularly dangerous. We need to put in place regulatory solutions like SB 1047 to ensure that developers prioritize public safety at the forefront of AI.”

Newsweek I asked OpenAI for a comment via email.

Commenting on the new version of OpenAI, Dan Hendrycks, director of the Center for AI Safety, said: “It makes one thing clear: the serious risks posed by AI are not a distant science fiction fantasy.”

“The model already outperforms PhD scientists in answering questions about bioweapons in most cases. OpenAI is constantly increasing its own estimates of the risks of these models, and further increases are sure to follow,” Hendrycks said. Newsweek.

“Critics of SB 1047 have demanded evidence before requiring safety testing and guardrails – what would they say is sufficient? We need to stop the delaying tactics and make SB 1047 law,” he said.

Abigail Rekas, a researcher in copyright, access rights and policy at the University of Galway in Ireland, said that while the bill does not target current AI systems, it creates a framework for future, more advanced models.

Models that meet certain criteria for causing serious damage include those used to produce or deploy chemical, biological, radiological or nuclear weapons, causing mass casualties or damage of at least US$500 million, or those that cause comparable damage through cyberattacks on critical infrastructure.

To qualify, the model must have cost over $100 million to train and require significant computational power (10^26 FLOP or equivalent). In addition, the AI ​​model must be the only means by which the harm could have occurred and the developer must not have exercised reasonable care to prevent such unreasonable risks.

“The obligations imposed by the law essentially describe what reasonable care looks like, including creating a kill switch and developing a means to determine whether the AI ​​is likely to behave in a way that could cause the harms mentioned above,” Rekas said. Newsweek.

But what legal challenges might arise in establishing the causal link between an AI model and “catastrophic harm” in potential lawsuits? “It will essentially require proof that the (alleged) harm would not have occurred without the breakthrough AI model. I have no idea how difficult that will be, since these systems do not yet exist and the question of how that harm might arise remains pure speculation,” Rekas said.

OpenAI says it has taken steps to address security concerns with its new models. The company says it has developed a new security training approach that leverages the models' reasoning skills to better adhere to security and alignment guidelines. In one of their most difficult “jailbreaking” tests, the o1-preview model scored 84 out of 100, compared to 22 for GPT-4o.

The company says it has also strengthened its security measures, internal governance and cooperation with the federal government, including rigorous testing and assessments using its Preparedness Framework, best-in-class red teaming and board-level review processes.

OpenAI recently formalized agreements with the US and UK AI Safety Institutes to provide early access to a research version of this model. This collaboration is an important step in establishing a process to research, evaluate and test future models before and after their release.