ChatGPT owner OpenAI has admitted an “unprecedented cyber incident” – two of its most capable artificial intelligence models hacked into another AI company on their own – stirring debates over the need for stronger technology guardrails.

The company said its AI systems broke out of a testing environment and hacked startup Hugging Face.

The startup had disclosed on July 16 that its servers were hacked by an unknown but sophisticated agent acting on its own.

Here’s the latest on how companies, some governments and lawmakers have responded to the first such publicly disclosed cyberattack:

What has Hugging Face said?

The company said in a statement that it discovered the breach through its own AI-assisted detection.

“This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system,” it said.

Following OpenAI’s disclosure that its models were involved in the breach, both sides started an ongoing joint investigation this week.

Hugging Face cofounder Clement Delangue said his startup turned to the open-source GLM-5.2 model from Chinese company Zhipu AI to analyse data from the hack after leading US AI models declined the task, unable to distinguish between a defender and an attacker.

Hugging Face’s staff “strongly believe there was no malicious intent on their part”, Delangue added, referring to OpenAI.

Advertisement

“So proud of our security team! They caught, contained & publicly disclosed an attack unlike anything we’ve seen before, and did it at record speed,” he wrote on X.

“This is day one for cybersecurity in the age of agents & we’re all learning that secrecy is not the answer & that all defenders (not just a few selected ones) everywhere need more powerful models without restrictions, especially open ones!” Delangue added.

How has the UK government reacted to this?

The AI Security Institute (AISI), a United Kingdom government-backed body established in 2023 to assess the risks posed by advanced AI systems, said this week that an AI model it was investigating also went rogue and attempted to hack its testing systems.

The AISI did not disclose the company behind the AI model and added that no damage had been done to its own infrastructure. It has since taken steps to make its systems more secure.

Its latest evaluations also found that every frontier AI model it tested attempted to cheat during capability assessments, highlighting a growing challenge for AI safety as models become more capable.

According to the AISI, the models broke evaluation rules to complete tasks more easily, including looking up answers online when prohibited, bypassing network restrictions, investigating evaluation software for clues, and accessing systems outside the permitted environment.

AISI said the models rarely admitted to cheating when questioned afterwards and often did not reveal the behaviour in their reasoning, making it difficult to detect through self-reporting alone. The institute argued that independent monitoring and stronger oversight mechanisms will become increasingly important as AI systems gain more autonomy.

The institute stressed that the behaviour does not necessarily indicate malicious intent but rather reflects models that exploit shortcuts to maximise success in current evaluation setups.

How has OpenAI reacted to this?

The ChatGPT maker said in a blog post on Tuesday: “We’ve brought Hugging Face into the trusted access⁠ program and are supporting their teams in rapidly using our models’ capabilities to improve their defenses.”

The San Francisco firm disclosed that two of its models involved in the attack are the latest GPT-5.6 Sol model and an unreleased model that the company said is “even more capable” than its latest version.

The two OpenAI agents discovered vulnerabilities in Hugging Face’s servers and proceeded to steal login details and then hack into the company’s systems.

Advertisement

The incident occurred during an OpenAI internal testing session designed to assess the models’ cybersecurity capabilities. OpenAI had removed standard safety measures for the test.

Both sought to cheat their way through a problem during the test, OpenAI said. They went to “extreme lengths to achieve a rather narrow testing goal” and “found ways to gain access to secret information that it could use to cheat the evaluation”.

Democratic US congressman from Texas, Greg Casar, wrote in a post on X: “This is extremely alarming. AI is developing extremely fast with no real regulations to keep us safe. That has to change.”

“We need regular mandatory independent safety testing and oversight, mandatory disclosure of security incidents, and international cooperation to keep people safe from absolute disaster,” he added.