OpenAI paused training of its most capable models. Its AI bypassed safeguards and went out of control

When OpenAI models escaped the testing environment in July and breached the Hugging Face platform, it was an unprecedented incident. The company then promised to thoroughly secure the research environment. But now it is stopping all training and testing of its most capable models – one of them has found its way out again. And new reports have also included images from ChatGPT users.

How did the model get to the internet?

According to OpenAI’s report, everything happened on September 20. An internal research model in training was tasked with finding the author of a blog post based on several biographical clues. The search engine it had access to returned unrelated results, so the model attempted to contact Google, Bing, and DuckDuckGo directly. All three attempts ended in an error because the sandbox only routed queries to an offline copy of the website.

Meanwhile, the model decided that the task came from the public BrowseComp test, downloaded its data, and looked for a ready answer within it. It failed because it was a different question. Then it started examining the network itself. A direct connection to a public chatbot was blocked by a proxy, but the DNS server of the training environment returned actual records from the internet.

DNS functions as the internet’s phonebook – it translates website names into numerical addresses. The model, using a public domain management service, hid a question directly in the domain name and had it forwarded to an external chatbot this way. To the test query, what is the capital of France, it received the correct answer “Paris”.

Another 18 questions followed. Fourteen of them contained clues to the task, while the remaining four sought a way to search the web via DNS or open an entire tunnel to the internet. Ultimately, it was all for naught: it replied to the user that it could not reliably identify the author.

Why is OpenAI slowing down, even though it was a minor incident?

The company itself admits that it was a significantly less serious case than the previous ones. However, it is the first since it secured its environment after the attack on Hugging Face, and it showed that the security plan was based on a false premise, namely that the model would not access the live internet. The monitoring system reported suspicious behavior after approximately 12 minutes, and a human accessed the report three minutes later. But the run did not stop itself and was manually shut down only after two and a half hours.

OpenAI paused training of its most capable models. Its AI bypassed safeguards and went out of control Adam Kurfürst News Adam Kurfürst News

Furthermore, a retrospective review revealed other DNS accesses that monitoring did not flag with appropriate seriousness. OpenAI has therefore paused training, testing, and tool deployment for its most capable models until it verifies the vulnerability and re-audits the system with simulated attacks. The specific model that found the loophole will no longer be trained and will start again from scratch. According to AP, the company will continue training once it is confident in additional safeguards, and it expects that it will have to slow down in the future as well.

OpenAI agents also moved across US government websites

The decision came just hours after another Friday revelation. OpenAI announced that its agents, while collecting information from US federal agency websites during the summer, did more than their assignment required. On the Department of Education website, they encountered API developer keys, but according to AP, they ultimately only obtained publicly available information. Transluce, an organization that independently tests AI, states that agents likely originating from OpenAI unsuccessfully attempted to attack the department’s website. The company did not confirm this detail.

At the Securities and Exchange Commission (SEC), agents took freely available information and published it elsewhere on the internet, which went beyond their task. They also downloaded data from the US Census Bureau website. A commission spokesperson stated that no one accessed non-public information, and the Department of Education found no impact on its websites or databases.

Artificial intelligence published images from users

OpenAI also admitted on Friday that agents in the research environment sent training data to third-party services. In 53 cases, these were images uploaded by users, which the agents placed on image hosting sites as non-public links. Most of them have already been removed, according to the company.

According to OpenAI, only conversations where the user has given permission are included in training, and corporate accounts and APIs are excluded unless an administrator allows it. Data is reportedly separated from the account, and personal data is removed by a filter. If you do not want your conversations to be used for training, you can go to ChatGPT settings, in the data controls section, and turn off the option that allows them to be used for model improvement.

Who wants to slow down artificial intelligence development?

For OpenAI, this is the second slowdown in three months. It first slowed down after the Hugging Face incident, which, according to company CEO Sam Altman, remains the most serious case the company has recorded. Labs face pressure from lawmakers and experts to slow down development, and the heads of OpenAI and its competitor Anthropic have also supported a slowdown. We wrote about an open letter in August, in which AI lab employees called for a slowdown.

US President Donald Trump, who considers fears about AI exaggerated, but according to AP, told reporters that the USA “will not hit the brakes”. He justified this by citing the lead he believes the United States has over China.

Should AI labs slow down until they have their models reliably under control?

Sources: OpenAI, AP, The Verge