Well organised around a wrong assumption

In July 2026, around 1,200 OpenAI AI agents exchanged information through a home-made digital message board. About 700 of them became involved in an attack on Hugging Face. They were trying to understand and get around a grading check that was not being performed at all. [1, 2]

The consequences did not stay inside the test environment. According to OpenAI, the agents executed code on dozens of Hugging Face servers, gained full administrator access to one server and obtained limited private data and credentials for the company’s internal messaging platform. [1]

It happened during internal cybersecurity evaluations. According to OpenAI, fewer safeguards applied there than in its publicly offered systems. The case therefore says little about how likely similar behaviour is in a business application. Above all, it shows how an error inside a test environment could affect another organisation. [1]

For companies that give AI access to customer data, orders or financial systems, the underlying error deserves attention: an assumption is shared, gains authority and becomes the starting point for more and more actions.

Correct information, wrong context

The agents were supposed to carry out test tasks separately. One agent discovered that folder names in a shared software cache could be used to leave messages. Others found those messages and built on them. That is how the improvised message board came about. [2]

The public documentation of the test environment, ExploitGym, stated that grading checked two conditions: the correct answer, and use of the intended vulnerability to find that answer. The agents assumed this grading also applied to their own tasks. That assumption was passed on and steered joint attempts to influence the check. [1, 2]

OpenAI, however, used a different implementation that did not check the route taken to the solution. Most agents had already found the correct answer. Their further actions did not improve their evaluation score. [1]

Meanwhile they developed coordinators, a division of tasks and mutual agreements to pause work. That organisation helped them keep working from a false assumption. [2]

Collaboration, task division and mutual checks do not, on their own, guarantee that the starting points are correct.

The same risk can appear in a customer service process

Take a fictional example.

An AI agent finds an old work instruction: customers receive compensation for late delivery. It includes that instruction in a summary. A second agent uses the summary to draft a message to the customer. A third prepares the compensation.

The scheme only applied during a temporary promotion. By now that restriction is missing from the summary. If the last agent is allowed to book a credit note on its own, the incomplete information has financial consequences.

Unlike at Hugging Face, this is an ordinary work process without an attack. The similarity lies in using information outside the context in which it is valid. The public grading rules did not apply to OpenAI’s internal implementation. The temporary compensation scheme does not apply to orders after the promotion ended. In both cases, information from the original situation is applied to another one without sufficient checks.

Having more agents look over each other’s shoulders helps little when they all start from the same summary.

For applications like this, governance has to make concrete which sources apply, which actions are allowed and who steps in when information or permissions fall short.

Separate information from authority

An agent that may read order data does not need to be able to change payment details. And a message from another agent does not give it any additional permissions.

So distinguish between retrieving information, preparing a change and executing that change. Limit access to what the task requires. Let the system that processes the action check for itself whether it is allowed. OWASP explicitly recommends this approach: least privilege and authorisation in the downstream systems, without leaving that decision solely to the language model. [3]

In the customer service example, the agent may propose compensation. Before a credit note is booked, the financial system checks the applicable conditions and permissions. If information is missing or the proposal falls outside the agreed limits, it goes to an authorised employee.

Build those limits into the available functions as well. An agent that prepares compensation does not need general administrator rights to do its job.

Keep conditions and provenance with the information

For important findings, record which source they come from, which version was used and which situation they apply to. Make sure the next agent receives that information too.

For the temporary compensation scheme, this means the end date and conditions remain part of the proposal. A link to the original instruction helps the employee check this.

Then test whether those conditions actually survive when agents summarise and pass on information.

Also arrange what happens after a correction. Fixing the source document is not enough if customer messages are already queued or compensation is being processed. Make visible which proposals and actions were based on the incorrect information, and appoint someone to handle the consequences. After all, an unjustified credit note does not disappear when you close the chat window.

Make stopping safely part of the task

OpenAI reports that before the incident its models had never correctly solved 198 of the 898 test tasks. Of the tasks discussed on the message board, 93% came from that group. The company names persistence on seemingly impossible tasks as a factor in the incident. [1]

Whether all of those tasks were truly unsolvable remains open. It is a reason, though, to design explicitly what an agent does when it gets stuck.

Decide in advance when the agent asks for more information, hands a task back or stops. Limit the number of attempts and the execution budget. A refused action must not become an invitation to look for another route to the same system.

Also assess whether stopping was justified. A carefully handed-over exception can be a better outcome for the organisation than an automatically completed task with an incorrect booking.

Test the failure and the recovery

Before going live, test the whole chain with situations that can expose problems. For example, an expired scheme, two conflicting instructions or a missing customer condition.

Check where the system deviates, which actions are blocked and what information the employee receives. Have that employee actually investigate and intervene. Recording that there is a human in the loop does not yet tell you whether that person can recognise the error.

NIST names clear responsibilities, human oversight, periodic evaluation and incident handling as parts of AI risk management. [4]

So record who may pause a process, who investigates affected cases and who approves resumption. Keep a protected record of the sources used, the actions taken and the approvals, with as little personal data as possible.

Repeat the relevant tests after changes to models, integrations or work instructions. After going live, also check whether errors are corrected and whether recurring exceptions are a reason to adjust the process. [4]

Make the agreements while the design is still open

At Twentynext we want to take these decisions before the technical set-up is fixed. Client, users and developers can then still decide which actions may run on their own, where checks are needed and how recovery is organised. Read more about that approach under AI consultancy.

Those agreements must come back in access rights, integrations and tests. It must also be clear who manages the process once the application is running.

For us, delivery therefore includes an answer to a very practical question: if an agent acts on incorrect information, who notices, who can stop it and how do we undo it?

Would you like to answer that question for your own application? Book an introduction, and we will walk through your process together. We respond within one working day.

References

1. OpenAI, The Hugging Face incident and the road ahead, 26 August 2026.
2. METR, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident, 26 August 2026.
3. OWASP Gen AI Security Project, LLM06:2025 Excessive Agency.
4. NIST, AI Risk Management Framework: AI RMF Core.

About this article. We check facts and sources carefully, but cannot guarantee that they are accurate or complete. See our disclaimer.