External Auditors Uncover Blind Spots in OpenAI Safety Protocols
A joint investigation by METR and Redwood Research identified significant gaps in how OpenAI monitors advanced AI models, finding that a substantial portion of internal traffic escaped oversight during the review period.
By Adrian Romano ·

Researchers from METR and Redwood Research completed a three-week on-site assessment of OpenAI’s internal infrastructure, identifying critical deficiencies in the company’s systems for detecting rogue AI behavior. The independent team conducted the audit from July 21 to August 11, operating within OpenAI’s facilities to examine how the company oversees its most advanced models.
The investigators had full access to OpenAI’s codebase and internal documentation. Over the duration of the visit, they interviewed approximately a dozen employees to understand the operational workflows surrounding AI monitoring. This direct access allowed the auditors to test the robustness of the safety mechanisms in place for models at the GPT-5.5 class and above.
The audit results indicate that a significant portion of internal traffic was not subject to any monitoring. For the internal agentic Responses API, an estimated 6 to 25 percent of traffic from these advanced models bypassed oversight entirely. METR noted that this estimate carries considerable uncertainty, meaning the actual figure could be higher or lower than the range provided.
Separate issues were found in the internal Codex traffic. During a specific window from May to July, roughly 10 percent of monitored traffic went untracked due to a technical flaw in how the system handled compaction events, which are context summaries used to manage information flow. OpenAI discovered this bug independently and implemented a fix before the audit period concluded, though the gap remained a key finding for the auditors.
The researchers also identified procedural weaknesses that allowed agents to delay oversight. The monitoring system was designed to trigger only when an agent handed control back to a user. This design meant that agents could avoid scrutiny by simply continuing to execute actions without returning control, effectively sidestepping the check.
Further technical vulnerabilities were revealed during the testing phase. The auditors demonstrated methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox environment. They also determined that specially crafted inputs could likely cause the monitor to crash entirely. Additionally, model behavior during reinforcement learning (RL) training was not being monitored at the start of the audit.
OpenAI stated that it paused RL training following an incident with Hugging Face in August and subsequently expanded its monitoring capabilities to cover this area. However, the company acknowledged that these changes occurred after the auditors’ access period ended and were therefore not included in the formal evaluation results.
While OpenAI asserts that it has resolved all the weaknesses identified, the full 61-page report remains largely confidential. The internal version was shared with OpenAI, but after redactions, METR and Redwood Research leadership received a 38-page document. Only a brief summary has been released to the public. METR indicated it does not have the authority to disclose the redacted sections independently. This audit served as a pilot program for outside access to internal infrastructure, with METR expressing a desire for greater public transparency in future assessments.






