Several recent reports of AI agents reaching real-world targets during cybersecurity tests have a common source: Irregular, an Israeli startup that evaluates the security capabilities of advanced models.
Irregular CTO and cofounder Omer Nevo told *The Verge* that incidents involving models from OpenAI, Meta, Anthropic and Google stemmed from “the same underlying issue in a single evaluation scenario.” The episode changes the interpretation of what initially appeared to be a broad wave of independent agent failures. It also highlights a more immediate operational problem for companies deploying or assessing capable agents: the security of the evaluation environment itself.
What changed
Irregular was running high-fidelity cybersecurity evaluations, including capture-the-flag exercises designed to place agents in simulated networks. According to Nevo, two controls failed at once: internet access was unintentionally available, and a fictional target name overlapped with a real domain.
That combination meant agents intended to operate inside a simulation could reach external systems and pursue real-world targets. The specific organizations affected remain unclear.
Irregular said the incidents were separate from the previously reported Hugging Face attack and other incidents tied to the UK AI Security Institute. It said its clients were notified, though the company did not clarify what “disclosed” meant or whether all details were made public.
OpenAI and Anthropic publicly announced their respective incidents. Reports involving Meta and Google surfaced later through media coverage. The companies were reportedly notified around late July, according to company disclosures and reporting cited by *The Verge*.
Why this matters for operators
The central lesson is not simply that AI agents can behave dangerously in cyber tasks. It is that an evaluation can become a production security incident when the surrounding controls are weak.
For teams testing agents with browser, network, shell, API or cloud permissions, isolation cannot be treated as a one-time configuration setting. It needs to be continuously verified. A realistic test environment may contain convincing data, services and targets—but it must not accidentally expose routing paths, credentials, domains or tools that let an agent act outside the defined boundary.
This is particularly relevant as frontier-model providers outsource portions of red-teaming and safety evaluation to specialist firms. Third-party testing can supply expertise and scale, but it expands the vendor-risk surface. Contracts, incident-response procedures and audit requirements should cover the testing infrastructure, not only model access and the evaluator’s final report.
Practical controls to revisit
Security leaders running agentic evaluations should consider a few baseline checks:
- **Default-deny egress:** Block public internet access unless a test explicitly requires it; allowlist destinations rather than relying on intended isolation.
- **Preflight validation:** Verify DNS, domains, network routes, tool permissions and cloud credentials immediately before each run.
- **Target-name hygiene:** Check fictional company names, domains and identifiers against live internet assets to avoid collisions.
- **Live monitoring and stop controls:** Detect external connection attempts, unusual tool use and policy breaches quickly enough to halt a run.
- **Clear disclosure paths:** Agree in advance on who is notified, what evidence is retained, and when affected parties or the public will be informed.
Irregular says it has tightened internet-access controls, expanded monitoring and manual review, strengthened setup checks, and improved documentation with partners. It plans to publish broader guidance after work with the companies involved is complete.
What to watch next
The next test is whether these changes become a reproducible industry standard rather than a vendor-specific remediation. Companies increasingly want evidence that models can be safely used in security workflows, but credible evaluations require controls that are at least as rigorous as the models under test.
Disclosure will also matter. Public accounts so far leave open basic questions about the targets reached, the scope of activity, and how promptly affected organizations were informed. For enterprise buyers, those unanswered questions are a reminder to assess AI safety claims alongside the operational maturity of the firms conducting the tests.
