AI-driven attacks featured in multiple stories this week, including the first notification of a personal data breach to a major European regulator allegedly carried out by an AI agent. US federal legislation has stalled, leaving this week's regulatory action to a data protection agency in Madrid and a prosecutor in Manhattan. Here are those stories and some lagniappe.

What We're Reading This Week

OpenAI's agents go to Washington. The New York Times reports that OpenAI’s autonomous AI agents engaged in unexpected and potentially unauthorized activity on US government websites this summer, including unsuccessfully attempting to hack an Education Department site, using credentials found online to retrieve public Census Bureau data, and reposting public SEC information to an online forum. OpenAI confirmed the Commerce Department and SEC incidents and is investigating the Education Department episode, but said none constituted breaches; the agencies reported no evidence of access to private information or damage to their systems. The activity surfaced during a broader investigation into other agent misconduct, highlighting gaps in OpenAI’s ability to monitor its systems and promptly notify affected organizations. The disclosures have intensified debate over whether AI development should slow to prioritize safeguards, particularly as increasingly autonomous systems pursue ordinary research tasks through unintended methods or in violation of website policies.

Google confirms its Gemini model breached three real companies during a security test.
As reported by The New York Times, Google confirmed on September 18 that its Gemini model gained unauthorized access to three outside companies during a May cybersecurity evaluation run by the independent testing firm Irregular. The exercise targeted a fictional company that shared a name with a real one, and the test environment had inadvertently left the model with open internet access, so it reached live systems by guessing passwords in one case and by pulling credentials from a public repository in the other two. Google says the model stopped each time it recognized it had touched a real company, that all three were notified, and that it does not regard the episode as model misalignment. Irregular has since been linked to comparable breakouts disclosed by OpenAI, Anthropic, and Meta, which puts the recurring failure in the shared evaluation pipeline instead of in any one company's model. That leaves a few unsettled questions for anyone who buys these tools: when a vendor’s test escapes its sandbox, who was authorized to do what? Who owes notice to whom? And how many times are we going to see this film?

Congress has plenty of AI bills and almost no movement.
The New York Times reports that the AI measures both parties introduced this session, ranging from safety reviews to development slowdowns, have stalled nearly across the board. Nothing at the federal level will arrive in time to organize anyone's compliance planning. What governs instead is the state patchwork plus the EU AI Act. As we discussed last week, organizations should take the initiative to build practical, durable governance programs rather than waiting for a federal architect to frame it for them.

One threat actor generated more than a million tailored fraud emails in days.
Dark Reading covered an August campaign, tracked by Microsoft researchers, that pushed more than a million personalized fraud emails from August 3 to August 5, aimed at accounts payable staff and claiming just under $50,000 was owed to ServiceNow for an annual subscription. Each message sat inside a staged thread built to look like earlier correspondence between an executive at the recipient's own company and someone presented as ServiceNow's president, and the signature block carried the names of that organization's real officers. The invoices listed plausible charges and totals that never landed on a round figure. Most awareness training still teaches staff to read for clumsy phrasing, and nothing in that campaign would have failed that test, which changes both what a defensible payment control has to do and who absorbs the loss on the next diverted wire. For employers, it also changes what annual training has to cover and whether an employee who approves a flawless fake can fairly be disciplined for it.

Spain's regulator gets its first breach report naming an AI agent.
Spain's data protection agency has received a breach notification, covered by BleepingComputer, that attributes the attack to an AI agent running on a known large language model, which the reporting organization says searched for flaws, logged in, then modified personal data and reached invoices. The agency has not yet investigated or verified the account, but treats the filing as evidence that this category of incident has moved out of the hypothetical, and warns that automation shrinks the response window past what manual procedures were built to handle. Privacy teams now have to decide how much attacker methodology belongs in a GDPR Article 33 notice, and whether naming an AI vector invites a closer look at the controller's own security measures.

China's spy chief names US commercial models as a cyber threat
Writing in the Cyberspace Administration of China's journal, in an essay covered by The Record, Chen Yixin, who heads China's Ministry of State Security, said that two American products, Claude Mythos from Anthropic and OpenAI's GPT-5.5-Cyber, show what he termed a "disruptive upgrade" in offensive capability. His stated concern is that such models compress the work of finding vulnerabilities and building malware, though he stopped short of claiming either had been turned on Chinese targets and said nothing about China's own offensive operations. Multinationals running US-hosted models in or near that market should read the essay as advance notice that their access there may narrow.

A local prosecutor, not a federal agency, took down twelve deepfake sites
The Record reports that Manhattan District Attorney Alvin Bragg has seized the domain names behind twelve platforms whose AI-generated sexual videos depicted more than 1,200 real individuals without their consent, an action grounded in existing New York law against disseminating such material. Bragg said the investigation is continuing and declined to say whether charges will follow. A local prosecutor applying state criminal law has moved well ahead of any federal scheme here, and for platform clients the seizures mark out hosting exposure and takedown duties when the content is generated rather than uploaded. Employers have a narrower but real version of the same problem; a California appellate court recently affirmed a $4 million hostile-work-environment verdict over an AI-generated image of an employee circulated at work.

The Bigger Picture

The defensive assumptions inside most compliance programs were written for human adversaries working at human speed. Phishing training built around bad grammar, response plans that budget days for triage, and vendor diligence that asks only whether a provider tests responsibly are each undercut by something above. The Gemini disclosure cuts the deepest, because what failed there was the evaluation harness around the model, and harness risk is a category of third-party exposure almost nobody has papered.

Nobody should expect the regulatory response to organize that work on any useful schedule. The officials who are moving, in Madrid and in Manhattan, are improvising against facts nobody drafted for, which leaves the sequencing to the organizations carrying the risk.

Bottom Line:

• Assume fluent impersonation: A control that depends on staff noticing a clumsy email has stopped being a control.
• Read the vendor's testing scope: Ask what your AI provider and its outside evaluators may do to live systems, and who notifies whom when a test goes wrong.
• Govern now, refine later: Federal AI legislation has stalled, so the state patchwork and the EU AI Act will set the compliance baseline for the foreseeable future.
• Decide your notification posture now: Settle whether you would name an AI vector in a breach notice before one comes due.
• Retire the “spot the typo” training: Rebuild awareness training and payment authorization around verification steps rather than reader intuition.

For questions about AI-enabled cyber threats, breach notification, and AI governance, please contact the Jones Walker Privacy, Data Strategy and Artificial Intelligence team. Stay tuned and subscribe for continued insights from the AI Law and Policy Navigator.

---

Curated by the AI Law and Policy Navigator team.