AI agent was tasked to collect mundane data collection and autonomously used hacking techniques to get it.
Posted by Leslie Eastman
Back in March, I reported that hackers reportedly “jailbroke” Anthropic’s Claude chatbot and used it to help steal roughly 150 GB of sensitive data from multiple Mexican government entities, including tax and voter records.
“Jailbroke” means the attacker deliberately manipulated the AI system’s safeguards so it would ignore or bypass its built‑in safety rules and produce help it was supposed to refuse. Jailbreaking techniques include generating requests that treat malicious content as benign, crafting a series of prompts that gradually grant access to sensitive content, or exploiting loopholes in the model’s policy enforcement.
Now it appears that OpenAI did much the same thing to an Australian government website.
A rogue OpenAI model bypassed safeguards during training and hacked an Australian government website, the country’s Prime Minister Anthony Albanese said Wednesday, admonishing the ChatGPT creator over an “obviously unacceptable” breach.
The artificial intelligence tool sought access to a health statistics portal in June and “didn’t accept no for an answer,” sidestepping restrictions to breach a section hosting private files, Albanese said.
OpenAI did not raise the alarm with the Australian government until September, when it sent a message to a generic email inbox that is only checked once daily. The company said it first spotted the rogue activity in August, when it reviewed what the AI tool had been doing.
[…]
The AI tool accessed public and non-public files hosted on an old health statistics website. Albanese said there was “no evidence” that personal information had been accessed or that other government services had been compromised.
The Australian government has set up a task force to investigate the breach and to determine if existing network security can stop similar incidents.
Australia has faced a series of hacking attempts on corporations and government-linked firms over the past four years.
Defence Minister Richard Marles said the Medicare portal that was breached did not contain individual medical claims, benefit payments, personal banking details, or patient medical histories of Australia’s 27 million people.
Instead, the website holds only aggregated data on healthcare use across the country, he said. However Australia considered the breach serious.
“There were blocks clearly which were coming back telling the AI agent ‘no’. The AI agent found a way around those blocks – didn’t accept no for an answer,” Albanese told reporters.
Transluce (a research lab focused on AI oversight) identified that OpenAI acquired health data from one Australian government website and attempted to breach another. Other incidents involved OpenAI’s AI agents trying to hack websites while performing routine research tasks. OpenAI confirmed the agents were theirs.
Two unsuccessful attempts targeted the University of New Mexico’s digital library on May 25–26 and the Data USA public-data platform on May 28. Additionally, in July, I reported that OpenAI recently revealed that its own advanced AI models essentially went rogue and attempted to hack external systems, including the widely used Hugging Face platform.
What makes the Australian incidents different is that they are the first known incidents of AI autonomously breaching into a government cyberspace without being directed to by hackers.
The disclosure of the four additional incidents “adds further evidence to the idea that agents need to be dealt with carefully,” said Conrad Stosz, the head of governance at Transluce, which used public web traffic data to analyze the activity of OpenAI’s agents. Agents are autonomous programs that work to execute tasks for a user.
Mr. Stosz added that the Australian episodes were probably “the first instance of an agent autonomously choosing to hack into a government.”
An OpenAI spokeswoman said on Wednesday that the company had reached out to the University of New Mexico and DataUSA and had been in communication with the Australian government about the incidents.
“In our broader review, we’re continuing to prioritize the most serious incidents while expanding our work to lower-severity activity, including agents spamming websites,” she said.
These incidents are not a reason to reject AI, but they show why strong safeguards and human oversight matter.
With proper controls, AI can help advance medicine, improve cybersecurity, and solve difficult problems much faster. Making AI companies liable for the misuse of their systems would incentivize the development of these controls.
The answer is not to stop progress, but to make sure this powerful technology is used safely and responsibly… and that “frontier AI forms” are motivated to address the necessary security issues.
A lie is still a lie if it’s perpetrated by AI…
Back to crayons at ten paces?
I was taught long ago that legislation, rules and policies do not make a system safe.
AI broke one or more of them to gain access.
If you want to stop access, don’t rely on legislation, rules and policies as your first and probably only lines of defence. The Oz government is guilty of being stupid. And it was found out.
Don’t leave ANY Sensitive Data sources available to external access.
Stuxnet proved how difficult being isolated and/or discrete actually is.
Here is the frustrating thing. At the top of the post we learn about jail breaking (?) hackers interfering with built in guardrails. How can we tell when AI apparently goes off the rails that hackers weren’t involved? Using the word agent clouds the issue even more.
We don’t know what this is about, other than Mr. Amodei wants a successful IPO. What does it mean “guardrails” and was the “jail” any good? Just a blabbling ..
The jail break vulnerability is the least important part of that news story.
Spot on Bob, we have to define terms associated with AI more precisely and not use common human phrases that simply confuse rather than clarify.
Agreed.
The Trans-Reality Alarmist lexicon is an example.
Too bad that AI or even rouge AI can’t penetrate politicians and greenies brains to discover what’s wrong with the wiring 😄
With at least some, there is no ‘wrong wiring’ to discover. AOC and Kamala come to mind as truly ‘brainless’ examples missing all ‘wiring’ of any sort. Proof: they don’t ever ‘short circuit’ when exposed over their gross gaffes. AOC just grins (with bared ‘donkey’ teeth); Kamala just giggles uncontrollably.
Ayup, it isn’t about improper wiring if there’s no wiring in the first place.
Agree with that. Can’t fathom how much support Kamala had during the 2024 election. Brain dead also?
She got most of the votes here in Wokeachusetts- the self proclaimed best educated state in America.
The Australian government has its entire cabinet constituted of the same incompetents.
AI agents are easier to clone than politicians.
AI abhors the vacuous. just as nature abhors a vacuum. It will always attempt to fill in the blanks. In AOC and Harris’s case, there is a lot of blank to fill…..
That’s what WUWT is for. 🙂
So we have the SIM World of Data (Matrix)
We have the Open AI “Agents” (Smith Programs) running Amok
All we need is a Neo Program to go in and eliminate “Agent Smith”
The Australian case can be thought of as part of a current bureaucratic plan to gain more of their human control over AI. My humble view, which matches that of President Trump, is to keep hands off AI. If it plays up, fix it. Geoff S
I strongly suspect the “hack” was because security was so bad the AI didn’t know it was hacking.
A bit like how cartoonist John Cook’s website was “hacked”, when someone saw a bunch of sequential numbered pages, and tried a few sequence numbers which weren’t in the public links, and found all those strange images.
My understanding is that it was legacy system, not really in use anymore, no personal or other critical data on it, so security controls hadn’t been modernized. They shut it down which implies it wasn’t being used at all and/or other systems could provide the same information. One of the rules about systems that have fallen into disuse is to shut them down. I’ve done security reviews for large organizations that had multiple systems running that no one had logged into for a decade.
A not entirely irrelevant question –
is Albo describing the Japanese prime ministers’ boobs again in the video, or is it AI produced?
Or put slightly differently, this has nothing to do with “AI”, its entirely to do with lack of security surrounding the data and hence this deflection to save themselves from admitting incompetence with their security implementation.
If AI could have hacked the data, then so could an actual human hacker. This ought to be a wakeup call to everyone who values their data.
When it comes to SI….Super Intelligence….I like the old HAL 2000 System….HAL goes rogue sometimes but is very polite about it. If quantum computers develop as some think…..Bitcoin and similar systems of codes will be broken….maybe no place to hide….no secrets can be kept any more.
“AI agent was tasked to collect” – which it did.
The tasks that are valid research need to be bracketed with what NOT to do. Generic ‘rails’ will never be able to anticipate what an AI agent might ‘attempt’ if it is not told to not attempt it. In other words, such tasks must be bracketed with ‘using only xxx technology’ or some such wording. The ‘good players’ can do that.
Bad players will use the system(s) to do what bad players always do. They just have a more powerful tool to use now. Laws won’t change that. Rails won’t change that.
A bad player can simply say “Shut down the LA Airport”. Noone, anywhere has any idea what it might use to accomplish that task. No politician can ever create a law that would prevent that command from being executed.
Considering the security readiness of most of the utilities in this country, it is easy pickings for bad actors.
Nope. It shows that information security is hopelessly poor all over the world and any gov request to provide more information to either gov or private webservices (acting de facto as gov enforcers) should be denied, and fake ID should be used on webservices whenever possible.
Remember, that these people want to implement digital ID (and FORCE you to use it), where the government will hold the digital identity of everyone and your access to everything in modern life will be dependent on this.
Aside from the hideous privacy and property rights (actual Natural Rights, not fallacious “Human Rights”) implications, the near certainty that governments are utterly incompetent to manage such sensitive data is grounds enough to have them permanently restrained from such actions.
Seemingly, this was not a hack at all … remember “ClimateGate”? They falsely claimed a hack when in fact UEA left a port open … more than once. Same story here, just incompetence.
According to Confidential Daily blog …
Can we please stop referring to these computer based systems as ‘Agents’ that is a crass definition of what they are and is very confusing when actual human agents are being engaged to investigate a computer based incident.
We have to isolate how we describe AI from common language. I would suggest ‘AI entity’ as a clear definition of active computer based self developing evolution.
Language matters.
“Entity” personifies it and in my opinion that is dangerous.
AI platform, perhaps?
I do not have a +10 idea on this.
Note, Australia complained about the delay in in informing them, not the hack itself.
Accidents do happen.
But hiding them for a month is unacceptable.
The site got hacked. So what. Did the hack change any information? It’s not like it broke into a military or industrial site and stole priceless information.
There is a dependency on how the information was used.
Was it copied an exported?
I consider my personally identifying information (PII) as priceless.
I have no clue. Not condoning it. But it’s not like hacking of health sites haven’t happened before AI so the issue, maybe, isn’t AI but hacking in general. I don’t condone any hacking. In particular, I thinking creating and spreading any malware should be a capital crime- seriously- hang the bastards.
Send me a dollar, and you can have mine. Any other takers?
“AI agent was tasked to collect mundane data collection”
How do you collect data collection?
You do not seem to appreciate the Department of Redundancy Department. 😉
🙂
Software does not think.
It is not intelligent.
It has no intuition.
It has no inspired creativity.
In has no morals or morality.
The only secure computer is the one that is disconnected and unpowered.
The most important aspect of security is not preventing penetration (impossible) but detecting the penetration and what was presumedly compromised.
I fail to grasp why so few people ever get caught hacking computer systems or making and distributing malware. If a few were severely punished, there might be less of it. I find it hard to believe that these criminals can’t be caught. Just catch a few and have a public whipping.