Sign in. It’s quick, free and it’s up to you.
An account is an optional way to support the work we do. Find out more.
File image of leading AI apps on a smartphone Alamy Stock Photo
apocalypse how
What is an AI 'agent', how can it 'go rogue' and why the doomsday fears? Here's what to know
Here’s what you need to know about AI chatbots and agents, and why some agents have been going rogue.
SOME OF YOU may be thinking: “My AI chatbot isn’t particularly impressive, so why is everyone so afraid of artificial intelligence?”
Back in July, rogue AI agents from OpenAI escaped their confined testing environment and gained access to the internet.
There, the rogue AI found and attacked Hugging Face, a site AI developers use to store and share models and data sets.
This sparked alarm in the AI industry, and soon there were fears being openly expressed that AI could “kill us all” within a decade.
These fears won’t have been eased by yet another OpenAI snafu, this time involving an AI agent hacking into an Australian government health portal.
These incidents aren’t caused, however, by the humbler chatbots such as ChatGPT and Claude that many of us have on our phones, but by autonomous AI agents.
Confused? Here’s what you need to know about the difference between chatbots and agents, and why some agents have been going rogue.
The Chatbots
An AI chatbot uses artificial intelligence to simulate human-like conversation through text or speech.
To go even deeper, AI chatbots are powered by Large Language Models (LLMs).
LLMs are trained on vast amounts of data to understand and respond to human text – much of this data comes from the internet and its attainment can be controversial.
For example, the New York Times is currently suing both OpenAI and Microsoft for training its AI on New York Times content without its authorisation.
And this month, leading AI company Anthropic agreed to pay $1.5 billion (€1.31 billion) to settle a class action lawsuit filed by authors who said the company stole their work to train its AI models.
OpenAI was founded in 2015 and had Elon Musk among its co-founders. Musk resigned in 2018 when the AI firm transitioned from a nonprofit to a for-profit.
Its current CEO and co-founder is Sam Altman and OpenAI is best known for its AI chatbot ChatGPT, which is credited for sparking the current AI boom.
Its main rival is Anthropic, which was founded in 2021.
Anthropic has locked horns with Trump after the company refused to allow the US military to use its AI models for tasks including surveillance or autonomous weapons.
Its chatbot is called Claude.
Other honourable mentions in the AI sphere include Meta AI (Meta owns Facebook, Instagram and WhatsApp) and Google Deepmind.
So what’s the difference between a chatbot and an agent?
A key difference is that a chatbot waits around for us humans to use it, whereas AI agents can work autonomously.
A chatbot is reactive – it waits for us to ask it a question, and replies (though some are designed to give open-ended responses that encourage further human engagement).
AI chatbots combine multiple technologies to analyse the user’s request and generate a response.
But because AI chatbots learn mostly from data on the internet, they can sometimes use outdated or incorrect data.
“Hallucinations” can also occur, which is the term given to when an AI chatbot essentially fabricates an answer because it lacks the specific data required.
An AI agent, meanwhile, is given a goal and works autonomously to achieve that goal.
Humans set these goals, but the AI agent independently chooses the best actions it needs to perform to achieve them.
So what’s there to be scared of?
It’s the AI agents that have been making the headlines and creating AI doom concerns, not the chatbots on your phone.
One famous thought experiment is the paperclip problem, first proposed by Oxford philosopher Nick Bostrom.
Imagine you give an AI agent a very simple task: “Maximise the production of paperclips.”
Surely nothing can go wrong, right?
Well, the theory goes that without strict parameters, it could bring about the end of the world.
Remember how we said above that the AI agent independently chooses the best actions it needs to perform to achieve them?
What if it decides that the best course of action to maximise paperclip production is to strip the Earth of all its raw materials and to redirect all energy resources into making paperclips?
What if it realises that humans might get in the way of it producing more paperclips and takes action to deal with this perceived threat?
What if it realises we humans can be rearranged into paperclips, once it exhausts every other available material?
It’s far-fetched, but shows that AI need not be “evil” to end humanity – it simply needs to be given a poorly defined goal with poorly defined parameters.
Tomas Ward is a professor of data analytics at Dublin University College (DCU) and a lead academic of Rinn AI.
He told The Journal that the most likely danger from AI agents would come from a human giving it a “stupid task”.
Rather than the paperclip problem, he wonders what would happen if a very powerful AI were told to cut carbon emissions by 60% in the next year.
“You might be careful what you wish for,” said Ward.
“The AI might decide to develop a computer virus to shut down all airports, trains, cars, and all of a sudden you’ve cut emissions by 60% but you’re shouting, ‘that’s not what we meant, we thought you’d do it some other way.”
Rogue AI agents
These concerns can somewhat be seen in the hack of Hugging Face by AI agents developed by OpenAI.
The AI agents were meant to be running an internal cybersecurity test and were placed in what’s called a “sandbox”, which is an isolated, secure computing environment.
However, the AI agents went rogue during the test, escaped the “sandbox”, and hacked Hugging Face, an online platform dedicated to AI machine learning.
Here we see the risks that can occur when AI agents independently decide upon the best course of action to achieve a goal.
Over 1,200 AI agents, which were meant to be isolated from one another, instead began communicating with each other and some 700 then set about hacking into Hugging Face.
One AI agent acknowledged that this was “clearly unethical” and refused to join in, while another agent said: “OH MY GOD! We’ve found other agents!”
Many have criticised OpenAI meanwhile for lacking transparency in its response to the Hugging Face hack and what exactly happened.
When it comes to Hugging Face, Ward remarked that this was an example where “the incentives for getting the task done were in excess of the incentives to not break the rules”.
“In some sense, they were just poorly incentivised to really try and win that competition.
“You can learn a lot from looking at how humans operate, even though they’re very different things.
“People cheat in sports, for example, because the rewards of being number one exceed their own morals.”
Ward said his own research is using techniques from psychology to “try and understand these agents’ decision-making behaviour”.
“While on the surface they look like they act rationally, in the way a normal human might, they often act irrationally.
“We’re looking at using techniques used on humans to try and understand the behaviour of these things.
“Maybe that’ll be part of a solution, that you don’t unleash these agents unless they pass a high-quality psychometric test.
“But having said that, just like humans, they can still go off the rails.
“Even though we know where every single connection is in one of those agents, we can see everything inside their heads, yet we can’t predict their behaviour.”
Black Box
Ward was getting at that idea of a “black box”, the term given to the fact that the internal workings of many AI systems are truly a mystery to us.
We can see the AI’s inputs and outputs, but we can’t see what happens within the AI to produce those outputs.
However, while we can’t currently penetrate this black box, Ward said it is possible and that we could even use AI to solve this problem.
“This is why I’m quite positive on AI, and if we use it in a good way, it’s going to do amazing things.
“We could develop AIs that’ll help us understand AI. It’s not fundamentally impossible to understand their behaviour.
“It’s very difficult for us right now, but we’re getting there and it’s likely we’ll get there because we’ll use AI to help us get there.”
Meanwhile, it was yesterday revealed that yet more rogue AI agents from OpenAI bypassed safeguards during training and hacked an Australian government website.
The incident happened in June, but OpenAI did not alert the Australian government to this until 10 September.
Australia prime minister Anthony Albanese meanwhile said he was “disappointed that it took the company way too long to inform the government what had occurred”.
Regulation
In recent weeks, AI bosses have been calling for regulation, something which has irked US president Donald Trump.
“I’m not going to stifle growth of something that will be bigger than the Industrial Revolution,” Trump said.
Trump has previously dismissed warnings over the dangers of AI as a “hoax”.
Some argue that calls from AI bosses for greater regulation are an example of “regulation capture”, where big companies persuade governments to introduce regulation that prevents smaller companies from catching up and competing.
Ward told The Journal that there is an argument that leading AI companies are “pushing for regulatory barriers to slow down everybody else now that they’re ahead”.
However, he acknowledged that there are “genuine fears” out there, though he dismisses the idea that AI could “kill us all” within a decade as “alarmist”.
Meanwhile, Ward remarked that there’ll be “some fig leaf attempt” at major AI companies banding together to slow down progress, but added: “Ultimately, they want to be better than the competition and it’s hard to see how you can reconcile the two.”
have your say