Advisor, confidant and friend (without benefits)
- Erna van Wyk
AI: Can we trust the advice that it gives us, and the motives behind the people who build the systems?
The warning on the opposite page sounds sensible enough, and yet these are precisely the high-stakes areas in which people are turning to generative AI chatbots such as ChatGPT, Claude and other large language models (LLMs).
The concern extends beyond the professional advice that people are seeking. They are also looking for life and relationship guidance from AI, drawn by its 24/7 availability, low cost and perceived security. Studies suggest that users mostly prefer chatbots that are agreeable, even sycophantic, in their offering of interpersonal advice – including when they affirm harmful or illegal behaviour.
Why trust a chatbot?
“Anonymity is another big factor, especially in the mental health and medical fields where stigma and prejudice still exist,” says Professor Sumaya Laher, a psychologist and Head of the µÚÒ»³Ô¹ÏÍø School of Human and Community Development. “The assumption is that if you ask an AI chatbot something, you’ll have anonymity and protection. The perception is that these are platforms that provide accurate, objective information, and what you get back will be more reliable than a friend’s support.”
She adds: “Unlike WebMD and similar platforms where people used to check their medical symptoms, GenAI is more interactive, almost like a doctor or therapist. Its responses sound reasonable and accurate.”
Laher says that people form a connection with an AI chatbot because it is accessible, affordable and anonymous. However, quite often people do not realise the harm that it can do precisely because AI is not always 100% accurate. “Given that much of what is fed into AI is developed in the Global North, it may miss cultural nuances that are relevant in countries like ours. Even though it feels like a very real connection, it isn’t.”
The cost of trust
In just four years, GenAI has made information and advice available to anyone with an internet connection.
“Many of my students have disclosed that they take their relationship advice from an AI chatbot instead of a friend, a counsellor or a therapist,” says Dr Martin Bekker, Lecturer in the School of Electrical and Information Engineering and Fellow in the µÚÒ»³Ô¹ÏÍø Machine Intelligence and Neural Discovery (MIND) Institute.
Yet trust in AI advice raises ethical questions beyond accuracy.
As an ethicist, Bekker is concerned with the question around autonomy, the principle that individuals have their own identity and make their own choices. “If we violate autonomy, if we manipulate people, such as when we use blackmail, encourage addiction or gaslight people, then we are normally violating some ethical norm. There are all sorts of ways in which we can take autonomy away from people, and the most obvious is lying,” he says.
“With AI chatbots there is an interesting infraction of autonomy in that we have to pretend that this is a real person, so we are already kind of fooling ourselves. Then we essentially ask it for guidance that influences our real behaviour in the world. Our autonomy is influenced by that and it’s a super tricky thing.”
Anthropomorphosis
Humans humanise almost everything: we see faces in clouds, we form attachments to inanimate objects and we believe that our animals understand what we say. “We cast human characteristics into everything,” says Bekker.
We are no different when it comes to AI chatbots, except that anthropomorphising is deliberately built into the design. AI systems are trained on large amounts of internet data to predict the next word. “However, what made them useful was reinforcement learning – a kind of feedback reward loop between human and machine – introduced with GenAI,” says Professor Steven James from the School of Computer Science and Applied Mathematics and Fellow in the µÚÒ»³Ô¹ÏÍø MIND Institute. “GenAIs are designed in ways that reinforce users’ beliefs and viewpoints, making them easier to trust. Humans generally prefer being told that they are correct and AI chatbots are trained to reinforce those beliefs.”
He adds: “Today you will see the little thumbs up or thumbs down emojis in the prompts and even if you do not use them, the fact that you are continuing the conversation indicates that you clearly enjoy talking to the AI chatbot. So, it's just going to reinforce that signal – that whatever it's doing, it should keep on doing. A lot of it boils down to the fact that it will continue to give the human what the human wants to hear.”
Upstream or downstream — where does responsibility swim?
Associate Professor Michele van Eck from the School of Law and Fellow in the µÚÒ»³Ô¹ÏÍø MIND Institute says that while AI systems and chatbots are wonderful resources if used correctly, there is “trust in thinking that these systems are producing accurate information”.
“We forget that we need to check, verify and make sure the information given to us is correct,” says van Eck.
She adds that technology companies are increasingly relying on transparency measures, disclaimers and ethical guidelines to build trust but that regulation is struggling to keep pace with the technology, leaving users exposed to growing privacy and security risks.
"Data is the new oil. When people share personal information with AI chatbots, it can be connected to their broader digital footprint, creating a powerful tool for targeted marketing and potential misuse. At worst, it creates opportunities for hackers and identity theft when different pieces of personal information are brought together."
It's not just about understanding terms and conditions – there is also a need for education.
“It's easy to say, 'just read the terms and conditions’. But how is the average person supposed to make sense of them, if half of the lawyers don’t understand them?” asks van Eck. “Even when they're written in plain language, most people still don't understand what they're agreeing to.”
Then what do we do? We paste the T&Cs into AI and ask it to explain them. But is that truly an objective engagement?
Professor Rennie Naidoo from the School of Business Sciences is just as concerned. “People are increasingly trusting AI with very personal aspects of their lives, whether it’s their personal data, ID numbers, thoughts or feelings. What concerns me is that this seems to be happening quite easily and without much thought. There appear to be very few guardrails.”
For Naidoo, AI systems may warn users not to share sensitive information but in many cases they still accept that information. “Users should not assume it will be automatically rejected, deleted or excluded from further processing,” he says.
Naidoo believes that designers and technology companies need to take responsibility for user safety.
"The designers of AI want us to keep using these systems. Like social media, there are business incentives behind them and that can shape how they're designed to keep users engaged. That's why the risks go beyond just advice and decision-making.”
Companion chatbots
One of the biggest concerns today is the rise of companion chatbots such as Replika, says Dr Victoria Williams, from the School of Computer Science and Applied Mathematics and Fellow in the µÚÒ»³Ô¹ÏÍø MIND Institute.
The loneliness epidemic that intensified after the Covid-19 pandemic is still a major concern today as more people are working remotely, living on their own and having little contact with other human beings.
“I’m very concerned about companion chatbots. When you go on to a companion chatbot, it doesn’t have any disclaimers. It doesn’t say, ‘please note that this is not a human’. People will log in and automatically feel connected and feel like, ‘oh, it understands me in ways people don’t’, as opposed to recognising that this is actually based on how it has been designed,” says Williams.
The danger is not merely that people believe that AI is accurate but that they begin to experience AI as a relationship.
- Erna van Wyk is Senior Communications Officer at µÚÒ»³Ô¹ÏÍø University.
- This article first appeared in CURIOS.TY, a research magazine produced by  and the .
- Read more in the 21st issue, themed #Trust — the invisible glue binding society, science and technology.