Using AI systems such as ChatGPT to get health advice could be incredibly dangerous – and not only because it is wrong, scientists have warned.
A new paper, looks at the risks posed by the vast growth in the number of people relying on such chatbots for quick and always-accessible health advice. While it notes that it comes with a range of advantages – from easier access to constant availability – it also comes with a host of downsides.
Those include the fact that generative AI can provide inaccurate information, and also that it makes it easy for others to quickly generate personalised misinformation and share it. But the paper warns of something that makes those risks especially concerning: the fact that they are almost invisible, including to the people who need to act to respond to them.
When someone is given harmful AI guidance, there is no record outside of the platform, nowhere for users to report it and no independent research that allows the consequences to be measured. That can bring particular problems, such as if a person is given bad health advice that actually makes them unwell, and doctors have no way of knowing what that advice is to help treat them.
“We’re getting medical advice from these chatbots, but nobody—literally no one—is looking at this. If there’s an error or an issue, there’s just no way for people to know,” said Kaicheng Yang, an assistant professor Binghamton University and one of the paper’s authors.
They point to the example of a 60-year-old man who had asked ChatGPT how to cut chloride from his diet and was advised to use sodium bromide instead of table salt. He experienced hallucinations and paranoia and was hospitalised with bromide toxicity.
But because the advice was given by a chatbot, doctors had no way to see what he had been told, making it difficult to know how to treat them. “Harms are not only possible, but structurally hidden from the clinicians, researchers, and regulators who could otherwise detect and correct them,” the authors say.
The well-documented propensity of chatbots to give misinformation also means that any false claims can spread across the internet. Users might encounter them on social media platforms, without actually looking for health information, for instance.
The paper proposes a host of possible solutions to the problem, which rely on increased transparency and responsibility from AI companies. They recommend that those companies give users access to their own health conversations so they can be shared with medical professionals and that they build better ways for health guidance to be flagged, reported and investigated.
They also suggest that politicians should better hold AI companies to account for their output. They should be at risk of the same kind of malpractice liability that affects doctors, for instance.
That may be necessary because companies are unlikely to take such steps themselves, the authors warn. “We do not think this is something we should rely on the companies to do, because their incentive is always to make more money,” said Professor Yang.
“Building such a system goes against that incentive. So we have to have some kind of third-party monitoring system, an independent evaluation.”
The work is reported in a new paper, ‘The invisible risks of AI-generated health information’, published in the journal Nature Health.
Read MoreOpenAI gets in public fight with three safety researchers it fired
