AI technology that listens to and transcribes patients’ consultations with doctors can put them at risk by getting the names of drugs and illnesses wrong, an NHS watchdog has warned.
In one case a woman was left badly shaken when the AI scribe’s summary of her conversation wrongly said she had demyelination – serious nerve damage that can lead to multiple sclerosis.
It was only when the patient, an NHS health professional, queried the AI tool’s record of the result of her MRI scan that the hospital corrected it to what it should have been – “null demyelination”.
“This was eventually corrected but was a very traumatising experience to be given an incorrect diagnosis because of AI and then be told it’s a typo,” said the woman, who asked not to be named.
Healthwatch England has highlighted the case and other mistakes by AI scribes to show that the technology, which the NHS is rolling out rapidly, is a potential threat to patient safety. Unless detected, errors could end up in patients’ medical records and affect their care, it said.
In another case, an AI scribe confused the drug the GP had prescribed with a different one of a similar name – a blunder which again the patient, rather than the doctor, identified. On another occasion an AI-generated summary letter did not say that the hospital consultant had told the patient to seek a repeat prescription from their GP for their migraine, which could have left them unable to get their medication.
Healthwatch, the statutory NHS patient champion, has heard “multiple stories from patients who have noticed these errors when a health professional hasn’t”, it said. “These inaccuracies may persist in their records if the patient doesn’t catch them.”
The government’s 10-year health plan for the NHS in England expects AI scribes to “liberate staff from their current burden of bureaucracy and administration, freeing up time to care and to focus on the patient”. It is central to the planned “big shift” for the NHS from being an analogue to a digital-based service.
A Healthwatch spokesperson said: “Healthcare has never been error-free. But our findings show the urgent need for clarity over how patients can report and get corrected any mistakes made by AI scribing tools or the professionals that use them.”
Rachel Power, chief executive of the Patients Association, said: “Trust and confidence in this technology depend on good communication and genuine partnership with patients and right now both are missing.”
Healthwatch added that it was “worrying” that the Medicines and Healthcare products Regulatory Agency has decided not to classify AI scribes as medical devices, which means there will be no England-wide oversight to ensure they are safe to use and effective.
GPs and hospital doctors in England are already using 27 different AI scribes.
Ministers have been warned that the NHS and medics could be sued over mistakes made by AI.
AI’s accuracy was in the spotlight recently when patients in Rotherham complained to their local Healthwatch that an AI receptionist used by some local GP practices did not understand their strong Yorkshire accents.
Dr Shier Ziser Dawood, a GP in London, last year warned in a leading medical journal that AI scribes may prove “a double-edged sword” for family doctors.
She recounted that an AI scribe said she had told her patient to “continue their Prozac” even though she had not prescribed or discussed that drug with them. That is an example of what are known as “hallucinations”, where AI scribes refer to something that was not raised during the consultation.
She warned that the need for doctors to review all transcripts in order to check for errors meant that AI tools are not yet proving time-saving.
Given the belief of NHS bosses that scribes will mean GPs no longer have to take notes during an appointment, family doctors may be expected to see two more patients every day, even though NHS GP consultation times are already some of the shortest in the world, Dawood added, writing in the British Journal of General Practice.
Dr Charlotte Blease, an expert in AI use in healthcare at Uppsala university in Sweden, said: “AI can and does make mistakes.”
Her research found that GPs who use ambient voice technology believe that errors are more likely to creep in when the consultation is with more than one person, with patients with a complex medical history, and with those whose first language is not English.
But, she added: “The fact is, doctors can and do make mistakes without AI. And it is certainly possible the error rate is worse.”
More than half the 1,003 UK GPs in her survey last year believed that their ambient AI records were more accurate than those produced themselves, Blease said.
