There’s an old joke in digital that ‘AI’ is anything that doesn’t work yet. Once it starts working we call it something else: ‘software’ or just ‘the system’. AI chatbots like ChatGPT and Co-Pilot have only recently arrived on our laptops and phones, but the NHS has been using forms of artificial intelligence for many years.

“Think of things we take for granted now: speech to text, that used to be AI; image recognition, that used to be AI,” says Paul Jones, chief digital and information officer at Leeds Teaching Hospitals. “But now they work, we give them a different title.”

Leeds has been running an AI-supported breast cancer screening programme—originally developed in response to a shortage of radiologists—for several years. Scans are first viewed by an AI system and a consultant radiologist, Jones explains. “If they agree, that’s fine. If not, then a second radiologist gets involved. So, you’re getting twice as much work out of this quite rare resource. That’s brilliant.” Nowadays, he adds, the AI is “about as good as the consultants but in slightly different ways”.

This is ‘machine learning’, a more ‘traditional’ form of AI that’s trained to recognise patterns in data and images. We tend to disapply the scary ‘AI’ label to these technologies because they’re tried, tested and trusted.

Many of the newer generation of AI tools in which the government places so much faith fall into a different category: ‘generative’ AI, which produces new content by ‘scraping’ existing information sources. In the NHS, this includes ambient voice technology (or ‘AI scribes’) which listen to consultations and produce notes, or ‘decision support’ systems aiming to help doctors diagnose conditions and recommend treatments.

Speaking to managers and clinicians over recent months, trust in these tools seems highly variable—ranging from a generalised mistrust of anything AI to the kind of breathless enthusiasm that could land you a job at the Tony Blair Institute.

Generative AI raises different safety and governance questions to traditional AI, says Felix Peckitt, assistant director of data architecture at West and North London ICB, and MiP National Committee rep for London. “The outputs… can be very plausible even to experts. So the problem is that something being plausible and looking right is no longer enough to tell us whether it’s true,” he explains.

This “plausible bullshit” will be familiar to anyone who’s taken Google’s ‘AI Summaries’ at face value. Then there’s ‘agentic’ AI’, a form of generative AI which actually does things, like sending emails, booking appointments or ordering supplies. “That also becomes a problem of accountability,” adds Peckitt. “When programmes are interacting with people, interacting with other software and doing things, you need to monitor that and have clear accountability.”

We have to solve these problems because, unlike drugs or traditional AI, everyone has access to generative AI tools. The increasing use of ‘shadow AI’—tools not tested or approved for the purposes they’re being used—in healthcare demands a national, organisational and managerial response.

“We’re increasingly seeing clinicians turn to ChatGPT because they use it for everything else,” says Jordan Fulcher, a clinical pharmacist and consultant for AI-powered solutions provider Wolters Kluwer. “The genie’s out the bottle… we know that giving people a slap on the wrist and saying, ‘You shouldn’t be doing this,’ won’t stop them using these tools.”

Tom Micklewright, GP and digital lead for Cheshire and Merseyside ICB, says some newly-trained GPs have been using ChatGPT for medical queries without realising the dangers.

“ChatGPT doesn’t use journal sources, it will be pulling in data from Reddit, blogs and everything else,” he explains. The data these GPs are inputting will be used to train the ChatGPT model, raising security risks too. “The data centres are all in the US and aren’t necessarily GDPR-compliant,” he says.

The big difference between this and just Googling medical information—which doctors have been doing for years—is that you have to use your doctor brain to make sense of search results, but ChatGPT pretends it actually has a doctor brain.

At Leeds, generative AI deployment is currently limited to the AI scribes used in emergency departments, Jones explains. The risk here is “that we train doctors to be lazy and just click on it without checking the notes”. But doctors have always known that not checking notes means “gambling with their professional registration”, he says. “I think [the risks] are mainly about training, culture and responsibility.”

Good governance of generative AI demands transparency about information sources and “making sure the underlying content is really trusted,” says Wolter Kluwer’s Jordan Fulcher. The firm recently developed an AI tool that sits on top of its ‘UpToDate’ library of peer-reviewed medical information, used by NHS doctors for over 30 years. It’s now being trialled by several NHS trusts.

Clinicians trust the system because “it gives you the recommendations you would want if you asked another doctor,” says Fulcher. Unlike ChatGPT and Co-Pilot, “it has the evidence and the references and you know where it’s coming from, because all the authors are clinicians.”

While individual clinicians must understand any AI tools they’re using to make clinical decisions, there’s also a duty on trusts and the NHS as a whole to “support them and make sure they’re [using] with tools that are trusted,” he adds.

UpToDate doesn’t serve up any AI-generated content, the AI tool is just a way of interacting with already-trusted information. But that’s not the case with the scores of mental health chatbots that have appeared in recent years—many downloadable, without any clinical oversight, from app stores.

“A whole bunch really have no trust because there’s nothing backing them up,” says Ross Harper, chief executive of Limbic, the only mental health chatbot licensed as a Class IIa medical device. “There’s no clinical evidence, no third party validation and no live deployments in clinical settings.”

Limbic is now used by around 60% of NHS Talking Therapy services. “Our entire strategy is to leverage trust,” Harper says. “Being peer-reviewed… is one way to build trust. Third party accreditations on information governance, data security and quality management is another. The best way to build adoption [of AI tools] is to be the most trusted solution.”

Proper governance, guardrails, building trust – these things take time. They can’t be rushed by politicians eager to find a crock of gold at the end of the AI rainbow. But could our light-touch regulatory approach and the associated lack of national guidance on using AI—generative AI in particular—actually be holding up deployment rather than speeding it up?

A commission led by Professor Alastair Denniston is reviewing the current regulatory regime, which is fragmented, confusing and relies heavily on self-reporting by tech firms. In May, the UK medical devices regulator proposed allowing approval of many generative AI tools without any independent scrutiny (see below). In the same week, the Health Service Journal reported that safety concerns with AI scribes used in dozens of NHS hospitals were going unreported because staff didn’t understand the regulatory system.

In a change of approach, NHS England published a list of approved AI scribes last year, but GP Tom Micklewright says it wasn’t as useful as hoped. “AI providers just self-certified to be on the list,” he says. “No one’s looked at their documentation. No one’s made sure they’re safe, which means it still falls on GPs or doctors to do it.”

Duplication with AI assurance is “burdening clinicians with more mind-numbing paperwork” and “wasting millions”, Micklewright claims. He wants to see AI treated like drugs, with “a central process that says, ‘Everything here is safe, approved, meets the standards. We’ll take care of that, you just make sure you’re using it correctly.”

Unsurprisingly, his view finds an echo in parts of the tech industry that have taken the trouble to meet independent quality and safety standards. “Proper regulation is the way forward,” but we need to redefine how it will work for generative AI because it’s a consumer-level product that anyone can access,” says Fulcher.

Limbic’s Ross Harper warns that good innovation will be stifled without a consistent regulatory regime and national contracts for developing AI tools. “It can’t work if every trust and region has to do their own evaluation—they’re just not set up for that,” he says. “If the evaluation markers are clear, transparent and definitive, everybody can get on board with why one solution was chosen above others.”

These problems can hamper deployment of more established forms of AI too. Frimley Health was only the second trust in England to develop an AI tool for echo imaging, ”and my goodness it has been really tricky to implement that,” says Kavitha Vimalesvaran. “The procurement and governance is often fragmented, leading to duplication, and it slows adoption down massively because everyone is working in silos.”

While many AI tools have regulatory approval, she explains, few can show hard evidence that “they impact on patient outcomes or staff workload, on health economics or inequalities. And demonstrating cost savings, she adds, “is very difficult”.

Leeds digital chief Paul Jones is sceptical about the need for new governance and regulatory frameworks specifically for AI. Clinicians have been finding new ways to treat patients “for donkey’s years”, he says, and the trust has well-established processes for ensuring new clinical procedures and treatments are safe, consistent and backed by evidence. “Why are we treating AI as something different? Just because it runs on a computer?” he asks.

Jones explains that the AI clinical governance team at Leeds reports into the trust’s quality committee—a clinical route to assurance which builds on existing governance structures. “It’s not a digital thing,” he says. He accepts “people have worries about AI” but the answer isn’t “to start from scratch”, he insists, but “to start from the things we’ve got.”

To succeed with any new technology, we need to harness the power while taming the risks — something NHS managers “do day-in, day-out”, says MiP National Committee member Felix Peckitt. “Being able to clearly specify a problem and the steps needed to resolve it is a core managerial competency — and also something that gets really good results with AI.”

It’s a huge challenge that managers, and MiP as their union, need to get to grips with. Managers are used to operating “in a highly regulated environment where the stakes are high”, Peckitt explains, and their skills in critical thinking, delegation and risk management, together with an understanding of accountability and governance, mean they’re “ideally placed” to take professional responsibility for the safe and effective implementation of AI.

Professional registration, already being introduced at the most senior levels, will be another advantage, Peckitt reckons. Without an “infrastructure of professional registration it could be really difficult to implement AI safely – you can’t be struck off for being a rogue software engineer,” he says. “And professional registration as a manager is one of the few marks of accountability which can’t be replaced by AI. //