Before You Press “Enable AI”: What Every Healthcare Leader Needs to Know

A colleague shows you an AI tool on their phone. It answers a clinical question in seconds, drafts a note, flags a drug interaction. It feels like the future in your palm. Then the governance officer in you asks the quieter question: whose data just left the building?

What AI looks like when it is done well

In our hospitals we are not theorising about AI; we are running it.

• AI scribes in consultations. Our pilot gave physicians their presence back: eye contact, undivided listening, while notes were produced in the background. Clinicians reported feeling less burdened and more connected to patients.
• AI meeting scribes. Committees, multidisciplinary reviews and board discussions now produce richer minutes and a searchable record of decisions. It is one of the most practical, lowest-risk uses of AI available, and almost nobody writes about it.
• AI in the call centre. Matching patients to the right physician, the next available slot and their insurance coverage turns a frustrating first contact into a service.
• Bespoke tools. We build and test small solutions for specific functions ourselves, inside governed pilots with clear criteria, timelines, clinical oversight and a willingness to stop.

The state of play

Shadow AI is already inside your hospital: a Wolters Kluwer Health survey found that 40% of healthcare staff had encountered unauthorised AI tools at work. Patients are not waiting either; tens of millions ask consumer chatbots health questions every day. ECRI named the misuse of AI chatbots the number one health technology hazard for 2026. And most medical AI is never reviewed by a regulator.

Six risks to govern

1. Confident and wrong. Duke University researchers analysed more than 11,000 real patient–chatbot conversations and found that real questions are emotional and ambiguous, and chatbots tend to agree. Never let AI have the final word on a clinical matter.

2. Privacy and data sovereignty. Uploading a patient summary into a consumer AI tool may breach data protection law. Every tool touching patient data needs the right agreements, data residency review and clarity on whether data is used for training.

3. Bias and the black box. Models trained on historical data inherit historical inequities. Demand explainability and demographic bias testing. Ask "accurate for whom?"

4. Model drift. AI quietly gets worse as populations and practice change. Treat AI as a living system with continuous monitoring, not a completed project.

5. Skill erosion. Clinicians who rely on AI for a task can lose proficiency without it. Preserve skills deliberately and keep a human pathway for every AI-dependent function.

6. Cost and complexity. Pressure for a quick return shortcuts governance. Slow down to go fast: phased, governed deployment with clear KPIs beats a rushed one every time.

A Connected Care framework for AI governance

Set up a multidisciplinary AI governance committee of management, IT and clinical leaders; no tool goes live on a vendor demo alone. Every tool must answer five questions: Does it work with our clinical systems? What data trained it, and does it represent our patients? Does it meet our privacy and residency obligations? Who integrates and supports it locally? Can it explain its reasoning so clinicians can challenge and override it? Then train before you deploy, monitor continuously, and always preserve a way for patients to reach a person.

Questions for boards and leadership teams

1. How much unsanctioned AI is being used in our hospital today, and what is our policy?
2. Which AI tools touch patient data, and do we have the right agreements for each?
3. Who monitors our AI tools for drift and bias after go-live?
4. Can every patient still reach a human being?

We should never close all the doors of human interaction.

First published in the Connected.Care newsletter on LinkedIn. Sources include ECRI's 2026 hazard report, Wolters Kluwer and Duke University.

Related reading: AI in hospitals: policy before pilot · Cybersecurity is patient safety