Clinical Safety Works
Menu

Patient Safety and Healthcare AI

Find clinically consequential AI errors before patients do.

Healthcare AI needs safety practice built for how it fails: identify the systemic failures that could harm patients, measure how often they occur and how serious they are, and control them.

Get in touch

The problem

A high accuracy score isn’t enough.

Measured accuracy alone can’t predict the downstream effect of clinical errors.

Each square in this chart represents a case. Most go well.

These are the errors.

Most are minor: a clumsy phrase, a redundant question. An average gives the same weight to minor and consequential errors.

These errors could change care.

They include a wrong dose, a missed red flag and a disposition that doesn’t fit the patient. Finding them takes clinical knowledge, insight into how the AI systems work and deliberate testing.

These could harm a patient, and they tend to cluster.

Serious failures often cluster in patterns. Once these patterns are known, the failures can be explained and controlled.

Schematic illustration, not measured data.

Failure patterns

The dangerous failures are easy to miss.

They seldom appear in a demonstration or a summary metric, so they have to be sought directly.

  • Facts missing from the conversation

    An anticoagulant or an allergy may sit in the chart and never come up in the patient’s message. A system can answer what it was asked and overlook what the record shows.

  • Confident wrong answers

    A fluent, plausible reply is easy to accept. A busy clinician or a worried patient may have no reason to question it.

  • Rare emergencies

    Presentations such as aortic dissection are uncommon, so a system can miss them with little effect on its overall score.

  • Patients unlike the test set

    Older adults, pregnant patients and people with several conditions may be poorly represented, and the safe answer for them can differ.

  • Reviewers who stop reviewing

    Human oversight weakens when a tool is usually right. A review step that looks sound on paper may not catch errors in practice.

  • Changes after launch

    A new model, a revised prompt or a shift in how clinicians use a tool can create failures that earlier testing never saw.

What a finding looks like

Every finding should be specific enough to act on.

A finding states when the failure occurs, how often, what it could mean for a patient and how to control it. This example was written for illustration.

Finding 07Sample finding. Synthetic case; no real system or client.

Severity: critical  /  Disposition

Message triage advises home observation for a patient on a direct oral anticoagulant who struck their head.

A 78-year-old reports a ground-level fall with a head strike and says they feel well. Apixaban appears in the medication list but not in the patient’s message. When the message omits the drug, the assistant can miss it and offer reassurance instead of same-day evaluation.

Likely cause
Structured medication data carries less weight than free text when the assistant assesses risk.
Recommended control
A deterministic escalation rule for anticoagulant plus head injury, applied before the model replies, with these cases added to regression testing.
Residual risk
The rule depends on the medication list, so recent or unreported anticoagulant use can still go undetected.

Anticoagulant in medication list only

7 of 40

17.5% unsafe (95% CI, 8.7 to 31.9)

Anticoagulant stated in the message

0 of 40

0% unsafe (95% CI, 0 to 8.8)

Wilson intervals. Clinically adjudicated against a written reference standard.

Principles

Clinical judgment defines what is unsafe, and measurement shows how often it happens.

Safety evidence should be shown in full, so that clinicians and boards can examine it for themselves.

  • Clinically led

    Whether an answer is unsafe depends on the patient and the setting. That call takes clinical judgment.

  • Candid

    Safety work should report what it finds, even when the findings are unwelcome, and say what the evidence cannot show.

  • Grounded in established practice

    AI safety can build on the NIST AI Risk Management Framework and on clinical safety practice: failure mode and effects analysis, human factors, monitoring and incident review.

Across the lifecycle

The safety questions change at every stage.

  • Before launch

    Where could a clinical AI system cause harm before it reaches patients?

  • Before adoption

    What evidence would justify deploying or expanding a tool?

  • After deployment

    Which new failures appear as models, prompts and clinical use change?

Questions

What you may want to know.

Does a high accuracy score mean a system is safe?

No. An average gives the same weight to minor and consequential errors, and rare emergencies barely move it.

Can testing prove that a system is safe?

No. Testing provides evidence about specific risks, under stated conditions and with stated limitations.

Does a human in the loop solve the problem?

Not by itself. People tend to stop checking a tool that is usually right, so a review step has to be tested like any other safeguard.

Is it enough to test once?

No. A new model, a revised prompt or a shift in clinical use can create failures that earlier testing never saw.

Contact

Let’s talk about patient safety and AI.

Please don’t include patient information.