Skip to content
Pulsar InteractivePulsar Interactive

← Blog / Insights

What Doctors Are Missing About AI (It Isn't the Medicine)

· Updated · 6 min read · Pulsar Interactive ·

The rooms at Siriraj x MIT Hacking Medicine 2026 were full of doctors, pharmacists and health researchers who had already started using ChatGPT. The curiosity was high. The experimentation was real. What was missing wasn't motivation. It was structure.

Our team mentored hackathon teams over the weekend and spent the conference days talking through AI use cases with clinicians. The gaps we saw weren't about medical knowledge or technical aptitude. They were practical: how these tools actually work, where data can safely go, and how to know when an answer is trustworthy.

That matches the numbers. In the American Medical Association's 2026 survey of nearly 1,700 physicians, 81% already use AI professionally, yet 88% say safety and efficacy validation is critical and 86% say the same of data privacy [1].

Bar chart: 81% of physicians use AI professionally; 88% say safety and efficacy validation is critical; 86% say data privacy is critical; 85% want a say in AI adoption decisions.
Adoption has run ahead of the guardrails. Source: AMA 2026 Physician Survey on Augmented Intelligence [1].

Here's what we think every clinician working with AI should understand, and what most workshops skip.

1. A mental model of how LLMs fail

Large language models don't retrieve facts. They predict the next most probable words based on patterns in their training data. That means they can produce confident, fluent, completely wrong answers.

The problem isn't that clinicians don't know this in theory. It's that the experience of using ChatGPT doesn't reinforce it. The output sounds authoritative. The formatting looks professional. "It sounded right" becomes the default validation, and that's dangerous in a clinical context.

Understanding that these models are probabilistic, not deterministic, changes how you use them. You design workflows that assume the model might be wrong, and you build in checks before any output touches a decision.

2. Knowing where data can and can't go

Many clinicians we spoke with were using consumer chatbots for work tasks. Some knew about the privacy risks. Fewer knew there was an alternative.

The rule is simple: patient data never goes into a general-purpose tool. Not into a free chatbot, and not into any service that might use your input for training or store it outside your control. In Thailand, health information is sensitive personal data under the Personal Data Protection Act, which sets a higher bar for collecting and using it [2].

Enterprise setups (Azure OpenAI, Google Vertex AI, AWS Bedrock and others) offer contractual terms on how your data is handled. Consumer tools don't. Training opt-outs help, but they aren't enough on their own. If it's patient data, it belongs in a system your institution controls, with an audit trail.

Table showing which data can go where: published guidelines are fine anywhere; internal protocols only in enterprise or hospital-controlled systems; de-identified data needs approval in enterprise tools; identifiable patient data only in hospital-controlled systems, with approval.
A conservative default. Your institution's policy always comes first. Not legal advice.

3. Retrieval vs memory

A request we hear often: "I want the AI to know our hospital's guidelines."

The instinct is to assume the model can memorise your protocols. It can't, not reliably, and not without mixing them up with everything else it was trained on.

The better approach is retrieval-augmented generation (RAG): the system searches your own documents first, then uses those passages to ground its answer. You're not asking the model what it "knows" about liver disease. You're asking it to summarise the relevant section of your internal guideline, and to show you where it found it. In a 2024 benchmark of medical question answering, adding retrieval improved accuracy across six models by up to 18% [3].

Diagram comparing two flows: asking a model from memory produces a fluent answer with unknown source; grounding it in your guidelines produces an answer linked to the exact section.
Grounding doesn't make a model infallible. It makes every answer checkable.

This also solves the cross-contamination problem. A well-designed system keeps knowledge partitioned by domain, so a query about liver function doesn't pull content from a lung protocol.

4. How to test a tool before trusting it

No one should deploy an AI assistant without testing it first. But most clinicians don't have a testing habit, because they don't have a method.

Here's a simple one: build a small question bank of ten to twenty questions that represent real use cases. Run them through the tool. Grade each answer as pass, warn or fail.

Then test it through personas. What does it say to a doctor? A nurse? A pharmacist? A patient? Does it refuse questions it shouldn't answer? Does it escalate appropriately?

Four-step loop: write 10 to 20 real questions, ask them as each persona, grade every answer pass, warn or fail, fix the sources or rules, then retest.
A one-hour testing habit. No data science team required.

This requires a structured hour and a willingness to treat the tool like any other clinical instrument: something you validate before you rely on it.

5. Governance is the real blocker, not the model

The technology works. What stops most healthcare organisations from using it isn't the model. It's governance.

Compliance teams want guarantees that LLMs can't give: zero risk, deterministic behaviour, and liability someone else will carry. None of that exists. The World Health Organization's guidance on large multi-modal models makes the same point from the other side, calling for these tools to be built for clearly defined tasks, with clinicians involved from the start [4]. As WHO Chief Scientist Dr Jeremy Farrar put it: "We need transparent information and policies to manage the design, development, and use of LMMs." [4]

The answer isn't to wait for a perfect model. It's to decide what the system refuses to do, who reviews its outputs, and how you keep an audit trail.

Three governance questions: scope (what will it do and refuse), human review (who checks before anyone acts), audit trail (who asked what and what it answered).
Governance, reduced to three questions a compliance team can sign off on.

6. The underrated win: a place to ask "stupid questions"

One theme we keep hearing from clinicians is the psychological value of a private AI assistant.

Clinicians are expected to know everything. Asking a colleague a basic question can feel like admitting ignorance. A private assistant gives them somewhere to ask without judgment.

That's not a trivial benefit. It closes knowledge gaps in real time, reinforces learning, and reduces the load of staying current. The tool doesn't replace expertise; it supports it. Or, in the AMA's words, AI should be "designed to enhance, not replace, physicians" [1].

What a good AI workshop for clinicians should cover

Most AI workshops for healthcare are either too technical (how transformers work) or too abstract (the future of medicine). Neither helps someone use AI safely tomorrow.

A practical workshop should cover:

  • A mental model: how LLMs generate output, why they hallucinate, what that means for validation.
  • Data rules: consumer vs enterprise, what counts as patient data, where the legal boundaries are.
  • Hands-on practice: using the tool with their own non-clinical documents (guidelines, protocols, research summaries).
  • Testing habit: building a question bank, grading outputs, testing across personas.
  • Governance basics: scope, human review, audit trails.

The goal isn't to turn clinicians into AI engineers. It's to give them enough structure to experiment safely, and enough judgment to know when to stop.

We build this

Pulsar runs practical AI workshops and builds governed assistants for teams in healthcare and other regulated fields. If your organisation is experimenting with AI but hasn't figured out the governance piece, we should talk.

No hype. No one-size-fits-all demos. Just tools that work, and training that sticks.

Sources

  1. American Medical Association. More than 80% of physicians use AI professionally: AMA survey. March 2026.
  2. Kingdom of Thailand. Personal Data Protection Act B.E. 2562 (2019), English translation, Section 26 (sensitive personal data).
  3. Xiong G, Jin Q, Lu Z, Zhang A. Benchmarking Retrieval-Augmented Generation for Medicine. Findings of ACL, 2024.
  4. World Health Organization. WHO releases AI ethics and governance guidance for large multi-modal models. 18 January 2024.