AI Hallucinations in Medicine — The Problem Behind the Miracle Headlines
Three in five adults now ask a chatbot medical questions — most not knowing it was validated for none of it, and that it can hand you a confidently wrong drug dose. How to tell a real medical AI from a dangerous one.
The Promise
- A class of narrow, FDA-cleared medical AI validated against real patients — the tools that genuinely earn the word miracle.
- The 'narrow, measured, monitored' test that separates a trustworthy medical tool from a dangerous one.
- Three questions to ask before you act on anything an AI tells you about your health.
The Risk
- General chatbots predict the next word instead of retrieving a fact — which is how one hands you a confident, wrong methotrexate dose without a flicker of doubt.
- Three in five adults now ask chatbots medical questions off-label, on tools cleared for none of it.
- The models fail hardest exactly where it's most dangerous: rare conditions, wrong-jurisdiction guidance, and patients who don't fit the average.
Two different things wearing the same name
Three in five adults have asked a chatbot a medical question. Almost none of them know the tool was validated for nothing of the sort.
There are two technologies both called “AI in healthcare,” and the headlines blur them on purpose. The first is narrow, FDA-cleared software — a model that reads a retinal scan or flags a stroke on a CT, tested against real patients and cleared for one specific task. That one earns the word miracle. The second is the general chatbot people open to ask about a rash, a dose, a symptom. Same label. Completely different risk.
Why a chatbot is confidently wrong
A general chatbot does not look up a fact. It predicts the next word. Most of the time the most probable next word is also the correct one, which is exactly what makes the failures so dangerous — they arrive in the same confident voice as the right answers. Ask it for a methotrexate dose and it can hand you a wrong number with no flicker of doubt, because doubt is not something the mechanism produces.
An FDA clearance does not fix this. A clearance validates one narrow task, not the technology. The model that reads the scan was never cleared to answer your follow-up question, and the chatbot answering your question was never cleared at all.
Where it fails hardest
The failures cluster exactly where the stakes are highest. Rare conditions, where the training data is thin. Wrong-jurisdiction guidance, where it confidently cites the wrong country’s drug rules. And the patients who don’t fit the average — the ones a model trained on the median quietly serves worst. The average patient gets a usable answer. The patient who needed real care is the one most likely to get a fluent, wrong one.
Three questions before you act
One. Was this specific tool validated for this specific use, or am I asking a general chatbot to do a cleared device’s job? Two. Is the answer something I can verify against a named, authoritative source before I act on it? Three. Am I the average case this model was built for, or the edge case where it fails?
The narrow, measured, monitored tools are real medicine and they are getting better. The chatbot you opened to ask about a dose is not one of them. The deciding factor for your health isn’t whether AI is impressive. It’s whether a human checked before anyone acted.