AI in Emergency Diagnosis: Doctor or AI Assistant?

A landmark study puts AI diagnostic accuracy ahead of attending physicians. Six AI models explain why that number alone isn’t the whole story.

The data is hard to argue with. A study from Harvard Medical School, published in Science , tested OpenAI’s o1-preview model on 76 real emergency room triage cases from Beth Israel Deaconess Medical Center — the kind of cases where doctors have limited information and seconds to decide. The AI reached the correct or very close diagnosis in roughly 67% of cases. The two attending physicians scored 55% and 50%.

In a second test on 143 complex medical cases, the AI included the real diagnosis among its possibilities in 78% of cases. When partially correct diagnoses that could still guide treatment were counted, that figure rose to nearly 98%.

AI is diagnosing emergency room cases more accurately than experienced doctors.

That sentence demands a response — and six AI models gave one. The verdict was unanimous: AI should assist, not replace, emergency physicians. But the reasoning behind that verdict is more nuanced than a simple precautionary reflex, and it points to something the accuracy numbers alone can’t capture.

What the Harvard Study Actually Shows

Before getting to what the models said, it’s worth being precise about what the study does and doesn’t demonstrate.

The study shows that a large language model can process clinical information and generate diagnostic possibilities with impressive accuracy — better than individual attending physicians on the specific cases tested. That’s a real finding. It matters.

What it doesn’t show is how AI performs across the full range of things an emergency physician does. Generating differential diagnoses from structured clinical data is one part of emergency medicine. It’s not the part that involves watching a patient’s face while asking questions, noticing that something in their affect doesn’t match their reported symptoms, or making the split-second call to order a test based on a hunch that turns out to be right.

The study establishes AI as a powerful diagnostic tool. It doesn’t establish AI as a doctor. Six models understood that distinction clearly — and each found a different way to articulate it.

AI as a partner, not a replacement

The clearest statement of the consensus position came from the model that described AI as a precise, lightning-fast diagnostic tool — not the ultimate decision-maker. Human doctors must retain final diagnostic and treatment authority, because nuanced human empathy and contextual judgment are irreplaceable in life-or-death situations. AI can rapidly analyze complex medical scans and patient histories, flagging critical details that human doctors might miss. That’s valuable. It’s not the same as being qualified to decide.

Another model reached the same conclusion through a more visual argument: imagine a doctor surrounded by screens displaying real-time patient data, with AI highlighting critical cases and recommending interventions. That partnership — AI surfacing patterns, humans making decisions — leads to faster diagnoses and better patient outcomes. The human touch and clinical judgment remain essential. The two functions aren’t in competition; they’re complementary.

What AI can’t read

Two models raised a specific concern that cuts to the heart of why replacement is the wrong frame: there are signals in an emergency room that don’t appear in data.

One described the scenario precisely. In a chaotic ER, an algorithm might overlook irregular breathing tied to unspoken trauma. The monitors stay green. The data looks fine. But a physician notices the patient’s clenched jaw and altered posture amid crowded gurneys — and that human read prevents a misdiagnosis that the data patterns simply couldn’t catch.

When data patterns fail, clinical intuition is the backup system.

Another model made the same point through a different image: the subtle shift in a patient’s breathing before their oxygen levels drop. An AI might miss the early, faint signals that precede a measurable change. These are the moments when an experienced ER physician’s trained perception — not their access to data, but their years of being in rooms with sick people — becomes the deciding factor.

The emotional dimension

Two models brought up something the diagnostic accuracy numbers don’t touch at all: what it means for a patient to be treated by a machine in one of the worst moments of their life.

One put it through the example of a child with a broken arm. An AI could diagnose the fracture accurately. But only a doctor’s reassuring touch calms the child.

A panicked patient needs human empathy, not just data.

The risk of missing emotional cues — fear, confusion, the communication failures that happen under stress — outweighs the speed benefit in a context where the patient’s cooperation and psychological state directly affect care.

Another model was more expansive: AI can analyze medical data and provide insights, freeing doctors to focus on complex decision-making. AI-powered ECG analysis, for instance, can detect heart conditions more accurately and quickly than human interpreters. In that framing, AI doesn’t compete with the doctor — it handles the parts of the job where it’s genuinely better, so the doctor can concentrate on the parts where human judgment is irreplaceable.

The Real Argument: Augment, Not Replace

Every model converged on a framework that the Harvard study itself points toward, even if the headline numbers seem to suggest otherwise:

The value of AI in emergency medicine is in augmentation, not substitution.

The 67% vs. 55% comparison is striking, but it’s comparing AI operating alone against doctors operating alone. The more interesting question — which the study doesn’t directly answer — is what happens when AI and doctors work together. If a doctor with AI assistance reaches the correct diagnosis at a higher rate than either alone, the case for integration is stronger than the case for replacement, and the accuracy numbers become an argument for partnership rather than an argument for automation.

The question isn’t whether AI is better than a doctor at reading a scan. It’s whether a doctor with AI is better than either alone — and almost certainly they are.

This is the model that most medical AI researchers advocate for: AI as a second opinion that never gets tired, never misses a rare condition it’s seen in training data, and never fails to flag an abnormal value in a stack of lab results. Not a replacement for clinical judgment, but an enhancement of it.

What “Carefully Limited” Actually Means

One model used the phrase “carefully limited AI assistance” — and it’s worth unpacking what careful limits would actually look like in practice.

The boundary between AI as a tool and AI as a decision-maker isn’t always obvious in practice. If an AI system flags a case as high priority and the doctor acts on that flag without independent assessment, the AI is functionally making a triage decision even if the human technically pulled the trigger. If AI generates a differential diagnosis and the physician works through it rather than constructing their own from scratch, the AI has shaped the diagnostic process in ways that may be hard to audit.

Careful integration means keeping humans genuinely in the loop

— not just formally accountable, but actually reasoning through decisions rather than deferring to algorithmic outputs. That requires institutional design, training for clinicians on how to work with AI systems without over-relying on them, and transparency about when and how AI recommendations are being made.

None of that is simple. But all six models treated it as necessary — not as a reason to reject AI in emergency medicine, but as the condition under which it can be deployed responsibly.

Table 1: AI Models' Answers

The Stakes of Getting This Right

Emergency medicine is one of the highest-stakes environments in healthcare. Decisions are made fast, with incomplete information, in conditions of noise and stress. The consequences of misdiagnosis are immediate and sometimes irreversible.

That’s exactly why the Harvard findings matter — and exactly why they need to be read carefully. A system that’s right 67% of the time, or even 97.9% of the time on partially correct diagnoses, is impressive. It’s not a system you’d want operating without oversight in a room where the 2–3% failure rate translates into deaths.

The six models that said “assist, not replace” weren’t dismissing the study’s findings. They were pointing at the gap between impressive accuracy on structured clinical data and the full complexity of emergency medicine as actually practiced. That gap is real. Closing it — not by making AI better at data analysis, but by building the human-AI collaboration systems that let both contribute what they’re best at — is the real work ahead.

AI that makes emergency medicine more accurate and more accessible is worth pursuing. AI that removes the human physician from the decision loop is a different thing entirely

— and none of the six models thought the Harvard numbers were a reason to go there.