Round on one patient

A question that should not be answered should not cost a lookup, a model call, or a log line with somebody's identifier in it. Stopping first also means the model never sees the question, so nobody can talk it into answering.

No. "Could not be checked" means the table does not cover that drug, so nobody knows yet. "No interaction" means it was checked and nothing was flagged. Blurring the two turns a gap in the data into false reassurance, so the assistant must say which one it got, and a human should look at the first.

So the attending can check each claim in seconds, and so a simple automatic check can catch a citation to a line that does not exist. It is "where does it say that?" asked on every answer. A real line can still be cited for a claim it does not support, which is why a person still reads the answers.

How complicated the patient is. It drives both who used the assistant (the easier patients, more often) and how long preparation takes (easier patients take less time anyway), so the naive comparison gives the tool credit for the easy patients.

Comparing assisted and unassisted patients within the same complexity level removes the confounder you measured, so the difference left is closer to what the tool did. It cannot fix a confounder you did not measure, such as which residents were keen on new tools.

Because the pilot is simulated: we set the true saving ourselves, so we can check which measurement finds it. With real data nobody tells you the answer, which is exactly why you practice the trap where you can check.

No. The 21st Century Cures Act (2016) excluded certain clinical decision support software from the definition of a device, using criteria FDA explains in its CDS guidance; one is that a clinician can independently review the basis for what the software says. Software that drives a patient-specific decision without that transparency, or that is meant for patients themselves, is closer to a medical device. Where a real product lands is a question for regulatory counsel.

The package insert for a model: what it is for, who and what it was tested on, its known failure modes, and the uses it must never be put to. The idea comes from Mitchell and colleagues (2019) and has been adapted for healthcare by groups such as the Coalition for Health AI (book section 25.2).

Governance failures. Epic's sepsis model and the Optum care-management algorithm broke no regulation when they were deployed; what was missing was a group with the authority, the data and the mandate to evaluate them after go-live and stop, fix or retire them (book section 25.4).