MMHA 6601 Week 5 AI Imaging Triage Appraisal Example

Faster for Whom? Appraising an AI Worklist Triage Tool for Head CT by Intended Use, Independent Validation and Turnaround

Student Name

Walden University

MMHA 6601: Technology and Innovations in Healthcare

Instructor Name

Month Day, Year

Faster for Whom? Appraising an AI Worklist Triage Tool for Head CT by Intended Use, Independent Validation and Turnaround

The Tool and the Claim

The partner hospital that reads imaging for Ridgeway Health Partners, the invented fourteen-practice group in this course's scenarios, is considering an artificial intelligence tool that analyzes head CT scans as they are completed and moves studies with suspected intracranial hemorrhage to the top of the radiologist's worklist. The vendor's materials claim high sensitivity, faster diagnosis and better outcomes. Ridgeway's leaders, whose patients are scanned in the hospital's emergency department and outpatient center, asked for an appraisal they could use to advise the hospital.

This paper separates the claim into three parts and tests each: what the tool is cleared to do, how well it performs on data it was not trained on and whether reordering the worklist actually shortens time to diagnosis for the patients who matter.

Intended Use

Tools of this kind are generally cleared by the Food and Drug Administration as computer-aided triage and notification devices. That category matters: the tool does not diagnose. It flags a study for earlier review, and the radiologist still reads every scan in full. Any claim that the tool improves diagnostic accuracy goes beyond its intended use. The proper claim is narrower: it may get suspected bleeds read sooner.

An analysis of FDA clearances for artificial intelligence devices found that most were evaluated on retrospective data, many at a single site, and few reported performance across demographic subgroups (Wu et al., 2021). The appraisal therefore asks for evidence beyond the clearance summary.

Validation Evidence

The systematic review found that only a small share of deep learning imaging studies validated their models externally, and that reporting was often poor (Liu et al., 2019). Performance on vendor data says little about performance on a community hospital's scanners, protocols and patients. A silent trial, in which the tool flags studies in the background without affecting the worklist, measures local sensitivity and the false-positive rate without risk. It also shows whether performance holds across the hospital's two scanner models and across patient groups, such as older adults with prior strokes or calcifications that might confuse the model. The trial requires the vendor's cooperation, a radiologist to adjudicate disagreements between the tool and the final report, and an analyst to compile results, but it costs little compared with the license.

Table 1

Evidence Considered

The most useful evidence is local. The appraisal recommends a silent trial before any worklist change.

Questions to Put to the Vendor

Before the trial, the hospital should ask the vendor five questions in writing. On what data was the model trained, and from how many institutions? Has it been validated externally, and are those results published? How does performance vary by scanner manufacturer, slice thickness and patient age? What happens when the model is updated: will the hospital be told, and will performance be rechecked? And who is responsible if a flag is missed? The answers will not settle the appraisal, but evasive answers would be a warning sign, and the update question matters because a model that changes silently could perform differently from the one the hospital tested.

Effect on Turnaround

Even a perfectly accurate flag helps only if the worklist is the bottleneck. At the partner hospital, emergency head CTs are already marked stat and read within a median of about twenty minutes during the day, an illustrative figure for this paper. The gain from reordering would be largest at night, when one radiologist covers several hospitals and the worklist is long, and for outpatient or inpatient scans not marked urgent, where an unexpected bleed may wait hours.

That points to where value lies. A health system study found that flagging reprioritized studies and shortened the time to diagnosis for unexpected hemorrhage among routine scans (Arbabshirani et al., 2018). The patients most likely to benefit are not those the emergency department already suspects of bleeding, but those whose scans were ordered for something else.

False positives have a cost too. Each false flag moves a normal study ahead of others, delaying patients whose scans were actually more urgent for other reasons, such as a suspected stroke without bleeding. If the false-positive rate is high, the radiologist learns to discount flags, which is the alert fatigue problem in a new form. Challenges in moving AI from accuracy to clinical impact include exactly these workflow effects (Kelly et al., 2019).

Recommendation

The hospital should run a 90-day silent trial, measuring local sensitivity, false-positive rate and the number of hemorrhages found on scans not marked stat. If local sensitivity is acceptable and false positives are below a level radiologists agree on in advance, the tool should go live for night and non-stat studies first, where turnaround gains are likely, with monthly review of time to read for flagged studies. Ridgeway should ask the hospital to share results, because its outpatients are among those who stand to benefit. A short report to the network's medical directors every quarter, covering flagged studies from Ridgeway patients and the time to final read, would let primary care physicians know what to expect when they order a head CT for an older patient after a fall.

Conclusion

The AI triage tool may help, but not in the way its marketing suggests. It is a reordering tool, not a diagnostic one; its published validation is mostly not independent; and its benefit depends on where the worklist is slow. A silent trial and a targeted launch let the hospital learn whether the promise holds on its own patients.

References

Arbabshirani, M. R., Fornwalt, B. K., Mongelluzzo, G. J., Suever, J. D., Geise, B. D., Patel, A. A., & Moore, G. J. (2018). Advanced machine learning in action: Identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. npj Digital Medicine, 1, 9. https://doi.org/10.1038/s41746-017-0015-z

Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019). Key challenges for delivering clinical impact with artificial intelligence. BMC Medicine, 17, 195. https://doi.org/10.1186/s12916-019-1426-2

Liu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., Ledsam, J. R., Schmid, M. K., Balaskas, K., Topol, E. J., Bachmann, L. M., Keane, P. A., & Denniston, A. K. (2019). A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: A systematic review and meta-analysis. The Lancet Digital Health, 1(6), e271-e297. https://doi.org/10.1016/S2589-7500(19)30123-2

Wu, E., Wu, K., Daneshjou, R., Ouyang, D., Ho, D. E., & Zou, J. (2021). How medical AI devices are evaluated: Limitations and recommendations from an analysis of FDA approvals. Nature Medicine, 27(4), 582-584. https://doi.org/10.1038/s41591-021-01312-x