PHLT 8062 Week 6 JBI Critical Appraisal Example

Critical Appraisal of One Composite Controlled Before-and-After Evaluation of a Birth Dose Order Set, Separating What the Design Could Establish From How Well It Was Carried Out, Using the JBI Checklist for Quasi-Experimental Studies

Student Name

Walden University

PHLT 8062: Dissertation Seminar II

Instructor Name

Month Day, Year

Critical Appraisal of One Composite Controlled Before-and-After Evaluation of a Birth Dose Order Set, Separating What the Design Could Establish From How Well It Was Carried Out, Using the JBI Checklist for Quasi-Experimental Studies

The Study and Why It Was Chosen

The evidence table showed that the review's hospital systems claim rests mainly on before-and-after reports from single hospitals. This appraisal examines one of them in depth, a composite evaluation that stands in for the strongest report the search found. Two hospitals in the same health system took part. At the intervention hospital, an urban facility with about 3,200 births a year, an admission order set making the hepatitis B birth dose a default order was introduced in January of the evaluation's second year. A second hospital in the same system, with about 2,600 births a year, made no change and served as the comparison. Birth dose administration before discharge, drawn from chart review of all eligible infants, was compared for twelve months before and twelve months after the change. Coverage at the intervention hospital rose from 78% to 92%; at the comparison hospital it moved from 80% to 82%. The authors concluded that the order set increased birth dose administration.

The appraisal uses the JBI checklist for quasi-experimental studies (Tufanaru et al., 2020) and keeps two questions apart throughout: what a controlled before-and-after design could establish at best, and how well this particular study was conducted.

What the Design Could Establish

A controlled before-and-after design sets the change seen in an exposed group beside the change seen over the same period in an unexposed group. At its best, it rules out explanations that would affect both groups equally, such as a national guidance update or a seasonal pattern, because those would show up in the comparison group as well (Shadish et al., 2002). It cannot rule out events that affect only one group, and it depends on the groups following similar paths before the intervention. A single comparison hospital also gives no estimate of how much hospitals vary from year to year on their own. So even a perfectly executed study of this design would support a conclusion of the form "coverage rose more at the intervention hospital than at a similar hospital without the change," not a general claim that order sets raise coverage everywhere.

How Well It Was Carried Out: The Checklist

Table 1

JBI Checklist for Quasi-Experimental Studies Applied to the Composite Evaluation

Note. Judgments follow the JBI checklist's wording; the study is a composite representing reports identified in the review.

The Two Most Consequential Weaknesses

Analysis That Ignores the Monthly Data

The authors had monthly coverage for 24 months at both hospitals but compared only the annual averages. Monthly data would have allowed a test of whether coverage at the intervention hospital was already rising before the order set, which would undercut the conclusion, and whether the rise began when the order set started. Without that analysis, a gradual improvement that predated the change cannot be distinguished from an effect of the order set. Reporting guidance for improvement studies asks authors to show data over time for exactly this reason (Ogrinc et al., 2016).

Unreported Co-Interventions

The paper does not say whether anything else changed at the intervention hospital, such as staff education, a new electronic health record module or a pharmacy stocking change. In improvement work, interventions are often bundled, and an order set introduced alongside training may receive credit for both. A comparison hospital in the same system partly protects against system-wide changes, but not against changes specific to the intervention hospital.

Verdict

The design could support a qualified conclusion that coverage rose more at the intervention hospital than at its comparison. The execution weakens even that: the analysis did not use the monthly data that could test for a preexisting trend, co-interventions were not reported, and the hospitals' similarity was not shown. The study counts as supportive but weak evidence for the review's hospital systems claim. In the review, it will be described as showing an association consistent with an order set effect, with the two weaknesses stated, and it will not be cited as establishing that order sets raise coverage.

What This Means for the Dissertation

The appraisal sharpens the dissertation's own design. Its comparison of periods at nine hospitals will use monthly data and test for trends before the July 2021 change, and it will record other changes at the hospitals during the study period wherever program staff can document them. The weaknesses found here are the ones a committee will look for in the dissertation.

References

Ogrinc, G., Davies, L., Goodman, D., Batalden, P., Davidoff, F., & Stevens, D. (2016). SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): Revised publication guidelines from a detailed consensus process. BMJ Quality & Safety, 25(12), 986-992. https://doi.org/10.1136/bmjqs-2015-004411

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

Tufanaru, C., Munn, Z., Aromataris, E., Campbell, J., & Hopp, L. (2020). Systematic reviews of effectiveness. In E. Aromataris & Z. Munn (Eds.), JBI manual for evidence synthesis. JBI.