NURS 8546 Week 10 Outcome Evaluation Design Example

Making the Promise Precise Enough to Be Wrong: An Outcome Evaluation Design for First-Attempt Medication Administration Checkoffs in a Flipped Skills Course

Student Name

College of Nursing, Walden University

NURS 8546: Strategies for Innovation in Nursing Education

Instructor Name

Month Day, Year

Making the Promise Precise Enough to Be Wrong: An Outcome Evaluation Design for First-Attempt Medication Administration Checkoffs in a Flipped Skills Course

The Promised Gain, Stated Precisely

The flipped skills course was proposed with a general promise: learners will be better prepared for clinical practice. That promise cannot be wrong, which means it cannot be evaluated. This design narrows it to one outcome: the first-attempt failure rate on the course's medication administration checkoff. In the last four terms, 29% of learners failed this checkoff on the first attempt, most often for missing a patient identification step or a calculation error. The flipped design gives learners more supervised practice time, so if it works as intended, fewer learners should fail the checkoff the first time. The promise, stated precisely, is that the first-attempt failure rate will fall from 29% to 20% or lower.

Comparison Structure

The obvious design, comparing this year's results with last year's, was weighed and set aside. Comparing this year's failure rate with last year's would credit the flipped design with any change caused by other factors, such as a different cohort, a new checkoff evaluator or a change in the admission requirements, all of which are planned or possible in the next year. Shadish et al. (2002) describe such alternative explanations as threats to internal validity, and a before-and-after design leaves most of them open.

The design instead uses concurrent sections. During the second stage of the rollout, four of the eight laboratory sections will be flipped and four will not, in the same term, with the same checkoff evaluators and the same cohort. Comparing their failure rates removes the cohort and calendar explanations. The cost is in scheduling: learners cannot choose their section on the basis of format, so sections must be assigned by the registrar's usual process without regard to format, and the program must accept that half the cohort receives the older design for one more term.

Measures

Three measures were chosen, each over a nearer alternative.

Measure 1: First-attempt failure on the medication administration checkoff. This is the outcome itself. A nearer alternative, learners' self-rated confidence in medication administration, was rejected because confidence often rises without any change in performance. The checkoff measures what learners can do. It cannot show whether they would perform the same way with a real patient.

Measure 2: Type of error on failed checkoffs. Evaluators already record the reason for each failure. Categorizing failures as identification, calculation or technique errors shows whether the flipped design reduces the errors it targets. A nearer alternative, the overall checkoff score, was rejected because it mixes minor and critical errors. This measure cannot detect errors that evaluators do not observe.

Measure 3: Medication administration performance in the first clinical course. Clinical instructors rate learners on a medication administration item during the first four weeks of the next semester's clinical course. This measure was chosen over a repeat checkoff because it reflects performance in practice. It cannot separate the effect of the skills course from differences between clinical instructors, units and patients. Kirkpatrick and Kirkpatrick (2016) describe this as the difference between learning and behavior: a checkoff shows what was learned, while clinical performance shows whether it was applied.

Measurement Intervals

Intervals follow the program's calendar. The checkoff is measured in week 12 of the skills course, when it is normally scheduled. Error types are recorded at the same time. Clinical performance is measured in weeks 1 to 4 of the following semester's clinical course. A fourth measurement, a repeat checkoff given without warning in week 10 of the clinical course, about six months after the skills course, tests whether any advantage decays. A practice change that improves performance at week 12 but not six months later would be worth less than it appeared.

Results will be reported to the course faculty after each measurement point rather than only at the end. The week 12 checkoff results will be available before the gate 2 decision, the clinical ratings before the end of the following semester and the six-month repeat checkoff before the program decides whether to keep the design in its consolidated form. Each report will show the flipped and non-flipped sections side by side with the number of learners in each.

Threats to Validity

Four threats are named. Two can be addressed. Evaluator bias could favor one format; it is reduced by having evaluators who do not teach in either section and do not know each learner's format. Unequal sections could arise if one section has more repeating learners; this is checked by comparing sections' prior grades.

Two cannot be resolved. Contamination will occur, because learners in non-flipped sections can watch the recorded demonstrations, which are available to all learners in the course; this would make the formats look more similar than they are. Instructor differences cannot be separated from format, because each section has one instructor, and the flipped sections in term 2 are taught by full-time faculty while some non-flipped sections are taught by adjuncts. The evaluation reports this limitation and does not attempt to adjust for it.

What Counts as No Effect

The design states its null result in advance. If the first-attempt failure rate in flipped sections is not at least 5 percentage points lower than in non-flipped sections in term 2, the evaluation will report that the flipped design produced no meaningful effect on this outcome. With about 50 learners in each format, a difference smaller than 5 points could easily occur by chance and would not justify the costs of stage 3.

Stating the null result matters here for a particular reason. The research on flipped teaching in nursing has produced inconsistent effects on performance (Betihavas et al., 2016), so a finding of no effect is a realistic possibility, not a remote one. If the program has not said in advance what no effect looks like, a small difference in the expected direction will be tempting to report as success. The 5-point threshold was chosen by asking the course faculty what difference would justify the cost of paid adjunct preparation and new equipment; they agreed that anything smaller would not. The evaluation will report the full results, including error types and clinical ratings, whatever the checkoff comparison shows, so that a null finding on the main outcome does not hide useful information about how the design worked.

References

Betihavas, V., Bridgman, H., Kornhaber, R., & Cross, M. (2016). The evidence for "flipping out": A systematic review of the flipped classroom in nursing education. Nurse Education Today, 38, 15-21. https://doi.org/10.1016/j.nedt.2015.12.010

Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation. ATD Press.

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.