NURS 6451 Week 10 evaluation plan example

Evaluation Plan: Secure Messaging on Four Medical-Surgical Units, From Approvals to a Report Written to the STARE-HI Standard

Student Name

College of Nursing, Walden University

NURS 6451: Evaluation Methods for Health Information Technology

Instructor Name

Month Day, Year

Evaluation Plan: Secure Messaging on Four Medical-Surgical Units, From Approvals to a Report Written to the STARE-HI Standard

Summary

Question: after the switch from pagers to the messaging app on four medical-surgical units, does a nurse wait more or less time for the prescriber covering a patient to acknowledge a routine request? Design: a non-randomized stepped-wedge comparison using the staggered go-live, with each unit's time series and concurrent pager units as comparisons. Primary measure: the acknowledgment interval, defined identically for observed pager periods and logged app periods. Decision the findings will inform: whether to fix the recipient directory and urgency settings before extending the app to the hospital's remaining units, and whether medical staff should agree new expectations for marking and answering messages. Reporting: a final report following the STARE-HI guideline for health informatics evaluation studies.

1. Approvals and Data Access

Owner: evaluation lead, the clinical informatics nurse. Prerequisite: sponsor approval of this plan. Work: obtain a determination from the institutional review board on whether the evaluation is quality improvement or research; obtain medical executive committee agreement to observe prescribers with results reported only by unit and time of day; agree with information services and the privacy officer on log fields, extraction schedule and de-identification of message threads; confirm export terms with the vendor. Produces: a signed data access agreement and an approvals record. Time: four weeks, starting immediately, because the baseline observation cannot begin without it.

2. Baseline Observation on Units 3 and 4

Owner: observation coordinator, a nurse educator trained in time-motion methods. Prerequisite: approvals. Work: two weeks of structured observation of non-urgent nurse requests and prescriber acknowledgments on each unit before its go-live, day and evening shifts, using the written definitions in the measurement plan, plus observation of prescribers during morning rounds. Produces: the pager-period baseline for the primary measure and the interruption measure. Time: two observers, about 60 hours each across four weeks. This step has a fixed deadline, since it must finish before each unit goes live.

3. Usability Round

Owner: evaluation lead with an information services analyst. Prerequisite: vendor test environment. Work: repeat the think-aloud sessions with six new nurses after the vendor changes the directory, urgency default and forwarding. Produces: comparison of misdirected messages and task completion with the first round. Time: one week, scheduled after the vendor's fix and before Unit 4's go-live.

4. Go-Live Waves With Measurement

Owner: evaluation lead. Prerequisite: baseline complete on the unit about to switch. Work: at each go-live, begin daily extraction of message logs for the unit; continue observation windows on units still on pagers; each month, watch a set of requests on the switched unit as they happen to check logged against observed intervals; run the downstream prompt to pharmacists, clerks and night coverage every two weeks. Produces: the stepped-wedge dataset. Time: through twelve weeks after the last unit switches.

5. Workflow Observation

Owner: observation coordinator. Prerequisite: at least eight weeks since go-live on a unit, so early novelty has faded. Work: time-motion observation of prescribers during rounds and interviews with clerks and pharmacists; monthly review of a sample of 50 de-identified threads per unit against the record for the balancing measure. Produces: interruption patterns and the share of decisions not reaching the chart.

6. Analysis

Owner: evaluation lead, with a statistician from the quality department. Prerequisite: complete dataset. Comparisons: acknowledgment intervals on app units against pager units in the same weeks, and each unit before against after, accounting for the trend over time. Stepped-wedge analyses must allow for clustering by unit and for secular trends, because later units spend more of the study in the intervention condition (Hemming et al., 2015). The statistician will specify the model, including how staffing and census are adjusted for, before any outcome data are examined, and that specification will be recorded. Secondary and balancing measures will be summarized by unit and period. Qualitative data from interviews and the downstream prompt will be analyzed for themes by two team members.

7. Data Handling

Message logs are extracted by information services with patient names and record numbers replaced by study codes; the key stays with information services. Message thread text is reviewed by one designated analyst in a secure hospital environment, de-identified before any other team member sees it, and never copied to personal devices. Observation notes record roles and times, not names. All evaluation data are stored on the hospital's approved research drive and destroyed twelve months after the final report unless the privacy officer approves longer retention. No result will name or single out an individual prescriber.

8. Reporting

The final report will follow STARE-HI, which lists what reports of health informatics evaluations should describe, including the setting, the system and its context, the study design and methods, the results and the limitations (Talmon et al., 2009). Committing to it now means the team records these details as work proceeds rather than reconstructing them afterward. Three products will be delivered: a two-page summary for the sponsor with the decision recommendation; unit-level reports for each nurse manager; and a configuration findings list for information services and the vendor. The CDC framework's final step, ensuring use and sharing lessons learned, is built into the delivery: the evaluation lead will present findings at the steering meeting where the extension decision is scheduled (Centers for Disease Control and Prevention [CDC], 1999).

9. Resources and Timeline

Staff time rather than cost is listed, since dollar figures would require budget data this plan does not have. Evaluation lead: about 8 hours a week for the duration. Two observers: about 60 hours each for baseline, 40 each for workflow observation. Statistician: about 30 hours. Information services analyst: about 4 hours a week during extraction. Privacy officer: about 6 hours for review. Timeline in weeks from plan approval: approvals weeks 1 to 4; baseline weeks 3 to 8 as unit schedules require; usability round week 9; go-live waves and measurement weeks 6 to 30; workflow observation weeks 14 to 30; analysis weeks 31 to 36; report week 38.

References

Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. MMWR Recommendations and Reports, 48(RR-11), 1-40.

Hemming, K., Haines, T. P., Chilton, P. J., Girling, A. J., & Lilford, R. J. (2015). The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting. BMJ, 350, h391. https://doi.org/10.1136/bmj.h391

Talmon, J., Ammenwerth, E., Brender, J., de Keizer, N., Nykänen, P., & Rigby, M. (2009). STARE-HI: Statement on reporting of evaluation studies in health informatics. International Journal of Medical Informatics, 78(1), 1-9. https://doi.org/10.1016/j.ijmedinf.2008.09.002