PHLT 8044 Week 7 Indicators and Credibility Plan Example

Measurement Plan for the School Water Access Evaluation: One Primary Indicator per Question, the Record Each Is Read From, and Whether the People Deciding Would Believe It

Student Name

Walden University

PHLT 8044: Advanced Program Evaluation

Instructor Name

Month Day, Year

Measurement Plan for the School Water Access Evaluation: One Primary Indicator per Question, the Record Each Is Read From, and Whether the People Deciding Would Believe It

Purpose

The six evaluation questions written earlier each need a measure. This plan assigns each question exactly one primary indicator, defines it precisely, names the record it will be read from and adds a judgment that most measurement plans omit: whether the people deciding the program's future would find the indicator credible. Indicators that are technically sound but distrusted by decision-makers can sink an otherwise good evaluation, and indicators that decision-makers trust but that measure the wrong thing can mislead them (Patton, 2008). The Program Evaluation Standards place valid and reliable information under accuracy and the credibility of evidence to users under utility, and this plan tries to satisfy both (Yarbrough et al., 2011). Secondary indicators are listed where a primary one leaves an important gap.

Indicators

Table 1

Primary Indicators by Evaluation Question

Note. Sugary drinks are defined as beverages with added sugar, including regular sodas, sports drinks, sweetened teas and fruit drinks; flavored milk is recorded separately.

Discussion of the Harder Choices

Purchases as the Primary Outcome

Cafeteria purchases are an imperfect measure of what the program aims to change, since they miss drinks students bring from home or buy before and after school. They were chosen as the primary outcome anyway because they are objective, available for every program and waiting-list school and already trusted by the budget office, which reads food service revenue reports every quarter. The credibility column explains the choice: an indicator the budget office questions would weaken the expansion decision however carefully it was gathered. The whole-day recall in question 5 covers the gap and serves as a check on whether lower school purchases are offset elsewhere.

Why Not Meter Counts

The station meters provide the program's headline figure, but they are not used as an outcome indicator. Meters exist only in program schools, so they cannot be compared with waiting-list schools, and they count all refills, including staff use and water that would have been drunk from fountains anyway. They serve instead as a secondary implementation indicator for question 3, showing whether stations are being used at all.

Self-Report and Its Credibility

Two indicators rely on student self-report. For question 1, the state survey's sugary drink item is used widely and allows comparison with earlier years. For question 5, a previous-day recall is more accurate than a usual-frequency question because it asks about a specific day, but students may still underreport sugary drinks if they know the survey concerns a water program. To reduce that bias, the recall will ask about all beverages without mentioning the program, and it will be given by the evaluation team rather than teachers.

Defining Sugary Drinks

How sugary drinks are defined changes the numbers. Flavored milk, for example, contains added sugar but is part of federally reimbursable school meals, and some stakeholders would object to counting it alongside soda. The plan records flavored milk separately so that results can be reported with and without it, avoiding a dispute that could otherwise overshadow the findings.

How the Indicators Will Be Reported

Each primary indicator will be reported in the same form every time it appears: the value for program schools, the value for waiting-list schools, the change in each group and the difference between those changes, with a confidence interval. Implementation indicators will be reported by school as well as overall, since principals need to see their own school's figures and the program manager needs to see where stations are failing. Indicators based on self-report will carry a note on their limits each time they appear in a table, so that a reader skimming a single exhibit is not misled. Consistent reporting matters because the budget office will compare figures across reports, and an indicator that changes form between the interim and final reports invites doubt about whether it has changed in substance.

Secondary Indicators

Three secondary indicators supplement the primary set: meter counts per enrolled student per week in program schools, as a use measure; the share of students reporting they trust school water, from the theory-and-design survey; and plain water purchases in cafeterias, since some schools sell bottled water and the program could reduce those sales as well. None is used to judge the program's success, but each helps explain the primary results.

Outside the Plan's Scope

The plan does not measure weight, for the reasons given in the evaluability assessment and evaluation questions: the program's design makes a weight effect unlikely within one year, and BMI data cover only a third of districts. Rossi et al. (2019) caution against measuring outcomes a program could not plausibly move within the evaluation period. Nor does the plan measure hydration status or academic outcomes, which some advocates mention, because no feasible indicator would be credible within the evaluation's resources.

References

Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). SAGE.

Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2019). Evaluation: A systematic approach (8th ed.). SAGE.

Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). SAGE.