PHLT 8044 Week 2 Water Program Evaluability Study Example

Evaluability Assessment of a State School Water Access Program: Wholey's Conditions Applied to Bottle-Filling Stations in Middle Schools, With a Split Verdict on What Can Be Judged Now

Student Name

Walden University

PHLT 8044: Advanced Program Evaluation

Instructor Name

Month Day, Year

Evaluability Assessment of a State School Water Access Program: Wholey's Conditions Applied to Bottle-Filling Stations in Middle Schools, With a Split Verdict on What Can Be Judged Now

Purpose

Before the state commits funds to an outcome evaluation of its school water access program, this assessment asks whether the program can be evaluated usefully at all. Wholey (2015) proposed that evaluability assessment check four conditions: whether program goals and priority information needs are well defined and agreed, whether the goals are plausible given the program's design and resources, whether relevant performance data can be obtained at reasonable cost, and whether intended users are willing and able to use evaluation findings. Its three components are filtered refill stations mounted in middle school buildings, a reusable bottle for every student and a two-week water campaign every fall. It has operated for two school years in 61 schools.

Methods

The assessment drew on four sources over six weeks. Program documents included the legislative authorization, the grant agreement with schools, installation records and quarterly reports. Interviews were conducted with eleven people: the state program manager, two state health department staff, four principals, two school nurses, a district facilities director and a food service director. Site visits to six schools, chosen to vary by size and income, observed station placement and use at lunch and between classes. A data inventory listed every existing record that bears on the program's goals, who holds it and how often it is updated.

Condition 1: Are Goals and Information Needs Defined and Agreed?

The authorizing statute states two goals: to increase students' consumption of water during the school day and to reduce consumption of sugar-sweetened beverages. The grant agreement adds a third, to support healthy weight among students. Interviews revealed that stakeholders weight these differently. The program manager and legislators' staff emphasized healthy weight, because it was the argument that secured funding. Principals and nurses emphasized hydration and fewer sugary drinks, which they see as achievable. The food service director emphasized beverage sales, since cafeteria revenue from drinks has fallen. Priority information needs are therefore partly agreed: everyone wants to know whether students drink more water, but not everyone expects or wants a judgment on weight.

Finding: partly met. Goals are written down, but stakeholders disagree about which one the evaluation should judge.

Condition 2: Are the Goals Plausible Given the Program's Design?

The program's design is plausible for its first two goals. Stations make cold, filtered water convenient; bottles make carrying it possible; promotion makes it visible. Evidence supports this chain: improving the convenience and appeal of water in cafeterias has increased students' water consumption (Kenney et al., 2015), and a review of school water strategies identified access and appeal as key barriers (Patel & Hampton, 2011). The third goal, healthy weight, is plausible only over a longer period and at a larger effect than water alone typically produces. Trials that found weight effects combined water provision with education and ran for a full school year or more (Muckelbauer et al., 2009). In this program, the promotion lasts two weeks, and many stations sit in hallways far from the cafeteria where drinks are chosen.

Finding: met for water intake and sugary drinks; doubtful for weight within the program's current design and time frame.

Condition 3: Can Relevant Data Be Obtained?

Data on water dispensed and on cafeteria purchases are available at low cost and cover all program schools. Self-reported intake is available but sparse. BMI data exist in only a third of districts and would require new agreements to obtain.

Finding: met for implementation, water dispensed and cafeteria purchases; partly met for self-reported intake; not met for weight without new data collection.

Table 1

Data Inventory: What Exists, Who Holds It and What It Can Show

Note. Holders and frequencies describe the composite state program as reported in interviews and documents.

Condition 4: Will Intended Users Use the Findings?

The program manager and the legislative budget office both said they would use findings to decide whether to expand the program to more schools next biennium, a concrete decision with a date. Principals said they would use school-level reports on station use to move poorly placed stations. Utilization depends on findings arriving before the budget cycle, which constrains what the evaluation can measure in time.

Finding: met, with a timing constraint.

Disagreement as a Finding

The disagreement over goals is itself one of the assessment's most useful results. Wholey (2015) treats agreement on what an evaluation should judge as a precondition, because an evaluation that answers one stakeholder's question will be dismissed by the others. Here, an evaluation built around weight would likely report no effect and be read by principals as unfair to a program they value, while an evaluation built only around hydration would leave the budget office without the evidence it asked for. Naming the disagreement now allows it to be settled before money is spent. The program manager has agreed to convene the legislature's staff, two principals and the food service director for a single meeting to agree on a primary question, with the assessment's evidence on plausibility and data availability as the starting point.

Verdict and Recommendations

The program is evaluable now for implementation, reach, water use and sugary drink purchases at school, and not yet for weight. Recommendations follow. First, agree with stakeholders that the evaluation's primary question concerns water and sugary drink consumption, with weight treated as a longer-term question. Second, use station meter data and cafeteria sales as primary data sources, supplemented by a short student survey in a sample of schools. Third, if the state wants a future answer on weight, negotiate data agreements with districts that already screen BMI and consider relocating some stations nearer cafeterias, which would make the weight goal more plausible. Fourth, time the first report to the budget cycle.

References

Kenney, E. L., Gortmaker, S. L., Carter, J. E., Howe, M. C. W., Reiner, J. F., & Cradock, A. L. (2015). Grab a cup, fill it up! An intervention to promote the convenience of drinking water and increase student water consumption during school lunch. American Journal of Public Health, 105(9), 1777-1783. https://doi.org/10.2105/AJPH.2015.302645

Muckelbauer, R., Libuda, L., Clausen, K., Toschke, A. M., Reinehr, T., & Kersting, M. (2009). Promotion and provision of drinking water in schools for overweight prevention: Randomized, controlled cluster trial. Pediatrics, 123(4), e661-e667. https://doi.org/10.1542/peds.2008-2186

Patel, A. I., & Hampton, K. E. (2011). Encouraging consumption of water in school and child care settings: Access, challenges, and strategies for improvement. American Journal of Public Health, 101(8), 1370-1379. https://doi.org/10.2105/AJPH.2011.300142

Wholey, J. S. (2015). Exploratory evaluation. In K. E. Newcomer, H. P. Hatry, & J. S. Wholey (Eds.), Handbook of practical program evaluation (4th ed., pp. 88-107). Jossey-Bass.