MSHR 6625 Week 10 Training Pilot Review Example

What a Three-Week Pilot With Two Crews Exposed: A Pilot Review of Asset-Field Training That Reports What Held, Interviews the Best and Worst Cases and Traces Each Shortfall to a Design Decision

Student Name

Walden University

MSHR 6625: Building Human Capital Through Training and Development

Instructor Name

Month Day, Year

What a Three-Week Pilot With Two Crews Exposed: A Pilot Review of Asset-Field Training That Reports What Held, Interviews the Best and Worst Cases and Traces Each Shortfall to a Design Decision

Pilot Summary

Mill Creek Water District piloted the asset-field program for three weeks with two crews: a hydrant and valve crew of seven technicians led by an experienced crew lead, and a main repair crew of six led by a crew lead newly promoted that year. Both crews completed all three in-truck sessions. Offline mode was enabled and paper tickets ended for the two crews before the pilot began.

The table follows the four levels of Kirkpatrick and Kirkpatrick (2006), with results still months away.

The design worked as intended with the hydrant and valve crew. All seven technicians met the learning standard, and asset-field completion on real orders rose sharply within two weeks. Locating asset IDs on real hydrants, the part of the program that most depended on in-truck practice, produced the largest improvement. The reaction ratings and comments praised practice on real equipment.

Table 1

Pilot Results by Evaluation Level

Composite utility. Figures are illustrative pilot results.

What Fell Short

The main repair crew's results fell short in three ways, and each traces to a design decision.

First, two of six technicians did not meet the learning standard for pipe materials. The design relied on pipe samples in a training kit, but the main repair crew's kit was missing the ductile iron sample, so the crew lead used photographs for the cast and ductile iron comparison, the hardest distinction in the program. This traces to the decision to rely on a physical kit without a check that each kit was complete.

Second, behavior on real orders did not improve. Main repair orders often involve emergency work at night, and the crew completed orders after leaving the site, from memory. This traces to the decision to time the sessions during routine work, which did not prepare technicians for the conditions under which most main repair orders are closed.

Third, the newly promoted crew lead skipped the ninth event, the bridge to that day's real orders, in two of three sessions, and did not do the same-day check on real orders. This traces to the decision to rely on the facilitator guide alone, without a short briefing for crew leads before they ran their first session.

Success Case Interviews

Brinkerhoff (2003) proposed the success case method: studying the most and least successful participants to learn what made the difference. The most successful technician, on the hydrant crew, said the turning point was finding the ID stamp on an old hydrant during the session, a make he had been guessing on for years. The least successful, on the main crew, said he understood the pipe materials in the session but "at two in the morning in a hole, you just pick the one you think." His account confirmed that the problem was the conditions of use, not understanding.

Changes Before Full Rollout

Three changes follow. Every training kit will be checked against a list before it leaves the yard. The main repair sessions will add a scenario completed under time pressure with low light, and crews will be asked to complete pipe material before leaving the hole, with the crew lead checking the next morning. Every crew lead will attend a one-hour briefing before running sessions, covering the events most often skipped. Salas et al. (2012) note that pilots and evaluation are most useful when they lead to specific design changes; these three are specific.

What the Pilot Changed in the Evaluation Plan

The pilot also showed that the reaction card was less useful than expected: the hydrant crew rated the sessions highly, and the main crew lower, but neither rating predicted behavior. The card will be kept, since it costs little, but the evaluation will lean on the behavior measure from the first wave.

What the Pilot Cannot Show

Two crews over three weeks is a small sample. The hydrant crew's success may partly reflect an experienced crew lead. Results at the fourth level, more accurate asset records and fewer wasted visits, will not be measurable for months.

References

Brinkerhoff, R. O. (2003). The success case method: Find out quickly what's working and what's not. Berrett-Koehler.

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). Berrett-Koehler.

Salas, E., Tannenbaum, S. I., Kraiger, K., & Smith-Jentsch, K. A. (2012). The science of training and development in organizations: What matters in practice. Psychological Science in the Public Interest, 13(2), 74-101. https://doi.org/10.1177/1529100612436661