HLTH 8451 Week 5 Criteria Weighting Protocol Example

Weights Before Scores: A Swing Weighting Protocol for Comparing the Heat Refuge Mandate, a Performance Standard and Guidance, Elicited From Five Named Roles

Student Name

Walden University

HLTH 8451: Public Policy Analysis

Instructor Name

Month Day, Year

Weights Before Scores: A Swing Weighting Protocol for Comparing the Heat Refuge Mandate, a Performance Standard and Guidance, Elicited From Five Named Roles

Purpose and Sequence

The Week 4 quick analysis compared three alternatives on four criteria without weighting them, so its recommendation rested on priorities nobody stated. This protocol sets out how weights will be elicited for the multi-criteria comparison that will sit beside the cost-benefit analysis. It is written in the future tense because none of the steps has been carried out. The protocol will be dated and filed with the client before any alternative is scored, so that a reader can confirm that weights were not adjusted to favor a result already known. The weights will be fixed on the date of filing. If evidence later changes a criterion's range, the affected weights will be re-elicited from all judges and both versions will be reported.

Criteria, Defined for Measurement

The four criteria from Week 4 will be refined and a fifth added. Each will be defined so that it has a measurable range across the alternatives, which swing weighting requires.

Table 1

Criteria, Indicators and Provisional Ranges Across the Alternatives

Ranges are provisional and will be replaced by Week 6 estimates before elicitation begins.

Why Swing Weights

Judges will not be asked how important each criterion is in general. Keeney (1992) identifies weights assigned without reference to the ranges of consequences as among the most common errors in decision analysis, since a criterion can matter a great deal in principle and very little in a particular choice. If all three alternatives were equally easy to survey, surveyor burden would deserve no weight however much a judge cares about inspectors. Swing weighting ties each weight to how much the alternatives actually differ (von Winterfeldt & Edwards, 1986).

The procedure will run as follows. Each judge will be shown a hypothetical alternative that sits at the worst level on all five criteria. The judge will be asked which single criterion they would most want to move from its worst to its best level. That swing receives 100 points. The judge will then be asked which criterion they would move next and how valuable that swing is compared with the first, as a number from 0 to 100, and so on through all five. Weights will be each criterion's points divided by the judge's total. As a consistency check, each judge will answer one trade-off question: how many additional surveyor hours per survey would exactly offset a gain of five deaths avoided per year? Answers inconsistent with the stated weights will be shown to the judge, who may revise them. Revisions will be recorded with their reasons.

The protocol will use an additive value model, which assumes that the value of doing well on one criterion does not depend on the level of another. Belton and Stewart (2002) note that this assumption should be checked rather than taken for granted. Judges will be asked whether, for example, flexibility matters less when health protection is low. If any judge says yes, the interaction will be reported.

Guarding Against Known Biases

Weight judgments are sensitive to how they are asked. Weber and Borcherding (1993) review evidence that weights shift with the elicitation method and that a criterion split into several subcriteria tends to receive more total weight than the same criterion left whole. Three safeguards follow. Cost will remain one criterion rather than being divided into capital, energy and testing costs, so that it does not gain weight from the way it is listed. The order in which criteria are presented will be randomized for each judge. The facilitator will read each range aloud before each judgment and will not comment on any answer, so that judges respond to the consequences rather than to cues from the person asking.

Judges, Specified by Role

Weights express whose values count, so judges will be chosen by role rather than convenience, and the reasons for each role will be stated.

Table 2

Judge Roles and Reasons for Inclusion

The roles are fixed in advance; persons will be recruited to fill them and will not be named in the report.

No one will serve in two roles. A resident or family member was considered as a sixth judge and is not included because the elicitation requires familiarity with the indicators that would make participation a burden. This is a limit of the protocol, and the Week 11 report will state it.

Recording Disagreement

Elicitation will take place in two rounds. In the first, each judge will set weights alone. In the second, judges will see the anonymous first-round weights and a summary of the reasons given, and each may revise. Both rounds will be reported. The five final weight sets will be kept separate and will not be averaged. An average would produce a set of weights no judge holds and would hide the conflict a regulator most needs to see, for example between an advocate who places most of the weight on health and an administrator who places it on cost and flexibility. The report will show the range of weights on each criterion and identify the judge at each end of it.

Use in Later Weeks

Scores for each alternative on each criterion will be set after the weights are filed, using Week 6 predictions converted to a 0 to 100 scale where the worst level is 0 and the best is 100. The protocol commits Week 9 to rerunning the comparison under each judge's weights separately and reporting which alternative ranks first under each. If the ranking changes with the judge, the client will be told whose values favor which alternative. That is part of the finding, not a failure of the method.

References

Belton, V., & Stewart, T. J. (2002). Multiple criteria decision analysis: An integrated approach. Kluwer Academic.

Keeney, R. L. (1992). Value-focused thinking: A path to creative decisionmaking. Harvard University Press.

von Winterfeldt, D., & Edwards, W. (1986). Decision analysis and behavioral research. Cambridge University Press.

Weber, M., & Borcherding, K. (1993). Behavioral influences on weight judgments in multiattribute decision making. European Journal of Operational Research, 67(1), 1-12. https://doi.org/10.1016/0377-2217(93)90318-H