HLTH 8451 Week 9 Regulatory Uncertainty Analysis Example

Does the Performance Standard Still Win? One-Way, Scenario, Judge-Weight and Monte Carlo Tests of the Heat Refuge Analysis

Student Name

Walden University

HLTH 8451: Public Policy Analysis

Instructor Name

Month Day, Year

Does the Performance Standard Still Win? One-Way, Scenario, Judge-Weight and Monte Carlo Tests of the Heat Refuge Analysis

Purpose

The central estimates from the cost-benefit model favor the performance standard: annualized net benefits of about $197 million against about $180 million for the refuge mandate, using a 7 percent discount rate and an unadjusted value per statistical life. This paper asks whether that verdict survives the uncertainty the earlier weeks identified. It applies the tests fixed in advance by the Week 6 sensitivity plan and the Week 5 weighting protocol, in order from simplest to most demanding, and closes by separating robust findings from those that rest on contested judgments. All parameter values are composite and labeled as such.

Parameters

Table 1

Uncertain Parameters, Central Values and Ranges Tested

The table lets a reader check every range before looking at any result.

One-Way Sensitivity

Each parameter was varied alone across its range while the others stayed at central values. Results for the performance standard are shown below in order of their effect, which is how a tornado diagram would display them.

Table 2

One-Way Results for the Performance Standard, Net Benefits in Millions of Dollars per Year

Bars in a tornado diagram would run from the low-end to the high-end value, widest at the top.

Only one parameter, the odds ratio, turns net benefits negative within its range. At the lower confidence limit from the Irma study, the excess risk of death after power loss is small enough that benefits fall below costs. Costs matter much less than benefits: the full range of capital costs moves net benefits by $64 million, while the range of the mortality effect moves them by $570 million. The share of facilities outside the rule's reach matters less than expected because both costs and benefits scale with the number of facilities affected, so it changes the size of net benefits more than their sign.

Scenarios From Week 3

Two formulations set aside in Week 3 describe different worlds rather than different parameter values, so they are run as scenarios. In the staffing scenario, cooling helps only when staff are available to move and monitor residents. The mandate's effectiveness falls to 55 percent, since every resident must be brought into the refuge, and the performance standard's falls to 60 percent, since some facilities cool residents in place. Net benefits fall to about $67 million for the mandate and $143 million for the performance standard. The performance standard's advantage widens.

In the grid reliability scenario, outages during heat emergencies occur twice as often as in the central case, at 5 percent a year. Net benefits rise to about $545 million for the mandate and $525 million for the performance standard. The ranking reverses: when exposure is frequent, the mandate's higher effectiveness outweighs its higher cost.

Judges' Weights From Week 5

The multi-criteria comparison was rerun under each judge's weight set, kept separate as the protocol required. Scores run from 0 to 100.

Table 3

Weighted Scores by Judge

Weight sets are composite and stand in for elicitation that the protocol describes.

Three judges favor the performance standard and two favor the mandate. The advocate's preference follows from heavy weight on health and on equal protection for rural residents, where the mandate scores best. The survey official's follows from the weight on inspection burden, where a measurable refuge room is easier to check than a facility's chosen means. Guidance comes close to first only under the administrator's and the economist's weights. No judge's weights put guidance first. The disagreement is a finding: the choice between the two regulatory options depends on whether health protection and ease of inspection count for more than cost and flexibility.

Monte Carlo Simulation

Distributions were assigned only to the three parameters for which a defensible shape exists. The log odds ratio was drawn from a normal distribution matching the published confidence interval. Outage probability was drawn from a triangular distribution from 1.5 to 4.0 percent with a mode of 2.5, reflecting expert-style bounds without a firm central tendency. The value per statistical life was drawn from a lognormal distribution centered on $12 million, since values cannot be negative and published estimates are skewed upward. Other parameters were held at central values, since assigning distributions without evidence would produce false precision (Morgan & Henrion, 1990; Briggs et al., 2012).

In 10,000 draws, net benefits for the performance standard were positive in 90 percent and for the mandate in 85 percent. The median for the performance standard was about $196 million, with a 90 percent interval from about -$45 million to $692 million. The wide interval is driven mostly by the odds ratio.

Discount Rates

Discount rates are normative choices, so they are reported rather than simulated. At 7 percent, net benefits are about $197 million for the performance standard and $180 million for the mandate. At 3 percent, they are $223 million and $219 million. At 2 percent, they are $229 million and $228 million. Lower rates reduce the annualized cost of equipment and narrow the gap between the alternatives until they are nearly equal.

Valuation Method

As Week 7 committed, a life-year approach was also run. Valuing each death avoided at about $730,000 rather than $12 million turns net benefits for the performance standard to about -$110 million. No parameter range reverses the verdict as decisively.

What Is Robust and What Is Not

The finding that some form of regulation produces positive net benefits is robust to most parameter ranges and to the staffing scenario, and holds in 90 percent of simulations. It is not robust to the valuation method or to the lowest plausible mortality effect. The preference for the performance standard over the mandate is weaker. It holds in the central case, the staffing scenario and at 7 percent, but reverses when outages are frequent, nearly disappears at low discount rates and depends on whose weights are applied. The client should be told that the case for acting rests on how deaths among frail residents are valued, and that the case for the performance standard over the mandate rests on how often outages are expected to occur.

References

Briggs, A. H., Weinstein, M. C., Fenwick, E. A. L., Karnon, J., Sculpher, M. J., & Paltiel, A. D. (2012). Model parameter estimation and uncertainty analysis: A report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Medical Decision Making, 32(5), 722-732. https://doi.org/10.1177/0272989X12458348

Morgan, M. G., & Henrion, M. (1990). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press.

Skarha, J., Gordon, L., Sakib, N., June, J., Jester, D. J., Peterson, L. J., Andel, R., & Dosa, D. M. (2021). Association of power outage with mortality and hospitalizations among Florida nursing home residents after Hurricane Irma. JAMA Health Forum, 2(11), e213900. https://doi.org/10.1001/jamahealthforum.2021.3900