MGMT 8525M Week 7 Prediction Case Example

What a Manager Was Entitled to Believe About Google Flu Trends, Week by Week: A Prediction Case Separating Input Drift, a Changing Phenomenon and Flaws Present From the Start, and Naming the Weekly Check That Would Have Caught the Miss

Student Name

Walden University

MGMT 8525M: Big Data Decision Making and Management

Instructor Name

Month Day, Year

What a Manager Was Entitled to Believe About Google Flu Trends, Week by Week: A Prediction Case Separating Input Drift, a Changing Phenomenon and Flaws Present From the Start, and Naming the Weekly Check That Would Have Caught the Miss

The Forecast as Issued

In November 2008 Google launched Flu Trends, and in February 2009 Ginsberg et al. (2009) described its method in Nature. The system estimated, week by week, what fraction of doctor visits involved flu-like symptoms in the United States, and in regions within it, from the volume of selected Google search queries. The authors selected 45 queries out of some 50 million candidates by how well their frequency tracked the Centers for Disease Control and Prevention's surveillance data from 2003 to 2007, and reported that the estimates matched the official figures closely and were available about one to two weeks earlier. For a hospital system planning staffing for the flu season, the promise was clear: near-real-time flu levels, ahead of the official count.

What Was Knowable Then

A manager reading the 2009 paper was entitled to believe that the model fit past seasons well and arrived earlier than official data. The manager was not entitled to believe that the fit would hold indefinitely. The model was a correlation between search behavior and doctor visits estimated over five seasons, with no mechanism linking the two beyond the assumption that sick people search about flu. Nothing in the method would detect if search behavior changed for reasons other than illness.

The Outcome Series

The first warning came within months. In the spring of 2009, the nonseasonal H1N1 pandemic produced a pattern of illness different from the seasonal flu on which the queries were selected, and the original model missed it; Google revised the model that autumn. The larger failure came later. Lazer et al. (2014) reported that in the 2012 to 2013 season the system estimated more than double the proportion of doctor visits for influenza-like illness that the CDC recorded, and that it had overestimated in 100 of 108 weeks beginning in August 2011.

Three Sources of a Miss

The case separates the miss into three sources. The first is drift in the inputs. Lazer and colleagues pointed to changes in Google's search engine itself, such as suggested searches and related-term features, which changed how often people typed flu-related queries for reasons unrelated to illness. The data-generating process behind the predictor moved, while the model assumed it was fixed.

The second is change in the phenomenon. The H1N1 pandemic was a real change in the pattern of illness. A model built on seasonal flu could not be expected to track it, and the 2009 miss belongs largely to this category.

The third is a flaw present from the start. Lazer and colleagues named part of the problem big data hubris: treating a large found dataset as a replacement for careful traditional measurement instead of an addition to it. Selecting 45 terms from 50 million by correlation with a few seasons of data invites overfitting: some selected terms tracked winter, not flu. Lazer and colleagues characterized the first version as partly a flu detector and partly a winter detector, which is what such a selection procedure would be expected to produce.

Against the Simple Rule

A useful benchmark for any forecast is a simple rule. Lazer and colleagues showed that a model projecting forward from CDC data a few weeks old performed better than Flu Trends in the period of overestimation, and that combining the two improved on either. A manager comparing Flu Trends each week with a lagged official figure would have seen the gap widen.

Proper Scoring

Flu Trends issued point estimates, not probabilities, so a proper scoring rule such as the one Brier (1950) proposed for probability forecasts cannot be applied to it directly. A manager could still have asked for, or built, a probabilistic version, for example the probability that next week's official figure would exceed a staffing threshold, and scored it with a Brier score over the season. A forecast that cannot be scored against outcomes in this way leaves its user unable to tell skill from luck.

Monitoring That Would Have Caught It

Three checks, run weekly, would have flagged the problem by early 2012. Compare the forecast with the official figure once it arrives and chart the running error. Compare it with a simple lagged rule and note when the simple rule wins. Track any announced changes to the input source, here Google's search features, and treat each as a reason to recheck the fit. None of these checks requires access to the model's internals.

References

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1-3. https://doi.org/10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2

Ginsberg, J., Mohebbi, M. H., Patel, R. S., Brammer, L., Smolinski, M. S., & Brilliant, L. (2009). Detecting influenza epidemics using search engine query data. Nature, 457(7232), 1012-1014. https://doi.org/10.1038/nature07634

Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014). The parable of Google Flu: Traps in big data analysis. Science, 343(6176), 1203-1205. https://doi.org/10.1126/science.1248506