Forecasting infectious disease incidence for public health

Throughout the course of the SARS-CoV-2 pandemic Dr Birrell and his team fulfilled a requirement both from within the UK Health Security Agency (including Public Health England) and externally through SPI-M, to produce pandemic nowcasts and medium-term ‘projections’ in real-time. This was achieved through the PHE-Cambridge real-time model (RTM), a deterministic compartmental model designed to assimilate an array of pandemic data streams to produce epidemic nowcasts and forecasts, whilst estimating key epidemic quantities. Such models need to accurately capture fluctuations in transmission arising from external determinants (eg changing government advice, the emergence of new variants). Ignoring this “environmental stochasticity" can lead to mis-calibrated models that underestimate uncertainty and produce biased predictions. This stochasticity was injected in the current RTM by modelling the infection hazard, βt, as a stochastic process, a discrete-time random-walk with coarse time-steps (Birrell et al, 2021). This, however, was a pragmatic choice, designed, above all, to alleviate computational burden. Here Dr Birrell will present alternative representations for βt, and frame the choice between them on the basis of predictive ability, investigating the performance in predicting deaths among individuals testing positive for SARS-CoV-2 over a three-week time horizon. In doing so he will introduce a novel approach using approximating pathwise-expansions of diffusion processes.

Dynamic models of infectious disease systems are often used to study the epidemiological characteristics of disease outbreaks, the ecological mechanisms affecting transmission, and the suitability of various mitigation and intervention strategies. In recent years these same models have been employed to generate probabilistic forecasts of infectious disease incidence at population scale. Here, Professor Shaman will discuss some of the issues impeding mechanistic infectious disease forecast accuracy and possible pathways for overcoming these challenges.

There are a variety of approaches to modelling and making predictions about the dynamics of infectious diseases. For instance, one can take a strategic/mechanistic approach that primarily concerns itself with determining what types of processes can cause certain patterns. Strategic models often require large amounts of data to parameterise them and make them useful for prediction. On the other extreme are tactical/phenomenological models, like regressions, that usually focus on fitting a pattern without elucidating why those patterns exist. Tactical models, while often conceptually simpler, can be poor for extrapolating beyond the range of the data. Thus, each approach has its strengths and weaknesses in terms of data needed to parameterise and validate the model and the types of predictions that we can make using them. Professor Johnson will talk about the open challenge of how to use data at different resolutions with models that incorporate both mechanistic and strategic elements to improve prediction. In particular, she will focus on building models to prediction the dynamics of vector-borne diseases at different spatial and temporal. 

The epidemic reproduction number has gained global attention during the COVID-19 pandemic. A wide range of approaches exist to estimate it, with varying levels of data requirements, model complexity, and computational cost. Dr Cori will present one of the simplest approaches, EpiEstim, and review some of the recent developments to improve it, and implications for epidemic forecasting. She will showcase selected examples from her team’s recent work including on influenza and COVID-19. Finally, Dr Cori will discuss what she considers to be priorities for future epidemic forecasting based on reproduction number estimation.

Throughout the COVID-19 pandemic, the SARS-CoV-2 virus has continued to evolve, with new variants outcompeting existing variants and often leading to different dynamics of disease spread. In this talk, Dr Del Valle will discuss how her team characterised variant transition differences with respect to speed, calendar timing, and magnitude of 13 SARS-CoV-2 variant waves/transitions for 215 countries and sub-country regions, between October 2020 and October 2022. Furthermore, she will discuss some of the associations contributing to the dynamics observed and how this work can inform epidemic forecasting models to ultimately increase accuracy.

Predictions from human forecasters can play a useful role, complementing model-based predictions when aiming to provide timely and accurate forecasts to decision-makers and the public. Human judgement can incorporate information not easily available to computer models; it can be elicited rapidly, and it can help answer questions that are hard to capture in a model. On the other hand, human judgement forecasting requires experience, subject matter knowledge, as well as time and, therefore, is not easily scalable. This talk will discuss strengths and weaknesses of human judgment forecasting compared to model-based predictions and explore ways to overcome these by combining the two approaches. Dr Bosse will present findings from a study eliciting crowd forecasts of COVID-19 in Germany and Poland using a newly created app, 'crowdforecastr', and a follow-up study with a larger pool of forecasters conducted in the UK. 

An epidemic incidence curve is inherently latent, with substantial uncertainty even in the far past. To forecast the future, we must first reduce and manage the uncertainty in the immediate past and present. One framework for doing that is ‘pancasting’ – the joint stochastic modelling of past, present and future. The temporal dynamic model is the same throughout, while the distribution of past curves is stochastically constrained by measurements to date. Professor Rosenfeld will discuss this framework and its use of the data revision triangle. A realistic model of the temporal dynamics of epidemics must account for the fact that viral transmissibility is not constant, but rather a complex function of population immunity, environmental factors, government policy, human behaviour, and viral evolution. These drivers of viral transmissibility are well known yet their effect is hard to quantify. The first year of the pandemic gave us a rare opportunity to make progress on this front, due to (1) no pre-existing immunity; (2) slow viral antigenic evolution; and (3) unprecedentedly detailed measurements. Professor Rosenfeld will discuss his team’s ongoing attempt to use this period to glean some insight into the true drivers of SARS-Cov-2 transmissibility. Time permitting, he will also discuss Delphi’s work on acquiring epidemiologically relevant data sources and extracting and publishing myriad epidemic signals in real-time and at fine geographic granularity.

Short-term forecasting of the COVID-19 pandemic is required to facilitate the planning of COVID-19 healthcare demand in hospitals. Over the course of the pandemic, Dr Paireau and her team developed different approaches to anticipate COVID-19 healthcare needs in France. In this talk, she will revisit the different steps that they followed to improve forecasts, moving from a single complex mechanistic model to an ensemble model relying on early predictors (such as the number of cases, mobility, or meteorological predictors) and a variety of statistical and machine learning methods. Dr Paireau will present the latest developments relative to this ensemble model, including the re-evaluation of the impact of mobility predictors in a period when non-pharmaceutical interventions restricting population movements were no longer the main driver of the epidemic dynamics. Now that we are moving towards endemicity, she will also present a new approach to forecast future COVID-19 waves based on the dynamics of past waves. Finally, Dr Paireau will discuss methodological challenges they faced in this forecasting exercise.

Probabilistic forecasts play an indispensable role in answering questions about the spread of newly emerged pathogens. However, uncertainties about the epidemiology of emerging pathogens can make it difficult to choose among alternative model structures and assumptions. To assess the potential for uncertainties about emerging pathogens to affect forecasts of their spread, we evaluated the performance of 16 forecasting models in the context of the 2015-2016 Zika epidemic in Colombia. Each model featured a different combination of assumptions about human mobility, spatiotemporal variation in transmission potential, and the number of virus introductions. Dr Oidtman and her team found that which model assumptions had the most ensemble weight changed through time. They additionally identified a trade-off whereby some individual models outperformed ensemble models early in the epidemic, but on average the ensembles outperformed all individual models. Their results suggest that multiple models spanning uncertainty across alternative assumptions are necessary to obtain robust forecasts for emerging infectious diseases. At the end of the presentation, Dr Oidtman will additionally consider the public health implications of this research and the methodological challenges and potential advancements moving forward. This research was completed as part of a large cross-functional collaboration among UNICEF, public health entities in Colombia (Instituto Nacional de Salud and Ministerio de Salud y Protección Social), multiple universities, and a telecommunication company (Telefónica).

The COVID-19 and FluSight Forecast Hubs have collected, disseminated, synthesised, and evaluated millions of real-time forecasts for reported COVID-19 cases, hospitalisations, and deaths and influenza hospitalisations at the national, state, and county (cases only) level from more than 100 different academic, industry, and independent research groups. This presentation will review several key findings from Forecast Hub evaluations and experience from the operation of these systems, including:

• An ensemble forecast that combined predictions from different teams was often one of the most accurate forecasts and provided a robust option for public communication by CDC.
• Forecasts, including the ensemble, did not reliably predict changes in trends.
• Reliability of ensemble forecasts for outcomes that occurred earlier in disease progression (eg, report of a case) was often lower than for outcomes that occurred later (eg, report of a death). This may be due to noise in the measurement of the former, and the availability of leading indicators for the latter.
• Collaborative, multi-team forecasting efforts and active collaboration and coordination between governmental public health agencies and forecasting teams were critical for creating a forecasting system that provided useful information to partners; encouraged model development, evaluation, and comparison; and fostered a community with an open science ethos.

Forecast evaluation is a crucial step in the development cycle of epidemiological prediction models, and a prerequisite for data-driven combination of different forecasts into an ensemble. Moreover, transparent evaluation is required to create trust in predictions and determine to which degree they can inform public health decision making. In this talk Dr Bracher will provide an overview of statistical evaluation methods, with a particular focus on probabilistic forecasts. He will introduce some central theoretical concepts including PIT histograms as well as consistency and propriety of scoring rules. The qualitative behaviour and practical applicability of different scoring rules will be contrasted, and score decompositions which enhance the interpretability of results will be discussed. He will then turn to specific challenges encountered in large-scale epidemiological forecasting projects. These include accounting for the discreteness of forecast targets, the comparison of models applied to incongruent sets of forecasting tasks, the role of naïve benchmark models and the appropriate choice of scale on which forecasts are evaluated. To conclude, Dr Bracher will touch upon the relationship between statistical evaluations and the perceived public health utility of different predictions.

Infectious disease forecasts may be useful for public health workers. For example, they can serve as a general tool for situational awareness to help inform communications with the public. They could also support more specific decision-making tasks such as site selection for clinical trials, implementation of non-pharmaceutical interventions, or planning the allocation of a limited supply of medical care resources such as ventilators. In settings where forecasts are used as an input to specific decisions, it may be helpful to evaluate the value of forecasts in the context of that decision-making exercise. For example, some decisions may require accurate information about the timing of the onset of a new wave of infections, while other decisions may require accurate forecasts of the timing of a peak in hospitalisations, and still other decisions may require accurate information about the relative size of peaks in hospitalisations across different jurisdictions. Different forecasters may achieve varying levels of accuracy for these different tasks. Professor Ray will illustrate these ideas using the example of allocating a limited supply of medical care resources to multiple jurisdictions. He has developed a proper scoring rule that formalises a measure of the value of a forecast in the context of this decision-making problem, as determined by the costs that would be incurred through misallocation if the forecast were used to set allocation levels. In his presentation, Professor Ray will explore how this scoring rule is related to the Weighted Interval Score (WIS) and will illustrate how it could be used to evaluate forecasts from the US COVID-19 Forecast Hub.

Over 24 months, the COVID-19 Scenario Modelling Hub (SMH) has provided 16 rounds of scenario projections to inform the US pandemic response. In each round, the SMH solicits and ensembles 3- to 12-month projections of cases, hospitalisations, and deaths at the state and national level from 6-10 independent modelling teams. Here we evaluate the SMH effort over these 16 rounds. However, the nature of scenario projections makes evaluation challenging. Scenario projections are conditional on a set of assumptions about the future, which will never unfold exactly as assumed. To address these challenges, we present a conceptual framework for understanding and interpreting the evaluation of a scenario projection. This framework rests on an evaluation of both the scenarios themselves (eg, scenario-specified vaccine uptake vs realised uptake) and the projections that were based on these scenarios (eg, projected incident hospitalisations vs observed hospitalisations). Key findings for SMH include: (1) in most cases, scenarios bracketed reality, but very few were close to the realised future; and (2) for scenarios that were “close” to reality, the ensemble was consistently more reliable than individual models. Building on the SMH experience, Dr Howerton will discuss further foundational work on scenario evaluation to advance the growing field of multi-model infectious disease scenario projections.

The primary task of a decision-maker during a disease outbreak is to decide on the best course of action. When making decisions in the face of uncertainty, such as in responding to emergency disease outbreaks, there exists a trade-off between information accrual and capitalising upon information by taking action. That is, as information accrues, such as via improved parameter estimates or model structure clarification, we become better informed as to the best course of action. However, the longer we wait to take action, the larger the potential opportunity cost from inaction. This work investigated this trade-off by updating model estimates and subsequent control action recommendations throughout the foot-and-mouth disease outbreaks of 2001 in the UK and 2010 in Miyazaki, Japan. Every four weeks throughout each outbreak, parameters were re-estimated based upon all available data and control actions re-evaluated. These recommendations were compared to those derived from parameters estimates using data from the whole outbreak (ie assuming one has all available information from the start of the outbreak). Overall, Dr Probert and his team found that while forecasts of outbreak severity in the early stages of an outbreak may be poor, the relative performance of different interventions were resolved early. He will reflect on comparable studies in other infectious disease systems.