Forecasting infectious disease incidence for public health

Throughout the course of the SARS-CoV-2 pandemic Dr Birrell and his team fulfilled a requirement both from within the UK Health Security Agency (including Public Health England) and externally through SPI-M, to produce pandemic nowcasts and medium-term ‘projections’ in real-time. This was achieved through the PHE-Cambridge real-time model (RTM), a deterministic compartmental model designed to assimilate an array of pandemic data streams to produce epidemic nowcasts and forecasts, whilst estimating key epidemic quantities. Such models need to accurately capture fluctuations in transmission arising from external determinants (eg changing government advice, the emergence of new variants). Ignoring this “environmental stochasticity" can lead to mis-calibrated models that underestimate uncertainty and produce biased predictions. This stochasticity was injected in the current RTM by modelling the infection hazard, βt, as a stochastic process, a discrete-time random-walk with coarse time-steps (Birrell et al, 2021). This, however, was a pragmatic choice, designed, above all, to alleviate computational burden. Here Dr Birrell will present alternative representations for βt, and frame the choice between them on the basis of predictive ability, investigating the performance in predicting deaths among individuals testing positive for SARS-CoV-2 over a three-week time horizon. In doing so he will introduce a novel approach using approximating pathwise-expansions of diffusion processes.

Dr Paul Birrell, UK Health Security Agency and MRC Biostatistics Unit University of Cambridge, UK

Dr Paul Birrell, UK Health Security Agency and MRC Biostatistics Unit University of Cambridge, UK

Dr Paul Birrell is a Senior Statistician at the UK Health Security Agency, leading on real-time pandemic modelling and a Senior Research Associate at the MRC Biostatistics Unit at the University of Cambridge. He sits on the Scientific Pandemic Influenza Group on Modelling (SPI-M) a permanent sub-group of the Scientific Advisory Group for Emergencies (SAGE) which provides decision-making support to government in the event of a national emergency. Paul began working on transmission modelling in response to the 2009 A/H1N1pdm outbreak and, whilst working at the University of Cambridge, developed a real-time modelling framework that was adopted by Public Health England as their official real-time model.

Dynamic models of infectious disease systems are often used to study the epidemiological characteristics of disease outbreaks, the ecological mechanisms affecting transmission, and the suitability of various mitigation and intervention strategies. In recent years these same models have been employed to generate probabilistic forecasts of infectious disease incidence at population scale. Here, Professor Shaman will discuss some of the issues impeding mechanistic infectious disease forecast accuracy and possible pathways for overcoming these challenges.

Professor Jeff Shaman, Carnegie Mellon University, USA

Professor Jeff Shaman, Carnegie Mellon University, USA

Jeffrey Shaman is a Professor of Environmental Health Sciences in the Columbia University Mailman School of Public Health and of Climate in the Columbia University Climate School. He also serves as Director of the Climate and Health Program at the Mailman School, Faculty Chair of the Earth Institute, and Vice Dean for Faculty Affairs at the Climate School. Professor Shaman holds a BA in Biology from the University of Pennsylvania and an MA and PhD in Climate Science from Columbia University. Current research foci include study of the survival, transmission and epidemiology of infectious agents, development and improvement of infectious disease forecasting systems, and investigation of contagion and transmission processes generally.

There are a variety of approaches to modelling and making predictions about the dynamics of infectious diseases. For instance, one can take a strategic/mechanistic approach that primarily concerns itself with determining what types of processes can cause certain patterns. Strategic models often require large amounts of data to parameterise them and make them useful for prediction. On the other extreme are tactical/phenomenological models, like regressions, that usually focus on fitting a pattern without elucidating why those patterns exist. Tactical models, while often conceptually simpler, can be poor for extrapolating beyond the range of the data. Thus, each approach has its strengths and weaknesses in terms of data needed to parameterise and validate the model and the types of predictions that we can make using them. Professor Johnson will talk about the open challenge of how to use data at different resolutions with models that incorporate both mechanistic and strategic elements to improve prediction. In particular, she will focus on building models to prediction the dynamics of vector-borne diseases at different spatial and temporal. 

Professor Leah Johnson, Virginia Tech, USA

Professor Leah Johnson, Virginia Tech, USA

Professor Leah Johnson is an Associate Professor of Statistics at Virginia Tech. She received her BS in Physics from the College of William and Mary (Virginia, US), and then her MS (Physics) and PhD (Physics/Applied Mathematics and Statistics) at the University of California Santa Cruz. Before joining Virginia Tech, she held research posts at the University of Cambridge, University of California Santa Barbara, and University of Chicago, and was an Assistant Professor of Integrative Biology at the University of South Florida. Her research interests are in statistical and mathematical biology, ecology, and epidemiology. In particular, she is interested in how individual behaviour and local interactions between individuals in a population, influence population level patterns as well as statistical methods for inference in mechanistic models of biological systems. Recently, her work has focused on understanding how environmental factors (particularly temperature) influence transmission of vector-borne pathogens.

The epidemic reproduction number has gained global attention during the COVID-19 pandemic. A wide range of approaches exist to estimate it, with varying levels of data requirements, model complexity, and computational cost. Dr Cori will present one of the simplest approaches, EpiEstim, and review some of the recent developments to improve it, and implications for epidemic forecasting. She will showcase selected examples from her team’s recent work including on influenza and COVID-19. Finally, Dr Cori will discuss what she considers to be priorities for future epidemic forecasting based on reproduction number estimation.

Dr Anne Cori, Imperial College London, UK

Dr Anne Cori, Imperial College London, UK

Dr Cori develops statistical methods and tools for the analysis of epidemic data. She is particularly interested in emerging infections, and contexts where data are scarce and statistical methods can be developed to answer important public health questions despite these limitations in data. She is the author of the R package EpiEstim. Her research mainly focuses on viruses including SARS-CoV-2, Ebola, MERS, influenza and HIV.

Throughout the COVID-19 pandemic, the SARS-CoV-2 virus has continued to evolve, with new variants outcompeting existing variants and often leading to different dynamics of disease spread. In this talk, Dr Del Valle will discuss how her team characterised variant transition differences with respect to speed, calendar timing, and magnitude of 13 SARS-CoV-2 variant waves/transitions for 215 countries and sub-country regions, between October 2020 and October 2022. Furthermore, she will discuss some of the associations contributing to the dynamics observed and how this work can inform epidemic forecasting models to ultimately increase accuracy.

Dr Sara Del Valle, Los Alamos National Laboratory, USA

Dr Sara Del Valle, Los Alamos National Laboratory, USA

Dr Sara Del Valle is a Senior Scientist at Los Alamos National Laboratory where she works on the development of mathematical, statistical, and computational models for the spread of infectious diseases and other applications in support of decision makers. Her team combines large-scale heterogeneous data streams with epidemiological models to understand, model, and forecast disease spread in real-time. Her work has been covered in several top media outlets such as National Geographic, NPR Science Friday, Popular Mechanics, and the New York Times. She received a PhD in Applied Mathematics from the University of Iowa.

Predictions from human forecasters can play a useful role, complementing model-based predictions when aiming to provide timely and accurate forecasts to decision-makers and the public. Human judgement can incorporate information not easily available to computer models; it can be elicited rapidly, and it can help answer questions that are hard to capture in a model. On the other hand, human judgement forecasting requires experience, subject matter knowledge, as well as time and, therefore, is not easily scalable. This talk will discuss strengths and weaknesses of human judgment forecasting compared to model-based predictions and explore ways to overcome these by combining the two approaches. Dr Bosse will present findings from a study eliciting crowd forecasts of COVID-19 in Germany and Poland using a newly created app, 'crowdforecastr', and a follow-up study with a larger pool of forecasters conducted in the UK. 

Dr Nikos Bosse, London School of Hygiene and Tropical Medicine, UK

Dr Nikos Bosse, London School of Hygiene and Tropical Medicine, UK

Nikos Bosse MD is a PhD student at the London School of Hygiene & Tropical Medicine working on infectious disease forecasting and forecast evaluation. For his PhD, he elicited forecasts from epidemiologists and lay people about the future trajectory of COVID-19 and compared these against model-based predictions. In addition to that, his work focuses on refining methods and building tools that help evaluate forecasts, such as the ‘scoringutils’ package in R.

An epidemic incidence curve is inherently latent, with substantial uncertainty even in the far past. To forecast the future, we must first reduce and manage the uncertainty in the immediate past and present. One framework for doing that is ‘pancasting’ – the joint stochastic modelling of past, present and future. The temporal dynamic model is the same throughout, while the distribution of past curves is stochastically constrained by measurements to date. Professor Rosenfeld will discuss this framework and its use of the data revision triangle. A realistic model of the temporal dynamics of epidemics must account for the fact that viral transmissibility is not constant, but rather a complex function of population immunity, environmental factors, government policy, human behaviour, and viral evolution. These drivers of viral transmissibility are well known yet their effect is hard to quantify. The first year of the pandemic gave us a rare opportunity to make progress on this front, due to (1) no pre-existing immunity; (2) slow viral antigenic evolution; and (3) unprecedentedly detailed measurements. Professor Rosenfeld will discuss his team’s ongoing attempt to use this period to glean some insight into the true drivers of SARS-Cov-2 transmissibility. Time permitting, he will also discuss Delphi’s work on acquiring epidemiologically relevant data sources and extracting and publishing myriad epidemic signals in real-time and at fine geographic granularity.

Professor Roni Rosenfeld, Carnegie Mellon University, USA

Professor Roni Rosenfeld, Carnegie Mellon University, USA

Roni Rosenfeld is head of the machine learning department and Professor of machine learning, language technologies, computer science, and computational biology in the School of Computer Science at Carnegie Mellon University, Pittsburgh, Pennsylvania. His current research interests are in tracking and forecasting epidemics. The Delphi Research group, which he co-founded and co-leads since 2012, has been playing a leading role in the development of epidemic forecasting technology in the US, and has been named a National Center for Epidemic Forecasting by the US CDC. Roni has previously worked on statistical language modelling, speech recognition, human machine speech interfaces, and the use of speech and language technologies to aid international developments. He received the Spira Teaching Excellence Award in 2017 and the Allen Newell Medal for Research Excellence in 1992 and again in 2022.

Short-term forecasting of the COVID-19 pandemic is required to facilitate the planning of COVID-19 healthcare demand in hospitals. Over the course of the pandemic, Dr Paireau and her team developed different approaches to anticipate COVID-19 healthcare needs in France. In this talk, she will revisit the different steps that they followed to improve forecasts, moving from a single complex mechanistic model to an ensemble model relying on early predictors (such as the number of cases, mobility, or meteorological predictors) and a variety of statistical and machine learning methods. Dr Paireau will present the latest developments relative to this ensemble model, including the re-evaluation of the impact of mobility predictors in a period when non-pharmaceutical interventions restricting population movements were no longer the main driver of the epidemic dynamics. Now that we are moving towards endemicity, she will also present a new approach to forecast future COVID-19 waves based on the dynamics of past waves. Finally, Dr Paireau will discuss methodological challenges they faced in this forecasting exercise.

Dr Juliette Paireau, Institut Pasteur, Santé publique France, France

Dr Juliette Paireau, Institut Pasteur, Santé publique France, France

Dr Juliette Paireau is a research engineer in the Mathematical Modelling of Infectious Diseases Unit at Institut Pasteur, Paris and in the Infectious Diseases Department at Santé publique France, Paris. Her work is at the interface between modelling and public health. She conducts various research projects investigating the spatial and temporal dynamics of infectious diseases to improve prevention and control strategies. In particular, she develops ensemble models to forecast the dynamics of COVID-19 and influenza in France at the regional level, to support public health decision-making.

Probabilistic forecasts play an indispensable role in answering questions about the spread of newly emerged pathogens. However, uncertainties about the epidemiology of emerging pathogens can make it difficult to choose among alternative model structures and assumptions. To assess the potential for uncertainties about emerging pathogens to affect forecasts of their spread, we evaluated the performance of 16 forecasting models in the context of the 2015-2016 Zika epidemic in Colombia. Each model featured a different combination of assumptions about human mobility, spatiotemporal variation in transmission potential, and the number of virus introductions. Dr Oidtman and her team found that which model assumptions had the most ensemble weight changed through time. They additionally identified a trade-off whereby some individual models outperformed ensemble models early in the epidemic, but on average the ensembles outperformed all individual models. Their results suggest that multiple models spanning uncertainty across alternative assumptions are necessary to obtain robust forecasts for emerging infectious diseases. At the end of the presentation, Dr Oidtman will additionally consider the public health implications of this research and the methodological challenges and potential advancements moving forward. This research was completed as part of a large cross-functional collaboration among UNICEF, public health entities in Colombia (Instituto Nacional de Salud and Ministerio de Salud y Protección Social), multiple universities, and a telecommunication company (Telefónica).

Dr Rachel Oidtman, Colorado Access, USA

Dr Rachel Oidtman, Colorado Access, USA

Dr Rachel Oidtman is an infectious disease modeler and data scientist at Colorado Access, a non-profit healthcare provider in Colorado, USA. Her research focuses on the development of mathematical and statistical models to understand and forecast infectious disease dynamics, with a specific interest in reducing the burden of diseases. She has worked on a range of applied research questions and a variety of pathogens, including arboviruses (including Zika and dengue) and respiratory viruses (including influenza and SARS-CoV-2). Dr Oidtman earned her PhD in Biological Sciences from University of Notre Dame in 2020 and her BSc in Applied Natural Resources and Statistics from Cornell University in 2015.

The COVID-19 and FluSight Forecast Hubs have collected, disseminated, synthesised, and evaluated millions of real-time forecasts for reported COVID-19 cases, hospitalisations, and deaths and influenza hospitalisations at the national, state, and county (cases only) level from more than 100 different academic, industry, and independent research groups. This presentation will review several key findings from Forecast Hub evaluations and experience from the operation of these systems, including:

• An ensemble forecast that combined predictions from different teams was often one of the most accurate forecasts and provided a robust option for public communication by CDC.
• Forecasts, including the ensemble, did not reliably predict changes in trends.
• Reliability of ensemble forecasts for outcomes that occurred earlier in disease progression (eg, report of a case) was often lower than for outcomes that occurred later (eg, report of a death). This may be due to noise in the measurement of the former, and the availability of leading indicators for the latter.
• Collaborative, multi-team forecasting efforts and active collaboration and coordination between governmental public health agencies and forecasting teams were critical for creating a forecasting system that provided useful information to partners; encouraged model development, evaluation, and comparison; and fostered a community with an open science ethos.

Dr Matthew Biggerstaff, US Centers for Disease Control and Prevention, USA

Dr Matthew Biggerstaff, US Centers for Disease Control and Prevention, USA

Matthew Biggerstaff, ScD, MPH, serves as Team Lead of the Applied Research and Modeling Team in CDC’s Influenza Division’s Epidemiology and Prevention Branch (EPB). In this role, he oversees team activities related to the following: characterizing flu season severity, disease burden, and influenza virus transmission in the population; integrating advanced analytics and mathematical modelling into flu-related public health response efforts; and improving the use and accuracy of influenza forecasting to better predict and mitigate the impact of flu. Dr Biggerstaff joined CDC in 2006 and has been with the Influenza Division since 2009. 

Forecast evaluation is a crucial step in the development cycle of epidemiological prediction models, and a prerequisite for data-driven combination of different forecasts into an ensemble. Moreover, transparent evaluation is required to create trust in predictions and determine to which degree they can inform public health decision making. In this talk Dr Bracher will provide an overview of statistical evaluation methods, with a particular focus on probabilistic forecasts. He will introduce some central theoretical concepts including PIT histograms as well as consistency and propriety of scoring rules. The qualitative behaviour and practical applicability of different scoring rules will be contrasted, and score decompositions which enhance the interpretability of results will be discussed. He will then turn to specific challenges encountered in large-scale epidemiological forecasting projects. These include accounting for the discreteness of forecast targets, the comparison of models applied to incongruent sets of forecasting tasks, the role of naïve benchmark models and the appropriate choice of scale on which forecasts are evaluated. To conclude, Dr Bracher will touch upon the relationship between statistical evaluations and the perceived public health utility of different predictions.

Dr Johannes Bracher, Karlsruhe Institute of Technology, Germany

Dr Johannes Bracher, Karlsruhe Institute of Technology, Germany

Johannes Bracher is a postdoctoral researcher at Karlsruhe Institute of Technology (KIT) and Heidelberg Institute of Theoretical Studies (HITS), Germany. He holds an MSc in Statistics from the University of Munich (2016) and a PhD in Biostatistics from the University of Zurich (2020). His research interests include probabilistic forecasting, in particular forecast evaluation and combination, as well as real-time modelling of infectious diseases and count time series methodology. In 2023 he will establish an Emmy Noether Independent Junior Research Group at KIT, funded by the German Research Foundation (DFG). 

Infectious disease forecasts may be useful for public health workers. For example, they can serve as a general tool for situational awareness to help inform communications with the public. They could also support more specific decision-making tasks such as site selection for clinical trials, implementation of non-pharmaceutical interventions, or planning the allocation of a limited supply of medical care resources such as ventilators. In settings where forecasts are used as an input to specific decisions, it may be helpful to evaluate the value of forecasts in the context of that decision-making exercise. For example, some decisions may require accurate information about the timing of the onset of a new wave of infections, while other decisions may require accurate forecasts of the timing of a peak in hospitalisations, and still other decisions may require accurate information about the relative size of peaks in hospitalisations across different jurisdictions. Different forecasters may achieve varying levels of accuracy for these different tasks. Professor Ray will illustrate these ideas using the example of allocating a limited supply of medical care resources to multiple jurisdictions. He has developed a proper scoring rule that formalises a measure of the value of a forecast in the context of this decision-making problem, as determined by the costs that would be incurred through misallocation if the forecast were used to set allocation levels. In his presentation, Professor Ray will explore how this scoring rule is related to the Weighted Interval Score (WIS) and will illustrate how it could be used to evaluate forecasts from the US COVID-19 Forecast Hub.

Professor Evan Ray, University of Massachusetts, USA

Professor Evan Ray, University of Massachusetts, USA

Professor Ray is a Research Assistant Professor of Biostatistics at the University of Massachusetts, Amherst, and co-director of the US COVID-19 Forecast Hub. His research focuses on methods for infectious disease forecasting.

Over 24 months, the COVID-19 Scenario Modelling Hub (SMH) has provided 16 rounds of scenario projections to inform the US pandemic response. In each round, the SMH solicits and ensembles 3- to 12-month projections of cases, hospitalisations, and deaths at the state and national level from 6-10 independent modelling teams. Here we evaluate the SMH effort over these 16 rounds. However, the nature of scenario projections makes evaluation challenging. Scenario projections are conditional on a set of assumptions about the future, which will never unfold exactly as assumed. To address these challenges, we present a conceptual framework for understanding and interpreting the evaluation of a scenario projection. This framework rests on an evaluation of both the scenarios themselves (eg, scenario-specified vaccine uptake vs realised uptake) and the projections that were based on these scenarios (eg, projected incident hospitalisations vs observed hospitalisations). Key findings for SMH include: (1) in most cases, scenarios bracketed reality, but very few were close to the realised future; and (2) for scenarios that were “close” to reality, the ensemble was consistently more reliable than individual models. Building on the SMH experience, Dr Howerton will discuss further foundational work on scenario evaluation to advance the growing field of multi-model infectious disease scenario projections.

Dr Emily Howerton, Penn State University, USA

Dr Emily Howerton, Penn State University, USA

Emily Howerton is a PhD candidate in the Department of Biology at Penn State University. She uses mathematical models to design strategies for controlling infectious diseases in the face of uncertainty, most recently focusing on the COVID-19 pandemic. For example, Emily has modelled the interaction between diagnostic testing and non-pharmaceutical interventions to identify optimal combination management strategies. Emily is also a member of COVID-19 Scenario Modelling Hub coordination team, where she develops methods to capture uncertainty from multiple models in 6-month projections of COVID-19 outcomes. In particular, she has adapted aggregation and evaluation methods from multiple fields to accommodate the unique needs of infectious disease scenario projections.

The primary task of a decision-maker during a disease outbreak is to decide on the best course of action. When making decisions in the face of uncertainty, such as in responding to emergency disease outbreaks, there exists a trade-off between information accrual and capitalising upon information by taking action. That is, as information accrues, such as via improved parameter estimates or model structure clarification, we become better informed as to the best course of action. However, the longer we wait to take action, the larger the potential opportunity cost from inaction. This work investigated this trade-off by updating model estimates and subsequent control action recommendations throughout the foot-and-mouth disease outbreaks of 2001 in the UK and 2010 in Miyazaki, Japan. Every four weeks throughout each outbreak, parameters were re-estimated based upon all available data and control actions re-evaluated. These recommendations were compared to those derived from parameters estimates using data from the whole outbreak (ie assuming one has all available information from the start of the outbreak). Overall, Dr Probert and his team found that while forecasts of outbreak severity in the early stages of an outbreak may be poor, the relative performance of different interventions were resolved early. He will reflect on comparable studies in other infectious disease systems.   

Dr Will Probert, Consultant, UK

Dr Will Probert, Consultant, UK

Will Probert currently works as a consultant within WHO's Global HIV, Hepatitis, and STI programmes. Will previously held postdoctoral research positions deploying individual-based models to inform public health, or veterinary, decision-making. While at Oxford University with Professor Christophe Fraser, projects included being co-lead developer of the COVID-19 individual-based model 'OpenABM-Covid19' and as a modeller on the HPTN 071 (PopART) trial of HIV prevention. At both the University of Warwick, with Professor Mike Tildesley, and the Pennsylvania State University, with Professor Katriona Shea and Professor Matt Ferrari, projects investigated the use of reinforcement learning algorithms to generate novel control strategies in individual-based models of foot-and-mouth disease.