Forecasting natural and social systems

Andreas Joseph will discuss a generic workflow for the use of machine learning models to inform decision making and to communicate modelling results with stakeholders. It involves three steps: (1) a comparative model evaluation, (2) a feature importance analysis and (3) statistical inference based on Shapley value decompositions. Joseph will cover the different steps of the workflow in detail and demonstrate each by forecasting changes in US unemployment one year ahead using the well-established FRED-MD dataset. He will also illustrate that universal function approximators from the machine learning literature, including gradient boosting and artificial neural networks, outperform more conventional linear models. This better performance is associated with greater flexibility, allowing the machine learning models to account for time-varying and nonlinear relationships in the data generating process. The Shapley value decomposition identifies economically meaningful nonlinearities learned by the models. Shapley regressions for statistical inference on machine learning models enable us to assess and communicate variable importance akin to conventional econometric approaches. While Joseph will also explore high-dimensional models, he will suggest that the best trade-off between interpretability and performance of the models is achieved when a small set of variables is selected by domain experts.

Dr Andreas Joseph, Bank of England, UK

Dr Andreas Joseph, Bank of England, UK

Andreas received his PhD from City University of Hong Kong in 2014 working on international trade and financial networks. He then briefly moved to the economics department of the European Central Bank before arriving at the Bank of England’s Advanced Analytics division in 2015. Since then, Andreas mostly worked on the development and use of machine and deep learning techniques in an economic policy context, such as model inference and interpretability, and more recently reinforcement learning in macroeconomics.

A longstanding goal of psychology is to predict or forecast the things people do and feel. Despite this, tools to accurately predict future behaviours and experiences have remained elusive. Dr Beck’s work indicates that one reason behavioural forecasting models may fail is the assumption that people should share psychological and situational antecedents of the same behaviours and experiences. In this talk, Dr Beck will present research from two different studies of younger and older adults to highlight the utility of person-specific machine learning methods. Using intensive longitudinal data and three machine learning forecasting approaches, she investigated the degree to which several behaviours and experiences (eg, loneliness, procrastination, physical pain) could be predicted from past psychological (ie personality and affective states), situational (ie objective situations and psychological situation cues), and time (ie trends, diurnal cycles, time of day, and day of the week) phenomena from a person-specific perspective. Rather than pitting persons against situations, such an approach allows psychological phenomena, situations, and time to jointly predict future behaviour and experiences. Dr Beck found (1) a striking degree of prediction accuracy across participants, (2) that most participants’ future behaviours are predicted by both person and situation features, and (3) that the most important features vary greatly across people. She concluded with future directions and challenges in psychological and behavioural forecasting.

Dr Emorie D Beck, University of California, Davis, USA

Dr Emorie D Beck, University of California, Davis, USA

Dr Emorie Beck is an Assistant Professor of Personality and Individual Differences at the University of California, Davis. She received her PhD (2020) and MA (2019) from Washington University in St. Louis and her BA (with honours; 2016) from Brown University. Dr Beck’s research focuses on what personality is, which has consequences for how we measure personality, what those measures predict both short- and long-term, how personality is thought to change. She studies how to understand the personality of an individual relative to only themself, relative to some others, and relative to all others. To do so, she uses a mix of methods, including experience sampling methods, passive sensing, survey data, panel data, cognitive tests, and more measured across time intervals from moments to years along with an array of statistical approaches, including time series analysis and forecasting, hierarchical modeling, machine learning, network psychometrics, structural equation modeling, and more.

PollyVote.com was launched in 2004 as a long-term project to demonstrate the value of forecasting principles, derived from the general forecasting literature, by applying them to the high-profile task of forecasting major elections in the United States, Germany, and France. As part of this project, Professor Graefe provided empirical evidence on the relative predictive accuracy of different forecasting methods (ie, intention-based polling, expectations-based judgment, and model-based forecasts). The evidence shows that a method’s past accuracy is a poor predictor of future accuracy, and that it is difficult to assess the direction of forecasting error a priori, let alone identify the most accurate forecast prior to the event. The data further shows that combining forecasts across different methods that incorporate different information yields substantial gains in accuracy and avoids large errors. Many of the lessons learned from nearly twenty years of election forecasting have implications for improving forecasting in other domains.

Professor Andreas Graefe, Macromedia University, Munich, Germany

Professor Andreas Graefe, Macromedia University, Munich, Germany

Dr Andreas Graefe is professor in the Faculty of Business at Macromedia University of Applied Sciences, Munich, Germany. Prior to that, he worked at Karlsuhe Intitute of Technology and LMU Munich, and was a visiting scholar at the University of Pennsylvania and University of Columbia. His research interests are forecasting, with a particular focus on political forecasting, decision-making, and algorithms. He is responsible for the PollyVote.com project for forecasting elections in different countries and currently serves as chair of APSA’s Political Forecasting Group.

In the past few years, humanitarian organisations have been exploring the potentials of anticipatory action, where scientific forecasts are used to trigger pre-agreed financing and action ahead of devastating climate shocks from floods to drought. These frameworks build upon existing climate models, such as seasonal tercile forecasts or global flood models, to predict when the worst shocks are expected to occur and act accordingly. However, applying models of climate phenomena with varying forecast outputs, lead times, uncertainty, or forecasting periods to forecast humanitarian impact presents numerous hurdles. The UN OCHA Centre for Humanitarian has been working to understand and overcome these challenges in our anticipatory action pilots. Through the collection of impact data or proxies, the validation of climate models’ performance in predicting impacts, and the development of simple frameworks around these models for decision makers, the Centre is developing or has implemented pilots in 14 countries for multiple different shocks. This talk will give an overview of the Centre’s approach in these pilots to translate climate science into actionable analysis for humanitarian decision makers. It will particularly cover the technical issues faced when validating model performance and how this performance and the uncertainty around it is communicated to decision makers. It will finish with a discussion on the open questions the Centre is currently working on as the humanitarian anticipatory action agenda continues to grow.

Mr Seth Caldwell, United Nations, USA

Mr Seth Caldwell, United Nations, USA

With over 7 years of experience in data and analytics in humanitarian response and a masters in analytics, Seth Caldwell is a data scientist at the UN Centre for Humanitarian Data. He and the rest of the predictive analytics team develop triggers and manage the technical work for OCHA’s anticipatory pilots on drought, floods, and other shocks. In the past year, Seth has developed drought prediction frameworks in the Sahel, applied flood forecast models in South Sudan, and led the Centre’s work in exploring the potential of conflict prediction models for anticipatory humanitarian action.

People often make mistakes, sometimes quite serious, when they try to predict the future. Examples include international politics, financial investments, public policies, and medical diagnoses. Past psychological research shows that human predictions are often worse than very simple mathematical rules or algorithms, although predictions based on more sophisticated techniques are now gaining some traction. The Intelligence Community needed predictions of global events, so IARPA (the research wing of the US Intelligence Community) conducted a series of geopolitical forecasting tournaments in which the team at the University of Pennsylvania competed with other university teams. Questions covered a wide range of topics, including events the Syrian civil war, the Russian invasion in Crimea and whether French or Swiss inquiries would find elevated polonium in Yasser Arafat's body. IARPA defined predictive success as the Brier scoring rule. All the teams, using the same scoring rule, were given the same questions during the same period, so science could proceed at a rapid pace. The team at the University of Pennsylvania won the tournaments by a wide margin each year for four years. Five drivers accounted for our success. They included: identifying forecasting talent; training forecasters in probabilistic reasoning; placing forecasters in teams rather than having them work alone; tracking them by putting the best forecasters on the same teams; and aggregating forecasts using simple new algorithms. Human forecasts could be predicted with a surprising degree of accuracy when predictions were viewed from multiple perspectives, including psychological, economic, political, and statistical standpoints.

Professor Barbara Mellers, University of Pennsylvania, USA

Professor Barbara Mellers, University of Pennsylvania, USA

Barbara Mellers is a cognitive psychologist at the University of Pennsylvania who studies how people make judgments and decisions. She has developed descriptive models of factors that influence human decisions but are not incorporated into classical economic models. These ‘nuisance’ variables include the context for a judgment (or the distribution of surrounding stimuli), the response mode (or how the question is asked), anticipated emotions about the outcomes of decisions, and human tendencies to cooperate despite opportunities to engage in self-interest. In the last few decades, Barbara has focused primarily on how to improve human judgments and decisions. For example, she has worked with friends and colleagues on ways to increase the accuracy of human forecasts of geopolitical events using large-scale field experiments with randomised control trials. She, and others, have tested the effectiveness of elicitation methods (i.e. prediction polls vs prediction markets), probability training techniques, team interactions and aggregation algorithms. Improving the accuracy of human forecasts is a critical step toward improving the overall quality of judgments and decisions.

Forecasting exercises in demography attempt to answer questions related to human populations. Example questions range from big general ones such as “How many people will there be globally in 2100?” to questions that are focused on smaller populations and specific indicators like “What will be the change in adolescent birth rates in a specific subnational area between now and 2030?”. Often, due to data limitations, nowcasting of demographic indicators is necessary for producing forecasts, and estimates of current levels and past trends are of interest in their own right. Examples include estimation of abortion rates currently in countries where abortions are illegal, or the assessment of past trends in maternal mortality in low-income countries without well-functioning registration systems. Typically, uncertainty associated with past trends and future predictions is substantial and needs to be taken into account when interpreting results. In this talk, Professor Alkema will introduce the use of Bayesian statistical modelling approaches to nowcasting and forecasting in demography. Illustrated by examples in the area of family planning, reproductive health, and fertility, she will present general modelling challenges for producing nowcasts and forecasts, and common features of Bayesian statistical models that overcome these challenges. Professor Alkema will end with some comments related to ongoing and future research in statistical demography to improve the quality and utility of model-based nowcasts and forecasts. 

Professor Leontine Alkema, University of Massachusetts, USA

Professor Leontine Alkema, University of Massachusetts, USA

Leontine Alkema, PhD, is Professor of Biostatistics at the University of Massachusetts Amherst, United States of America. Dr. Alkema’s research focuses on the development of statistical models to assess trends and differentials in reproductive, maternal, and child health outcomes worldwide. She has developed estimation and projection methods for key global indicators, such as the under-five mortality rate, maternal mortality, usage of and unmet need for contraceptive methods, abortion incidence, and total fertility. She collaborates with various United Nations agencies, the World Health Organization, the FP2030 community, and Guttmacher Institute to help make available improved estimation methods, software and resulting model-based estimates to diverse international audiences.

Floods are major natural disasters that can be responsible for life losses and severe economic damages. Flood forecasting and early warning systems are needed to anticipate the arrival of these events and mitigate their impacts. Flood impacts are in great part related to changes in societal vulnerability, such as increased exposure to extreme events and increased density of risk-prone urbanised areas. Climate change is also likely playing a role in increasing flood risk, by exacerbating the frequency and intensity of extreme rainfall events. When forecasting floods, these two factors must be considered: the weather, and the characteristics of the river basin. Understanding and tackling the many uncertainties that affect our ability to foresee when and where a flood event will occur, and how strong it will be, from the atmospheric to the land and river conditions, is important to inform decision-makers, civil protection authorities and the population at risk. We often assume that supplied with information, people will take better risk-based decisions. However, the efficient communication of uncertain flood forecasts also requires understanding how forecast information is perceived and used in real time. New technologies and data have been increasingly used to achieve more accurate, timely and reliable flood forecasts and better predict their impacts. Lessons learned from past flood events have shown that we can still improve our ability to communicate and act based on uncertain flood forecasts, by increasing anticipation and confidence in the forecasts. 

Dr Maria-Helena Ramos, INRAE, France

Dr Maria-Helena Ramos, INRAE, France

Maria-Helena Ramos is a research scientist in hydrology and hydrometeorology. She was awarded a BSc in Civil engineering in 1993 and an MSc in Environment and water resources in 1998 in Brazil. In France, she was awarded a PhD in Atmospheric and Earth Sciences in 2002. She carried out research on hydrometeorological forecasting at the Joint Research Center of the European Commission in Italy, working with the European Flood Awareness System, before joining INRAE in France in 2007. Her research focuses on methods to improve flood and drought forecasting, to communicate forecasts and to inform decision making in water resources management under present and future climate conditions. She is currently the president of the Hydrological Sciences Division of the European Geosciences Union (EGU), member of the Scientific Steering Committee of the World Weather Research Programme (WMO), and a Fellow of the European Centre for Medium-range Weather Forecasts (ECMWF).

Earthquakes are emergent phenomena, arising in complex fault systems where tectonic stresses cause spontaneous ruptures across a vast range of spatiotemporal scales. The complexity of such systems requires that forecasting models treat seismic activity as a stochastic process. An earthquake forecast estimates the probability that a future earthquake greater than a certain magnitude will occur in specified space-time window. The most common type of forecast, widely used in seismic hazard analysis, is the time-independent Poisson model, which assumes earthquakes occur randomly in time with spatially variable magnitude-frequency distributions but stationary rates. Forecasts can be improved by introducing time-dependent probabilities that account past seismic activity. On long time scales of decades to centuries, earthquake activity is modulated by the accumulation and release of stress on major faults, which can be modelled as a Reid renewal process. On short time scales of days to months, earthquake sequences show clustering in space and time, as indicated by aftershock activity, which can be described by a Hawkes self-exciting process. Long-term forecasts, such as the US National Seismic Hazard Model, are currently the most important forecasting tools for civil protection against earthquake damage, because they guide the seismic safety provisions of building codes and performance-based design. Properly applied, short-term forecasts have operational utility in anticipating the increased seismicity that follows large earthquakes; however, while the short-term probabilities of large, potentially damaging earthquakes can vary over several orders of magnitude, they typically remain low in an absolute sense (< 1% per day). Translating such low-probability forecasts into effective decision-making is a difficult challenge.

Professor Thomas Jordan, University of Southern California, USA

Professor Thomas Jordan, University of Southern California, USA

Thomas H. Jordan is a University Professor at the University of Southern California and holds the W. M. Keck Foundation Chair of Earth Sciences. His current research is focused on system-level models of earthquake processes, earthquake forecasting, and structural seismology. He received his PhD from the Caltech in 1972, was head of MIT’s Department of Earth, Atmospheric and Planetary Sciences from 1988 to 1998, and moved from MIT to USC in 2000. From 2002 to 2017, he was the director of the Southern California Earthquake Center, where he coordinated an international research program in earthquake system science. He is a member of the US National Academy of Sciences.

Wildland fires are dynamic, complex events that generate transient phenomena including fire whirls, blow-ups, bursts of flame ahead of the fire line, deep thunderstorms, and wind speeds among atmospheric extremes. Public fire spread predictions are not yet a standard forecast product and, despite emergency alert systems and improving weather forecasts, communities may be tragically unprepared for complex and explosive fire behaviour. Fire behaviour models have been developed and applied since the 1970s. Early models – steady-state kinematic formulae or empirical relationships based on theory, laboratory fire beds, or small-scale prescribed fire experiments – were primarily used to estimate a fire’s rate of expansion. Ad hoc calibration prolonged their use though recent years. As complexities of wildfire behaviour became recognised, models evolved from simple algorithms to computational fluid dynamics modelling systems, where fluid or weather models are coupled to algorithms parameterising fire behaviour or combustion processes. Compared to legacy tools, computational modelling systems increase cost and complexity but have produced breakthrough understanding of the mechanisms underlying outlier wildfire events such as fine-scale extreme winds that can draw ignitions from the electric grid and the production of fire-induced winds. When combined with remotely sensed active fire detection data, coupled weather-fire models have been configured to not only forecast a fire’s growth but expand our ability to anticipate when fires may bifurcate, merge, or change directions, and currently broach prediction of phenomena like large firenadoes. Coen will discuss case studies of recent extreme events and the challenges and limitations in our remote sensing systems, fire prediction tools, and meteorological models that add to wildfires’ mystery and apparent unpredictability.

Dr Janice Coen, National Center for Atmospheric Research, USA

Dr Janice Coen, National Center for Atmospheric Research, USA

Dr Janice Coen holds positions of Project Scientist in the Mesoscale and Microscale Meteorology Laboratory at the National Center for Atmospheric Research in Boulder, Colorado, and Senior Research Scientist at the University of San Francisco in San Francisco, California. She received a BSc in Engineering Physics from Grove City College and an MSc and PhD from the Department of Geophysical Sciences at the University of Chicago. She studies fire behaviour and its interaction with weather using coupled weather-fire CFD models and flow analysis of high-speed IR fire imagery. Her recent work investigated the mechanisms leading to extreme wildfire events, fine-scale wind extrema that lead to ignitions by the electric grid, and integration of coupled models with satellite active fire detection data to forecast the growth of landscape-scale wildfires.

Most extant assessments of long-term risks of weather hazards rely mostly or entirely on historical records. In many cases, such records are too short to capture the all-important tail risks and are often seriously flawed owing to inadequate measurements. Moreover, even if there were excellent and long records of weather hazards, the application of such records to assessment of current risks would be seriously compromised by climate change that has already occurred. For this reason, Emanuel advocates for the application of computational weather models to long term risks and, in his talk, will discuss such an application to the specific risks posed by tropical cyclones. The main challenge posed by the application of computational models to long-term risk is the need of enough computer power to simultaneously resolve the phenomena in question and run the model for approximately 1,000 years of simulated time to define the all-important 100-year events. Emanuel will discuss how this challenge can be addressed through a deep understanding of the physics of the weather hazards.

Emeritus Professor Kerry Emanuel ForMemRS, Massachusetts Institute of Technology, USA

Emeritus Professor Kerry Emanuel ForMemRS, Massachusetts Institute of Technology, USA

Dr Kerry Emanuel is the Cecil and Ida Green emeritus professor of atmospheric science at the Massachusetts Institute of Technology, where he was on the faculty from 1981 to 2022. Before that he was on the faculty of UCLA from 1978 to 1981.

Emanuel’s initial focus was on the dynamics of rain and snow banding in winter storms, but his interests gradually migrated to the meteorology of the tropics and to climate change. His specialty is hurricane physics and he was the first to investigate how long-term climate change might affect hurricane activity, an issue that continues to occupy him today. His interests also include cumulus convection, and advanced methods of sampling the atmosphere in aid of numerical weather prediction. 

Emanuel is the author or co-author of over 200 peer-reviewed scientific papers, and three books, including Divine Wind: The History and Science of Hurricanes, published by Oxford University Press and aimed at a general audience, and What We Know about Climate Change, published by the MIT Press and now entering its third edition. He was a co-founder and co-director of MIT’s Lorenz Center, a climate think tank devoted to basic, curiosity-driven climate research.