Forecasting natural and social systems

Andreas Joseph will discuss a generic workflow for the use of machine learning models to inform decision making and to communicate modelling results with stakeholders. It involves three steps: (1) a comparative model evaluation, (2) a feature importance analysis and (3) statistical inference based on Shapley value decompositions. Joseph will cover the different steps of the workflow in detail and demonstrate each by forecasting changes in US unemployment one year ahead using the well-established FRED-MD dataset. He will also illustrate that universal function approximators from the machine learning literature, including gradient boosting and artificial neural networks, outperform more conventional linear models. This better performance is associated with greater flexibility, allowing the machine learning models to account for time-varying and nonlinear relationships in the data generating process. The Shapley value decomposition identifies economically meaningful nonlinearities learned by the models. Shapley regressions for statistical inference on machine learning models enable us to assess and communicate variable importance akin to conventional econometric approaches. While Joseph will also explore high-dimensional models, he will suggest that the best trade-off between interpretability and performance of the models is achieved when a small set of variables is selected by domain experts.

A longstanding goal of psychology is to predict or forecast the things people do and feel. Despite this, tools to accurately predict future behaviours and experiences have remained elusive. Dr Beck’s work indicates that one reason behavioural forecasting models may fail is the assumption that people should share psychological and situational antecedents of the same behaviours and experiences. In this talk, Dr Beck will present research from two different studies of younger and older adults to highlight the utility of person-specific machine learning methods. Using intensive longitudinal data and three machine learning forecasting approaches, she investigated the degree to which several behaviours and experiences (eg, loneliness, procrastination, physical pain) could be predicted from past psychological (ie personality and affective states), situational (ie objective situations and psychological situation cues), and time (ie trends, diurnal cycles, time of day, and day of the week) phenomena from a person-specific perspective. Rather than pitting persons against situations, such an approach allows psychological phenomena, situations, and time to jointly predict future behaviour and experiences. Dr Beck found (1) a striking degree of prediction accuracy across participants, (2) that most participants’ future behaviours are predicted by both person and situation features, and (3) that the most important features vary greatly across people. She concluded with future directions and challenges in psychological and behavioural forecasting.

PollyVote.com was launched in 2004 as a long-term project to demonstrate the value of forecasting principles, derived from the general forecasting literature, by applying them to the high-profile task of forecasting major elections in the United States, Germany, and France. As part of this project, Professor Graefe provided empirical evidence on the relative predictive accuracy of different forecasting methods (ie, intention-based polling, expectations-based judgment, and model-based forecasts). The evidence shows that a method’s past accuracy is a poor predictor of future accuracy, and that it is difficult to assess the direction of forecasting error a priori, let alone identify the most accurate forecast prior to the event. The data further shows that combining forecasts across different methods that incorporate different information yields substantial gains in accuracy and avoids large errors. Many of the lessons learned from nearly twenty years of election forecasting have implications for improving forecasting in other domains.

In the past few years, humanitarian organisations have been exploring the potentials of anticipatory action, where scientific forecasts are used to trigger pre-agreed financing and action ahead of devastating climate shocks from floods to drought. These frameworks build upon existing climate models, such as seasonal tercile forecasts or global flood models, to predict when the worst shocks are expected to occur and act accordingly. However, applying models of climate phenomena with varying forecast outputs, lead times, uncertainty, or forecasting periods to forecast humanitarian impact presents numerous hurdles. The UN OCHA Centre for Humanitarian has been working to understand and overcome these challenges in our anticipatory action pilots. Through the collection of impact data or proxies, the validation of climate models’ performance in predicting impacts, and the development of simple frameworks around these models for decision makers, the Centre is developing or has implemented pilots in 14 countries for multiple different shocks. This talk will give an overview of the Centre’s approach in these pilots to translate climate science into actionable analysis for humanitarian decision makers. It will particularly cover the technical issues faced when validating model performance and how this performance and the uncertainty around it is communicated to decision makers. It will finish with a discussion on the open questions the Centre is currently working on as the humanitarian anticipatory action agenda continues to grow.

People often make mistakes, sometimes quite serious, when they try to predict the future. Examples include international politics, financial investments, public policies, and medical diagnoses. Past psychological research shows that human predictions are often worse than very simple mathematical rules or algorithms, although predictions based on more sophisticated techniques are now gaining some traction. The Intelligence Community needed predictions of global events, so IARPA (the research wing of the US Intelligence Community) conducted a series of geopolitical forecasting tournaments in which the team at the University of Pennsylvania competed with other university teams. Questions covered a wide range of topics, including events the Syrian civil war, the Russian invasion in Crimea and whether French or Swiss inquiries would find elevated polonium in Yasser Arafat's body. IARPA defined predictive success as the Brier scoring rule. All the teams, using the same scoring rule, were given the same questions during the same period, so science could proceed at a rapid pace. The team at the University of Pennsylvania won the tournaments by a wide margin each year for four years. Five drivers accounted for our success. They included: identifying forecasting talent; training forecasters in probabilistic reasoning; placing forecasters in teams rather than having them work alone; tracking them by putting the best forecasters on the same teams; and aggregating forecasts using simple new algorithms. Human forecasts could be predicted with a surprising degree of accuracy when predictions were viewed from multiple perspectives, including psychological, economic, political, and statistical standpoints.

Forecasting exercises in demography attempt to answer questions related to human populations. Example questions range from big general ones such as “How many people will there be globally in 2100?” to questions that are focused on smaller populations and specific indicators like “What will be the change in adolescent birth rates in a specific subnational area between now and 2030?”. Often, due to data limitations, nowcasting of demographic indicators is necessary for producing forecasts, and estimates of current levels and past trends are of interest in their own right. Examples include estimation of abortion rates currently in countries where abortions are illegal, or the assessment of past trends in maternal mortality in low-income countries without well-functioning registration systems. Typically, uncertainty associated with past trends and future predictions is substantial and needs to be taken into account when interpreting results. In this talk, Professor Alkema will introduce the use of Bayesian statistical modelling approaches to nowcasting and forecasting in demography. Illustrated by examples in the area of family planning, reproductive health, and fertility, she will present general modelling challenges for producing nowcasts and forecasts, and common features of Bayesian statistical models that overcome these challenges. Professor Alkema will end with some comments related to ongoing and future research in statistical demography to improve the quality and utility of model-based nowcasts and forecasts. 

Floods are major natural disasters that can be responsible for life losses and severe economic damages. Flood forecasting and early warning systems are needed to anticipate the arrival of these events and mitigate their impacts. Flood impacts are in great part related to changes in societal vulnerability, such as increased exposure to extreme events and increased density of risk-prone urbanised areas. Climate change is also likely playing a role in increasing flood risk, by exacerbating the frequency and intensity of extreme rainfall events. When forecasting floods, these two factors must be considered: the weather, and the characteristics of the river basin. Understanding and tackling the many uncertainties that affect our ability to foresee when and where a flood event will occur, and how strong it will be, from the atmospheric to the land and river conditions, is important to inform decision-makers, civil protection authorities and the population at risk. We often assume that supplied with information, people will take better risk-based decisions. However, the efficient communication of uncertain flood forecasts also requires understanding how forecast information is perceived and used in real time. New technologies and data have been increasingly used to achieve more accurate, timely and reliable flood forecasts and better predict their impacts. Lessons learned from past flood events have shown that we can still improve our ability to communicate and act based on uncertain flood forecasts, by increasing anticipation and confidence in the forecasts. 

Earthquakes are emergent phenomena, arising in complex fault systems where tectonic stresses cause spontaneous ruptures across a vast range of spatiotemporal scales. The complexity of such systems requires that forecasting models treat seismic activity as a stochastic process. An earthquake forecast estimates the probability that a future earthquake greater than a certain magnitude will occur in specified space-time window. The most common type of forecast, widely used in seismic hazard analysis, is the time-independent Poisson model, which assumes earthquakes occur randomly in time with spatially variable magnitude-frequency distributions but stationary rates. Forecasts can be improved by introducing time-dependent probabilities that account past seismic activity. On long time scales of decades to centuries, earthquake activity is modulated by the accumulation and release of stress on major faults, which can be modelled as a Reid renewal process. On short time scales of days to months, earthquake sequences show clustering in space and time, as indicated by aftershock activity, which can be described by a Hawkes self-exciting process. Long-term forecasts, such as the US National Seismic Hazard Model, are currently the most important forecasting tools for civil protection against earthquake damage, because they guide the seismic safety provisions of building codes and performance-based design. Properly applied, short-term forecasts have operational utility in anticipating the increased seismicity that follows large earthquakes; however, while the short-term probabilities of large, potentially damaging earthquakes can vary over several orders of magnitude, they typically remain low in an absolute sense (< 1% per day). Translating such low-probability forecasts into effective decision-making is a difficult challenge.

Wildland fires are dynamic, complex events that generate transient phenomena including fire whirls, blow-ups, bursts of flame ahead of the fire line, deep thunderstorms, and wind speeds among atmospheric extremes. Public fire spread predictions are not yet a standard forecast product and, despite emergency alert systems and improving weather forecasts, communities may be tragically unprepared for complex and explosive fire behaviour. Fire behaviour models have been developed and applied since the 1970s. Early models – steady-state kinematic formulae or empirical relationships based on theory, laboratory fire beds, or small-scale prescribed fire experiments – were primarily used to estimate a fire’s rate of expansion. Ad hoc calibration prolonged their use though recent years. As complexities of wildfire behaviour became recognised, models evolved from simple algorithms to computational fluid dynamics modelling systems, where fluid or weather models are coupled to algorithms parameterising fire behaviour or combustion processes. Compared to legacy tools, computational modelling systems increase cost and complexity but have produced breakthrough understanding of the mechanisms underlying outlier wildfire events such as fine-scale extreme winds that can draw ignitions from the electric grid and the production of fire-induced winds. When combined with remotely sensed active fire detection data, coupled weather-fire models have been configured to not only forecast a fire’s growth but expand our ability to anticipate when fires may bifurcate, merge, or change directions, and currently broach prediction of phenomena like large firenadoes. Coen will discuss case studies of recent extreme events and the challenges and limitations in our remote sensing systems, fire prediction tools, and meteorological models that add to wildfires’ mystery and apparent unpredictability.

Most extant assessments of long-term risks of weather hazards rely mostly or entirely on historical records. In many cases, such records are too short to capture the all-important tail risks and are often seriously flawed owing to inadequate measurements. Moreover, even if there were excellent and long records of weather hazards, the application of such records to assessment of current risks would be seriously compromised by climate change that has already occurred. For this reason, Emanuel advocates for the application of computational weather models to long term risks and, in his talk, will discuss such an application to the specific risks posed by tropical cyclones. The main challenge posed by the application of computational models to long-term risk is the need of enough computer power to simultaneously resolve the phenomena in question and run the model for approximately 1,000 years of simulated time to define the all-important 100-year events. Emanuel will discuss how this challenge can be addressed through a deep understanding of the physics of the weather hazards.