Symbolic regression in the physical sciences

After a broad overview of the symbolic regression landscape, I will present the ambitious programme of searching function space exhaustively with a single, objective metric that combines accuracy and simplicity. At low complexity, an exhaustive search is afforded by our algorithm Exhaustive Symbolic Regression (ESR), which generates all possible simple functions produced from a user-defined basis set of operators. Single-objective optimisation is possible by means of the Minimum Description Length (MDL) principle, which puts function accuracy (log-likelihood of the data) and complexity (both structural and parametric) on the same scale (number of nats) to allow them to be directly traded off. I will showcase the ESR+MDL combination on several hot topics in astrophysics, and discuss potential future developments.

Dr Harry Desmond, University of Portsmouth, UK

Dr Harry Desmond, University of Portsmouth, UK

Harry Desmond is a University Research Fellow at the Institute of Cosmology and Gravitation, University of Portsmouth. His interests in astrophysics include galaxy dynamics & phenomenology, the galaxy-halo connection, constrained cosmological simulations, stellar structure, modified gravity and dark matter. He is also interested in astrostatistics, particularly Bayesian methods and field-level inference. In symbolic regression he was a developer of the Exhaustive Symbolic Regression algorithm, which also introduced description length as a joint measure of functions’ accuracy and complexity. He is keen both on methodological developments in symbolic regression – for example developing hybrids between the exhaustive and stochastic (eg genetic programming) approaches – and on applications in astrophysics and cosmology. This includes creating more principled fitting functions, or even learning new physics.

In this talk I will discuss symbolic regression as a tool for studying new ideas in high energy physics. Many proposals for new physics that goes beyond the Standard Model yield a high dimensional parameter space that is difficult to examine by conventional means. Traditional exhaustive search methods are often time consuming and are possible for only the simplest of theories. This situation holds across surprisingly many activities, from confronting theories with experimental data, to model building in string theory. In this talk I will describe ongoing work that brings symbolic regression to bear on such problems. The first example I will discuss is one in which a phenomenological proposal for new physics confronts experimental data. The model I consider is the so-called Constrained Minimal Supersymmetric Standard Model, which has a four-dimensional parameter space. We provide a set of analytical expressions that reproduce low-energy observables of interest in terms of the fundamental parameters of the theory. These observables are traditionally derived through a long chain of difficult computation, but we will see that this process can be significantly short-cut using symbolic regression, which provides a set of powerful formulae that can be used in analyses instead. The second problem I will discuss occurs in string theory, where one would like to determine the Kähler metric which is crucial for determining the effective 4D physics. Typically it is a very difficult task to find analytic formula for the Kahler metric, although approximations can often be found using for example the Donaldson algorithm. I will discuss how symbolic regression could play a crucial role in providing the desired analytic expressions. The talk will aim to provide pedagogical descriptions of the physical problems.  

Professor Steven Abel, Durham University, UK

Professor Steven Abel, Durham University, UK

Steven Abel is a Professor of Mathematical Sciences and a member of the Institute of Particle Physics Phenomenology in Durham University. Before coming to Durham he held research positions at Oxford, Brussels, Cern and Paris. His main areas of interest are  phenomenology Beyond the Standard Model, focussing on string theory motivated approaches, heuristic search methods such as genetic algorithms and symbolic regression, and applications of quantum computing methods to quantum field theory.

Process-structure-property relationships are fundamental in material science and engineering and key to developing new and better materials. Symbolic regression is a powerful tool for discovering mathematical models that describe these relationships. It can automatically generate equations that predict material behaviour under given manufacturing conditions and optimise its performance, e.g. strength and elasticity. Unlike the commonly used “black-box” machine learning models, the results are interpretable and provide insights into the studied system, e.g. the influence of individual alloying elements or processing parameters like temperature. This method thus facilitates the design of new materials with desired properties, the optimization of manufacturing processes, and the acceleration of material design without relying on predefined models or assumptions. The forthcoming lecture will illustrate how symbolic regression can derive the constitutive laws that describe the behaviour of various metallic alloys during plastic deformation. Constitutive modelling is a vital mathematical framework for understanding the relationship between stress and strain in materials under different loading conditions. It clarifies how materials respond to external forces by employing established material laws or equations. Such models are indispensable for predicting material behaviour in engineering applications, including structural analysis, design, and simulations.

Dr Evgeniya Kabliman, Technical University of Munich, Germany

Dr Evgeniya Kabliman, Technical University of Munich, Germany

Dr Evgeniya Kabliman studied applied physics and mathematics at the South Urals State University. In 2011, she finished her PhD studies at the Technical University of Vienna and worked as a Scientist at the Austrian Institute of Technology for 9 years. In 2021, she started as a group leader for Materials Computation at the Chair of Materials Engineering of Additive Manufacturing at the Technical University of Munich. Dr Kabliman’s research focuses on applying computational methods to materials science to describe the material's behaviour at multiple length scales, including the application of machine learning. In particular, she applied symbolic regression in modelling plastic deformation of metallic materials.

Symbolic Regression, a form of supervised learning, attempts to solve the NP-hard problem of searching both function form and its free parameters. It is useful when the objective is a concise and accurate mathematical description of the data.

An often considered alternative, decision trees can be easy to interpret but grow exponentially, and mathematical functions can represent many non-linearities otherwise challenging to interpret.

We propose Brush, a multi-objective symbolic regression framework that combines the decision tree’s split-wise structures with flexible conditionals to build mathematical expressions.

The inclusion of split-wise operations enables the combination of the interpretability of decision trees with the flexibility of mathematical expressions.

To complement the evolutionary loop, we test static and dynamic, contextual and context-free, multi-armed bandits that adapt the sampling process for symbols during search.

We benchmark Brush against several state-of-the-art algorithms over 250 datasets, reporting R2, training time, recovery rate of ground-truth equations, and final expressions.

Brush showed Pareto-optimal performance in R2 and model size ranks, and multi-armed bandits shifted the trade-off between accuracy and expression size along the optimal curve.

Notably, training time was shorter than that of zero-shot, transformer-based approaches on a single core.

Our experiments highlight how Brush effectively navigates a much larger search space by incorporating split-wise equations.

This tool shows promise as both a first-principles modelling technique and for learning interpretable prediction models, enhancing the practical applications of symbolic regression in various fields.

Dr William G La Cava, Boston Children's Hospital, USA

Dr William G La Cava, Boston Children's Hospital, USA

William La Cava is an Assistant Professor at the Computational Health Informatics Program (CHIP) at Boston Children’s Hospital and Harvard Medical School. He received his PhD from UMass Amherst with a focus on interpretable modelling of dynamical systems. Prior to joining CHIP, he was a post-doctoral fellow and research associate in the Institute for Biomedical Informatics at the University of Pennsylvania.

We present a mathematical formulation for machine learning of (1) functions on symmetric matrices that are invariant with respect to the action of permutations by conjugation, and (2) functions on point clouds that are invariant with respect to rotations, reflections, and permutations of the points. To achieve this, we provide a general construction of generically separating invariant features using ideas inspired by Galois theory. We construct O(n^2) invariant features derived from generators for the field of rational functions on n × n symmetric matrices that are invariant under joint permutations of rows and columns. We show that these invariant features can separate all distinct orbits of symmetric matrices except for a measure zero set; such features can be used to universally approximate invariant functions on almost all weighted graphs. For point clouds in a fixed dimension, we prove that the number of invariant features can be reduced, generically without losing expressivity, to O(n), where n is the number of points. We combine these invariant features with DeepSets to learn functions on symmetric matrices and point clouds with varying sizes. We empirically demonstrate the feasibility of our approach on molecule property regression and point cloud distance prediction. 

Assistant Professor Soledad Villar, Johns Hopkins University, USA

Assistant Professor Soledad Villar, Johns Hopkins University, USA

Soledad Villar is an Assistant Professor of Applied Mathematics and Statistics at Johns Hopkins University. Her research is in mathematical theory for machine learning. In particular, she focuses on techniques from algebra, statistics, and optimisation, to design and analyse machine learning models for scientific applications. 

Scientists aim to create mathematical models that accurately describe observed phenomena. In the past, models were manually created from domain knowledge and subsequently validated using data. More recently, models are automatically extracted from large datasets using machine learning algorithms. However, finding meaningful models from data remains an ongoing challenge. In this talk I will explore two novel approaches for deriving scientific laws that exploit both experimental data and axiomatic knowledge, enabling the creation of interpretable models with minimal data requirements: (1) AI-Descartes utilizes symbolic regression to extract models from data and then applies logical reasoning to select those that best align with established axioms; (2) AI-Hilbert, combines polynomial optimization and logical constraints ensuring both theoretically consistency and empirical validation at the same time. Together, these methods offer a new perspective on how to integrate data and logic, establishing the new paradigm of derivable scientific discovery. More information can be found at ai-descartes.github.io and ai-hilbert.github.io.

Dr Cristina Cornelio, Samsung AI, UK

Dr Cristina Cornelio, Samsung AI, UK

Cristina Cornelio is a research scientist at Samsung Research, specialising in neuro-symbolic integration, which combines machine learning with traditional reasoning techniques. After completing her PhD at the University of Padua, Italy, she spent five years as a research scientist at IBM Research, contributing to projects at both the TJ Watson Research Center in New York and the IBM Zurich Lab. In 2021, she joined the Samsung AI Center in Cambridge, UK. During her doctoral studies, she co-developed a kidney exchange algorithm now used nationwide in Italy. She has published her research in top-tier conferences and journals, including Nature Communications, with her work spanning neuro-symbolic AI applications in robotics, scientific discovery, reasoning, and agent preferences.

Sensing is a universal task in science and engineering. Downstream tasks from sensing include learning dynamical models, inferring full state estimates of a system (system identification), control decisions, and forecasting. These tasks are exceptionally challenging to achieve with limited sensors, noisy measurements, and corrupt or missing data. Existing techniques typically use current (static) sensor measurements to perform such tasks and require principled sensor placement or an abundance of randomly placed sensors. In contrast, we propose a SHallow REcurrent Decoder (SHRED) neural network structure which incorporates (i) a recurrent neural network (LSTM) to learn a latent representation of the temporal dynamics of the sensors, and (ii) a shallow decoder that learns a mapping between this latent representation and the high-dimensional state space.  The latent space is critical for understanding the symbolic dynamics, which provides an on-the-fly compression for modelling physical and engineering systems. Forecasting in the latent space symbolic representation is also achieved from the sensor time-series data alone, producing an efficient paradigm for predicting temporal evolution with an exceptionally limited number of sensors. In the example cases explored, including turbulent flows, complex spatio-temporal dynamics can be characterized with exceedingly limited sensors that can be randomly placed with minimal loss of performance.

Professor Jose Nathan Kutz, University of Washington, USA

Professor Jose Nathan Kutz, University of Washington, USA

J Nathan Kutz received the BS degree in physics and mathematics from the University of Washington, Seattle, WA, USA, in 1990, and the PhD degree in applied mathematics from Northwestern University, Evanston, IL, USA, in 1994. He is currently the Director of the AI Institute in Dynamics Systems, the Boeing Professor of AI and Data-Driven Engineering, Professor of applied mathematics and electrical and computer engineering, and a Senior Data Science Fellow with the eScience Institute, University of Washington. He works at the interaction of machine learning and the dynamics of complex systems.