Understanding images in biological and computer vision

Many insects have excellent navigational skills, covering distances, conditions and terrains that are still a challenge for robotics. The primary sense they use is vision, both to obtain self-motion information for path integration, and to establish visual memories of their surroundings to guide homing and route following. Insect vision is relatively low resolution, but exploits a combination of sensory tuning and behavioural strategies to solve complex problems. For example, by filtering for ultraviolet light in an omnidirectional view, segmentation of the shape of the horizon between sky and ground becomes both simple and highly consistent. These shapes can then be used for recognition of location, even under different weather conditions or variations in pitch and tilt. Insect brains are also relatively small and low powered, yet able to produce efficient and effective solutions to complex problems. Through computational modelling, it is becoming possible to link the insights from field experiments to neural data, and thus test hypotheses regarding these brain processing algorithms. For example, we have shown the circuitry of the insect mushroom body neuropil has been shown to be sufficient to support memory of hundreds of images, allowing rapid assessment of familiarity which can be used to guide the animal along previously experienced routes.

Professor Barbara Webb, University of Edinburgh, UK

Professor Barbara Webb, University of Edinburgh, UK

Barbara Webb grew up in Sydney, Australia, and studied Psychology at the University of Sydney, obtaining a 1st class BSc Honours degree and university medal in 1988. She moved to Edinburgh to study Artificial Intelligence for her PhD, graduating in 1993; her thesis work to build a robotic cricket was featured in Scientific American. She is a leading researcher in the field of Biorobotics, using robots to model biological systems. She held lectureships in Psychology at the University of Edinburgh (1993-1995), the University of Nottingham (1995-1998) and the University of Stirling (1999-2003), before joining the School of Informatics at the University of Edinburgh as a Reader in 2003. She became a full Professor of Biorobotics in 2010.

In the production of visually guided, goal-directed behaviour two types of action may be distinguished. Sampling actions involve shifting gaze in order to acquire information from the visual environment. Manipulative actions involve using effectors in order to alter the state of the environment. Both types of actions are coordinated in a cycle in which noisy information from the environment is used to infer the state of that environment. Given this estimated state, an appropriate course of action has to be selected. Decision theoretic models have been used to account for the selection between possible competing actions within each class. A common feature of these models is that noisy sensory evidence in favour of different action choices is accumulated over time until a decision threshold is reached. The utility of these models will be evaluated for both types of actions. Specifically, the following open questions will be addressed. Is saccade target selection controlled by a "race to threshold" between competing motor programmes in the presence of a foveal load? To what extent is the temporal trigger signal controlled by the foveal processing demands and the selection of the next target? Do the dynamics of real world manipulative actions (reaching and grasping) reflect the underlying decision process? What is the linking function between the temporally evolving decision variable and the different components of manipulative actions?

Dr Casimir Ludwig, University of Bristol, UK

Dr Casimir Ludwig, University of Bristol, UK

Casimir was trained as a neuropsychologist at the Radboud University Nijmegen (The Netherlands). He moved to the UK in 1998 and spent almost a year working at the MRC – Cognition and Brain Sciences Unit, before moving on to a PhD at the University of Bristol. After a succession of post-doctoral positions and a 5-year EPSRC Advanced Research Fellowship, he was appointed Reader in Experimental Psychology in August 2012. Casimir's research is focused on the way gaze is used to extract information from the environment, and how the sampled information is then used to infer the state of the environment and guide action. He uses a variety of approaches, including eye tracking, visual psychophysics, motion capture, and computational modelling.

Investigation of vision in the context of ongoing behaviour has contributed a number of insights by highlighting the importance of behavioural goals, and focusing attention on how vision and action play out in time. In this context, humans make continuous sequences of sensory-motor decisions to satisfy current goals, and the role of vision is to provide the relevant information for making good decisions in order to achieve those goals. Professor Hayhoe will review the factors that control gaze in natural behaviour, including evidence for the role of the task, which defines the immediate goals, the rewards and costs associated with those goals, uncertainty about the state of the world, and prior knowledge. Visual computations are often highly task-specific, and evaluation of task relevant state is a central factor necessary for optimal action choices. This governs a very large proportion of gaze changes, which reveal the information sampling strategies of the human visual system. When reliable information is present in memory, the need for sensory updates is reduced, and humans can rely instead on memory estimates, depending on their precision, and combine sensory and memory data according to Bayesian principles. It is suggested that visual memory representations are critically important not only for choosing, initiating and guiding actions, but also for predicting their consequences, and separating the visual effects of self-generated movements from external changes.

Professor Mary Hayhoe, University of Texas, USA

Professor Mary Hayhoe, University of Texas, USA

Mary Hayhoe received her PhD from UC San Diego and has served on the faculty at the University of Rochester and University of Texas at Austin, where she is currently a member of the Center for Perceptual Systems. She served on the Board of Directors of the Vision Sciences Society and was President in 2014-2015. She is on the editorial board of the Journal of Vision and was awarded the Davida Teller Award in 2017. She has pioneered the study of behaviour in in both real and virtual environments and her work addresses how visual perception and action work together to generate behaviour. She has made contributions to understanding human eye movements in natural environments, especially how gaze behavior relates to attention, working memory, and cognitive goals.

Robotics captures the attention like few other fields of research, however to move beyond controlled laboratory settings dynamic robots need flexible, redundant and trustworthy sensing which pairs with their control systems. In this talk Dr Fallon will discuss the state of research in perception for dynamic robots: specifically robots which move quickly and rely on visual understanding to make their way in the world namely walking and flying robots. Dr Fallon will outline the development of state estimation, mapping and navigation algorithms for humanoid and quadruped robots and describe how they have been demonstrated in real fielded systems. The talk will also overview current limitations, both computational and sensory, and describe some prototype sensing systems which are biologically inspired. The first topic will overview the adaption of accurate registration methods to the Boston Dynamics Atlas and NASA Valkyrie robots where explored the challenge of localization over long baselines and with low sensory overlap. Secondly Dr Fallon will explore how these methods can be fuse with Vision for a dynamic quadruped trotting and crawling in challenging lighting conditions. Lastly, he will present ongoing research in probabilistically fusing proprioceptive state estimation with dense visual mapping to allow a humanoid robot to build a rich dense map while overcoming dynamics, moving objects and challenging lighting conditions.

Dr Maurice Fallon, University of Oxford, UK

Dr Maurice Fallon, University of Oxford, UK

Maurice Fallon is a Royal Society University Research Fellow at the Oxford Robotics Institute (University of Oxford). His research is focused on probabilistic methods for localization and mapping. He has also make made research contributions to state estimation for legged robots and is also interested in high dimensional motion planning and manipulation for dynamic robots. He received his PhD from the University of Cambridge and went on to a post doc position in MIT where he work on the problem of Simultaneous Localization and Mapping or SLAM. From 2013-2015, he was the perception lead of MIT's DARPA Robotics Challenge team - a multi-year competition developing technologies for semi-autonomous humanoid exploration and manipulation in disaster situations. After a period in the School of Informatics in the University of Edinburgh, he joined Oxford in 2017.

Without light, there would be no colour. But, in human vision, colour is not simply determined by light. Rather, colour is the result of complex interactions between light, surfaces, eyes and brains. Embedded in these are the neural mechanisms of colour constancy, which stabilise object colours under changes in the illumination spectrum. Yet, colour constancy is not perfect, and predicting the colour appearance of a particular object for a particular individual, when viewed under a particular illumination, is not simple. Yellow bananas may remain ever yellow even under fluorescent lamps, but blue dresses may turn white under ambiguous lights. Various factors affect colour appearance: the shape of the object, whether it is 2D or 3D, of a recognisable form or not; its other surface properties, whether it is glossy or matte, textured or uniform; and individual variations in visual processing. The shape of the illumination spectrum also affects colour appearance: metamerism in contemporary lighting provides a new challenge to colour constancy. How to measure colour – a subjective experience – most reliably is another challenge for vision scientists. This talk will describe psychophysical experiments and theory addressing these considerations in our understanding of object colour perception by humans.

Professor Anya Hurlbert, Newcastle University, UK

Professor Anya Hurlbert, Newcastle University, UK

Anya Hurlbert is Professor of Visual Neuroscience, Director of the Centre for Translational Systems Neuroscience and Dean of Advancement at Newcastle University. She trained as a physicist (BA, Princeton University), physiologist (MA, Cambridge University), neuroscientist (PhD, Brain and Cognitive Sciences, MIT), and physician (MD, Harvard Medical School). Her main research interest is in understanding the human brain, through understanding the human visual system, and in particular, colour perception; she is also interested in applied areas such as digital imaging and novel lighting technologies. She co-founded the Institute of Neuroscience at Newcastle, and was Institute Director for 10 years. Professor Hurlbert is active in the public understanding of science, and has developed several science-based art exhibitions, most recently an interactive installation at the National Gallery, London, for its 2014 summer exhibition "Making Colour". She is past Chairman of the Colour Group (GB), and currently Scientist Trustee of the National Gallery.

Shape inferences from images, or line drawings, are classical ill-posed inverse problems. Computational researchers mainly seek 'priors' for regularisation, e.g. regarding the light source, or scene restrictions for training neural networks, such as indoor rooms. While of technical interest, such solutions differ in two fundamental ways from human perception: (i) our inferences are largely robust across lighting and scene variations; and (ii) individuals only perceive qualitatively, not quantitatively, the same shape from a given image. Importantly, we know from psychophysics that similarities across individuals concentrate near certain configurations, such as ridges and boundaries, and it is these configurations that are often represented in line drawings. Professor Zucker will introduce a method for inferring qualitative 3D shape from shading that is consistent with these observations. For a given shape, certain shading patches become equivalent to “line drawings” in a well-defined shading-to-contour limit. Under this limit, and invariantly, the contours partition the surface into meaningful parts using the Morse-Smale complex. Critical contours are the (perceptually) stable parts of this complex and are invariant over a wide class of rendering models. The result provides a topological organisation of the surface into 'bumps' and 'dents' from the underlying shading geometry, and provides an invariant linking image gradient flows to surface organisation.

Professor Steven Zucker, Yale University, USA

Professor Steven Zucker, Yale University, USA

Steven W Zucker works in computational vision, computational neuroscience, and computational biology. He is the David and Lucile Packard Professor of Computer Science at Yale University, and also Professor of Biomedical Engineering. He is a member of the Program in Applied Mathematics, which he directed from 2003 to 2009, and a member of the Interdepartmental Neuroscience Program. Currently he co-directs the Swartz Center, which fosters collaborative research and interdisciplinary training in computational and systems neuroscience. Zucker was elected a Fellow of the Royal Society of Canada, the Canadian Institute for Advanced Research, a Fellow of the IEEE, and (by)Fellow of Churchill College, Cambridge. He won the Siemens Award and was recently named a Distinguished Investigator by the Paul G Allen Family Foundation. 

In computer vision, illumination is considered to be a problem that needs to be ‘solved’. The colour bias due to illumination is removed to support colour-based image recognition, stable tracking (in and out of shadows) amongst other tasks. In this talk Professor Finlayson will review historical and current algorithms for illumination estimation. In the classical approach, the illuminant colour is estimated by an - ever more sophisticated - analysis of simple image summary statistics. More recently, the full power - and much higher complexity - of deep learning has been deployed (where, effectively, the definition of the image statistics of interest are found as part of the overall optimisation). Professor Finlayson will challenge the orthodoxy of deep learning i.e. that it is the obvious solution to illuminant estimation. Instead he will propose that the estimates made by simple algorithms are biased and this bias can be corrected to deliver leading performance. The key new observation in our method - bias correction has been tried before with limited success - is that the bias must be corrected in an exposure invariant way. 

Professor Graham Finlayson, University of East Anglia, UK

Professor Graham Finlayson, University of East Anglia, UK

Graham Finlayson is a professor of computer science at the University of East Anglia where he leads the Colour and Imaging Lab. Graham's research spans colour image processing, physics-based computer vision and visual perception. He is interested in taking the creative spark of an idea, developing the underlying theory and algorithms and then implementing and commercialising the technology. Graham's IP ships in 10s of millions of products.

A key goal of biological image understanding is to extract useful perceptual representations of the external world from the images that are formed on the retina. This is challenging for a variety of reasons, one of which is that multiple configurations of the external world can produce the same retinal image, making the image-understanding problem fundamentally under constrained. None-the-less, biological systems are able to combine constraints provided by the retinal image with those provided by statistical regularities in the natural environment to produce representations that are well correlated with physical properties of the external world. One example of this is provided by our ability to perceive object colour and material properties. These percepts are correlated with object spectral surface and geometric surface reflectance, respectively. But the retinal image formed of an object depends not only on object surface reflectance, but also on the spatial and geometric properties of the illumination. This dependence in turn leading to ambiguity that perceptual processing must resolve. To understand how such processing works, it is necessary to measure how perception is used to identify object colour and material, and how such identification depends on object surface reflectance (both spatial and spectral) as well as on object-extrinsic factors such as the illumination.  Classically, such measurements have been made using indirect techniques, such as matching by adjustment or naming. This talk will introduce a novel measurement method that uses object selection directly, together with a model of underlying perceptual representation, to study the stability of object colour across changes in illumination, as well as how object colour and material trade off in identification.

Professor David H Brainard, University of Pennsylvania, USA

Professor David H Brainard, University of Pennsylvania, USA

David H Brainard is the RRL Professor of Psychology at the University of Pennsylvania. He received an AB in Physics (Magna Cum Laude) from Harvard University (1982) and an MS (Electrical Engineering) and PhD (Psychology) from Stanford University in 1989. He is currently the RRL Professor of Psychology at the University of Pennsylvania and his research focuses on colour vision and colour image processing. He is a fellow of the Optical Society, ARVO and the Association for Psychological Science. At present, he directs Penn's Vision Research Center, co-directs Penn's Computational Neuroscience Initiative, is on the Board of the Vision Sciences Society, and is a member of the editorial board of the Journal of Vision.