Understanding images in biological and computer vision

Many insects have excellent navigational skills, covering distances, conditions and terrains that are still a challenge for robotics. The primary sense they use is vision, both to obtain self-motion information for path integration, and to establish visual memories of their surroundings to guide homing and route following. Insect vision is relatively low resolution, but exploits a combination of sensory tuning and behavioural strategies to solve complex problems. For example, by filtering for ultraviolet light in an omnidirectional view, segmentation of the shape of the horizon between sky and ground becomes both simple and highly consistent. These shapes can then be used for recognition of location, even under different weather conditions or variations in pitch and tilt. Insect brains are also relatively small and low powered, yet able to produce efficient and effective solutions to complex problems. Through computational modelling, it is becoming possible to link the insights from field experiments to neural data, and thus test hypotheses regarding these brain processing algorithms. For example, we have shown the circuitry of the insect mushroom body neuropil has been shown to be sufficient to support memory of hundreds of images, allowing rapid assessment of familiarity which can be used to guide the animal along previously experienced routes.

In the production of visually guided, goal-directed behaviour two types of action may be distinguished. Sampling actions involve shifting gaze in order to acquire information from the visual environment. Manipulative actions involve using effectors in order to alter the state of the environment. Both types of actions are coordinated in a cycle in which noisy information from the environment is used to infer the state of that environment. Given this estimated state, an appropriate course of action has to be selected. Decision theoretic models have been used to account for the selection between possible competing actions within each class. A common feature of these models is that noisy sensory evidence in favour of different action choices is accumulated over time until a decision threshold is reached. The utility of these models will be evaluated for both types of actions. Specifically, the following open questions will be addressed. Is saccade target selection controlled by a "race to threshold" between competing motor programmes in the presence of a foveal load? To what extent is the temporal trigger signal controlled by the foveal processing demands and the selection of the next target? Do the dynamics of real world manipulative actions (reaching and grasping) reflect the underlying decision process? What is the linking function between the temporally evolving decision variable and the different components of manipulative actions?

Investigation of vision in the context of ongoing behaviour has contributed a number of insights by highlighting the importance of behavioural goals, and focusing attention on how vision and action play out in time. In this context, humans make continuous sequences of sensory-motor decisions to satisfy current goals, and the role of vision is to provide the relevant information for making good decisions in order to achieve those goals. Professor Hayhoe will review the factors that control gaze in natural behaviour, including evidence for the role of the task, which defines the immediate goals, the rewards and costs associated with those goals, uncertainty about the state of the world, and prior knowledge. Visual computations are often highly task-specific, and evaluation of task relevant state is a central factor necessary for optimal action choices. This governs a very large proportion of gaze changes, which reveal the information sampling strategies of the human visual system. When reliable information is present in memory, the need for sensory updates is reduced, and humans can rely instead on memory estimates, depending on their precision, and combine sensory and memory data according to Bayesian principles. It is suggested that visual memory representations are critically important not only for choosing, initiating and guiding actions, but also for predicting their consequences, and separating the visual effects of self-generated movements from external changes.

Robotics captures the attention like few other fields of research, however to move beyond controlled laboratory settings dynamic robots need flexible, redundant and trustworthy sensing which pairs with their control systems. In this talk Dr Fallon will discuss the state of research in perception for dynamic robots: specifically robots which move quickly and rely on visual understanding to make their way in the world namely walking and flying robots. Dr Fallon will outline the development of state estimation, mapping and navigation algorithms for humanoid and quadruped robots and describe how they have been demonstrated in real fielded systems. The talk will also overview current limitations, both computational and sensory, and describe some prototype sensing systems which are biologically inspired. The first topic will overview the adaption of accurate registration methods to the Boston Dynamics Atlas and NASA Valkyrie robots where explored the challenge of localization over long baselines and with low sensory overlap. Secondly Dr Fallon will explore how these methods can be fuse with Vision for a dynamic quadruped trotting and crawling in challenging lighting conditions. Lastly, he will present ongoing research in probabilistically fusing proprioceptive state estimation with dense visual mapping to allow a humanoid robot to build a rich dense map while overcoming dynamics, moving objects and challenging lighting conditions.

Without light, there would be no colour. But, in human vision, colour is not simply determined by light. Rather, colour is the result of complex interactions between light, surfaces, eyes and brains. Embedded in these are the neural mechanisms of colour constancy, which stabilise object colours under changes in the illumination spectrum. Yet, colour constancy is not perfect, and predicting the colour appearance of a particular object for a particular individual, when viewed under a particular illumination, is not simple. Yellow bananas may remain ever yellow even under fluorescent lamps, but blue dresses may turn white under ambiguous lights. Various factors affect colour appearance: the shape of the object, whether it is 2D or 3D, of a recognisable form or not; its other surface properties, whether it is glossy or matte, textured or uniform; and individual variations in visual processing. The shape of the illumination spectrum also affects colour appearance: metamerism in contemporary lighting provides a new challenge to colour constancy. How to measure colour – a subjective experience – most reliably is another challenge for vision scientists. This talk will describe psychophysical experiments and theory addressing these considerations in our understanding of object colour perception by humans.

Shape inferences from images, or line drawings, are classical ill-posed inverse problems. Computational researchers mainly seek 'priors' for regularisation, e.g. regarding the light source, or scene restrictions for training neural networks, such as indoor rooms. While of technical interest, such solutions differ in two fundamental ways from human perception: (i) our inferences are largely robust across lighting and scene variations; and (ii) individuals only perceive qualitatively, not quantitatively, the same shape from a given image. Importantly, we know from psychophysics that similarities across individuals concentrate near certain configurations, such as ridges and boundaries, and it is these configurations that are often represented in line drawings. Professor Zucker will introduce a method for inferring qualitative 3D shape from shading that is consistent with these observations. For a given shape, certain shading patches become equivalent to “line drawings” in a well-defined shading-to-contour limit. Under this limit, and invariantly, the contours partition the surface into meaningful parts using the Morse-Smale complex. Critical contours are the (perceptually) stable parts of this complex and are invariant over a wide class of rendering models. The result provides a topological organisation of the surface into 'bumps' and 'dents' from the underlying shading geometry, and provides an invariant linking image gradient flows to surface organisation.

In computer vision, illumination is considered to be a problem that needs to be ‘solved’. The colour bias due to illumination is removed to support colour-based image recognition, stable tracking (in and out of shadows) amongst other tasks. In this talk Professor Finlayson will review historical and current algorithms for illumination estimation. In the classical approach, the illuminant colour is estimated by an - ever more sophisticated - analysis of simple image summary statistics. More recently, the full power - and much higher complexity - of deep learning has been deployed (where, effectively, the definition of the image statistics of interest are found as part of the overall optimisation). Professor Finlayson will challenge the orthodoxy of deep learning i.e. that it is the obvious solution to illuminant estimation. Instead he will propose that the estimates made by simple algorithms are biased and this bias can be corrected to deliver leading performance. The key new observation in our method - bias correction has been tried before with limited success - is that the bias must be corrected in an exposure invariant way. 

A key goal of biological image understanding is to extract useful perceptual representations of the external world from the images that are formed on the retina. This is challenging for a variety of reasons, one of which is that multiple configurations of the external world can produce the same retinal image, making the image-understanding problem fundamentally under constrained. None-the-less, biological systems are able to combine constraints provided by the retinal image with those provided by statistical regularities in the natural environment to produce representations that are well correlated with physical properties of the external world. One example of this is provided by our ability to perceive object colour and material properties. These percepts are correlated with object spectral surface and geometric surface reflectance, respectively. But the retinal image formed of an object depends not only on object surface reflectance, but also on the spatial and geometric properties of the illumination. This dependence in turn leading to ambiguity that perceptual processing must resolve. To understand how such processing works, it is necessary to measure how perception is used to identify object colour and material, and how such identification depends on object surface reflectance (both spatial and spectral) as well as on object-extrinsic factors such as the illumination.  Classically, such measurements have been made using indirect techniques, such as matching by adjustment or naming. This talk will introduce a novel measurement method that uses object selection directly, together with a model of underlying perceptual representation, to study the stability of object colour across changes in illumination, as well as how object colour and material trade off in identification.