Saliency and Attention: rePresentation, Interpretation and EmergeNce

Attention is a complex cognitive function essential for explaining human behaviour that allows us to select the most relevant events or items in our environment in order to focus our sensory and cognitive resources on them. It can be modulated either by bottom-up sensory-driven factors, or top-down task-specific goals. In the former case it is also referred to as salience or saliency.

Understanding saliency and attention is a highly challenging scientific endeavour and creating artificial machines capable of imitating them, a remarkable technological step forward. Despite the substantial advances in the field, including our own under our previous MinEco funded project SAMURAI (Saliency and Attention: MUltimodality, context-awaReness, self-Adaptation and bio-Inspiration) and others, the challenge is still far from being overcome.

In the meantime, two prominent technologies spanning this and several other disciplines have caused a profound impact on this research agenda: the deep learning paradigm in machine learning, and the maturity of sensor devices. On both these lines our research group has accumulated significant expertise. In this light we have identified the following key directions for advancing this technology:

1. Representation. How to measure and describe attention at various levels of detail is a hard problem due to the limited ability of the measuring devices to capture the phenomenon and inter-subject variability. From the data modeling perspective, this problem is two-fold: first, there is an underspecification of the target labels and second, a lack of appropriateness in input features, specially to model the dynamics and multimodality of phenomena. In this line, our proposal with SAPIENS is to explore unsupervised, weakly supervised and multilabeling methods with an emphasis on the representation of multimodal streams to take advantage of cross-modal synergies. This is fully aligned with the open lines of research in the machine learning community and therefore current methods are likely to evolve in this direction.

2. Interpretation. With the advent and spread of deep learning techniques, a growing concern is their lack of interpretability, in essense, their being conceived as black-boxes. This undermines their applicability, on the one hand, as a tool for scientific understanding of phenomena and, on the other, as an aid for experts in which they are likely to be used as modules inside a more complex system, possibly including human interaction in the loop. SAPIENS will adopt the Exploratory Data Analysis framework, explore information-theoretical evaluation methods for their characterization and search for bioinspired mathematical models behind current state-of-the-art machine learning methods.

3. Emergence. The emergence of order and organization in systems composed of many autonomous entities is a very basic process but very difficult to model or explain. In the phenomenon of attention we observe how a complex task-driven response arises from low level sensory inputs. In SAMURAI we demonstrated the capability to acquire knowledge through the discovery of latent classes or topics in top-down visual systems but not with aural or multimodal data streams. Within this wider framework SAPIENS aims at providing a more generally applicable solution making use of the wealth of scientific results available on this matter.

SAPIENS considers several real applications to test the theoretical advances in these three directions.


(Open access in our Institutional Repository:

[bibtex file= key=FernandezMartinez2018,rod:pel:gal:18b,val:pel:18c,tor:gon:dia:19,Molina-Moreno2018,pla:gon:mar:dia:19,val:pel:20,val:pel:19]


[bibtex file= key=Gonzalez2018,val:pel:cab:cor:oje:18b,gal:mon:19,val:pel:cor:oje:19]


[bibtex file= key=tesis6,tesis7]

Fechas de ejecución: 01/01/2018-31/12/2020

Financiado por: FEDER/Ministerio de Ciencia, Innovación y Universidades – Agencia Estatal de Investigación/TEC2017-84395-P

Comments are closed.