Esam Ghaleb

Selected Publications

1. 

   Joint Modelling of Audio-visual Cues Using Attention Mechanism for Emotion Recognition

   Esam Ghaleb, Jan Niehues, and Stylianos Asteriadis

   Multimedia Tools and Applications Jul 2022

   Abs Bib HTML PDF

   Emotions play a crucial role in human-human communications with complex socio-psychological nature. In order to enhance emotion communication in human-computer interaction, this paper studies emotion recognition from audio and visual signals in video clips, utilizing facial expressions and vocal utterances. Thereby, the study aims to exploit temporal information of audio-visual cues and detect their informative time segments. Attention mechanisms are used to exploit the importance of each modality over time. We propose a novel framework that consists of bi-modal time windows spanning short video clips labeled with discrete emotions. The framework employs two networks, with each one being dedicated to one modality. As input to a modality-specific network, we consider a time-dependent signal deriving from the embeddings of the video and audio modalities. We employ the encoder part of the Transformer on the visual embeddings and another one on the audio embeddings. The research in this paper introduces detailed studies and meta-analysis findings, linking the outputs of our proposition to research from psychology. Specifically, it presents a framework to understand underlying principles of emotion recognition as functions of three separate setups in terms of modalities: audio only, video only, and the fusion of audio and video. Experimental results on two datasets show that the proposed framework achieves improved accuracy in emotion recognition, compared to state-of-the-art techniques and baseline methods not using attention mechanisms. The proposed method improves the results over baseline methods by at least 5.4%. Our experiments show that attention mechanisms reduce the gap between the entropies of unimodal predictions, which increases the bimodal predictions’ certainty and, therefore, improves the bimodal recognition rates. Furthermore, evaluations with noisy data in different scenarios are presented during the training and testing processes to check the framework’s consistency and the attention mechanism’s behavior. The results demonstrate that attention mechanisms increase the framework’s robustness when exposed to similar conditions during the training and the testing phases. Finally, we present comprehensive evaluations of emotion recognition as a function of time. The study shows that the middle time segments of a video clip are essential in the case of using audio modality. However, in the case of video modality, the importance of time windows is distributed equally.

   @article{ghaleb2022joint,
     title = {Joint Modelling of Audio-visual Cues Using Attention Mechanism for Emotion Recognition},
     author = {Ghaleb, Esam and Niehues, Jan and Asteriadis, Stylianos},
     journal = {Multimedia Tools and Applications},
     year = {2022},
     publisher = {Springer},
     keywords = {journal},
     month = jul,
     doi = {10.1007/s11042-022-13557-w},
     url = {https://link.springer.com/article/10.1007/s11042-022-13557-w},
   }

2. 

   Skeleton-Based Explainable Bodily Expressed Emotion Recognition Through Graph Convolutional Networks

   Esam Ghaleb, André Mertens, Stylianos Asteriadis, and 1 more author

   In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) Jul 2021

   Abs Bib HTML PDF

   Much of the focus on emotion recognition has gone into the face and voice as expressive channels, whereas bodily expressions of emotions are understudied. Moreover, current studies lack the explainability of computational features of body movements related to emotional expressions. Perceptual research on body parts’ movements shows that features related to the arms’ movements are correlated the most with human perception of emotions. In this paper, our research aims at presenting an explainable approach for bodily expressed emotion recognition. It utilizes the body joints of the human skeleton, representing them as a graph, which is used in Graph Convolutional Networks (GCNs). We improve the modelling of the GCNs by using spatial attention mechanisms based on body parts, i.e. arms, legs and torso. Our study presents a state-of-the-art explainable approach supported by experimental results on two challenging datasets. Evaluations show that the proposed methodology offers accurate performance and explainable decisions. The methodology demonstrates which body part contributes the most in its inference, showing the significance of arm movements in emotion recognition.

   @inproceedings{ghalebskeleton,
     title = {Skeleton-Based Explainable Bodily Expressed Emotion Recognition Through Graph Convolutional Networks},
     author = {Ghaleb, Esam and Mertens, Andr{\'e} and Asteriadis, Stylianos and Weiss, Gerhard},
     booktitle = {2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)},
     year = {2021},
     organization = {IEEE},
   }