Esam Ghaleb

Selected Publications

1. 

   Co-Speech Gesture Detection through Multi-phase Sequence Labeling (to appear)

   Esam Ghaleb, Ilya Burenko, Marlou Rasenberg, and 6 more authors

   In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024

   Abs Bib HTML PDF

   Gestures are integral components of face-to-face communication. They unfold over time, often following predictable movement phases of preparation, stroke, and retraction. Yet, the prevalent approach to automatic gesture detection treats the problem as binary classification, classifying a segment as either containing a gesture or not, thus failing to capture its inherently sequential and contextual nature. To address this, we introduce a novel framework that reframes the task as a multi-phase sequence labeling problem rather than binary classification. Our model processes sequences of skeletal movements over time windows, uses Transformer encoders to learn contextual embeddings, and leverages Conditional Random Fields to perform sequence labeling. We evaluate our proposal on a large dataset of diverse co-speech gestures in task-oriented face-to-face dialogues. The results consistently demonstrate that our method significantly outperforms strong baseline models in detecting gesture strokes. Furthermore, applying Transformer encoders to learn contextual embeddings from movement sequences substantially improves gesture unit detection. These results highlight our framework’s capacity to capture the fine-grained dynamics of co-speech gesture phases, paving the way for more nuanced and accurate gesture detection and analysis.

   @inproceedings{ghaleb2023co,
     title = {Co-Speech Gesture Detection through Multi-phase Sequence Labeling (to appear)},
     author = {Ghaleb, Esam and Burenko, Ilya and Rasenberg, Marlou and Pouw, Wim and Uhrig, Peter and Holler, Judith and Toni, Ivan and {\"O}zy{\"u}rek, Asl{\i} and Fern{\'a}ndez, Raquel},
     booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
     year = {2024},
     pubstate = {prepublished}
   }

2. 

   Joint Modelling of Audio-visual Cues Using Attention Mechanism for Emotion Recognition

   Esam Ghaleb, Jan Niehues, and Stylianos Asteriadis

   Multimedia Tools and Applications 2023

   Abs Bib HTML PDF

   Emotions play a crucial role in human-human communications with complex socio-psychological nature. In order to enhance emotion communication in human-computer interaction, this paper studies emotion recognition from audio and visual signals in video clips, utilizing facial expressions and vocal utterances. Thereby, the study aims to exploit temporal information of audio-visual cues and detect their informative time segments. Attention mechanisms are used to exploit the importance of each modality over time. We propose a novel framework that consists of bi-modal time windows spanning short video clips labeled with discrete emotions. The framework employs two networks, with each one being dedicated to one modality. As input to a modality-specific network, we consider a time-dependent signal deriving from the embeddings of the video and audio modalities. We employ the encoder part of the Transformer on the visual embeddings and another one on the audio embeddings. The research in this paper introduces detailed studies and meta-analysis findings, linking the outputs of our proposition to research from psychology. Specifically, it presents a framework to understand underlying principles of emotion recognition as functions of three separate setups in terms of modalities: audio only, video only, and the fusion of audio and video. Experimental results on two datasets show that the proposed framework achieves improved accuracy in emotion recognition, compared to state-of-the-art techniques and baseline methods not using attention mechanisms. The proposed method improves the results over baseline methods by at least 5.4%. Our experiments show that attention mechanisms reduce the gap between the entropies of unimodal predictions, which increases the bimodal predictions’ certainty and, therefore, improves the bimodal recognition rates. Furthermore, evaluations with noisy data in different scenarios are presented during the training and testing processes to check the framework’s consistency and the attention mechanism’s behavior. The results demonstrate that attention mechanisms increase the framework’s robustness when exposed to similar conditions during the training and the testing phases. Finally, we present comprehensive evaluations of emotion recognition as a function of time. The study shows that the middle time segments of a video clip are essential in the case of using audio modality. However, in the case of video modality, the importance of time windows is distributed equally.

   @article{ghaleb2023joint,
     title = {Joint Modelling of Audio-visual Cues Using Attention Mechanism for Emotion Recognition},
     author = {Ghaleb, Esam and Niehues, Jan and Asteriadis, Stylianos},
     journal = {Multimedia Tools and Applications},
     volume = {82},
     number = {8},
     pages = {11239--11264},
     year = {2023},
     publisher = {Springer},
     doi = {10.1007/s11042-022-13557-w},
     url = {https://link.springer.com/article/10.1007/s11042-022-13557-w},
     keywords = {journal}
   }

3. 

   Skeleton-Based Explainable Bodily Expressed Emotion Recognition Through Graph Convolutional Networks

   Esam Ghaleb, André Mertens, Stylianos Asteriadis, and 1 more author

   In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) 2021

   Abs Bib HTML PDF

   Much of the focus on emotion recognition has gone into the face and voice as expressive channels, whereas bodily expressions of emotions are understudied. Moreover, current studies lack the explainability of computational features of body movements related to emotional expressions. Perceptual research on body parts’ movements shows that features related to the arms’ movements are correlated the most with human perception of emotions. In this paper, our research aims at presenting an explainable approach for bodily expressed emotion recognition. It utilizes the body joints of the human skeleton, representing them as a graph, which is used in Graph Convolutional Networks (GCNs). We improve the modelling of the GCNs by using spatial attention mechanisms based on body parts, i.e. arms, legs and torso. Our study presents a state-of-the-art explainable approach supported by experimental results on two challenging datasets. Evaluations show that the proposed methodology offers accurate performance and explainable decisions. The methodology demonstrates which body part contributes the most in its inference, showing the significance of arm movements in emotion recognition.

   @inproceedings{ghalebskeleton,
     title = {Skeleton-Based Explainable Bodily Expressed Emotion Recognition Through Graph Convolutional Networks},
     author = {Ghaleb, Esam and Mertens, Andr{\'e} and Asteriadis, Stylianos and Weiss, Gerhard},
     booktitle = {2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)},
     year = {2021},
     organization = {IEEE},
   }