Esam Ghaleb

Selected Publications

1. 

   Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation

   Esam Ghaleb, Bulat Khaertdinov, Wim Pouw, and 4 more authors

   In International Conference on Multimodal Interaction 2024

   Abs Bib PDF

   In face-to-face dialogues, the form-meaning relationship of co-speech gestures varies depending on contextual factors such as what the gestures refer to and the individual characteristics of speakers. These factors make co-speech gesture representation learning challenging. How can we learn meaningful gestures representations considering gestures’ variability and relationship with speech? This paper tackles this challenge by employing self-supervised contrastive learning techniques to learn gesture representations from skeletal and speech information. We propose an approach that includes both unimodal and multimodal pre-training to ground gesture representations in co-occurring speech. For training, we utilize a face-to-face dialogue dataset rich with representational iconic gestures. We conduct thorough intrinsic evaluations of the learned representations through comparison with human-annotated pairwise gesture similarity. Moreover, we perform a diagnostic probing analysis to assess the possibility of recovering interpretable gesture features from the learned representations. Our results show a significant positive correlation with human-annotated gesture similarity and reveal that the similarity between the learned representations is consistent with well-motivated patterns related to the dynamics of dialogue interaction. Moreover, our findings demonstrate that several features concerning the form of gestures can be recovered from the latent representations. Overall, this study shows that multimodal contrastive learning is a promising approach for learning gesture representations, which opens the door to using such representations in larger-scale gesture analysis studies.

   @inproceedings{Ghaleb2024le,
     title = {Learning Co-Speech Gesture Representations in Dialogue through Contrastive Learning: An Intrinsic Evaluation},
     author = {Ghaleb, Esam and Khaertdinov, Bulat and Pouw, Wim and Rasenberg, Marlou and Holler, Judith and {\"O}zy{\"u}rek, Asl{\i} and Fern{\'a}ndez, Raquel},
     booktitle = {International Conference on Multimodal Interaction},
     volume = {26},
     number = {26},
     year = {2024},
     isbn = {9798400704628},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3678957.3685707},
     doi = {10.1145/3678957.3685707},
     pages = {274–283},
     numpages = {10},
     keywords = {Gesture analysis, diagnostic probing., face-to-face dialogue, intrinsic evaluation, representation learning},
     location = {San Jose, Costa Rica},
     series = {ICMI '24},
   }

2. 

   Leveraging Speech for Gesture Detection in Multimodal Communication

   Esam Ghaleb, Ilya Burenko, Marlou Rasenberg, and 7 more authors

   arXiv preprint arXiv:2404.14952 2024

   Abs Bib HTML PDF

   Gestures are inherent to human interaction and often complement speech in face-to-face communication, forming a multimodal communication system. An important task in gesture analysis is detecting a gesture’s beginning and end. Research on automatic gesture detection has primarily focused on visual and kinematic information to detect a limited set of isolated or silent gestures with low variability, neglecting the integration of speech and vision signals to detect gestures that co-occur with speech. This work addresses this gap by focusing on co-speech gesture detection, emphasising the synchrony between speech and co-speech hand gestures. We address three main challenges: the variability of gesture forms, the temporal misalignment between gesture and speech onsets, and differences in sampling rate between modalities. Our approach leverages a sliding window technique to handle variability in gestures’ form and duration, using Mel-Spectrograms for acoustic speech signals and spatiotemporal graphs for visual skeletal data. We investigate extended speech time windows and employ separate backbone models for each modality to address the temporal misalignment and sampling rate differences. We utilize Transformer encoders in cross-modal and early fusion techniques to effectively align and integrate speech and skeletal sequences. The study results show that combining visual and speech information significantly enhances gesture detection performance. Our findings indicate that expanding the speech buffer beyond visual time segments improves performance and that multimodal integration using cross-modal and early fusion techniques outperforms baseline methods using unimodal and late fusion methods. Additionally, we find a correlation between the models’ gesture prediction confidence and low-level speech frequency features potentially associated with gestures. Overall, the study provides a better understanding and detection methods for co-speech gestures, facilitating the analysis of multimodal communication

   @article{ghaleb2024leveraging,
     title = {Leveraging Speech for Gesture Detection in Multimodal Communication},
     author = {Ghaleb, Esam and Burenko, Ilya and Rasenberg, Marlou and Pouw, Wim and Toni, Ivan and Uhrig, Peter and Wilson, Anna and Holler, Judith and {\"O}zy{\"u}rek, Asl{\i} and Fern{\'a}ndez, Raquel},
     journal = {arXiv preprint arXiv:2404.14952},
     year = {2024},
     pubstate = {prepublished},
   }

3. 

   Speakers align both their gestures and words not only to establish but also to maintain reference to create shared labels for novel objects in interaction

   Sho Akamine, Esam Ghaleb, Marlou Rasenberg, and 3 more authors

   In Proceedings of the Annual Meeting of the Cognitive Science Society 2024

   Abs Bib

   When we communicate with others, we often repeat aspects of each other’s communicative behavior such as sentence structures and words. Such behavioral alignment has been mostly studied for speech or text. Yet, language use is mostly multimodal, flexibly using speech and gestures to convey messages. Here, we explore the use of alignment in speech (words) and co-speech gestures (iconic gestures) in a referential communication task aimed at finding labels for novel objects in interaction. In particular, we investigate how people flexibly use lexical and gestural alignment to create shared labels for novel objects and whether alignment in speech and gesture are related over time. The present study shows that interlocutors not only establish shared labels multimodally but also keep aligning in words and iconic gestures over the interaction. We also show that the amount of lexical alignment positively correlates with the amount of gestural alignment over time, suggesting a close relationship between alignment in the vocal and manual modalities.

   @inproceedings{Akamine2024sp,
     title = {Speakers align both their gestures and words not only to establish but also to maintain reference to create shared labels for novel objects in interaction},
     author = {Akamine, Sho and Ghaleb, Esam and Rasenberg, Marlou and Fern{\'a}ndez, Raquel and Meyer, Antje and {\"O}zy{\"u}rek, Asl{\i}},
     booktitle = {Proceedings of the Annual Meeting of the Cognitive Science Society},
     volume = {45},
     number = {45},
     year = {2024},
     doi = {10.1145/3678957.3685707}
   }

4. 

   Analysing Cross-Speaker Convergence in Face-to-Face Dialogue through the Lens of Automatically Detected Shared Linguistic Constructions

   Esam Ghaleb, Marlou Rasenberg, Wim Pouw, and 4 more authors

   In Proceedings of the Annual Meeting of the Cognitive Science Society 2024

   Abs Bib PDF

   Conversation requires a substantial amount of coordination between dialogue participants, from managing turn taking to negotiating mutual understanding. Part of this coordination effort surfaces as the reuse of linguistic behaviour across speakers, a process often referred to as \textitalignment. While the presence of linguistic alignment is well documented in the literature, several questions remain open, including the extent to which patterns of reuse across speakers have an impact on the emergence of labelling conventions for novel referents. In this study, we put forward a methodology for automatically detecting shared lemmatised constructions—expressions with a common lexical core used by both speakers within a dialogue—and apply it to a referential communication corpus where participants aim to identify novel objects for which no established labels exist. Our analyses uncover the usage patterns of shared constructions in interaction and reveal that features such as their frequency and the amount of different constructions used for a referent are associated with the degree of object labelling convergence the participants exhibit after social interaction. More generally, the present study shows that automatically detected shared constructions offer a useful level of analysis to investigate the dynamics of reference negotiation in dialogue.

   @inproceedings{ghaleb2024an,
     title = {Analysing Cross-Speaker Convergence in Face-to-Face Dialogue through the Lens of Automatically Detected Shared Linguistic Constructions},
     author = {Ghaleb, Esam and Rasenberg, Marlou and Pouw, Wim and Toni, Ivan and Holler, Judith and {\"O}zy{\"u}rek, Asl{\i} and Fern{\'a}ndez, Raquel},
     booktitle = {Proceedings of the Annual Meeting of the Cognitive Science Society},
     volume = {45},
     number = {45},
     year = {2024},
     url = {https://escholarship.org/uc/item/43h970fc}
   }

5. 

   Co-Speech Gesture Detection through Multi-phase Sequence Labeling (to appear)

   Esam Ghaleb, Ilya Burenko, Marlou Rasenberg, and 6 more authors

   In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024

   Abs Bib HTML PDF

   Gestures are integral components of face-to-face communication. They unfold over time, often following predictable movement phases of preparation, stroke, and retraction. Yet, the prevalent approach to automatic gesture detection treats the problem as binary classification, classifying a segment as either containing a gesture or not, thus failing to capture its inherently sequential and contextual nature. To address this, we introduce a novel framework that reframes the task as a multi-phase sequence labeling problem rather than binary classification. Our model processes sequences of skeletal movements over time windows, uses Transformer encoders to learn contextual embeddings, and leverages Conditional Random Fields to perform sequence labeling. We evaluate our proposal on a large dataset of diverse co-speech gestures in task-oriented face-to-face dialogues. The results consistently demonstrate that our method significantly outperforms strong baseline models in detecting gesture strokes. Furthermore, applying Transformer encoders to learn contextual embeddings from movement sequences substantially improves gesture unit detection. These results highlight our framework’s capacity to capture the fine-grained dynamics of co-speech gesture phases, paving the way for more nuanced and accurate gesture detection and analysis.

   @inproceedings{ghaleb2023co,
     title = {Co-Speech Gesture Detection through Multi-phase Sequence Labeling (to appear)},
     author = {Ghaleb, Esam and Burenko, Ilya and Rasenberg, Marlou and Pouw, Wim and Uhrig, Peter and Holler, Judith and Toni, Ivan and {\"O}zy{\"u}rek, Asl{\i} and Fern{\'a}ndez, Raquel},
     booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
     year = {2024},
     doi = {10.1109/WACV57701.2024.00396},
     pages = { 3995-4003},
   }

6. 

   Joint Modelling of Audio-visual Cues Using Attention Mechanism for Emotion Recognition

   Esam Ghaleb, Jan Niehues, and Stylianos Asteriadis

   Multimedia Tools and Applications 2023

   Abs Bib HTML PDF

   Emotions play a crucial role in human-human communications with complex socio-psychological nature. In order to enhance emotion communication in human-computer interaction, this paper studies emotion recognition from audio and visual signals in video clips, utilizing facial expressions and vocal utterances. Thereby, the study aims to exploit temporal information of audio-visual cues and detect their informative time segments. Attention mechanisms are used to exploit the importance of each modality over time. We propose a novel framework that consists of bi-modal time windows spanning short video clips labeled with discrete emotions. The framework employs two networks, with each one being dedicated to one modality. As input to a modality-specific network, we consider a time-dependent signal deriving from the embeddings of the video and audio modalities. We employ the encoder part of the Transformer on the visual embeddings and another one on the audio embeddings. The research in this paper introduces detailed studies and meta-analysis findings, linking the outputs of our proposition to research from psychology. Specifically, it presents a framework to understand underlying principles of emotion recognition as functions of three separate setups in terms of modalities: audio only, video only, and the fusion of audio and video. Experimental results on two datasets show that the proposed framework achieves improved accuracy in emotion recognition, compared to state-of-the-art techniques and baseline methods not using attention mechanisms. The proposed method improves the results over baseline methods by at least 5.4%. Our experiments show that attention mechanisms reduce the gap between the entropies of unimodal predictions, which increases the bimodal predictions’ certainty and, therefore, improves the bimodal recognition rates. Furthermore, evaluations with noisy data in different scenarios are presented during the training and testing processes to check the framework’s consistency and the attention mechanism’s behavior. The results demonstrate that attention mechanisms increase the framework’s robustness when exposed to similar conditions during the training and the testing phases. Finally, we present comprehensive evaluations of emotion recognition as a function of time. The study shows that the middle time segments of a video clip are essential in the case of using audio modality. However, in the case of video modality, the importance of time windows is distributed equally.

   @article{ghaleb2023joint,
     title = {Joint Modelling of Audio-visual Cues Using Attention Mechanism for Emotion Recognition},
     author = {Ghaleb, Esam and Niehues, Jan and Asteriadis, Stylianos},
     journal = {Multimedia Tools and Applications},
     volume = {82},
     number = {8},
     pages = {11239--11264},
     year = {2023},
     publisher = {Springer},
     doi = {10.1007/s11042-022-13557-w},
     url = {https://link.springer.com/article/10.1007/s11042-022-13557-w},
     keywords = {journal}
   }

7. 

   Skeleton-Based Explainable Bodily Expressed Emotion Recognition Through Graph Convolutional Networks

   Esam Ghaleb, André Mertens, Stylianos Asteriadis, and 1 more author

   In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) 2021

   Abs Bib HTML PDF

   Much of the focus on emotion recognition has gone into the face and voice as expressive channels, whereas bodily expressions of emotions are understudied. Moreover, current studies lack the explainability of computational features of body movements related to emotional expressions. Perceptual research on body parts’ movements shows that features related to the arms’ movements are correlated the most with human perception of emotions. In this paper, our research aims at presenting an explainable approach for bodily expressed emotion recognition. It utilizes the body joints of the human skeleton, representing them as a graph, which is used in Graph Convolutional Networks (GCNs). We improve the modelling of the GCNs by using spatial attention mechanisms based on body parts, i.e. arms, legs and torso. Our study presents a state-of-the-art explainable approach supported by experimental results on two challenging datasets. Evaluations show that the proposed methodology offers accurate performance and explainable decisions. The methodology demonstrates which body part contributes the most in its inference, showing the significance of arm movements in emotion recognition.

   @inproceedings{ghalebskeleton,
     title = {Skeleton-Based Explainable Bodily Expressed Emotion Recognition Through Graph Convolutional Networks},
     author = {Ghaleb, Esam and Mertens, Andr{\'e} and Asteriadis, Stylianos and Weiss, Gerhard},
     booktitle = {2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)},
     year = {2021},
     organization = {IEEE},
   }