Publications | Esam Ghaleb

2022

MTAP
Joint Modelling of Audio-visual Cues Using Attention Mechanism for Emotion Recognition

Esam Ghaleb, Jan Niehues, and Stylianos Asteriadis

Multimedia Tools and Applications Jul 2022

Abs Bib HTML PDF

Emotions play a crucial role in human-human communications with complex socio-psychological nature. In order to enhance emotion communication in human-computer interaction, this paper studies emotion recognition from audio and visual signals in video clips, utilizing facial expressions and vocal utterances. Thereby, the study aims to exploit temporal information of audio-visual cues and detect their informative time segments. Attention mechanisms are used to exploit the importance of each modality over time. We propose a novel framework that consists of bi-modal time windows spanning short video clips labeled with discrete emotions. The framework employs two networks, with each one being dedicated to one modality. As input to a modality-specific network, we consider a time-dependent signal deriving from the embeddings of the video and audio modalities. We employ the encoder part of the Transformer on the visual embeddings and another one on the audio embeddings. The research in this paper introduces detailed studies and meta-analysis findings, linking the outputs of our proposition to research from psychology. Specifically, it presents a framework to understand underlying principles of emotion recognition as functions of three separate setups in terms of modalities: audio only, video only, and the fusion of audio and video. Experimental results on two datasets show that the proposed framework achieves improved accuracy in emotion recognition, compared to state-of-the-art techniques and baseline methods not using attention mechanisms. The proposed method improves the results over baseline methods by at least 5.4%. Our experiments show that attention mechanisms reduce the gap between the entropies of unimodal predictions, which increases the bimodal predictions’ certainty and, therefore, improves the bimodal recognition rates. Furthermore, evaluations with noisy data in different scenarios are presented during the training and testing processes to check the framework’s consistency and the attention mechanism’s behavior. The results demonstrate that attention mechanisms increase the framework’s robustness when exposed to similar conditions during the training and the testing phases. Finally, we present comprehensive evaluations of emotion recognition as a function of time. The study shows that the middle time segments of a video clip are essential in the case of using audio modality. However, in the case of video modality, the importance of time windows is distributed equally.
@article{ghaleb2022joint, title = {Joint Modelling of Audio-visual Cues Using Attention Mechanism for Emotion Recognition}, author = {Ghaleb, Esam and Niehues, Jan and Asteriadis, Stylianos}, journal = {Multimedia Tools and Applications}, year = {2022}, publisher = {Springer}, keywords = {journal}, month = jul, doi = {10.1007/s11042-022-13557-w}, url = {https://link.springer.com/article/10.1007/s11042-022-13557-w}, }

Dynamic Temperature Scaling in Contrastive Self-supervised Learning for Sensor-based Human Activity Recognition

Bulat Khaertdinov, Stylianos Asteriadis, and Esam Ghaleb

IEEE Transactions on Biometrics, Behavior, and Identity Science Jul 2022

@article{khaertdinov2022dynamic,
  title = {Dynamic Temperature Scaling in Contrastive Self-supervised Learning for Sensor-based Human Activity Recognition},
  author = {Khaertdinov, Bulat and Asteriadis, Stylianos and Ghaleb, Esam},
  journal = {IEEE Transactions on Biometrics, Behavior, and Identity Science},
  year = {2022},
  publisher = {IEEE},
  keywords = {journal}
}

2021

FG21
Skeleton-Based Explainable Bodily Expressed Emotion Recognition Through Graph Convolutional Networks

Esam Ghaleb, André Mertens, Stylianos Asteriadis, and 1 more author

In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) Jul 2021

Abs Bib HTML PDF

Much of the focus on emotion recognition has gone into the face and voice as expressive channels, whereas bodily expressions of emotions are understudied. Moreover, current studies lack the explainability of computational features of body movements related to emotional expressions. Perceptual research on body parts’ movements shows that features related to the arms’ movements are correlated the most with human perception of emotions. In this paper, our research aims at presenting an explainable approach for bodily expressed emotion recognition. It utilizes the body joints of the human skeleton, representing them as a graph, which is used in Graph Convolutional Networks (GCNs). We improve the modelling of the GCNs by using spatial attention mechanisms based on body parts, i.e. arms, legs and torso. Our study presents a state-of-the-art explainable approach supported by experimental results on two challenging datasets. Evaluations show that the proposed methodology offers accurate performance and explainable decisions. The methodology demonstrates which body part contributes the most in its inference, showing the significance of arm movements in emotion recognition.
@inproceedings{ghalebskeleton, title = {Skeleton-Based Explainable Bodily Expressed Emotion Recognition Through Graph Convolutional Networks}, author = {Ghaleb, Esam and Mertens, Andr{\'e} and Asteriadis, Stylianos and Weiss, Gerhard}, booktitle = {2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)}, year = {2021}, organization = {IEEE}, }
UM

Bimodal emotion recognition through audio-visual cues

Esam Ghaleb

Jul 2021

HTML PDF

BNAIC21

Explainable and Interpretable Features of Emotion in Human Body Expressions

André Mertens, Esam Ghaleb, and Stylianos Asteriadis

In BNAIC/BeneLearn 2021 Jul 2021

Bib

@inproceedings{maarten2021explainable,
  title = {Explainable and Interpretable Features of Emotion in Human Body Expressions},
  author = {Mertens, André and Ghaleb, Esam and Asteriadis, Stylianos},
  booktitle = {BNAIC/BeneLearn 2021},
  year = {2021}
}

IJCB21

Contrastive Self-supervised Learning for Sensor-based Human Activity Recognition

Bulat Khaertdinov, Esam Ghaleb, and Stylianos Asteriadis

In 2021 IEEE International Joint Conference on Biometrics (IJCB) Jul 2021

Bib

@inproceedings{khaertdinov2021contrastive,
  title = {Contrastive Self-supervised Learning for Sensor-based Human Activity Recognition},
  author = {Khaertdinov, Bulat and Ghaleb, Esam and Asteriadis, Stylianos},
  booktitle = {2021 IEEE International Joint Conference on Biometrics (IJCB)},
  pages = {1--8 (\textbf{second runner up award})},
  year = {2021},
  organization = {IEEE}
}

PerCom

Deep Triplet Networks with Attention for Sensor-based Human Activity Recognition

Bulat Khaertdinov, Esam Ghaleb, and others

In 2021 IEEE International Conference on Pervasive Computing and Communications (PerCom) Mar 2021

Bib

@inproceedings{khaertdinov2021deep,
  title = {Deep Triplet Networks with Attention for Sensor-based Human Activity Recognition},
  author = {Khaertdinov, Bulat and Ghaleb, Esam and others},
  booktitle = {2021 IEEE International Conference on Pervasive Computing and Communications (PerCom)},
  pages = {1--10},
  year = {2021},
  month = mar,
  organization = {IEEE}
}

2020

BNAIC20

Deep, dimensional and multimodal emotion recognition using attention mechanisms

Jan Lucas, Esam Ghaleb, and Stylianos Asteriadis

In BNAIC/BeneLearn 2020 Mar 2020

Bib

@inproceedings{lucas2020deep,
  title = {Deep, dimensional and multimodal emotion recognition using attention mechanisms},
  author = {Lucas, Jan and Ghaleb, Esam and Asteriadis, Stylianos},
  booktitle = {BNAIC/BeneLearn 2020},
  pages = {130},
  year = {2020}
}

ICIP20

Multimodal Attention-Mechanism For Temporal Emotion Recognition

Esam Ghaleb, Jan Niehues, and Stylianos Asteriadis

In 2020 IEEE International Conference on Image Processing (ICIP) Mar 2020

Bib

@inproceedings{ghaleb2020multimodal,
  title = {Multimodal Attention-Mechanism For Temporal Emotion Recognition},
  author = {Ghaleb, Esam and Niehues, Jan and Asteriadis, Stylianos},
  booktitle = {2020 IEEE International Conference on Image Processing (ICIP)},
  pages = {251--255},
  year = {2020},
  organization = {IEEE}
}

FG20

Temporal triplet mining for personality recognition

Dario Dotti, Esam Ghaleb, and Stylianos Asteriadis

In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) Mar 2020

2019

ACII19

Multimodal and temporal perception of audio-visual cues for emotion recognition

Esam Ghaleb, Mirela Popa, and Stylianos Asteriadis

In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII) Mar 2019

Bib

@inproceedings{ghaleb2019multimodal,
  title = {Multimodal and temporal perception of audio-visual cues for emotion recognition},
  author = {Ghaleb, Esam and Popa, Mirela and Asteriadis, Stylianos},
  booktitle = {2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII)},
  pages = {552--558},
  year = {2019},
  organization = {IEEE}
}

IEEE Multimedia

Metric learning-based multimodal audio-visual emotion recognition

Esam Ghaleb, Mirela Popa, and Stylianos Asteriadis

Ieee Multimedia Mar 2019

Bib

@article{ghaleb2019metric,
  title = {Metric learning-based multimodal audio-visual emotion recognition},
  author = {Ghaleb, Esam and Popa, Mirela and Asteriadis, Stylianos},
  journal = {Ieee Multimedia},
  volume = {27},
  number = {1},
  pages = {37--48},
  year = {2019},
  publisher = {IEEE},
  keywords = {journal}
}

Exploiting sensing devices availability in AR/VR deployments to foster engagement

Nicholas Vretos, Petros Daras, Stylianos Asteriadis, and 7 more authors

Virtual Reality Mar 2019

Bib

@article{vretos2019exploiting,
  title = {Exploiting sensing devices availability in AR/VR deployments to foster engagement},
  author = {Vretos, Nicholas and Daras, Petros and Asteriadis, Stylianos and Hortal, Enrique and Ghaleb, Esam and Spyrou, Evaggelos and Leligou, Helen C and Karkazis, Panagiotis and Trakadas, Panagiotis and Assimakopoulos, Kostantinos},
  journal = {Virtual Reality},
  volume = {23},
  number = {4},
  pages = {399--410},
  year = {2019},
  publisher = {Springer},
  keywords = {journal}
}

2018

ICMLA18

Towards Affect Recognition through Interactions with Learning Materials

Esam Ghaleb, Mirela Popa, Enrique Hortal, and 2 more authors

In Machine Learning and Applications (ICMLA), 2018 17th International Conference on Mar 2018

Bib

@inproceedings{ghaleb2018towards,
  title = {Towards Affect Recognition through Interactions with Learning Materials},
  author = {Ghaleb, Esam and Popa, Mirela and Hortal, Enrique and Asteriadis, Stylianos and Weiss, Gerhard},
  booktitle = {Machine Learning and Applications (ICMLA), 2018 17th International Conference on},
  pages = {76--79},
  year = {2018},
  organization = {IEEE}
}

IEEE IS

DEEP REPRESENTATION AND SCORE NORMALIZATION FOR FACE RECOGNITION UNDER MISMATCHED CONDITIONS

Esam Ghaleb, Gokhan Ozbulak, Hua Gao, and 1 more author

Mar 2018

Bib

@article{ghaleb2018deep,
  title = {DEEP REPRESENTATION AND SCORE NORMALIZATION FOR FACE RECOGNITION UNDER MISMATCHED CONDITIONS},
  author = {Ghaleb, Esam and Ozbulak, Gokhan and Gao, Hua and Ekenel, Hazim Kemal},
  issuetitle = {Trends and Controversies, IEEE Intelligent Systems},
  volume = {33},
  issue = {3},
  year = {2018},
  pages = {43-46},
  publisher = {IEEE},
  keywords = {journal}
}

2017

SMAP17

High-performance and lightweight real-time deep face emotion recognition

Justus Schwan, Esam Ghaleb, Enrique Hortal, and 1 more author

In Semantic and Social Media Adaptation and Personalization (SMAP), 2017 12th International Workshop on Mar 2017

Bib

@inproceedings{schwan2017high,
  title = {High-performance and lightweight real-time deep face emotion recognition},
  author = {Schwan, Justus and Ghaleb, Esam and Hortal, Enrique and Asteriadis, Stylianos},
  booktitle = {Semantic and Social Media Adaptation and Personalization (SMAP), 2017 12th International Workshop on},
  pages = {76--79},
  year = {2017},
  organization = {IEEE}
}

IntelliSys17

Multimodal fusion based on information gain for emotion recognition in the wild

Esam Ghaleb, Mirela Popa, Enrique Hortal, and 1 more author

In Intelligent Systems Conference (IntelliSys), 2017 Mar 2017

Bib

@inproceedings{ghaleb2017multimodal,
  title = {Multimodal fusion based on information gain for emotion recognition in the wild},
  author = {Ghaleb, Esam and Popa, Mirela and Hortal, Enrique and Asteriadis, Stylianos},
  booktitle = {Intelligent Systems Conference (IntelliSys), 2017},
  pages = {814--823},
  year = {2017},
  organization = {IEEE}
}

2015

ICMR15

Accio: A data set for face track retrieval in movies across age

Esam Ghaleb, Makarand Tapaswi, Ziad Al-Halah, and 2 more authors

In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval Mar 2015

Bib

@inproceedings{ghaleb2015accio,
  title = {Accio: A data set for face track retrieval in movies across age},
  author = {Ghaleb, Esam and Tapaswi, Makarand and Al-Halah, Ziad and Ekenel, Hazim Kemal and Stiefelhagen, Rainer},
  booktitle = {Proceedings of the 5th ACM on International Conference on Multimedia Retrieval},
  pages = {455--458},
  year = {2015},
  organization = {ACM}
}

2014

IFIP14

A Face Recognition Based Multiplayer Mobile Game Application

Ugur Demir, Esam Ghaleb, and Hazım Kemal Ekenel

In IFIP International Conference on Artificial Intelligence Applications and Innovations Mar 2014

Bib

@inproceedings{demir2014face,
  title = {A Face Recognition Based Multiplayer Mobile Game Application},
  author = {Demir, Ugur and Ghaleb, Esam and Ekenel, Haz{\i}m Kemal},
  booktitle = {IFIP International Conference on Artificial Intelligence Applications and Innovations},
  pages = {214--223},
  year = {2014},
  organization = {Springer}
}

SIU14

An energy efficient routing technique and implementation in WSNs

Abdullah Aydeger, Esam Ghaleb, and Sema Oktug

In Signal Processing and Communications Applications Conference (SIU), 2014 22nd Mar 2014

Bib

@inproceedings{aydeger2014energy,
  title = {An energy efficient routing technique and implementation in WSNs},
  author = {Aydeger, Abdullah and Ghaleb, Esam and Oktug, Sema},
  booktitle = {Signal Processing and Communications Applications Conference (SIU), 2014 22nd},
  pages = {1359--1362},
  year = {2014},
  organization = {IEEE}
}