- Title
- Multimodal emotion recognition based on speech and physiological signals using deep neural networks
- Creator
- Bakhshi, Ali; Chalup, Stephan
- Relation
- International Conference on Pattern Recognition (ICPR). Proceedings of 25th International Conference on Pattern Recognition Workshops, ICPR 2020 (Virtual 10-11 January, 2021) p. 289-300
- Publisher Link
- http://dx.doi.org/10.1007/978-3-030-68780-9_25
- Publisher
- Springer
- Resource Type
- conference paper
- Date
- 2021
- Description
- A suitable combination of data in a multimodal emotion recognition model allows conveying and combining each channel's information to achieve a better recognition of the encoded emotion than would be possible using only a single modality and channel. In this paper, we focus on combining speech and physiological signals to predict the arousal and valence levels of the emotional states of a person. We designed a neural network that can use the information from raw audio signals, electrocardiograms, heart rate variability, electro-dermal activity, and skin conductance levels, to predict emotional states. The proposed deep neural network architecture works as an end-to-end process, which means, neither any pre-processing of the input data nor post-processing of the prediction of the network was applied. Using the data of the modalities available in the publicly accessible part of the RECOLA database, we achieved results comparable to other state-of-the-art approaches.
- Subject
- multimodal data; emotion recognition; deep neural network
- Identifier
- http://hdl.handle.net/1959.13/1433671
- Identifier
- uon:39319
- Identifier
- ISBN:9783030687793
- Language
- eng
- Reviewed
- Hits: 432
- Visitors: 431
- Downloads: 0
| Thumbnail | File | Description | Size | Format |
|---|