Volume 27, Issue 4 (10-2024)
J Arak Uni Med Sci 2024, 27(4): 194-204 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Hosseini S S, Yamaghani M R. MultiModal Emotional Recognition by Artificial Intelligence and its Application in Psychology. J Arak Uni Med Sci 2024; 27 (4) :194-204
URL: http://jams.arakmu.ac.ir/article-1-7585-en.html
URL: http://jams.arakmu.ac.ir/article-1-7585-en.html
1- Malayer University, Malayer, Iran
2- Department of Computer Engineering and Information Technology, Islamic Azad University, Lahijan Branch, Lahijan, Iran ,o_yamaghani@liau.ac.ir
2- Department of Computer Engineering and Information Technology, Islamic Azad University, Lahijan Branch, Lahijan, Iran ,
Abstract: (534 Views)
Introduction: Nowadays, the use of artificial intelligence and machine learning has impacted all fields of study. Utilizing these methods for identifying individuals' emotions through integrating audio, text, and image data has shown higher accuracy than conventional methods, presenting various applications for psychologists and human-machine interaction. Identifying human emotions and individuals' reactions is crucial in psychology and psychotherapy. Emotional identification has traditionally been conducted individually and by analyzing facial expressions, speech patterns, or handwritten responses to stimuli and events. However, depending on the subject's conditions or the analyst's circumstances, this approach may lack the required accuracy. This paper aimed to achieve high-precision emotional recognition from audio, text, and image data using artificial intelligence and machine learning methods.
Methods: This research employs a correlation-based approach between emotions and input data, utilizing machine learning methods and regression analysis to predict a criterion variable based on multiple predictor variables (the emotional category as the criterion variable and the features, audio, image, and text variables as predictors). The statistical population of this study is the IEMOCAP dataset, and the data type of this research is a mixed quantitative-qualitative approach.
Results: The results indicated that combining audio, image, and text data for multi-modal emotional recognition significantly outperformed the recognition of emotions from each data alone, exhibiting a precision of 82.9% in the baseline dataset.
Conclusions: The results demonstrate a considerably acceptable precision in identifying human emotions through audio integration, text, and image data compared to individual data when using machine learning and artificial intelligence methods.
Methods: This research employs a correlation-based approach between emotions and input data, utilizing machine learning methods and regression analysis to predict a criterion variable based on multiple predictor variables (the emotional category as the criterion variable and the features, audio, image, and text variables as predictors). The statistical population of this study is the IEMOCAP dataset, and the data type of this research is a mixed quantitative-qualitative approach.
Results: The results indicated that combining audio, image, and text data for multi-modal emotional recognition significantly outperformed the recognition of emotions from each data alone, exhibiting a precision of 82.9% in the baseline dataset.
Conclusions: The results demonstrate a considerably acceptable precision in identifying human emotions through audio integration, text, and image data compared to individual data when using machine learning and artificial intelligence methods.
References
1. Khobdeh SB, Yamaghani MR, Sareshkeh SK. Basketball action recognition based on the combination of YOLO and a deep fuzzy LSTM network. J Supercomput. 2024;80:3528-53. doi: 10.1007/s11227-023-05611-7
2. Nadim M, Ahmadifar H, Mashkinmojeh M. Application of image processing techniques for quality control of mushroom. Caspian J Health Res. 2019;4(3):72-5. doi: 10.29252/cjhr.4.3.72
3. Langeroudi MK, Yamaghani MR, Khodaparast S. "FD-LSTM A Fuzzy LSTM Model for Chaotic Time-Series Prediction. IEEE Intelligent Systems. 2022;37(4):70-8. doi: 10.1109/MIS.2022.3179843
4. Yamaghani M, Zargari F. Classification and retrieval of radiology images in H.264/AVC compressed domain. SIViP. 2017;11:573-80. doi: 10.1007/s11760-016-0996-0
5. Wang JZ, Zhao S, Wu C, Adams RB, Newman MG, Shafir T, et al. Unlocking the emotional world of visual media an overview of the science, research, and impact of understanding emotion drawing insights from psychology, engineering, and the arts, this article provides a comprehensive overview of the field of emotion analysis in visual media and discusses the latest research, systems, challenges, ethical implications, and potential impact of artificial emotional intelligence on society. Proc IEEE Inst Electr Electron Eng. 2023;111(10):1236-86. doi: 10.1109/JPROC.2023.3273517. pmid: 37859667
6. Bagheri Sheykhangafshe F, Abolghasemi A, Kafi Masouleh SM. Predicting Resilience Based on Dark Triad Personality and Psychological Wellbeing in Athletes Students [in Persian]. J Arak Uni Med Sci. 2021;24(2):230-45. doi: 10.32598/jams.24.2.6151.1
7. Anvarjon T, Mustaqeem, Kwon S. Deep-Net A Lightweight CNN-Based Speech Emotion Recognition System Using Deep Frequency Features. Sensors (Basel). 2020;20(18) 5212. doi: 10.3390/s20185212. pmid: 32932723
8. Bharti SK, Varadhaganapathy S, Gupta RK, Shukla PK, Bouye M, Hingaa SK, et al. Text-Based Emotion Recognition Using Deep Learning Approach. Comput Intell Neurosci. 2022;2645381. doi: 10.1155/2022/2645381. pmid: 36052029
9. Bharti SK, Varadhaganapathy S, Gupta RK, Shukla PK, Bouye M, Hingaa SK, et al. Text-based emotion recognition using deep learning approach. Comput Intell Neurosci. 2022;2022:2645381. doi: 10.1155/2022/2645381. pmid: 36052029
10. Rumpf HJ, Browne D, Brandt D, Rehbein F. Addressing taxonomic challenges for Internet Use Disorders in light of changing technologies and diagnostic classifications. J Behav Addict. 2020;9(4):942-4. doi: 10.1556/2006.2020.00094. PMID: 33289695.
11. Kong X, Zhang K. A novel text sentiment analysis system using improved depthwise separable convolution neural networks. PeerJ Comput Sci. 2023;9:e1236. doi: 10.7717/peerj-cs.1236. pmid: 37346624.
12. Ventura-Bort C, Wendt J, Weymar M. The role of interoceptive sensibility and emotional conceptualization for the experience of emotions. Front Psychol. 2021;12:0712418. doi: 10.3389/fpsyg.2021.712418. pmid: 34867591
13. Ventura-Bort C, Wendt J, Weymar M. The role of interoceptive sensibility and emotional conceptualization for the experience of emotions. Front Psychol. 2021;12:712418. doi: 10.3389/fpsyg.2021.712418. pmid: 34867591
14. Harmon TG. Understanding and addressing the individualized emotional impact of aphasia a framework for speech-language pathologists. Semin Speech Lang. 2023. doi: 10.1055/s-0043-1776418. Epub ahead of print. pmid: 37992735
15. Tavabie S, Bass S, Minton O. Emotional intelligence in palliative medical education. Br J Hosp Med (Lond). 2020;81(12):1-5. Doi: 10.12968/hmed.2020.0573. Epub 2020 Dec 22. PMID: 33377833.
16. Steidl S. (5 March 2011). "FAU Aibo Emotion Corpus". Pattern Recognition Lab. Available from: https://www5.cs.fau.de/en/our-team/steidl-stefan/fau-aibo-emotion-corpus/
17. Hao M, Cao WH, Liu ZT, Wu M, Xiao P. Visual-audio emotion recognition based on multi-task and ensemble learning with multiple features, Neurocomputing. 2020;391:42–51. doi: 10.1016/j.neucom.2020.01.048
18. Rauschert S, Raubenheimer K, Melton PE, Huang RC. Machine learning and clinical epigenetics a review of challenges for diagnosis and classification. Clin Epigenetics. 2020;12(1):51. doi: 10.1186/s13148-020-00842-4. pmid: 32245523
19. Das A, Mock J, Huang Y, Golob E, Najafirad P. Interpretable Self-Supervised Facial Micro-Expression Learning to Predict Cognitive State and Neurological Disorders. Proc AAAI Conf Artif Intell. 2021;35(1):818-26. doi: 10.1609/aaai.v35i1.16164 pmid: 34221694
20. Ozkanca Y, Öztürk MG, Ekmekci MN, Atkins DC, Demiroglu C, Ghomi RH. Depression screening from voice samples of patients affected by Parkinson's disease. Digit Biomark. 2019;3(2):72-82. doi: 10.1159/000500354. pmid: 31872172
21. Dong J, Wu Z, Xu H, Ouyang D. FormulationAI a novel web-based platform for drug formulation design driven by artificial intelligence. Brief Bioinform. 2023 Nov 22;25(1) bbad419. doi: 10.1093/bib/bbad419. pmid: 37991246.
22. Lee YH, Lee SHB, Chung JY. Research on how emotional expressions of emotional labor workers and perception of customer feedbacks affect turnover intentions emphasis on moderating effects of emotional intelligence. Front Psychol. 2019;9:2526. doi: 10.3389/fpsyg.2018.02526. pmid: 30662415
23. Zhai Y, Song X, Chen Y, Lu W. A study of mobile medical app user satisfaction incorporating theme analysis and review sentiment tendencies. Int J Environ Res Public Health. 2022;19(12):7466. doi: 10.3390/ijerph19127466. pmid: 35742713
24. Terhürne P, Schwartz B, Baur T, Schiller D, Eberhardt ST, André E, et al. Validation and application of the Non-Verbal Behavior Analyzer An automated tool to assess non-verbal emotional expressions in psychotherapy. Front Psychiatry. 2022;13:1026015. doi: 10.3389/fpsyt.2022.1026015. pmid: 36386975
25. Kameyama M, Umeda-Kameyama Y. Applications of artificial intelligence in dementia. Geriatr Gerontol Int. 2024;24(Suppl 1):25-30. doi: 10.1111/ggi.14709. PMID: 37916614
26. Owen S, Maratos FA. Recognition of subtle and universal facial expressions in a community-based sample of adults classified with intellectual disability. J Intellect Disabil Res. 2016;60(4):344-54. doi: 10.1111/jir.12253. pmid: 26857692
27. Wirries A, Geiger F, Hammad A, Redder A, Oberkircher L, Ruchholtz S, et al. Combined artificial intelligence approaches analyzing 1000 conservative patients with back pain-a methodological pathway to predicting treatment
28. efficacy and diagnostic groups. Diagnostics (Basel). 2021;11(11):1934. doi: 10.3390/diagnostics11111934. pmid: 34829286
29. Parekh AE, Shaikh OA, Simran, Manan S, Hasibuzzaman MA. Artificial intelligence (AI) in personalized medicine AI-generated personalized therapy regimens based on genetic and medical history short communication. Ann Med Surg (Lond). 2023;85(11):5831-3. doi: 10.1097/MS9.0000000000001320. pmid: 37915639
30. Khawaja Z, Bélisle-Pipon JC. Your robot therapist is not your therapist understanding the role of AI-powered mental health chatbots. Front Digit Health. 2023;5:1278186. doi: 10.3389/fdgth.2023.1278186. pmid: 38026836;
31. Ramzani Shahrestani M, Motamed S, Yamaghani MR. Recognition of Facial and Vocal Emotional Expressions by SOAR Model. Journal of Information Systems and Telecommunication (JIST). 2023; 11(3):209-21. doi: 10.61186/jist.39828.11.43.209
32. von Klipstein L, Riese H, van der Veen DC, Servaas MN, Schoevers RA. Using person-specific networks in psychotherapy challenges, limitations, and how we could use them anyway. BMC Med. 2020;18(1):345. doi: 10.1186/s12916-020-01818-0. pmid: 33222699
33. Ross ED. Differential Hemispheric Lateralization of Emotions and Related Display Behaviors Emotion-Type Hypothesis. Brain Sci. 2021;11(8):1034. doi: 10.3390/brainsci11081034. pmid: 34439653
34. Gweon H, Fan J, Kim B. Socially intelligent machines that learn from humans and help humans learn. Philos Trans A Math Phys Eng Sci. 2023;381(2251):20220048. doi: 10.1098/rsta.2022.0048. pmid: 37271177
35. Jain R, Rai RS, Jain S, Ahluwalia R, Gupta J. Real time sentiment analysis of natural language using multimedia input. Multimed Tools Appl. 2023;82:41021-36. doi: 10.1007/s11042-023-15213-3. pmid: 37362666
36. Huang F, Li X, Yuan C, Zhang S, Zhang J, Qiao S. Attention-Emotion-Enhanced Convolutional LSTM for Sentiment Analysis. IEEE Trans Neural Netw Learn Syst. 2022;33(9):4332-4345. doi: 10.1109/TNNLS.2021.3056664. pmid: 33600326
37. Jaksic C, Schlegel K. Accuracy in judging others' personalities the role of emotion recognition, emotion understanding, and trait emotional intelligence. J Intell. 2020;8(3):34. doi: 10.3390/jintelligence8030034. pmid: 32961916
38. Nourbakhsh A, Hoseinpour MM. Multiple feature extraction and multiple classifier systems in face recognition. In: Silhavy R, Silhavy P, Prokopova Z. (editors) Cybernetics Approaches in Intelligent Systems. Springer, Cham: CoMeSySo; 2017.
39. Mittal T, Bhattacharya U, Chandra R, Bera A, Manocha D. M3ER Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues, Proc. AAAI Conf. Artif Intell. 34(2020):1359–67. doi:10.1609/aaai.v34i02.5492
40. Hosseini SS, Yamaghani MR, Poorzaker Arabani S. Multimodal modelling of human emotion using sound, image and text fusion. SIViP. 2023;18:71-9. doi: 10.1007/s11760-023-02707-8
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
