Home / Current Issue / Paper 1713848
An Emotion Aware AI-Based Virtual Therapy System Using Text and Speech Analysis
Subject area: Science,Engineering and Technology · Area of research: Artificial Intelligence
DOI: https://doi.org/10.64388/IREV9I7-1713848
Abstract
Mental health support systems based on conventional chatbots often fail to provide emotionally appropriate responses due to their limited understanding of user emotions. To address these limitations, this paper presents an emotion-aware virtual therapy system that utilizes both textual and speech input to detect the emotional state of users and generate context sensitive therapeutic responses. The proposed system integrates natural language processing techniques for text-based emotion recognition and deep learning models for speech emotion analysis using acoustic features such as melfrequency cepstral coefficients. A multimodal architecture is designed to improve overall accuracy and reliability. The system is implemented as a mobile application to ensure accessibility and real time interaction. Experimental evaluation conducted on benchmark emotion datasets demonstrates that the proposed approach achieves higher performance compared to unimodal emotion detection methods. The results indicate that incorporating emotion awareness significantly enhances the effectiveness of virtual therapy systems by enabling personalized and empathetic user interactions. This work highlights the potential of multimodal artificial intelligence in scalable and intelligent mental health support solutions.
Keywords
Emotion Detection, Mental Health Support, Virtual Therapy System, Natural Language Processing, Speech Emotion Recognition, Multimodal Deep Learning, AI Chatbot
References
[1] B. Liu, “Sentiment analysis and opinion mining,” Morgan & Claypool Publishers, 2012.
[2] S. Poria, E. Cambria, and A. Gelbukh, “Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis,” EMNLP, 2015.
[3] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” NAACL-HLT, 2019.
[4] Z. Zhao, Z. Bao, Z. Zhang, and N. Cummins, “Automatic speech emotion recognition using deep neural networks,” IEEE ICASSP, 2014.
[5] S. Livingstone and F. Russo, “The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS),” PLoS ONE, 2018.
[6] S. Poria, E. Cambria, D. Hazarika, and N. Majumder, “Multimodal sentiment analysis: Addressing key issues and setting up baselines,” IEEE Intelligent Systems, 2018.
[7] M. Inkster, S. Sarda, and V. Subramanian, “An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being,” JMIR mHealth and uHealth, 2018.
[8] A. Abd-Alrazaq et al., “Effectiveness and safety of using chatbots to improve mental health: systematic review,” Journal of Medical Internet Research, 2020.
[9] D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “GoEmotions: A dataset of fine-grained emotions,” in Proc. 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, pp. 4040–4054.
[10] S. Livingstone and F. Russo, “The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS),” PLoS ONE, vol. 13, no. 5, 2018.
How to cite this paper
@article{1713848,
author = {Anushka Bohra},
title = {An Emotion Aware AI-Based Virtual Therapy System Using Text and Speech Analysis},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {9},
number = {7},
pages = {2041-2045},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1713848.pdf},
abstract = {Mental health support systems based on conventional chatbots often fail to provide emotionally appropriate responses due to their limited understanding of user emotions. To address these limitations, this paper presents an emotion-aware virtual therapy system that utilizes both textual and speech input to detect the emotional state of users and generate context sensitive therapeutic responses. The proposed system integrates natural language processing techniques for text-based emotion recognition and deep learning models for speech emotion analysis using acoustic features such as melfrequency cepstral coefficients. A multimodal architecture is designed to improve overall accuracy and reliability. The system is implemented as a mobile application to ensure accessibility and real time interaction. Experimental evaluation conducted on benchmark emotion datasets demonstrates that the proposed approach achieves higher performance compared to unimodal emotion detection methods. The results indicate that incorporating emotion awareness significantly enhances the effectiveness of virtual therapy systems by enabling personalized and empathetic user interactions. This work highlights the potential of multimodal artificial intelligence in scalable and intelligent mental health support solutions.},
keywords = {Emotion Detection, Mental Health Support, Virtual Therapy System, Natural Language Processing, Speech Emotion Recognition, Multimodal Deep Learning, AI Chatbot},
month = {January},
doi = {https://doi.org/10.64388/IREV9I7-1713848}
}