Multimodal retrieval-augmented generation with sentiment-aware responses for depression

Abstract

In view of the rapid growth in conversational interfaces, there is a demand for more adaptive retrieval systems. Traditional chatbots, built on pre-trained models, struggle to understand and respond to a wide range of real-world user queries. In view of this, a framework based on multimodal retrieval-augmented generation with sentiment-aware responses (MRAG-SAR) for depression that addresses the challenges and strengthens conversational interface performance has been developed. The proposed system can process unstructured data in multiple formats of inputs, such as audio, image, and text, using a multimodal late-fusion mechanism, making it suitable for managing diverse, domain-specific information. It also leverages powerful embedding models for capturing contextual similarity and scalable vector databases for efficient retrieval. The user queries are augmented to provide precise and nuanced context-related responses. The proposed framework retrieves content that captures the dimensions of the topic to enhance the relevance of the output responses. Introduction of sentiment-aware response generation allows the system to respond in a tone based on the emotional context of the user queries, which is essential in making use of personalized conversations. Relevant bidirectional encoder representations from transformers score (BERTScore) and other metrics highlight the framework's effectiveness in delivering high-quality responses.