Real-time intelligent virtual assistant based on retrieval augmented generation

Abstract

Improving user experience in accessing information on organizational websites remains a challenge. Users often face complex navigation and multi step searches that slow information retrieval. This study introduces the real time intelligent virtual assistant (RIVA), which integrates large language models (LLMs) with the retrieval-augmented generation (RAG) framework to support real-time interaction with website content. The system was implemented on the Universitas Pendidikan Ganesha (Undiksha) website using a WordPress content management system (CMS) and developed following the design science research (DSR) approach, which includes six stages: problem identification, solution objectives, design and development, demonstration, evaluation, and communication. The retrieval-augmented generation assessment (RAGAS) evaluation indicated that the combined model of text-embedding-ada-002 and semantic chunking yielded the best results, with context precision=0.83, context recall=0.90, response relevancy=0.91, faithfulness=0.83, and answer correctness=0.85. User experience questionnaire (UEQ) testing performed well, particularly in the novelty and stimulation dimensions. These results demonstrate that RIVA can provide users with access to relevant and engaging information. As a result, future research will focus on improving retrieval and developing adaptive semantic chunking for structured and complex data.