Effectiveness of artificial intelligence-driven chatbot responses in diabetes knowledge: a readability and reliability assessment

Abstract

Patient education is vital in diabetes management, empowering patients with necessary knowledge and skills to manage condition effectively. However, traditional educational methods often face challenges such as limited access to healthcare professionals and variability in information quality. This study aimed to assess the reliability and readability of artificial intelligence (AI)-driven chatbot responses in disseminating diabetes knowledge. Technically, the diabetes knowledge questionnaire (DKQ-24) was administered to evaluate the effectiveness of AI-driven chatbot in disseminating diabetes-related information. Responses were evaluated for reliability and quality applying the modified DISCERN (mDISCERN) scale and global quality scale (GQS), and readability was assessed using the Flesch reading ease (FRE) score, Flesch-Kincaid grade level (FKGL), gunning fog index (GFI), Coleman-Liau index (CLI), and simple measure of gobbledygook (SMOG). The mean mDISCERN score was 31.50±2.89, indicating generally reliable responses. The median GQS score was 4, reflecting the high overall quality. The readability assessment revealed a mean FRE score of 66.30, indicating that the text was fairly easy to read. FKGL mean score was 6.54±3.19, suggesting the text was suitable for readers at a sixth-grade level. In conclusion, AI-driven chatbot provides reliable and high-quality information on the diabetes self-management, but it requires improvements to enhance accessibility.