Generative artificial intelligence as an evaluator and feedback tool in distance learning: a case study on law implementation

Abstract

The development of generative artificial intelligence (GAI) has impacted various fields, including higher education. This research examines the use of GAI as an evaluator and feedback provider in distance legal education. This study tested five GAI models: ChatGPT, Perplexity, Gemini, Bing, and You, using a sample of 20 students and evaluations from legal experts. Descriptive statistical analysis and non-parametric tests, including Wilcoxon, intraclass correlation coefficient (ICC), Kappa, and Kendall's W, were used to assess accuracy, feedback quality, and usability. The results showed that ChatGPT was the most effective GAI, with the highest mean scores of 4.22 from experts and 4.12 from students, followed by Gemini with scores of 4.15 and 4.07. In terms of binary judgement accuracy, Gemini scored 80%, ChatGPT 60%, while Perplexity, Bing, and You had lower scores. Statistical analysis showed moderate agreement (ICC=0.439) and low alignment (Kappa=-0.058) between the GAIs and expert evaluations, with a Kendall's W value of 0.576 indicating moderate consistency in judgements. These findings emphasize the importance of selecting effective GAIs such as ChatGPT and Gemini to improve academic evaluation and learning in legal education, and pave the way for further innovations in the use of AI.