The effectiveness of ChatGPT in extracting architectural patterns and tactics

Abstract

This work investigates the potential of ChatGPT, a cutting-edge large language model (LLM), for software design analysis specifically in detecting architectural patterns and tactics. The evaluation involves comparing ChatGPT’s performance with that of Archie, a traditional Eclipse plugin designed for architectural analysis. The study uses the source code of five open-source software systems as the testing ground. Results reveal that ChatGPT achieves noteworthy performance in both pattern and tactic detection tasks. Specifically, for pattern detection, ChatGPT demonstrates an accuracy of up to 47.06%, while for tactic detection, it achieves a precision of 28.25%. While ChatGPT’s current capabilities are not yet a replacement for specialized tools like Archie, it offers significant potential as a complementary tool in architectural analysis workflows. By bridging the gap between natural language understanding and software engineering, ChatGPT could pave the way for more intelligent and automated solutions in the field. However, a key limitation is its difficulties in handling foundational or traditional tactics, resulting in a lower detection rate in certain areas. This research contributes valuable insights into the application of LLMs in software engineering, highlighting both the strengths and the limitations of ChatGPT in addressing complex architectural tasks.