LLM-FS-Agent: a deliberative role-based large language model architecture for transparent feature selection

Abstract

High-dimensional data can reduce interpretability and increase computational cost in machine learning (ML) pipelines. Large language models (LLMs) offer a promising approach for dimensionality reduction through feature selection (FS), but existing LLM-based methods lack structured reasoning and do not provide clear justifications for their choices. This study propose LLM-FS-Agent, a multi-agent architecture where specialized LLM roles collaborate through a deliberative process to evaluate and rank features. Each selection decision is accompanied by an explicit rationale. We tested our method on the CIC-DIAD 2024 internet of things (IoT) intrusion detection dataset and compared it against LLM-select and traditional techniques like principal component analysis (PCA) across multiple subset sizes. LLM-FS-Agent achieves comparable or better accuracy while reducing classifier training time by 46% on average (statistically significant, 0.094 s, p = 0.028 for extreme gradient boosting (XGBoost)). These results show that multi-agent deliberation improves transparency and computational efficiency in FS.