Deep attention transformer models for chronic kidney disease stage prediction

Abstract

Chronic kidney disease (CKD) is a growing health problem that demands accurate stage prediction for timely clinical intervention. Conventional machine learning (ML) models, although widely used, often struggle to capture complex, non-linear relationships in clinical data. To address this limitation, this study proposes a deep learning (DL) framework based on two tabular architectures: attention network (AttentionNet) and tabular transformer (TabTransformer). AttentionNet assigns higher importance to clinically relevant features through an attention mechanism, while TabTransformer uses self-attention to model dependencies among clinical variables. These models were evaluated against baseline approaches, including residual multilayer perceptron (ResMLP), wide and deep (WnD), and a standard multilayer perceptron (MLP). The dataset comprised 1,446 patient records with 26 validated features after leakage removal, spanning five CKD stages. Preprocessing involved cleaning, categorical encoding, normalization, and evaluation using 10-fold cross-validation. Results showed that ResMLP and TabTransformer consistently achieved the highest accuracy and F1-scores, both exceeding 99%, with minimal variation across folds. Ablation analysis revealed the critical role of glomerular filtration rate (GFR) and serum creatinine (SC), as their removal led to notable performance degradation. High area under the curve (AUC) values further confirmed model robustness. Overall, DL-based tabular models offer a reliable foundation for CKD stage prediction and clinical decision support systems.