Emotion recognition of electroencephalogram using hybrid convolutional neural networks and vision transformer

Abstract

Electroencephalogram (EEG) signal-based emotion classification faces challenges in spatio-temporal complexity and inter-channel redundancy. This paper proposes a 2D convolutional neural network (CNN) and vision transformer (ViT) approach. Relevant emotions require signal extraction in the 4-40 Hz frequency band using the discrete wavelet transform (DWT). This study uses the SEED dataset from 15 subjects, with 62 channels reduced to 12. Every 5-second segment is decomposed using DWT into four frequency bands, which are then mapped to a 2D spatial representation for CNNs and ViTs. Experiment results show that the DWT-2D CNN-ViT achieves the best accuracy of 87% with a final loss of 0.087. In the meantime, the CNN-only model (85%), the ViT-only model (77%), and the DWT-only model (47%). Testing with several optimizers also shows that AdamW provides the highest performance with the fastest training time of 12.34 minutes. These results demonstrate that this integration can produce more efficient and stable learning. These findings indicate that the combination of DWT and the CNN-ViT hybrid architecture is effective for accurate, stable EEG-based emotion recognition and is potentially applicable to real-time emotion-monitoring systems.