Indian sign language video generation using attention-enhanced generative adversarial network

Abstract

Sign language (SL) is the primary mode of communication for the Deaf signers. Despite advancements in deep learning, SL recognition, translation, and video generation face challenges like blurriness and inconsistencies. This research proposes a novel sign language translation (SLT) approach for Indian sign language (ISL) using an attention-driven generative adversarial network (GAN). The preprocessing pipeline includes video frame extraction, skeletal joint coordinate detection via OpenPose, and dynamic time warping (DTW) for pose data refinement. The squeeze and excitation (SE) attention mechanism enhances 2D convolutional layers, allowing the generator to focus on relevant skeletal pose sequences. A motion discriminator refines motion authenticity. Performance evaluation using two SL datasets demonstrates significant improvements in structural similarity index measure (SSIM), peak signal-to-noise ratio (PSNR), and temporal consistency metric (TCM) scores, achieving 99.60 (%) as SSIM, 31.10 dB as PSNR, and 0.9111 as TCM. The proposed model outperforms standard GAN and dynamic GAN in SL video generation.