Multilingual signs recognition using recurrent neural network

Abstract

Recognition of sign language is a crucial step towards providing individuals with hearing and speech impairment meaningful communication, but the fact that there are a number of distinct sign languages and gestures remain complex makes it a challenge to the current automated systems. The present paper describes a real-time multilingual sign language recognition system that is based on a recurrent neural network with long short-term memory (RNN-LSTM) with hand landmark MediaPipe-based hand landmark detection to successfully receive spatial and temporal gesture features. The proposed system was trained and tested over a self-collected set of alphabet gestures of the Chinese, American, and Indian sign language, including one hand and two-hand gestures, and was run with Keras with extensive performance evaluation metrics. The strength and generalization abilities of the suggested approach as part of different gesture patterns and variations in users are confirmed by experimental outcomes that indicate high recognition rates of 99.58%, 99.62%, and 99.63% of the Chinese, American, and Indian sign languages, respectively. These results demonstrate the promise of the given framework as a dedicated assistive system of communication and give it a solid base to continue its development to the point of the system of the continuous sign language recognition (CSLR) and multimodal translators.