Sign language emotion and alphabet recognition with hand gestures using convolution neural network

Abstract

American sign language (ASL) is a special means of interaction for hard-of-hearing individuals and has precise conventional rules. Since the general public does not know these sign language protocols, there is a need to have an efficient automatic sign-emotion recognition system. The objective of this paper is, to develop a framework that recognizes standard hand gestures. The gesture represents emotions and alphabet. This paper covers the methodology, results and performance factors, for experimentations. This experimentation of ASL-based alphabet and emotion recognition is novel as till now many efforts of alphabets categorization are done but this is the new direction of research where emotions, such as together’, ‘happy’, ‘peace, ’sad’, ‘confused’, and ‘love’ are captured and automatically classified with hand signs. We mention our approach to increase ‘accuracy’, wherein we capture images and regions of interest (ROI). In this article, a specifically designed convolution neural network (CNN), is used to identify emotions from hand gestures and the addition of ROI enhances accuracy. The captured hand gesture dataset of the size of 94,000 images. “peace” sign emotion has the highest recognition rate (‘98.95%’). Alphabet’s “P” and “Q” sign ASL alphabets have the maximum recognition rate of signs. In all, very impressive accuracy of “92%” and above is detected. The limits of the experimentation are as mentioned i) there is no repeatability of accuracy for the same hand gesture; ii) The distance and angle of hand gestures with camera are crucial factors for an experiment; and iii) the alphabet recognition system is not working for the alphabets “J” and “Z”.