Sign Language Emotion and Alphabet Recognition with Hand Gestures Using CNN

American Sign Language (ASL) is a special means of interaction for hard-of-hearing individuals and has precise conventional rules. Since the general public does not know these sign language protocols, there is a need to have an efficient automatic sign-emotion recognition system.

The objective of this paper is to develop a framework that recognizes standard hand gestures. The gesture represents emotions and alphabets. This paper covers the methodology, results, and performance factors for experimentations.

This experimentation of ASL-based alphabet and emotion recognition is novel, as till now, many efforts of alphabet categorization are done, but this is the new direction of research where emotions such as ‘together’, ‘happy’, ‘peace’, ‘sad’, ‘confused’, and ‘love’ are captured and automatically classified with hand signs.

We mention our approach to increase ‘accuracy’, wherein we capture images and regions of interest (ROI). In this article, a specifically designed convolution neural network (CNN) is used to identify emotions from hand gestures, and the addition of ROI enhances accuracy.

The captured hand gesture dataset is of the size of 94,000 images. The “peace” sign emotion has the highest recognition rate (98.95%). Alphabets “P” and “Q” in ASL alphabets have the maximum recognition rate of signs. In all, very impressive accuracy of 92% and above is detected.

The limits of the experimentation are as mentioned: i) There is no repeatability of accuracy for the same hand gesture; ii) The distance and angle of hand gestures with the camera are crucial factors for an experiment; and iii) The alphabet recognition system is not working for the alphabets “J” and “Z”.