Temporal convolutional network for Indonesian word by word sign language translator on MediaPipe framework

Abstract

Sign language is a language created to help the deaf-mute communicate with others. In Indonesia, there are two sign languages: Indonesian sign language system or sistem isyarat bahasa Indonesia (SIBI) and Indonesian sign language or bahasa isyarat Indonesia (BISINDO), but the deaf community prefers to use BISINDO. Deaf-mute people use sign language to communicate, but normal people usually avoid learning it because they feel it's unnecessary for daily communication. This will certainly be a problem for the deaf-mute because people who do not learn sign language will certainly have difficulty understanding what they are trying to convey. Therefore, an application that can translate sign language is needed to help overcome communication problems between the deaf-mute and normal people. This research focuses on BISINDO translation using a temporal convolutional network (TCN) with MediaPipe methods. The amount of data used consists of 2,700 videos representing 27 word-level signs in BISINDO; this number increases to 32,400 after undergoing the data augmentation process. The model built was tested in translating BISINDO at the word level in simulation, and a thorough comparison between TCN and bidirectional long short-term memory (BiLSTM); both of them were running on the MediaPipe platform separately. From the experiment, it is shown that the performance of TCN exceeds BiLSTM with an accuracy rate as high as 88.64%.