# Introduction ccording to the latest statistics of the World Health Organization, 5% of the world population cannot hear a single word [1]. It is a tragedy, which leads them to mutism, since they can-not hear or learn words to speak. In conditions such as Apraxia of Speech, Cerebral Palsy and Aphasia people suffering from the inability to speak. To reduce this gap between mute and non-mute people, sign language will act as a bridge between them. Sign language is the main communication method for those who cannot express their voice. In such situations, they will need a communication mechanism to express their ideas with each other. As a non-mute person, sharing ideas with a mute person will cause a difficult situation for both. Because of one person cannot hear any sound and other one cannot understand the hand gestures. In such situations, society may have to make sure that deaf mute people can understand them very well. The more non-mute people do not understand them via sign language, the more they will avoid having human interaction with the community. There are 6909 distinct spoken languages in the world today [2]. Same as that, 125 sign languages use around the globe in different countries [3]. As an example of the nature of sign language, American Sign Language (ASL) is quite different from British Sign Language (BSL), even though Author ? ? ? ?: e-mail: it16133228@my.sliit.lk English is the spoken language of both countries [2]. Within these 125 sign languages in the world, Sinhala sign language (SSL) also stands with other languages. With the "Smart community" concept, people tend to use technology to transform their natural lifestyle into more productive and positive ways. These devices sense and record user activities, predict their future behavior, and prepare everything one step ahead according to the user's preference or needs, giving him/her the most convenience, comfort, efficiency, and security. [4] 1. Computer vision-based method 2. Sensor-based method Computer vision-based gesture recognition [5]can be less accurate and less comfortable to the end user since, it involves many aspects such as motion modeling, motion analysis, pattern recognition and machine learning [6]. Outdoor light states act a key role in vision-based gesture identification since, without proper light conditions, cameras will not be able to recognize the gestures and method through the image processing mechanisms. Considering the sensor-based hand gesture recognition mechanisms, different sensors provide set of data according to the joints and finger separation that characterizes a hand gesture. By obtaining these data, any movement can be represented as a sequence of frames. The remainder of the paper is arranged as follows: Section II elaborates the background survey including a theoretical comparison of existing SIGN LANGUAGE INTERPRETERS. Section III explains the approach of implementing the Glove and whole environment in detail. Section IV explains the Test results from the process and finally, Section V concludes the paper conclusion. # II. # Background Review Researches in the sign language recognition systems are mainly based on computer vision and sensor based recognition mechanisms. The image processing techniques [7] using the camera to capture the image/video. Examinations of the data with static images and identify the image using various algorithms and create sentences for that into the display [7] Camera place to direct the place that captures highest available hand movements, higher resolution camera take up more calculation time and hold more memory space. A deaf person always need a high performance camera permanently and cannot use in a public place. Though, computer vision technologies restricted in terms of their functionalities, many high performance techniques required with more expensive sensors which not only made the application more complex and expensive. The system was limited to an accurate background without any noise or disturbance. Another research determines sign language recognition system using a hand glove. [8] [9] In this design, mute person need to wear a glove consist of 5 flex sensor for each finger and motion tracker. Data are directly coming from every 5 sensors and process sensor data with static data to produce sentences. It's using a neural network to increase the completion of the system. The main advantage of this system is the fast response in real-time applications. Its movable device and cost of the complete device is higher since the hardware used are expensive [9]. In another research, researches developed a sign language recognition system using a portable Accelerometer (ACC) and Surface Electro Myogram (sEMG) [10]these sensors are used to detect the hand gesture. ACC used to capture movement information of hand and arms. These Sensor output signals are input to the computer and process to identify the hand gesture and provide speech and text with both [10]. But none of the above methods provides users with two-way communication and as well as a graphical picture of each sign. Our proposed system will be capable of delivering the two-way conversation with visualizing pictures of relevant signs in the app with a user-friendly manner. Other than using flex sensors, a team of researchers used Potentiometer to extract the data from fingers [11]. Above system design was created to work with virtual reality applications like replacing the conventional input devices like joysticks in video games with the data glove. Also, the Robot control system to regulate machine activity at remote sensitive sites. As per above-mentioned readings, it is clear that hand gesture recognition system for mute people is very essential to the current society, yet it has many issues to be addressed such like the mobility of the device, identifying and express full sentences other than words, power consumption and develop as a smart device. III. # Methodology Proposed system is designed to identify and translate the hand gestures into a digital voice as final outcome. In implementation, system is consist of a software module and a hardware module. Accelerometer, flex sensors and printed circuit board includes in hardware module. As the initial step, system will capture the flex sensor and accelerometer readings. Record it one by one for each sign by using a pushbutton remote (push button). A push button is used to input the boundaries of a single data frame corresponding to a certain sign in the data stream. Number of data sets for one sign can be obtained and save them in a CSV file format. In the same manner different data sets for different signs (100) can be collected. These signs referred as mean data set. As the next step get overall CSV file set and calculate the mean value for each sign. After getting the mean values, collection of single CSV files set for each different sign. For example, if there are 10 signs, there will be x axis data for each 10 signs and saved in one CSV file. Similarly, now there are 8 CSV files for for X,Y,Z z-axis and 5 flex sensors. In addition to that, recorded data set also included in the system. As previous this also has 8 CSV files for X,Y,Z axis and also for the 5 flex. The only difference here is it is not necessary to calculate the mean value. The next step is to identify the mean squared error (MSE). Using recorded data and the actual data set can calculate the MSE value. Up to this moment there are two different data sets available in the system (actual sign data set and recorded data set). According to the graph here, it visualize our data set which consists of the actual gesture capture with their mean values. In addition to that, data set which consists of the recorded data set also available. Now according to the figure 2 graph, comparison of 2 graphs can be made by shifting them. As above, shifting the frame one by one will lead to calculate the MSE value. As a assumption, assume recorded data set and it has 120 data points. Initially it's necessary to fix the actual sign size. It will range 0 to 80 and then check it. Then shift by one value and next 1 to 81 ranges and check it. Likewise shifting the actual sign graph on our recorded sign graph and calculate the MSE value. As a last step of first phase, data can be stored in an array. After comparing the graphs by shifting method MSE value array can be generated. After checking the complete graph, point of lowest value in graph can be identified as the point where consist the lowest error. # Fig. 3: Shifting the graph Identification of fix value is essential since it used to compare the MSE values. To calculate this fix value, same two Graphs will be used. Actual one (for a one sign) and calculate the MSE value of it. In a perfect error-less scenario outcome should be 0.00. But obviously there is are range differences between same sign graphs. Because of that calculating this in several times we have to select a fix one. As the final results for the fix value we got the answer as 0.05. After getting fix value now it is easy to scale out our MSE values as following. 1. Ex-maximum error value = 0.05 2. If our minimum MSE value is (E) ? 0.05 This result can be accepted, since mean of the recorded data set is equal to the exact sign and very similar to the sign that we used to shift (Actual data set). We have eight CSV file up to now (both actual and recorded). This need to done to the same e CSV files also and we have to collect the results in several time for same graphs also to get a idea of the pattern that will take(ex R1, R1.1 R1.2). After getting results it can be visualize as below. After getting data from the sensors and calculating the MSE values, cost function will provide the processed data in to a Long short-term memory network (LSTM). Once the data received, LSTM will try to recognize a pattern with input data and give an output. If we consider 1,2,3,4 & 5 are five different signs, according to the below graph, output is identified as 1,2,3 &4. This is a perfect, error less situation since none of the other signs identified by accelerometer axis or flex sensors other than the original sign. It comes only if accelerometer axis and flex sensors identify the correct sign without any interference with other signs. If we check the output 5, it clearly visible all 8 sensors identify the output as "5" yet Y,Z axis and F3 sensor identified "5" with 3,4 signs as well. This is an error, yet it can be normalized, since once the same pattern identified by the LSTM network, it will keep the pattern in the memory. Whenever a similar pattern or pattern with minor changes identified by LSTM network it will give the output as pattern with most similarities. Once a pattern is recognized through LSTM, processed data will feed in to a another LSTM network for smooth the outcome of the sentence. Smoothing the outcome is essential because from the first LSTM network we only get identified word series. Processing is essential for give a user-friendly outcome to the end user. After getting a complete, meaningful sentence from second LSTM network it will transmitted over WiFi to a mobile application. From mobile application identified sentence will be expose to the end user in voice format. The designed Sign language recognition system has the capability of training an inexperienced user to the system with inbuilt training mode. Once a new user registered through the mobile application first time, user will be directed to the training mode. According to the given instructions user may complete the training in predefined time. Once the training session is completed, accuracy of new users hand gestures will be calculated and provide with percentage. # PCB Design Customized PCB was designed to obtain the signals from 5 flex sensors and accelerometer with optimized space usage to reduce the weight of the device. Other than using whole modules, this PCB is designed with separate ICs, sensors and SMD components to reduce the space usage. To build the serial communication we connect the CP2102 through a micro USB port. In such cases like, WiFi failure or battery power decrease we can directly connect the board and the Raspberry Pi via a micro USB cable. Also, we provide power, through the micro USB port. Data is directly coming from 5 flex sensors and MPU 6050 sensor. Through WiFi connection, data will transfer to the Raspberry Pi for processing. # Esp wroom 32D IC When we take the data in live data will store in an array that is coming from module. This array size is depending on the largest size count on the mean data set. When the array count is fully then it will send for the process. Again for the same process before it storing 50% of data elements from the array will be deleted. Then the data will shift to the first elements and also adding. Then as in previous when the array count is fully then it will send for the process. This whole process will be evaluated when we catch live data. # IV. # Test Results Experiments were mainly conducted with the test graph results. For example, observe the sign 'Good morning'. For the 'Good morning' sign, we should obtain the mean data set and recorded data set. It will show in figure 6 and 7 here (For a one Axis). # Discussion Study referencing shows the World Health Organization, Over 5% of the world's population or 466 million people have disabling hearing loss (432 million adults and 34 million children) [1]. In children under 15 years of age, 60% of hearing loss is attributable to preventable causes [1]. This figure is greater in low-and ( ) A Year 2020 middle-income countries (75%) as matched to highincome countries (49%) [1]. Overall, preventable causes of childhood hearing loss include: 1. Infections such as mumps, measles, rubella, meningitis, cytomegalovirus infections, and chronic otitis media (31% [1]. 2. Complications at the time of birth, such as birth asphyxia, low birth weight, prematurity, and jaundice (17%) [1]. 3. Use of ototoxic medicines in expecting mothers and babies (4%) [1]. Others (8%) Taking these matters of the impact of hearing problems in the world, it has been discovered that a solution to identify these sign language communicate and predict how two-way communication is done and how the new user familiar with the sign language using the training mode. Hence this research study is based on using machine learning to predict the variation of the signs and neural network to identify the specific signs by using the data. Two types of sensors are used to capture the data of a hand gesture. The flex sensor and accelerometer are used to capture the readings from a hand gesture in a multidimensional way. Five flex sensors are used to capture the finger movements using the resistance of the flex sensors located on every finger of the glove. Five flex sensors will be used for a single hand since the project focuses on hand gesturing of American Sign Language. The American Sign Language has been selected since the gesture of the language is only based mostly on a single hand. Furthermore, the accelerometer (GY-521) positioned on the top of the glove will be used to measure the acceleration force of the hand gesture. The data is taken by the sensor will be sent ESP wroom 32D sensor for further processing as well. Captured data transmitted via MPU 6050 Module to Raspberry-Pi and it will be processed in artificial neural network [12]. In this process ANN's output will be a collection of words, letters or numbers which will not give a proper sense to the end user. To overcome this obstacle, Natural Language Processing mechanism can be used in the proposed system. The overall application will be design from using Android studio and adobe XD for design interfaces. When considering the structure of mobile application, it has many interfaces to illustrate information to the user which is included different features; ? The device is connected to a mobile application through Bluetooth When open the application firstpage display connect tab and home tab then press the connect button application relates to the main control system. Provide text and animation for illustrate sign language Normal person is talking with the deaf person that voices are converting to the hand gestures and that gestures are display through the mobile application with text Suggestion 1. X-axis -Data points 2. Y-axis -Processing time This graph visualize the data processing speed of Intel core i7 processor and Raspberry Pi 4B.Green color graph illus-trates the data input without processing. Orange color graph illustrates the data processed through Intel core i7 (1065G7 CPU @ 1.50 GHz) processor. Blue color graph illustrates the Data processed through Raspberry Pi 4B (Broadcom BCM2711 SoC 1.5 GHz 64-bit quad-core ARM Cortex-A72). With the proposing method, input data rate will be reduced and sign graphs are checked with ¼ part of the complete sign in the beginning. Once the first ¼ of the sign is identified with cost function, rest of the sign will be checked. data points, MSE value should be calculated 120 times. In proposed method, a sign signal break into ¼ of full length. Once the mean value start to analyze the sign, first it check the ¼ of signal and verify whether it can identify the signal. If the first quarter can be identified, rest of the sign will be processed. Considering this method efficiency can be increased. # Conclusion End product of this project is useful for handicapped mute community, which will develop a bridge between those who comprehend sign language and those who do not. Initial version of this product support the ASL. We described the method for obtaining hand gestures by several sensors including flex sensors. With use of NLP mechanisms data will be processed and trained to give a more accurate output. During our project we faced several challenges and problems including obtaining and processing data, yet we give our full strength to minimize the errors, since it will lead to minimize the communication gap among the disable community. 1![Fig. 1: System overview](image-2.png "Fig. 1 :") 2![Fig. 2: Mean error](image-3.png "Fig. 2 :") 4![Fig. 4: LSTM Result](image-4.png "Fig. 4 :") 67![Fig. 6: Mean data set graph](image-5.png "Fig. 6 :Fig. 7 :") 8![Fig. 8: MSE Algorithm After getting the mean values, recorded data set in the 'Good morning' shift the frame, one by one and calculate the MSE. Before that, we calculate the fix MSE value previous (mentioned in the methodology). 1. Ex-maximum error value = 0.05 2. If our minimum MSE value is (E) ? 0.05 If the results are agreed to above criteria, we can accept the results. It visualize, recorded data set the exact sign is very similar to the sign that we used to shift ('Good morning' sign). ? 1/N -Divide by the total number of Data points ? yi -Actual output value ? y?i -Predicted output value ? yi-y?i -The absolute value of the residual V.](image-6.png "Fig. 8 :") 9![Fig. 9: Processing speed testing According to the current method, once a sign reached to the cost function, complete sign's mean squared error will be calculate. If sign consist with 120](image-7.png "Fig. 9 :") 10![Fig. 10: Suggested method VI.](image-8.png "Fig. 10 :") application interacts with deaf persons. It providesclear interfaces for user.? User can select modes.1. User Mode2. Training Mode3. Battery Level? Provide user-friendly interfaces for understanding sign language when considering this mobile application; it should be a special one because this © 2020 Global JournalsIoT Based Sign Language Recognition System * Prevalence of hearing impairment among primary school children in the kilimanjaro region within tanzania NaalsundSolvang Iselin Ertzgaard TønderKristin Sofie Hansen Giske HaganTobiasSindberg MnyanyiBang Cosmas Tronstad Tron NkyaVedul ØygardenAslam Jon International Journal of Pediatric Otorhinolaryngology 130 109797 2020 * Anatomical analysis between two languages alphabets: Visually typographic test transformation in mor-phological approaches MizanurRahman MdSalah Uddin ApurbaMd Samaun Hasan Ghosh Sadia Afrin ArifBoby ShahAhmed ShaikhMuhammad Sadiur Rahman Muhammad Allayear International Conference on Advances in Computing and Data Sciences Springer 2020 * CatalinaDolors Rodr ´?guez-Mart´?n AnnaRodr´?guez-Garc´?a Falco´-Pegueroles * Ethnographic analysis of communication and the deaf com-munity's rights in the clinical context Contemporary nurse 54 2 2018 * Smart community: an internet of things application XuLi RongxingLu XiaohuiLiang XueminShen JimingChen XiaodongLin IEEE Communications magazine 49 11 2011 * Vision-based gesture recognition: A review YingWu SThomas Huang International Gesture Workshop Springer 1999 * Tinker-ing with machine learning: The legality and consequences of online surveillance of students AmyCyphert Nevada Law Journal 20 2020 * Real-time malaysian sign language translation using colour segmentation and neural network RiniAkmeliawati MelaniePo-Leen Ye ChowOoi Kuang IEEE Instrumentation & Measurement Technology Conference IMTC 2007. 2007. 2007 IEEE * Data gloves for sign language recognition system PriyankaLokhande RiyaPrajapati SandeepPansare International Journal of Computer Applications 975 8887 2015 * Exploring sensor gloves for teaching children sign language KirstenEllis JanCarloBarca Advances in Human-Computer Interaction 2012. 2012 * A sign-component-based framework for chinese sign language recognition using accelerometer and semg data YunLi XiangChen XuZhang KongqiaoWang Z JaneWang IEEE transactions on biomedical engineering 59 10 2012 * Microcontroller based hand gesture recognition system using flex sensor for disabled people AaishaParveen UMRohitha International Journal of Computer Applications 975 8887 * Neural networks for optimization and signal processing ACochocki RolfUnbehauen 1993 John Wiley & Sons, Inc