SignAssist: Sign Language Interpreter using Deep Learning

References

[1] Chen, J. CS231A Course Project Final Report Sign Language Recognition with Unsupervised Feature Learning 2012. Available online: http://vision.stanford.edu/teaching/cs231a_autumn1213_internal/project/final/writeup/distributable/Chen_Paper.pdf (accessed on 15 March 2022).

[2] Deepsign: Sign Language Detection and Recognition Using Deep Learning Deep Kothadiya, Chintan Bhatt, Krenil Sapariya, Kevin Patel, Ana-Belén Gil-González, and Juan M. Corchado

[3] Sign Language Recognition using Deep Learning, Authors: Dhruv Sood, https://www.ijraset.com/research-paper/sign-language-recognition-using-deep-learning

[4] Adaloglou, N.; Chatzis, T. A Comprehensive Study on Deep Learning-based Methods for Sign Language Recognition. IEEE Trans.Multimed. 2022,24, 1750–1762.

[5] Jiang, S.; Sun, B.; Wang, L.; Bai, Y.; Li, K.; Fu, Y. Skeleton aware multi-modal sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 21–24 June 2021; pp. 3413–3423.

[6] Wadhawan, A.; Kumar, P. Sign language recognition systems: A decade systematic literature review. Arch. Comput. Methods Eng.2021,28, 785–813.

[7] Yang, D.; Martinez, C.; Visuña, L.; Khandhar, H.; Bhatt, C.; Carretero, J. Detection and Analysis of COVID-19 in medical imagesusing deep learning techniques. Sci. Rep. 2021,11, 19638.

[8] Likhar, P.; Bhagat, N.K.; Rathna, G.N. Deep Learning Methods for Indian Sign Language Recognition. In Proceedings of the 2020IEEE 10th International Conference on Consumer Electronics (ICCE-Berlin), Berlin, Germany, 9–11 November 2020; pp. 1–6.

[9] De Coster, M.; Herreweghe, M.V.; Dambre, J. Sign Language Recognition with Transformer Networks. In Proceedings of the Conference on Language Resources and Evaluation (LREC 2020), Marseille, France, 13–15 May 2020; pp. 6018–6024.

[10] Aparna, C.; Geetha, M. CNN and Stacked LSTM Model for Indian Sign Language Recognition. Commun. Comput. Inf. Sci.2020,1203, 126–134.

[11] Mittal, A.; Kumar, P.; Roy, P.P.; Balasubramanian, R.; Chaudhuri, B.B. A Modified LSTM Model for Continuous Sign LanguageRecognition Using Leap Motion. IEEE Sens. J. 2019,19, 7056–7063.

[12] Yan, S. Understanding LSTM and Its Diagrams. Available online: https://medium.com/mlreview/understanding-lstm- and-its-diagrams-37e2f46f1714

[13] Szegedy, C.; Ioffe, S.; Vanhoucke, V.; Alemi, A.A. Inception-v4, Inception-ResNet and the Impact of Residual Connections onLearning. arXiv 2016, arXiv:1602.07261.

[14] Muthu Mariappan, H.; Gomathi, V. Real-Time Recognition of Indian Sign Language. In Proceedings of the International Conference on Computational Intelligence in Data Science, Haryana, India, 6–7 September 2019.14.Mittal, A.; Kumar, P.; Roy, P.P.; Balasubramanian, R.; Chaudhuri, B.B. A Modified LSTM Model for Continuous Sign LanguageRecognition Using Leap Motion. IEEE Sens. J. 2019,19, 7056–7063.

[15] Camgoz, N.C.; Hadfield, S.; Koller, O.; Ney, H.; Bowden, R. Neural Sign Language Translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2018, Salt Lake City, UT, USA, 18–22 June 2018; IEEE: Piscataway, NJ, USA, 2018.

[16] Shivashankara, S.; Srinath, S. American Sign Language Recognition System: An Optimal Approach. Int. J. Image Graph. SignalProcess. 2018,10, 18–30

[17] Wazalwar, S.S.; Shrawankar, U. Interpretation of sign language into English using NLP techniques. J. Inf. Optim. Sci.2017,38, 895–910.

[18] Manware, A.; Raj, R.; Kumar, A.; Pawar, T. Smart Gloves as a Communication Tool for the Speech Impaired and Hearing Impaired.Int. J. Emerg. Technol. Innov. Res. 2017,4, 78–82.

[19] V. Bheda and D. Radpour, “Using deep convolutional networks for gesture recognition in American sign language,” arXiv:1710.06836, 2017.

[20] Sharma, M.; Pal, R.; Sahoo, A. Indian sign language recognition using neural networks and KNN classifiers. J. Eng. Appl. Sci.2014,9, 1255–1259.

[21] Thomas, J.; McDonagh, D. Shared language: Towards more effective communication. Australas. Med. J. 2013, 6, 46–54

[22] Chen, J.K. Sign Language Recognition with Unsupervised Feature Learning; CS229 Project Final Report; Stanford University: Stanford, CA, USA, 2011.

[23] Mekala, P.; Gao, Y.; Fan, J.; Davari, A. Real-time sign language recognition based on neural network architecture. In Proceedings of the IEEE 43rd Southeastern Symposium on System Theory, Auburn, AL, USA, 14–16 March 2011.

[24] Nandy, A.; Prasad, J.; Mondal, S.; Chakraborty, P.; Nandi, G. Recognition of Isolated Indian Sign Language Gesture in Real Time.Commun. Comput. Inf. Sci. 2010,70, 102–107.

[25] Efthimiou, E.; Fotinea, S.E.; Vogler, C.; Hanke, T.; Glauert, J.; Bowden, R.; Braffort, A.; Collet, C.; Maragos, P.; Segouat, J. Sign

[26] Language Recognition, Generation, and Modelling: A Research Effort with Applications in Deaf Communication. In UAHCI

[27] 2009: Universal Access in Human-Computer Interaction. Addressing Diversity; Lecture Notes in Computer Science; Stephanidis, C., Ed.; Springer: Berlin/Heidelberg, Germany, 2009; Volume 5614._3.

[28] Gao, W.; Fang, G.; Zhao, D.; Chen, Y. A Chinese sign language recognition system based on SOFM/SRN/HMM. Pattern Recognit. 2004, 37, 2389–2402.

[29] Liang, R.-H.; Ouhyoung, M.I. A Sign Language Recognition System Using Hidden Markov Model and Context Sensitive Search. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, Hong Kong, China, 1–4 July 1996.

[30] Chiarelli, B. The Origin of Human Language, Studies in Language Origins; Publisher John Benjamins: Amsterdam, The Netherlands,1991; 35p.

@article{1704651,
      author = {Karan Kharbanda, Utsav Sachdeva, Prof. Dr. Anu Rathee},
      title = {SignAssist: Sign Language Interpreter using Deep Learning},
      journal = {Iconic Research And Engineering Journals},
      year = {2023},
      volume = {6},
      number = {12},
      pages = {387-393},
      issn = {2456-8880},
      url = {https://www.irejournals.com/formatedpaper/1704651.pdf},
      abstract = {Effective communication is essential for individuals to express their ideas and emotions. However, persons with speech or hearing disabilities often face significant communication barriers. To address this issue, deep learning models, specifically LSTM and GRU, are proposed to recognize and translate signs from isolated American Sign Language (ASL) video frames. In this research, transfer learning and data augmentation techniques are utilized to develop a deep learning model for the ASL dataset. The proposed models achieve up to 95% accuracy in recognizing signs from ASL datasets. This research aims to develop a more natural and efficient way of communication for persons with hearing impairments and promote collaboration with people not trained in sign language. Overall, this study demonstrates the potential of deep learning models to reduce communication barriers and promote inclusivity.},
      keywords = {American Sign Language, SignAssist, Deep Learning, Sign Recognition, Gesture Recognition},
      month = {June},
  }