Current Volume 10
Temporal And Multimodal Enhancement of Semantically Interpretable Attention for Online Hand Gesture Recognition
Abstract
Hand gesture recognition plays significant role in HCI applications and also it has been incorporated in the fields of Virtual Reality (VR), Augmented Reality (AR) and Robotics. Gestures in the real world are continuous and under these conditions, any recognition online would be very difficult since there are two types of gesture classes to be recognized as well as the temporal limits of these classes of gesture. Current state-of-the-art deep learning based on sliding windows and attention mechanisms yield piece-wise and “jagged” predictions resulting in more false alarms as well as incorrect gesture boundaries. Single-modality methods are not beneficial for recognition purposes because they do not return full-fledged gestures of the gesture. It is a reproduction of a cross-attention-based online hand gesture recognition model based on Joint Collection Distance (JCD) and Frame Vector (FV) that reproduces attention-based models. Taking the temporal refinement approach is proposed to reduce the noise and enhance the detection of the boundaries. To extend this into multimodal, visual appearance features of video frames are added in through an existing CNN which can learn structural and appearance features. Experimental findings of IPN Hand Gesture Dataset indicate that the Temporal refinement enhanced the Detection Rate of 0.9184 to 0.9288, False Positives of 39 to 14, and Mean IoU of 0.7375 to 0.8153 which showed that the gesture recognition and temporal localization were more accurate.
Keywords
Hand Gesture Recognition, Multimodal Learning, Attention Mechanism, Temporal Refinement, Joint Collection Distance (JCD), Frame Vector (FV), Deep Learning, Sliding Window.
Citations
IRE Journals:
Saiba Teja Sri, Sanaboina Chandra Sekhar "Temporal And Multimodal Enhancement of Semantically Interpretable Attention for Online Hand Gesture Recognition" Iconic Research And Engineering Journals Volume 9 Issue 12 2026 Page 3661-3674 https://doi.org/10.64388/IREV9I12-1719448
IEEE:
Saiba Teja Sri, Sanaboina Chandra Sekhar
"Temporal And Multimodal Enhancement of Semantically Interpretable Attention for Online Hand Gesture Recognition" Iconic Research And Engineering Journals, 9(12) https://doi.org/10.64388/IREV9I12-1719448
About IRE Journals
IRE Journals is an open access online journals established with an aim to publish high quality of research work in various diciplines. We have more than 6 years in publications. Over 2500+ papers already published.
Important Links
For Authors
Volume 9:
Issue 1, Issue 2, Issue 3, Issue 4, Issue 5, Issue 6, Issue 7, Issue 8, Issue 9, Issue 10, Issue 11, Issue 12Volume 8:
Issue 1, Issue 2, Issue 3, Issue 4, Issue 5, Issue 6, Issue 7, Issue 8, Issue 9, Issue 10, Issue 11, Issue 12Volume 7:
Issue 1, Issue 2, Issue 3, Issue 4, Issue 5, Issue 6, Issue 7, Issue 8, Issue 9, Issue 10, Issue 11, Issue 12Volume 6:
Issue 1, Issue 2, Issue 3, Issue 4, Issue 5, Issue 6, Issue 7, Issue 8, Issue 9, Issue 10, Issue 11, Issue 12Volume 5:
Issue 1, Issue 2, Issue 3, Issue 4, Issue 5, Issue 6, Issue 7, Issue 8, Issue 9, Issue 10, Issue 11, Issue 12Volume 4:
Issue 1, Issue 2, Issue 3, Issue 4, Issue 5, Issue 6, Issue 7, Issue 8, Issue 9, Issue 10, Issue 11, Issue 12Volume 3:
Issue 1, Issue 2, Issue 3, Issue 4, Issue 5, Issue 6, Issue 7, Issue 8, Issue 9, Issue 10, Issue 11, Issue 12