Home / Current Issue / Paper 1716845
Multimodal Sensor-Agnostic Gesture- Controlled Gaming Interface with Adaptive Depth Integration
Subject area: Science,Engineering and Technology · Area of research: Computer Science
DOI: https://doi.org/10.64388/IREV9I10-1716845
Abstract
Despite the fact that HCI through gestures has gained significant popularity due to their intuitiveness in recent years, the current gesture recognition systems lack robustness since they rely on hardware support, use a single modality as input, and have few practical applications in real-world settings. In this paper, a gesture recognition algorithm that does not require any hardware support except for the camera sensor is introduced, taking into account RGB data and Microsoft Kinect depth sensors. In particular, the gesture recognizer is based on the detection of the joints of a skeleton and hand landmarks. The mathematical formula used for the proposed multimodal representation is shown below. F_t = [S_t || H_t || D_t]. These experiments are carried out using our own customized gesture dataset that includes 4,000 gestures from 4 categories from 10 different subjects. Recognition accuracy is 91.6% and latency is 34 ms in real-time operation mode, which are significantly better than the outcomes of the baselines based on individual modality.
Keywords
Gesture Recognition, Human Computer Interaction, Multimodal Fusion, LSTM, MediaPipe, Depth Sensing
How to cite this paper
@article{1716845,
author = {Sagar Gangal, Rahul Nelogi, Sachidananda K},
title = {Multimodal Sensor-Agnostic Gesture- Controlled Gaming Interface with Adaptive Depth Integration},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {9},
number = {10},
pages = {3184-3191},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1716845.pdf},
abstract = {Despite the fact that HCI through gestures has gained significant popularity due to their intuitiveness in recent years, the current gesture recognition systems lack robustness since they rely on hardware support, use a single modality as input, and have few practical applications in real-world settings. In this paper, a gesture recognition algorithm that does not require any hardware support except for the camera sensor is introduced, taking into account RGB data and Microsoft Kinect depth sensors. In particular, the gesture recognizer is based on the detection of the joints of a skeleton and hand landmarks. The mathematical formula used for the proposed multimodal representation is shown below. F_t = [S_t || H_t || D_t]. These experiments are carried out using our own customized gesture dataset that includes 4,000 gestures from 4 categories from 10 different subjects. Recognition accuracy is 91.6% and latency is 34 ms in real-time operation mode, which are significantly better than the outcomes of the baselines based on individual modality.},
keywords = {Gesture Recognition, Human Computer Interaction, Multimodal Fusion, LSTM, MediaPipe, Depth Sensing},
month = {April},
doi = {https://doi.org/10.64388/IREV9I10-1716845}
}