Previous talks at the SCCS Colloquium

Ashish Khanal: Hand pose estimation and gesture detection from webcam images

SCCS Colloquium |


Hand pose estimation and gesture detection has been challenging but has garnered a lot of recognition recently as it allows an interesting way for us to interact with our devices.This thesis explores how hand gestures and poses can be detected using webcam images.This can significantly improve user experience and accessibility for people with special needs. This thesis looks at several methods for hand pose estimation such as Hidden Markov models/Statistical method, curve fitting method using Bezier curves, Eigenspace method and compares it with the Neural network method by looking into MediaPipe.Media pipe uses Single shot detection (SSD), an object detection algorithm to detect the palm. The key points from the palm detection model will then be used to detect the whole hand using regression.the Deep learning method using Media pipe. Media pipe uses Single shot detection(SSD), an object detection algorithm to detect the palm and extrapolates the information from this to detect the fingers using regression. We will explore the classical computer vision models for hand pose estimation and compare it with a deep neural network solution called Mediapipe. We will apply the best models and create an application so that we can use it in case there’s . We will look at sift/ surf features for classical models while we will look at ssd for palm detection and use regression to to detect the fingers. And we will also discuss the applications where we use the hand gestures to move the map. We are going to use a combination of two models to detect the hand. We will begin with using single shot detection (SSD) to detect the palm. The key points from the palm detection model will then be used to detect the whole hand using regression. There are classical computer vision techniques available now that detect hand gestures and positions, however, these methods are not as accurate as we would like them to be. Therefore, we want to incorporate Neural network in our posture detection task to help us achieve better accuracy.Additionally, SSD is also faster than other object detection models such as Faster RCNN.Thus, this will significantly improve the usability in its applications.

Master's thesis talk. Ashish is advised by Felix Dietrich.