Sign language recognition has been an active research field for almost two decades. From early electric signal based sign language recognition to modern day recognition using deep learning techniques, researchers all over the world have tried to automate this task. While sign language recognition could be seen as a naive gesture recognition problem, sign language does not translate to spoken language word by word. In this thesis, this translation issue of Sign Languages is addressed and several solution approaches are demonstrated. We mainly aim to carry out key point detection based sign language recognition (SLR) to infer the meaning that the speaker wants to communicate by generating captions. We work with American Sign Language (ASL), specifically, MS ASL image data set and How2sign data set of ASL videos.