Lip-Reader

We aim to build a web-based app that can perform visual lip reading to help vocally impaired people communicate.

For this project, we are employing HCNN(Hahn-based Convolutional Neural Networks) in learning lip shape to character translation (letters and digits). Hahn orthogonal moments were used for feature extraction and input layer before being passed to the convolution layers

In this version, we use Python 3.9, PyTorch 1.9.0 and dlib 19.22.1

Install Libraries

pip install requirements.txt

Install FFmpeg

For MacOS

Once you have Homebrew installed install ffmpeg from the terminal with the following:

brew install ffmpeg

For Windows

Install FFmpeg on Windows

For Linux

sudo apt install ffmpeg

Download Facial Landmark Shape Predictor

Using Kaggle

kaggle datasets download -d codebreaker619/face-landmark-shape-predictor

Copy file to lip-reading-service/src/ALR/resources

References

Hahn Discrete Orthogonal Polynomials

1. Mesbah, A., Berrahou, A., Hammouchi, H., Berbia, H., Qjidaa, H., & Daoudi, M. (2019). Lip Reading with Hahn Convolutional Neural Networks. Image and Vision Computing. doi:10.1016/j.imavis.2019.04.010

2. Hongqing Zhu, Huazhong Shu, Jian Zhou, Limin Luo, Jean-Louis Coatrieux. Image analysis by discrete orthogonal dual Hahn moments. Pattern Recognition Letters, Elsevier, 2007, 28 (13), pp.1688-1704. ff10.1016/j.patrec.2007.04.013ff. ffinserm-00189813f

3. Nikiforov, Arnold F.; Uvarov, Vasilii B. (1988). Special Functions of Mathematical Physics || . , 10.1007/978-1-4757-1595-8(), –. doi:10.1007/978-1-4757-1595-8

4. Gamma function