Pytorch-Handwritten-Mathematical-Expression-Recognition

Update in 2019/3/27:

1、Now, this program is running in Pytorch0.4.
2、Use pretrained Densenet weights.(You can download here)
3、Solve some BUGs. (sacc is always 0).
4、Improve the accuracy of recognition.

• WER loss: 24.097%
  ExpRate: 32.216%

5、Any discussion and questions are welcome to contact me ([email protected]).

Original

This program uses Attention and Coverage to realize HMER (HandWritten Mathematical Expression Recognition) and written by Hongyu Wang.

Notice:
This program is writting with reference to the work of Dr. Jianshu Zhang from USTC.

@article{zhang2017watch,
  title={Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition},
  author={Zhang, Jianshu and Du, Jun and Zhang, Shiliang and Liu, Dan and Hu, Yulong and Hu, Jinshui and Wei, Si and Dai, Lirong},
  journal={Pattern Recognition},
  volume={71},
  pages={196--206},
  year={2017},
  publisher={Elsevier}
}

@article{zhang2018multi,
  title={Multi-Scale Attention with Dense Encoder for Handwritten Mathematical Expression Recognition},
  author={Zhang, Jianshu and Du, Jun and Dai, Lirong},
  journal={arXiv preprint arXiv:1801.03530},
  year={2018}
}

Requirements

Python 3.6
Pytorch 0.4 (This is important!)

Training and Testing

1. Install Requirements.
2. Decompression files in off_image_train and off_image_test, and this will be your training data and testing data.
3. python 'gen_pkl.py'. This python file will compress your training pictures or testing pictures into a '.pkl' file. Moreover, you should write the correct location of your data files.
4. python 'Train.py' for training.
5. python 'Densenet_testway.py' for testing.

Experiment

• This model is testing in CROHME14 dataset.

• The best result of this model is:

• WER loss: 25.715%
  ExpRate: 28.216%

• The HMER V2.0

• Visualization of results

• Visualization of Attention

Input image

step by step