nlp-tutorial

nlp-tutorial is a tutorial for who is studying NLP(Natural Language Processing) using TensorFlow and Pytorch.

• Most of the models in NLP were implemented with less than 100 lines of code.(except comments or blank lines)

Curriculum

1. Basic Embedding Model - (Example Purpose)

• 1-1. NNLM(Neural Network Language Model) - Predict Next Word
  • Paper - A Neural Probabilistic Language Model(2003)
• 1-2. Word2Vec(Skip-gram) - Embedding Words and Show Graph
  • Paper - Distributed Representations of Words and Phrases and their Compositionality(2013)
• 1-3. FastText(Application Level) - Sentence Classification
  • Site : http://fasttext.cc
  • Paper : Bag of Tricks for Efficient Text Classification(2016)
  • Usage : Google Colab

2. CNN(Convolutional Neural Network)

• 2-1. TextCNN - Binary Sentiment Classification
  • Paper - Convolutional Neural Networks for Sentence Classification(2014)
• 2-2. DCNN(Dynamic Convolutional Neural Network)

3. RNN(Recurrent Neural Network)

• 3-1. TextRNN - Predict Next Step
  • Paper - Finding Structure in Time(1990)
• 3-2. TextLSTM - Autocomplete
  • Paper - LONG SHORT-TERM MEMORY(1997)
• 3-3. Bi-LSTM - Predict Long Next Step Word

4. Attention Mechanism

• 4-1. Sequence2Sequence - Change Word
  • Paper - Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation(2014)
• 4-2. Seq2Seq with Attention - Translate
  • Paper - Neural Machine Translation by Jointly Learning to Align and Translate(2014)
• 4-3. Bi-LSTM with Attention - Binary Sentiment Classification
• 4-4. The Transformer - Translate
  • Paper - Attention Is All You Need(2017)

5. Model based on Transformer

• 6-1. BERT

Depencencies

• Python 2.7 or 3.5+
• Tensorflow 1.12.0
• Pytorch 0.4.1

Author

• Tae Hwan Jung(Jeff Jung) @graykode
• Email : [email protected]