Skip to content

Commit

Permalink
add all links to concepts
Browse files Browse the repository at this point in the history
  • Loading branch information
pfliu-nlp committed Jan 13, 2020
1 parent aab33e5 commit 18feecd
Showing 1 changed file with 98 additions and 97 deletions.
195 changes: 98 additions & 97 deletions concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,163 +24,164 @@ and no other tags need to be applied.
* Adam: [`optim-adam`](http://pfliu.com/pl-nlp2019/bs/optim-adam.html) (implies `optim-sgd`)
* Adagrad: [`optim-adagrad`](http://pfliu.com/pl-nlp2019/bs/optim-adagrad.html) (implies `optim-sgd`)
* Adadelta: [`optim-adadelta`](http://pfliu.com/pl-nlp2019/bs/optim-adadelta.html) (implies `optim-sgd`)
* Adam with Specialized Transformer Learning Rate ("Noam" Schedule): `optim-noam` (implies `optim-adam`)
* SGD with Momentum: `optim-momentum` (implies `optim-sgd`)
* AMS: `optim-amsgrad` (implies `optim-sgd`)
* Adam with Specialized Transformer Learning Rate ("Noam" Schedule): [`optim-noam`](http://pfliu.com/pl-nlp2019/bs/optim-noam.html) (implies `optim-adam`)
* SGD with Momentum: [`optim-momentum`](http://pfliu.com/pl-nlp2019/bs/optim-momentum.html) (implies `optim-sgd`)
* AMS: [`optim-amsgrad`](http://pfliu.com/pl-nlp2019/bs/optim-amsgrad.html) (implies `optim-sgd`)

### Initialization

* Glorot/Xavier Initialization: `init-glorot`
* He Initialization: `init-he`
* Glorot/Xavier Initialization: [`init-glorot`](http://pfliu.com/pl-nlp2019/bs/init-glorot.html)
* He Initialization: [`init-he`](http://pfliu.com/pl-nlp2019/bs/init-he.html)

### Regularization

* Dropout: `reg-dropout`
* Word Dropout: `reg-worddropout` (implies `reg-dropout`)
* Norm (L1/L2) Regularization: `reg-norm`
* Early Stopping: `reg-stopping`
* Patience: `reg-patience` (implies `reg-stopping`)
* Weight Decay: `reg-decay`
* Label Smoothing: `reg-labelsmooth`
* Dropout: [`reg-dropout`]
* Word Dropout: [`reg-worddropout`](http://pfliu.com/pl-nlp2019/bs/reg-worddropout.html) (implies `reg-dropout`)
* Norm (L1/L2) Regularization: [`reg-norm`](http://pfliu.com/pl-nlp2019/bs/reg-norm.html)
* Early Stopping: [`reg-stopping`](http://pfliu.com/pl-nlp2019/bs/reg-stopping.html)
* Patience: [`reg-patience`](http://pfliu.com/pl-nlp2019/bs/reg-patience.html) (implies `reg-stopping`)
* Weight Decay: [`reg-decay`](http://pfliu.com/pl-nlp2019/bs/reg-decay.html)
* Label Smoothing: [`reg-labelsmooth`](http://pfliu.com/pl-nlp2019/bs/reg-labelsmooth.html)

### Normalization

* Layer Normalization: `norm-layer`
* Batch Normalization: `norm-batch`
* Gradient Clipping: `norm-gradient`
* Layer Normalization: [`norm-layer`](http://pfliu.com/pl-nlp2019/bs/norm-layer.html)
* Batch Normalization: [`norm-batch`](http://pfliu.com/pl-nlp2019/bs/norm-batch.html)
* Gradient Clipping: [`norm-gradient`](http://pfliu.com/pl-nlp2019/bs/norm-gradient.html)

### Loss Functions (other than cross-entropy)

* Canonical Correlation Analysis (CCA): `loss-cca`
* Singular Value Decomposition (SVD): `loss-svd`
* Margin-based Loss Functions: `loss-margin`
* Contrastive Loss: `loss-cons`
* Noise Contrastive Estimation (NCE): `loss-nce` (implies `loss-cons`)
* Triplet loss: `loss-triplet` (implies `loss-cons`)
* Canonical Correlation Analysis (CCA): [`loss-cca`](http://pfliu.com/pl-nlp2019/bs/loss-cca.html)
* Singular Value Decomposition (SVD): [`loss-svd`](http://pfliu.com/pl-nlp2019/bs/loss-svd.html)
* Margin-based Loss Functions: [`loss-margin`](http://pfliu.com/pl-nlp2019/bs/loss-margin.html)
* Contrastive Loss: [`loss-cons`](http://pfliu.com/pl-nlp2019/bs/loss-cons.html)
* Noise Contrastive Estimation (NCE): [`loss-nce`](http://pfliu.com/pl-nlp2019/bs/loss-nce.html) (implies `loss-cons`)
* Triplet loss: [`loss-triplet`](http://pfliu.com/pl-nlp2019/bs/loss-triplet.html) (implies `loss-cons`)

### Training Paradigms

* Multi-task Learning (MTL): `train-mtl`
* Multi-lingual Learning (MLL): `train-mll` (implies `train-mtl`)
* Transfer Learning: `train-transfer`
* Active Learning: `train-active`
* Data Augmentation: `train-augment`
* Multi-task Learning (MTL): [`train-mtl`](http://pfliu.com/pl-nlp2019/bs/train-mtl.html)
* Multi-lingual Learning (MLL): [`train-mll`](http://pfliu.com/pl-nlp2019/bs/train-mll.html) (implies `train-mtl`)
* Transfer Learning: [`train-transfer`](http://pfliu.com/pl-nlp2019/bs/train-transfer.html)
* Active Learning: [`train-active`](http://pfliu.com/pl-nlp2019/bs/train-active.html)
* Data Augmentation: [`train-augment`](http://pfliu.com/pl-nlp2019/bs/train-augment.html)


## Sequence Modeling Architectures

### Activation Functions

* Hyperbolic Tangent (tanh): `activ-tanh`
* Rectified Linear Units (RelU): `activ-relu`
* Hyperbolic Tangent (tanh): [`activ-tanh`](http://pfliu.com/pl-nlp2019/bs/activ-tanh.html)
* Rectified Linear Units (RelU): [`activ-relu`](http://pfliu.com/pl-nlp2019/bs/activ-relu.html)

### Pooling Operations

* Max Pooling: `pool-max`
* Mean Pooling: `pool-mean`
* k-Max Pooling: `pool-kmax`
* Max Pooling: [`pool-max`](http://pfliu.com/pl-nlp2019/bs/pool-max.html)
* Mean Pooling: [`pool-mean`](http://pfliu.com/pl-nlp2019/bs/pool-mean.html)
* k-Max Pooling: [`pool-kmax`](http://pfliu.com/pl-nlp2019/bs/pool-kmax.html)

### Recurrent Architectures

* Recurrent Neural Network (RNN): `arch-rnn`
* Bi-directional Recurrent Neural Network (Bi-RNN): `arch-birnn` (implies `arch-rnn`)
* Long Short-term Memory (LSTM): `arch-lstm` (implies `arch-rnn`)
* Bi-directional Long Short-term Memory (LSTM): `arch-bilstm` (implies `arch-birnn`, `arch-lstm`)
* Gated Recurrent Units (GRU): `arch-gru` (implies `arch-rnn`)
* Bi-directional Gated Recurrent Units (GRU): `arch-bigru` (implies `arch-birnn`, `arch-gru`)
* Recursive Neural Network (RecNN): `arch-recnn`
* Tree-structured Long Short-term Memory (TreeLSTM): `arch-treelstm` (implies `arch-recnn`)
* Graph Neural Network (GNN): `arch-gnn`
* Graph Convolutional Neural Network (GCNN): `arch-gcnn` (implies `arch-gnn`)
* Recurrent Neural Network (RNN): [`arch-rnn`](http://pfliu.com/pl-nlp2019/bs/arch-rnn.html)
* Bi-directional Recurrent Neural Network (Bi-RNN): [`arch-birnn`](http://pfliu.com/pl-nlp2019/bs/arch-birnn.html) (implies `arch-rnn`)
* Long Short-term Memory (LSTM): [`arch-lstm`](http://pfliu.com/pl-nlp2019/bs/arch-lstm.html) (implies `arch-rnn`)
* Bi-directional Long Short-term Memory (LSTM): [`arch-bilstm`](http://pfliu.com/pl-nlp2019/bs/arch-bilstm.html) (implies `arch-birnn`, `arch-lstm`)
* Gated Recurrent Units (GRU): [`arch-gru`](http://pfliu.com/pl-nlp2019/bs/arch-gru.html) (implies `arch-rnn`)
* Bi-directional Gated Recurrent Units (GRU): [`arch-bigru`](http://pfliu.com/pl-nlp2019/bs/arch-bigru.html) (implies `arch-birnn`, `arch-gru`)
* Recursive Neural Network (RecNN): [`arch-recnn`](http://pfliu.com/pl-nlp2019/bs/arch-recnn.html)
* Tree-structured Long Short-term Memory (TreeLSTM): [`arch-treelstm`](http://pfliu.com/pl-nlp2019/bs/arch-treelstm.html) (implies `arch-recnn`)
* Graph Neural Network (GNN): [`arch-gnn`](http://pfliu.com/pl-nlp2019/bs/arch-gnn.html)
* Graph Convolutional Neural Network (GCNN): [`arch-gcnn`](http://pfliu.com/pl-nlp2019/bs/arch-gcnn.html) (implies `arch-gnn`)

### Other Sequential Architectures

* Convolutional Neural Networks (CNN): `arch-cnn`
* Attention: `arch-att`
* Self Attention: `arch-selfatt` (implies `arch-att`)
* Convolutional Neural Networks (CNN): [`arch-cnn`](http://pfliu.com/pl-nlp2019/bs/arch-cnn.html)
* Attention: [`arch-att`](http://pfliu.com/pl-nlp2019/bs/arch-att.html)
* Self Attention: [`arch-selfatt`](http://pfliu.com/pl-nlp2019/bs/arch-selfatt.html) (implies `arch-att`)

### Architectural Techniques

* Residual Connections (ResNet): `arch-residual`
* Gating Connections, Highway Connections: `arch-gating`
* Memory: `arch-memo`
* Copy Mechanism: `arch-copy`
* Bilinear, Biaffine Models: `arch-bilinear`
* Residual Connections (ResNet): [`arch-residual`](http://pfliu.com/pl-nlp2019/bs/arch-residual.html)
* Gating Connections, Highway Connections: [`arch-gating`](http://pfliu.com/pl-nlp2019/bs/arch-gating.html)
* Memory: [`arch-memo`](http://pfliu.com/pl-nlp2019/bs/arch-memo.html)
* Copy Mechanism: [`arch-copy`](http://pfliu.com/pl-nlp2019/bs/arch-copy.html)
* Bilinear, Biaffine Models: [`arch-bilinear`](http://pfliu.com/pl-nlp2019/bs/arch-bilinear.html)

### Standard Composite Architectures

* Transformer: `arch-transformer` (implies `arch-selfatt`, `arch-residual`, `arch-layernorm`, `optim-noam`)
* Transformer: [`arch-transformer`](http://pfliu.com/pl-nlp2019/bs/arch-transformer.html) (implies `arch-selfatt`, `arch-residual`, `arch-layernorm`, `optim-noam`)


## Model Combination

* Ensembling: `comb-ensemble`
* Ensembling: [`comb-ensemble`](http://pfliu.com/pl-nlp2019/bs/comb-ensemble.html)

## Search Algorithms

* Greedy Search: `search-greedy`
* Beam Search: `search-beam`
* A* Search: `search-astar`
* Viterbi Algorithm: `search-viterbi`
* Ancestral Sampling: `search-sampling`
* Gumbel Max: `search-gumbel` (implies `search-sampling`)
* Greedy Search: [`search-greedy`](http://pfliu.com/pl-nlp2019/bs/search-greedy.html)
* Beam Search: [`search-beam`](http://pfliu.com/pl-nlp2019/bs/search-beam.html)
* A* Search: [`search-astar`](http://pfliu.com/pl-nlp2019/bs/search-astar.html)
* Viterbi Algorithm: [`search-viterbi`](http://pfliu.com/pl-nlp2019/bs/search-viterbi.html)
* Ancestral Sampling: [`search-sampling`](http://pfliu.com/pl-nlp2019/bs/search-sampling.html)
* Gumbel Max: [`search-gumbel`](http://pfliu.com/pl-nlp2019/bs/search-gumbel.html) (implies `search-sampling`)

## Prediction Tasks

* Text Classification (text -> label): `task-textclass`
* Text Pair Classification (two texts -> label: `task-textpair`
* Sequence Labeling (text -> one label per token): `task-seqlab`
* Extractive Summarization (text -> subset of text): `task-extractive` (implies `text-seqlab`)
* Span Labeling (text -> labels on spans): `task-spanlab`
* Language Modeling (predict probability of text): `task-lm`
* Conditioned Language Modeling (some input -> text): `task-condlm` (implies `task-lm`)
* Sequence-to-sequence Tasks (text -> text, including MT): `task-seq2seq` (implies `task-condlm`)
* Cloze-style Prediction, Masked Language Modeling (right and left context -> word): `task-cloze`
* Context Prediction (as in word2vec) (word -> right and left context): `task-context`
* Relation Prediction (text -> graph of relations between words, including dependency parsing): `task-relation`
* Tree Prediction (text -> tree, including syntactic and some semantic semantic parsing): `task-tree`
* Graph Prediction (text -> graph not necessarily between nodes): `task-graph`
* Lexicon Induction/Embedding Alignment (text/embeddings -> bi- or multi-lingual lexicon): `task-lexicon`
* Word Alignment (parallel text -> alignment between words): `task-alignment`
* Text Classification (text -> label): [`task-textclass`](http://pfliu.com/pl-nlp2019/bs/task-textclass.html)
* Text Pair Classification (two texts -> label: [`task-textpair`](http://pfliu.com/pl-nlp2019/bs/task-textpair.html)
* Sequence Labeling (text -> one label per token): [`task-seqlab`](http://pfliu.com/pl-nlp2019/bs/task-seqlab.html)
* Extractive Summarization (text -> subset of text): [`task-extractive`](http://pfliu.com/pl-nlp2019/bs/task-extractive.html) (implies `text-seqlab`)
* Span Labeling (text -> labels on spans): [`task-spanlab`](http://pfliu.com/pl-nlp2019/bs/task-spanlab.html)
* Language Modeling (predict probability of text): [`task-lm`](http://pfliu.com/pl-nlp2019/bs/task-lm.html)
* Conditioned Language Modeling (some input -> text): [`task-condlm`](http://pfliu.com/pl-nlp2019/bs/task-condlm.html) (implies `task-lm`)
* Sequence-to-sequence Tasks (text -> text, including MT): [`task-seq2seq`](http://pfliu.com/pl-nlp2019/bs/task-seq2seq.html) (implies `task-condlm`)
* Cloze-style Prediction, Masked Language Modeling (right and left context -> word): [`task-cloze`](http://pfliu.com/pl-nlp2019/bs/task-cloze.html)
* Context Prediction (as in word2vec) (word -> right and left context): [`task-context`](http://pfliu.com/pl-nlp2019/bs/task-context.html)
* Relation Prediction (text -> graph of relations between words, including dependency parsing): [`task-relation`](http://pfliu.com/pl-nlp2019/bs/task-relation.html)
* Tree Prediction (text -> tree, including syntactic and some semantic semantic parsing): [`task-tree`
* Graph Prediction (text -> graph not necessarily between nodes): [`task-graph`](http://pfliu.com/pl-nlp2019/bs/task-graph.html)
* Lexicon Induction/Embedding Alignment (text/embeddings -> bi- or multi-lingual lexicon): [`task-lexicon`](http://pfliu.com/pl-nlp2019/bs/task-lexicon.html)
* Word Alignment (parallel text -> alignment between words): [`task-alignment`](http://pfliu.com/pl-nlp2019/bs/task-alignment.html)

## Composite Pre-trained Embedding Techniques

* word2vec: `pre-word2vec` (implies `arch-cbow`, `task-cloze`, `task-context`)
* GloVe: `pre-glove`
* Paragraph Vector (ParaVec): `pre-paravec`
* Skip-thought: `pre-skipthought` (implies `arch-lstm`, `task-seq2seq`)
* ELMo: `pre-elmo` (implies `arch-bilstm`, `task-lm`)
* BERT: `pre-bert` (implies `arch-transformer`, `task-cloze`, `task-textpair`)
* word2vec: [`pre-word2vec`](http://pfliu.com/pl-nlp2019/bs/pre-word2vec.html) (implies `arch-cbow`, `task-cloze`, `task-context`)
* GloVe: [`pre-glove`](http://pfliu.com/pl-nlp2019/bs/pre-glove.html)
* Paragraph Vector (ParaVec): [`pre-paravec`](http://pfliu.com/pl-nlp2019/bs/pre-paravec.html)
* Skip-thought: [`pre-skipthought`](http://pfliu.com/pl-nlp2019/bs/pre-skipthought.html) (implies `arch-lstm`, `task-seq2seq`)
* ELMo: [`pre-elmo`](http://pfliu.com/pl-nlp2019/bs/pre-elmo.html) (implies `arch-bilstm`, `task-lm`)
* BERT: [`pre-bert`](http://pfliu.com/pl-nlp2019/bs/pre-bert.html) (implies `arch-transformer`, `task-cloze`, `task-textpair`)

## Structured Models/Algorithms

* Hidden Markov Models (HMM): `struct-hmm`
* Conditional Random Fields (CRF): `struct-crf`
* Context-free Grammar (CFG): `struct-cfg`
* Combinatorial Categorical Grammar (CCG): `struct-ccg`
* Hidden Markov Models (HMM): [`struct-hmm`](http://pfliu.com/pl-nlp2019/bs/struct-hmm.html)
* Conditional Random Fields (CRF): [`struct-crf`](http://pfliu.com/pl-nlp2019/bs/struct-crf.html)
* Context-free Grammar (CFG): [`struct-cfg`](http://pfliu.com/pl-nlp2019/bs/struct-cfg.html)
* Combinatorial Categorical Grammar (CCG): [`struct-ccg`](http://pfliu.com/pl-nlp2019/bs/struct-ccg.html)

## Relaxation/Training Methods for Non-differentiable Functions

* Complete Enumeration: `nondif-enum`
* Straight-through Estimator: `nondif-straightthrough`
* Gumbel Softmax: `nondif-gumbelsoftmax`
* Minimum Risk Training: `nondif-minrisk`
* REINFORCE: `nondif-reinforce`
* Complete Enumeration: [`nondif-enum`](http://pfliu.com/pl-nlp2019/bs/nondif-enum.html)
* Straight-through Estimator: [`nondif-straightthrough`](http://pfliu.com/pl-nlp2019/bs/nondif-straightthrough.html)
* Gumbel Softmax: [`nondif-gumbelsoftmax`](http://pfliu.com/pl-nlp2019/bs/nondif-gumbelsoftmax.html)
* Minimum Risk Training: [`nondif-minrisk` ](http://pfliu.com/pl-nlp2019/bs/nondif-minrisk.html)
* REINFORCE: [`nondif-reinforce` ](http://pfliu.com/pl-nlp2019/bs/nondif-reinforce.html)

## Adversarial Methods

* Generative Adversarial Networks (GAN): `adv-gan`
* Adversarial Feature Learning: `adv-feat`
* Adversarial Examples: `adv-examp`
* Adversarial Training: `adv-train` (implies `adv-examp`)
* Generative Adversarial Networks (GAN): [`adv-gan`](http://pfliu.com/pl-nlp2019/bs/adv-gan.html)
* Adversarial Feature Learning: [`adv-feat`](http://pfliu.com/pl-nlp2019/bs/adv-feat.html)
* Adversarial Examples: [`adv-examp`](http://pfliu.com/pl-nlp2019/bs/adv-examp.html)
* Adversarial Training: [`adv-train`](http://pfliu.com/pl-nlp2019/bs/adv-train.html) (implies `adv-examp`)

## Latent Variable Models

* Variational Auto-encoder (VAE): `latent-vae`
* Topic Model: `latent-topic`
* Variational Auto-encoder (VAE): [`latent-vae`](http://pfliu.com/pl-nlp2019/bs/latent-vae.html)
* Topic Model: [`latent-topic`](http://pfliu.com/pl-nlp2019/bs/latent-topic.html)

## Meta Learning

* Meta-learning Initialization: `meta-init`
* Meta-learning Optimizers: `meta-optim`
* Meta-learning Loss functions: `meta-loss`
* Neural Architecture Search: `meta-arch`
* Meta-learning Initialization: [`meta-init`](http://pfliu.com/pl-nlp2019/bs/meta-init.html)
* Meta-learning Optimizers: [`meta-optim`](http://pfliu.com/pl-nlp2019/bs/meta-optim.html)
* Meta-learning Loss functions: [`meta-loss`](http://pfliu.com/pl-nlp2019/bs/meta-loss.html)
* Neural Architecture Search: [`meta-arch`](http://pfliu.com/pl-nlp2019/bs/meta-arch.html)

0 comments on commit 18feecd

Please sign in to comment.