Text Generation

Presentation: Slides

• Word2Vec
• Universal Sentence Encoder
• Sentiment
• LSTM/CharRNN
• GPT-2
• How to train a GPT-2 with our own dataset
• GPT-3
• ChatGPT
• GPT-4

Exploring sessions

• Github copilot
• ChatGPT
• GPT 3/4 API
• Word2Vec with ml5js
• Sentiment with ml5js
• CharRNN with ml5js
  • Train your own CharRNN, instruction
• GPT-2 with Runway
  • Train your own GPT-2
    • Training GPT-2 in Runway, video
    • Training GPT-2 with Colab, expires in 12 hours, remember to download the model before the session expires.
    • GPT-2 playground with Google Colab

Where to find data

• Free eBooks - Project Gutenberg
• All the news - Kaggle
• Wikipedia API, example query: https://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&exintro&explaintext&redirects=1&titles=Stack%20Overflow
• Enron (600,000+ employee emails made public during fraud investigation)
• Scrapping Reddit data

Code Examples

• https://editor.p5js.org/yining/sketches/cnlmIOoL9
• Sentiment
• Word2Vec
• CharRNN_Interactive
• CharRNN_Text
• CharRNN_Text_Stateful
• GPT-2 in p5 web editor, need to run Runway GPT-2 model at localhost:8000

Resources

• Training text generator with Runway
• Better Language Models and Their Implications: GPT-2 Blog
• Talk to Transformer: GPT-2 Demo
• Hugging face transformers
• CharRNN from ml5js
• Train CharRNN ml5js model
• Learning about Transformer models on Huggingface

RNN Related Projects:

• 🍿 Sunspring
• 🎨 Double Agent by Simon Biggs (Drawing)
• 🎨 Four Experiments in Handwriting with a Neural Network (Drawing)
• 📖 10 things artificial intelligence did in 2018 by Janelle Shane (Text)
• 📖 Writing with the Machine
• 🎶 Magenta: Make Music and Art Using Machine Learning
• 🎨 Handwriting Generation with RNN and p5.js
• 🎶 RNN for generating Baroque Music video
• 📖 Let's Read a Story by Itay Niv

Homework