Prompt Engineering Guide

This guide contains a set of recent papers, learning guides, and tools related to prompt engineering. The repo is intended as a research and educational reference for practitioners and developers.

Announcements:

• 📣 Full lecture + notebook + exercises on the ~15th of Feb (announcement will happen on Twitter)

• 📣 Examples and explainers coming soon!

• Join our Discord to discuss more about prompt engineering

---

Table of Contents

• Guides
• Papers
• Tools & Libraries
• Datasets
• Blog, Guides, Tutorials and Other Readings

---

Guides 🔮

The following are a set of guides on prompt engineering developed by us. Guides are work in progress.

• Prompt Engineering - Introduction
• Prompt Engineering - Basic Usage
• Prompt Engineering - Advanced Usage
• Prompt Engineering - Adversarial Prompts
• Prompt Engineering - Miscellaneous Topics

Papers

(Sorted by Release Date)

• Surveys / Overviews:

  • A Survey for In-context Learning (Dec 2022)
  • Towards Reasoning in Large Language Models: A Survey (Dec 2022)
  • Emergent Abilities of Large Language Models (Jun 2022)
  • A Taxonomy of Prompt Modifiers for Text-To-Image Generation (Apr 2022)
  • Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (Jul 2021)

• Approaches/Techniques:

  • Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery (Feb 2023)
  • Multimodal Chain-of-Thought Reasoning in Language Models (Feb 2023)
  • Large Language Models Can Be Easily Distracted by Irrelevant Context (Feb 2023)
  • Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models (Feb 2023)
  • Progressive Prompts: Continual Learning for Language Models (Jan 2023)
  • Batch Prompting: Efficient Inference with LLM APIs (Jan 2023)
  • On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (Dec 2022)
  • Constitutional AI: Harmlessness from AI Feedback (Dec 2022)
  • Successive Prompting for Decomposing Complex Questions (Dec 2022)
  • Discovering Language Model Behaviors with Model-Written Evaluations (Dec 2022)
  • Structured Prompting: Scaling In-Context Learning to 1,000 Examples (Dec 2022)
  • PAL: Program-aided Language Models (Nov 2022)
  • Large Language Models Are Human-Level Prompt Engineers (Nov 2022)
  • Ignore Previous Prompt: Attack Techniques For Language Models (Nov 2022)
  • Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods (Nov 2022)
  • Teaching Algorithmic Reasoning via In-context Learning (Nov 2022)
  • Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference (Nov 2022)
  • Ask Me Anything: A simple strategy for prompting language models (Oct 2022)
  • ReAct: Synergizing Reasoning and Acting in Language Models (Oct 2022)
  • Prompting GPT-3 To Be Reliable (Oct 2022)
  • Decomposed Prompting: A Modular Approach for Solving Complex Tasks (Oct 2022)
  • Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought (Oct 2022)
  • Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples (Sep 2022)
  • Promptagator: Few-shot Dense Retrieval From 8 Examples (Sep 2022)
  • On the Advance of Making Language Models Better Reasoners (June 2022)
  • Large Language Models are Zero-Shot Reasoners (May 2022)
  • MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning (May 2022)
  • Toxicity Detection with Generative Prompt-based Inference (May 2022)
  • The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning (May 2022)
  • A Taxonomy of Prompt Modifiers for Text-To-Image Generation (Apr 2022)
  • PromptChainer: Chaining Large Language Model Prompts through Visual Programming (Mar 2022)
  • Self-Consistency Improves Chain of Thought Reasoning in Language Models (March 2022)
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (Feb 2022)
  • Chain of Thought Prompting Elicits Reasoning in Large Language Models (Jan 2022)
  • Show Your Work: Scratchpads for Intermediate Computation with Language Models (Nov 2021)
  • Generated Knowledge Prompting for Commonsense Reasoning (Oct 2021)
  • Reframing Instructional Prompts to GPTk's Language (Sep 2021)
  • Design Guidelines for Prompt Engineering Text-to-Image Generative Models (Sep 2021)
  • Making Pre-trained Language Models Better Few-shot Learners (Aug 2021)
  • Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity (April 2021)
  • BERTese: Learning to Speak to BERT (April 2021)
  • The Power of Scale for Parameter-Efficient Prompt Tuning (April 2021)
  • Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (Feb 2021)
  • Calibrate Before Use: Improving Few-Shot Performance of Language Models (Feb 2021)
  • Prefix-Tuning: Optimizing Continuous Prompts for Generation (Jan 2021)
  • AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (Oct 2020)
  • Language Models are Few-Shot Learners (May 2020)
  • How Can We Know What Language Models Know? (July 2020)

• Applications:

  • Prompting for Multimodal Hateful Meme Classification (Feb 2023)
  • PLACES: Prompting Language Models for Social Conversation Synthesis (Feb 2023)
  • Commonsense-Aware Prompting for Controllable Empathetic Dialogue Generation (Feb 2023)
  • Crawling the Internal Knowledge-Base of Language Models (Jan 2023)
  • Legal Prompt Engineering for Multilingual Legal Judgement Prediction (Dec 2022)
  • Investigating Prompt Engineering in Diffusion Models (Nov 2022)
  • Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (Sep 2022)
  • Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language (Oct 2022)
  • Piloting Copilot and Codex: Hot Temperature, Cold Prompts, or Black Magic? (Oct 2022)
  • Plot Writing From Scratch Pre-Trained Language Models (July 2022)

• Collections:

  • Chain-of-ThoughtsPapers
  • Papers with Code
  • Prompt Papers

Tools & Libraries

(Sorted by Name)

• AI Test Kitchen
• betterprompt
• DreamStudio
• DUST
• Dyno
• EveryPrompt
• GPT Index
• GPTTools
• hwchase17/adversarial-prompts
• Interactive Composition Explorer
• LangChain
• LearnGPT
• Lexica
• loom
• Metaprompt
• OpenAI Playground
• OpenPrompt
• Playground
• Prodia
• Prompt Base
• Prompt Engine
• Prompt Generator for OpenAI's DALL-E 2
• Promptable
• PromptInject
• Prompts.ai
• PromptSource
• Scale SpellBook
• sharegpt
• ThoughtSource
• Visual Prompt Builder

Datasets

(Sorted by Name)

• Anthropic's Red Team dataset, (paper)
• Awesome ChatGPT Prompts
• DiffusionDB
• Midjourney Prompts
• P3 - Public Pool of Prompts
• PartiPrompts
• Real Toxicity Prompts
• Stable Diffusion Dataset
• WritingPrompts

Blog, Guides, Tutorials and Other Readings

(Sorted by Name)

• 3 Principles for prompt engineering with GPT-3
• A beginner-friendly guide to generative language models - LaMBDA guide
• A Complete Introduction to Prompt Engineering for Large Language Models
• A Generic Framework for ChatGPT Prompt Engineering
• AI Content Generation
• Awesome ChatGPT Prompts
• Best 100+ Stable Diffusion Prompts
• Best practices for prompt engineering with OpenAI API
• ChatGPT, AI and GPT-3 Apps and use cases
• CMU Advanced NLP 2022: Prompting
• Curtis64's set of prompt gists
• DALL·E 2 Prompt Engineering Guide
• DALLE Prompt Book
• Diffusion Models: A Practical Guide
• Exploiting GPT-3 Prompts
• Exploring Prompt Injection Attacks
• Extrapolating to Unnatural Language Processing with GPT-3's In-context Learning: The Good, the Bad, and the Mysterious
• Generative AI with Cohere: Part 1 - Model Prompting
• Giving GPT-3 a Turing Test
• GPT3 and Prompts: A quick primer
• How to Draw Anything
• How to get images that don't suck
• How to write good prompts
• Language Models and Prompt Engineering: Systematic Survey of Prompting Methods in NLP
• Learn Prompting
• Methods of prompt programming
• Mysteries of mode collapse
• NLP for Text-to-Image Generators: Prompt Analysis
• Notes for Prompt Engineering by sw-yx
• OpenAI Cookbook
• OpenAI Prompt Examples for several applications
• Pretrain, Prompt, Predict - A New Paradigm for NLP
• Prompt Engineering 101 - Introduction and resources
• Prompt Engineering 101: Autocomplete, Zero-shot, One-shot, and Few-shot prompting
• Prompt Engineering 101
• Prompt Engineering - A new profession ?
• Prompt Engineering by co:here
• Prompt Engineering by Microsoft
• Prompt engineering davinci-003 on our own docs for automated support (Part I)
• Prompt Engineering Guide: How to Engineer the Perfect Prompts
• Prompt Engineering in GPT-3
• Prompt Engineering Template
• Prompt Engineering Topic by GitHub
• Prompt Engineering: From Words to Art
• Prompt injection attacks against GPT-3
• Prompt injection to read out the secret OpenAI API key
• Prompting in NLP: Prompt-based zero-shot learning
• Prompting Methods with Language Models and Their Applications to Weak Supervision
• Prompts as Programming by Gwern
• Reverse Prompt Engineering for Fun and (no) Profit
• Simulators
• Start with an Instruction
• Talking to machines: prompt engineering & injection
• the Book - Fed Honeypot
• The ChatGPT Prompt Book
• Using GPT-Eliezer against ChatGPT Jailbreaking

Lecture + Tutorial

Full tutorial and lecture coming soon!

---

Feel free to open a PR if you think something is missing here. Always welcome feedback and suggestions.