DecryptPrompt

持续更新以下内容，Star to keep updated~

1. Prompt和LLM相关论文按细分方向梳理
2. AIGC相关应用
3. Prompt指南和教程
4. ChatGPT及AGI相关解读
5. 开源大模型
6. ChatGPT相关商业应用 [WIP]

My blogs

• 解密Prompt系列1. Tunning-Free Prompt：GPT2 & GPT3 & LAMA & AutoPrompt
• 解密Prompt系列2. 冻结Prompt微调LM： T5 & PET & LM-BFF
• 解密Prompt系列3. 冻结LM微调Prompt: Prefix-tuning & Prompt-tuning & P-tuning

Resources

paper List

• https://github.com/dongguanting/In-Context-Learning_PaperList
• https://github.com/thunlp/PromptPapers
• https://github.com/Timothyxxx/Chain-of-ThoughtsPapers

Recommend Blog

• OpenAI ChatGPT Intro
• OpenAI InstructGPT intro
• AllenAI ChatGPT能力解读：How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources ⭐
• Huggingface ChatGPT能力解读：The techniques behind ChatGPT: RLHF, IFT, CoT, Red teaming, and more
• Stephen Wolfram ChatGPT能力解读: What Is ChatGPT Doing and Why Does It Work?
• Chatgpt相关解读汇总
• 麻省理工科技采访OpenAI工程师
• AGI历史与现状

Tools & Tutorial

• ClickPrompt: 为各种prompt加持的工具生成指令包括Difussion，chatgpt
• Prompt-Engineer-Guide: 如何写prompt的系列教学指南 ⭐
• ChatGPT ShortCut：提供各式场景下的Prompt范例，支持搜索
• PromptPerfect:用魔法打败魔法，输入关键词，模型创建条理清晰的最美提示词
• openAI: ChatGPT出API啦, 价格下降10倍！
• OpenAI Cookbook: 提供OpenAI模型使用示例 ⭐

AIGC playground

• AI Topiah: 聆心智能AI角色聊天，和路飞唠了两句，多少有点中二之魂在燃烧
• chatbase: 情感角色聊天，还没尝试
• Vana: virtual DNA, 通过聊天创建虚拟自己！概念很炫
• New Bing：需要连外网否则会重定向到bing中国，需要申请waitlist
• WriteSonic：AI写作，支持对话和定向创作如广告文案，商品描述, 支持Web检索是亮点，支持中文
• copy.ai: WriteSonic竞品，亮点是像论文引用一样每句话都有对应网站链接，可以一键复制到右边的创作Markdown，超级好用！ :star1:
• NotionAI：智能Markdown，适用真相！在创作中用command调用AI辅助润色，扩写，检索内容，给创意idea
• Jasper: 同上，全是竞品哈哈
• ChatExcel: 指令控制excel计算，对熟悉excel的有些鸡肋，对不熟悉的有点用
• ChatPaper: 根据输入关键词，自动在arxiv上下载最新的论文，并对论文进行摘要总结，可以在huggingface上试用！
• copy.down: 中文的营销文案生成，只能定向创作，支持关键词到文案的生成
• Copilot: 要付费哟
• Fauxpilot: copilot本地开源替代
• CodeGex: 国内替代品，还没试过
• dreamstudio.ai: 开创者，Stable Difussion， 有试用quota
• midjourney: 开创者，艺术风格为主
• Dall.E: 三巨头这就凑齐了
• ControlNet: 为绘画创作加持可控性
• GFPGAN: 照片修复

开源模型

国外

• OPT-IML: Meta复刻GPT3，up to 175B, 不过效果并不及GPT3
• Bloom：BigScience出品，规模最大176B, 感觉应该对标text-davinci-002
• T0: BigScience出品，3B~11B的在T5进行指令微调的模型
• LLaMA:Meta开源指令微调LLM，规模70 亿到 650 亿不等
• ChatLLaMA: 基于RLHF微调了LLaMA
• MetaLM: 微软开源的大规模自监督预训练模型
• Alpaca: 斯坦福开源的使用52k数据在7B的LLaMA上微调得到，据说效果类似text-davinci-003, 模型不久后会发布

国内

• 国内开源模型魔塔社区：https://www.modelscope.cn/home
• PromptCLUE: 多任务Prompt语言模型
• Chatyuan：基于PromptCLUE训练的对话模型
• PLUG: 阿里达摩院发布的大模型，提交申请会给下载链接
• CPM2.0: 智源发布CPM2.0
• Moss: 复旦发布的大模型
• GLM: 清华发布的中英双语130B大模型

Papers

Survey

• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing ⭐
• Paradigm Shift in Natural Language Processing
• Pre-Trained Models: Past, Present and Future

LLM Ability Analysis & Probing

• How does in-context learning work? A framework for understanding the differences from traditional supervised learning
• Why can GPT learn in-context? Language Model Secretly Perform Gradient Descent as Meta-Optimizers
• Emerging Ability of Large Language Models
• Rethinking the Role of Demonstrations What Makes incontext learning work?
• Can Explanations Be Useful for Calibrating Black Box Models

Tunning Free Prompt

• GPT2: Language Models are Unsupervised Multitask Learners
• GPT3: Language Models are Few-Shot Learners ⭐
• LAMA: Language Models as Knowledge Bases?
• AutoPrompt: Eliciting Knowledge from Language Models

Fix-Prompt LM Tunning

• T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• PET-TC(a): Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference ⭐
• PET-TC(b): PETSGLUE It’s Not Just Size That Matters Small Language Models are also few-shot learners
• GenPET: Few-Shot Text Generation with Natural Language Instructions
• LM-BFF: Making Pre-trained Language Models Better Few-shot Learners ⭐
• ADEPT: Improving and Simplifying Pattern Exploiting Training

Fix-LM Prompt Tunning

• Prefix-tuning: Optimizing continuous prompts for generation
• Prompt-tunning: The power of scale for parameter-efficient prompt tuning ⭐
• P-tunning: GPT Understands Too ⭐
• WARP: Word-level Adversarial ReProgramming

LM + Prompt Tunning

• P-tunning v2: Prompt Tuning Can Be Comparable to Fine-tunning Universally Across Scales and Tasks
• PTR: Prompt Tuning with Rules for Text Classification
• PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains

Instruction Tunning LLMs

• Flan: FINETUNED LANGUAGE MODELS ARE ZERO-SHOT LEARNERS ⭐
• Flan-T5: Scaling Instruction-Finetuned Language Models
• Instruct-GPT: Training language models to follow instructions with human feedback star:
• T0: MULTITASK PROMPTED TRAINING ENABLES ZERO-SHOT TASK GENERALIZATION
• k-INSTRUCT: SUPER-NATURALINSTRUCTIONS: Generalization via Declarative Instructions on 1600+ NLP Tasks

Train for Dialogue

• LaMDA: Language Models for Dialog Applications
• Sparrow: Improving alignment of dialogue agents via targeted human judgements star:
• BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
• How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

Chain of Thought

• Chain of Thought Prompting Elicits Reasoning in Large Language Models ⭐
• COMPLEXITY-BASED PROMPTING FOR MULTI-STEP REASONING
• SELF-CONSISTENCY IMPROVES CHAIN OF THOUGHT REASONING IN LANGUAGE MODELS
• Large Language Models are Zero-Shot Reasoners
• PaLM: Scaling Language Modeling with Pathways

RLHF

• Deep reinforcement learning from human preferences
• PPO: Proximal Policy Optimization Algorithms ⭐
• InstrutGPT序作：learning to summarize from human feedback
• InstructGPT: Training language models to follow instructions with human feedback ⭐
• RL4LM：IS REINFORCEMENT LEARNING (NOT) FOR NATURAL LANGUAGE PROCESSING BENCHMARKS