Pre-train, Prompt, Predict

**************************** Updates ****************************

• 10/10: Add 30 new papers (including NLP and CV) ([68] - [97])
• 8/25: Add a new papers. ([67])
• 8/15: Add three new papers. ([64], [65], [66])
• 8/02: Add Several missing references.
• 8/01: Fix one incorrect reference. (Thank Junyang Lin for pointing it out.)
• 7/28: We released our paper. Check it out!

Typology | Prompt Paper | Interactive Paperlist | Pretrained LM Family | Survey and Website | Copyright

Feel free to let us know the missing papers (issue or pull request)

We highly recommend using the interactive paperlist for prompt-based learning.

Typology of Prompting Methods

• Pre-trained Models (Detailed Description)

  • Left-to-right Language Model
  • Masked Language Model
  • Prefix Language Model
  • Encoder-Decoder

• Prompt Engineering (Detailed Description)

  • Shape
    • Cloze prompt: [7] [53]
    • Prefix prompt: [17] [26]
  • Human Effort
    • Hand-crated
    • Automated
      • Discrete: [24] [27]
      • Continuous: [29] [59]

• Answer Engineering (Detailed Description)

  • Shape
    • Token: [53] [7]
    • Span: [19] [22]
    • Sentence: [17] [29]
  • Human Effort
    • Hand-crated: [2] [53]
    • Automated
      • Discrete: [24] [13]
      • Continuous: [28]

• Multi-Prompt Learning (Detailed Description)

  • Prompt Ensemble: [11] [57]
  • Prompt Augmentation: [48] [47]
  • Prompt Composition: [52]
  • Prompt Decomposition: [53]

• Prompt-based Training Strategies (Detailed Description)

  • Parameter Updating
    • Promptless Fine-tuning
    • Tuning-free Prompting: [17] [7]
    • Fixed-LM Prompt Tuning: [29] [28]
    • Fixed-prompt LM Tuning: [26] [13]
    • Prompt+LM Tuning: [33] [37]
  • Training Sample Size
    • Zero-shot: [34] [43]
    • Few-shot: [19] [27]
    • Full-data: [21] [29]

Copyright and Acknowledgement

• If you aim to use the picture from our github repo or survey, please state the source of these pictures.
• These pictures are drawn by Weizhe and co-designed by Pengfei.