Skip to content

Large Language Models

Jump to bottom
Brandon Ruffridge edited this page Jun 29, 2023 · 17 revisions

Prompt Engineering

  • Good prompt engineering tips from the creator of ChatGPT in the State of GPT youtube video
  • ChatGPT Prompt Engineering Short Course for Developers
  • Metaprompt Generator
  • LangChain
  • PromptSource
  • LMOps
    • Prompting scripts with editable parameters
    • Large language models have exhibited intriguing in-context learning capability, achieving promising zero- and few-shot performance without updating the parameters. However, conventional in-context learning is usually restricted by length constraints, rendering it ineffective to absorb supervision from a large number of examples.
    • In order to go beyond few shots, we introduce structured prompting that breaks the length limit and scales in-context learning to thousands of examples. Specifically, demonstration examples are separately encoded with well-designed position embeddings, and then they are jointly attended by the test example using a rescaled attention mechanism. So we can scale the number of exemplars with linear complexity instead of quadratic complexity with respect to length. Experimental results on a diverse set of tasks show that our approach improves end-task performance and reduces evaluation variance over conventional in-context learning as the number of demonstration examples increases.
  • Open AI examples
  • Prompt engineering 101
  • 8 insider tips
  • Learn prompting
  • Magic phrases
  • Guide by OpenAI Engineer - Lilian Weng
  • How to write an effective GPT-3 prompt
    • Test your prompt
      • Temperature → how creative the AI can be. Temp == 0 means same output for each run.
      • Max len → how long the combined prompt and response should be.
    • Offer Context
      • Basic prompt → "Write about productivity."
      • Better prompt → "Write a blog post about the importance of productivity for small businesses."
      • Basic prompt → "Write about how to house train a dog."
      • Better prompt → "As a professional dog trainer, write an email to a client who has a new 3-month-old Corgi about the activities they should do to house train their puppy."
    • Include helpful information upfront
      • Reid's resume: [paste full resume here]
      • Given the above information, write a witty speaker bio about Reid.
    • Give examples
    • Tell it the length of the response you want
      • Basic prompt: "Summarize this article."
      • Better prompt: "Write a 500-word summary of this article."
    • Define the expected formats
      • [Insert full text of an interview transcript]
      • Output the above interview in HTML.
    • Use some of these handy expressions
      • "Let's think step by step"
        • This makes the AI think logically and can be specifically helpful with math problems.
      • "Thinking backwards"
        • This can help if the AI keeps arriving at inaccurate conclusions.
      • "In the style of [famous person]"
        • This will help match styles really well.
      • "As a [insert profession/role]"
        • This helps frame the bot's knowledge, so it knows what it knows—and what it doesn't.
  • Streamline your GPT-3 prompt engineering workflow with Promptable's advanced tools and features
    • Organize prompts
    • Track changes
    • Evaluate prompts
    • Deploy prompts
    • (Uses OpenAI)
  • Promptify
    • Make GPT-3 easier to use with python code
    • Easily add one shot, two shot, or few shot examples to the prompt
    • Output always provided as a Python object
  • Prompt Engineering Guide
    • exhaustive list of tools, tutorials, datasets, and research papers. Probably too much. Need to sort out which ones are the most relevant to PeTaL.
  • The effects of irrelevant or misleading prompts
  • "Let's think step by step"
  • Self ask for multi-hop questions
  • Connecting LLMs with reasoning and external knowledge through MRKL
  • ReAct: Synergizing Reasoning and Acting in Language Models
  • Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models

Fine-tuning

Other tools/research

  • Cohere
  • LLM research to find better model
    • there is more to creating a good language model than size
    • MT-NLG had 530 billion params, but did not necessarily do better than GPT-3
    • Larger pre-training corpus can yield significant improvements (Gopher)
    • Model-size and amount of pretraining data are equally important when scaling a model
    • Chinchila is smaller than GPT-3, but can yield better results with more pre-training
    • Slightly better results with shallower, wider networks
  • Language Models are Changing AI. We Need to Understand Them
    • Many language models exist, but they are not compared on a unified standard, and even when language models are evaluated, the full range of societal considerations (e.g., fairness, robustness, uncertainty estimation, commonsense knowledge, disinformation) have not be addressed in a unified way.
    • They propose a new benchmark HELM to deal with this.
  • Take on LLMs from Andrew Ng
    • "Thus search, which can point us to pages from all corners of the web, can answer many questions that an LLM with fixed memory can't.That said, I see significant potential in another technology, retrieval augmented generation. Rather than relying on a fixed LLM to deliver the answer to a query, if we first find relevant documents (online or elsewhere) and then use an LLM to process the query and the documents into an answer, this could provide an alternative to current web search."
  • Catalog of every LLM improvement technique
  • ROUGE metric
  • Smaller multimodal chain-of-thought models can yield high accuracy

What PeTaL Actually Is

Clone this wiki locally