Skip to content

Latest commit

 

History

History
39 lines (38 loc) · 4.41 KB

llm_day_panel.md

File metadata and controls

39 lines (38 loc) · 4.41 KB

Paradigm Shifts in the Era of LLMs: Opportunities and Challenges in Academia, Industry, and Society.

  • Panelists: Ed Chi (Google DeepMind), Vedanuj Goswami (Meta FAIR), Jaime Teevan (Microsoft), Vy Vo (Intel Labs), Denny Zhou (Google DeepMind)
  • What's one perspective on LLMs you have?
    • Denny Zhou: Happy that many people know the importance of reasoning. Reasoning is all you need.
    • Vy Vo: Comparing human brains vs. LLMs. Comparative intelligence approaches will be more important going forward.
    • Ed: Has a background in HCI and cognitive science
    • Vedanuj: On the panel because he helped training LLaMA2 models.
    • Jaime: On the panel because she brings organizational diversity. Knowledge is increasingly embedded in conversations. How do we setup people to have successful conversations?
  • Some researchers in certain fields fear their fields will be eliminated because of LLMs. What fields will disappear?
    • Lawrence: There's an analog to work/jobs
    • Jaime: Should be excited to reimagine how research is conducted. How knowledge is disseminated. We have an amazing new tool on our hands now.
    • Ed: People working on methods are more likely to be out of a job, people working on data are more likely to be safe. 40-50% of questions touched on knowledge graphs and injecting them into LLMs - this community spent years on turning unstructured data into a knowledge graph. But a LLM is probably more capable than a knowledge graph. Turn knowledge graphs back into text that can be used to train LLMs.
    • Vy: Information is represented in multiple forms. Don't think that language is the only useful way to represent data/information.
      • We still don't understand how LLMs are doing what they are doing
      • Emerging field to understand machine intelligence. Will be able to steer it.
      • Grapple with the comparative approach.
    • Denny: LLMs are much weaker than what we thought.
      • Inconsistency: GPT-4 will make mistakes on elementary school math problems.
      • Where does reasoning come from? If it doesn't come from language, can we use synthetic language to understand reasoning.
  • What are some research directions to understand how LLMs are working?
    • Ed
      • Geoff Hinton said if an ML is making a good prediction, why do we care how he or she is making it?
      • Humans are a black box. Cognitive science - humans make decision and justify their decisions post-hoc.
      • You can interrogate the LLM directly for why it made a prediction/decision.
    • Vy: Deep learning framework for neuroscience. All animal brains are black boxes. Paper said what is being optimized for in the biological brain and AI systems. Look at the loss function that the model is being trained on.
  • Is it responsible for human beings to use that much energy? What is the future of AI in this regard?
    • 5 watts to think
    • 1-2 watts to generate a word
    • 540B or 175B parameter model - 400W per GPU
    • Jaime: Not trying to abdicate human intelligence. But like any tech, it's expensive in the beginning. Will expect it to improve in efficiency going forward.
  • Are there smarter ways to do research in LLMs without 4,000 GPUs?
    • Denny: Few companies are going to build LLMs. But research community can try to understand HOW LLMs work.
    • Ed: 30% of job is to how to organize research and set research agendas. Decompose the problem into 4 dimensions: data, compute, people and structure. Think about whether you have an advantage in these 4 areas. When you look at industry, startups and academics, it's hard to compete on data and compute but they can compete with people and structure. It's not just all about scale. The reasoning breakthroughs in LLMs came up by understanding a little bit about cognitive science and education.
  • How do I define my core strengths as a researcher?
    • Jaime: We don't know what we don't know. We are trained to figure that. What are the skills that a human needs? These are big hard questions. Whether you are a big tech company, startup or an 18-year old you have to figure stuff out.
    • Ed: Two alternative futures. Living in the Bay Area, encounter investors who are investing into startups for AI for enterprise use cases.
      1. Treat a LLM as a generic human. Augment it with a retrieval data, it will work as an assistant. As a VC I will be very happy. If you believe in Denny's work, this is Scenario 1.
      2. Train the model on a custom corpus, RAG only will not get to you human-level performance. As a VC I will be unhappy.