demo_apps/HelloLlamaLocal.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b7cabb96-2715-492e-825a-8f4ff161dc3b",
   "metadata": {},
   "source": [
    "## This demo app shows:\n",
    "* How to run Llama2 locally on a Mac using llama-cpp-python and the llama-cpp's quantized Llama2 model\n",
    "* How to use LangChain to ask Llama general questions\n",
    "* How to use LangChain to load a recent PDF doc - the Llama2 paper pdf - and ask questions about it. This is the well known RAG (Retrieval Augmented Generation) method to let LLM such as Llama2 be able to answer questions about the data not publicly available when Llama2 was trained, or about your own data. RAG is one way to prevent LLM's hallucination"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22450267",
   "metadata": {},
   "source": [
    "We start by installing necessary requirements and import packages we will be using in this example.\n",
    "- [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) a simple Python bindings for [llama.cpp](https://github.com/ggerganov/llama.cpp) library\n",
    "- pypdf gives us the ability to work with pdfs\n",
    "- sentence-transformers for text embeddings\n",
    "- chromadb gives us database capabilities \n",
    "- langchain provides necessary RAG tools for this demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2922732e-29e8-4ea7-8828-53364f5bf6fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# install all the required packages for the demo\n",
    "!CMAKE_ARGS=\"-DLLAMA_METAL=on\" FORCE_CMAKE=1 pip install llama-cpp-python\n",
    "!pip install pypdf sentence-transformers chromadb langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26bc4912",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.llms import LlamaCpp\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.callbacks.manager import CallbackManager\n",
    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
    "from langchain.prompts import PromptTemplate"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73df46d9",
   "metadata": {},
   "source": [
    "Next, initialize the langchain `CallBackManager`. This handles callbacks from Langchain and for this example we will use token-wise streaming so the answer gets generated token by token when Llama is answering your question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01fe5b9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for token-wise streaming so you'll see the answer gets generated token by token when Llama is answering your question\n",
    "callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8536c352",
   "metadata": {},
   "source": [
    "\n",
    "Set up the Llama 2 model. \n",
    "\n",
    "Replace `<path-to-llama-gguf-file>` with the path either to your downloaded quantized model file [here](https://drive.google.com/file/d/1afPv3HOy73BE2MoYCgYJvBDeQNa9rZbj/view?usp=sharing), \n",
    "\n",
    "or to the `ggml-model-q4_0.gguf` file built with the following commands:\n",
    "\n",
    "```bash\n",
    "git clone https://github.com/ggerganov/llama.cpp\n",
    "cd llama.cpp\n",
    "python3 -m pip install -r requirements.txt\n",
    "python convert.py <path_to_your_downloaded_llama-2-13b_model>\n",
    "./quantize <path_to_your_downloaded_llama-2-13b_model>/ggml-model-f16.gguf <path_to_your_downloaded_llama-2-13b_model>/ggml-model-q4_0.gguf q4_0\n",
    "\n",
    "```\n",
    "For more info see https://python.langchain.com/docs/integrations/llms/llamacpp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dff6aa6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "llm = LlamaCpp(\n",
    "    model_path=\"<path-to-llama-gguf-file>\"\n",
    "    temperature=0.0,\n",
    "    top_p=1,\n",
    "    n_ctx=6000,\n",
    "    callback_manager=callback_manager, \n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2cae215",
   "metadata": {},
   "source": [
    "With the model set up, you are now ready to ask some questions. \n",
    "\n",
    "Here is an example of the simplest way to ask the model some general questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e78549c-9c93-4bc2-b525-38d578a94fae",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "question = \"who wrote the book Innovator's dilemma?\"\n",
    "answer = llm(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545cb6aa",
   "metadata": {},
   "source": [
    "Alternatively, you can use LangChain's `PromptTemplate` for some flexibility in your prompts and questions.\n",
    "\n",
    "For more information on LangChain's prompt template visit this [link](https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7305c5b-6f55-4664-9206-2d7467653498",
   "metadata": {},
   "outputs": [],
   "source": [
    "# a more flexible way to ask Llama general questions using LangChain's PromptTemplate and LLMChain\n",
    "prompt = PromptTemplate.from_template(\n",
    "    \"who wrote {book}?\"\n",
    ")\n",
    "chain = LLMChain(llm=llm, prompt=prompt)\n",
    "answer = chain.run(\"innovator's dilemma\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "189de613",
   "metadata": {},
   "source": [
    "Now, let's see how Llama2 hallucinates, because it did not have knowledge about Llama2 at the time it was trained. \n",
    "By default it behaves like a know-it-all expert who will not say \"I don't know\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ba66a29-77e9-4149-9523-63a09545584e",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "prompt = PromptTemplate.from_template(\n",
    "    \"What is {what}?\"\n",
    ")\n",
    "chain = LLMChain(llm=llm, prompt=prompt)\n",
    "answer = chain.run(\"llama2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37f77909",
   "metadata": {},
   "source": [
    "One way we can fix the hallucinations is to use RAG, to augment it with more recent or custom data that holds the information for it to answer correctly.\n",
    "\n",
    "First we load the Llama2 paper using LangChain's [PDF loader](https://python.langchain.com/docs/modules/data_connection/document_loaders/pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3ebc261",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from langchain.document_loaders import PyPDFLoader\n",
    "loader = PyPDFLoader(\"llama2.pdf\")\n",
    "documents = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "302eaa54",
   "metadata": {},
   "outputs": [],
   "source": [
    "# quick check on the loaded document for the correct pages etc\n",
    "print(len(documents), documents[0].page_content[0:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c4ede5b",
   "metadata": {},
   "source": [
    "Next we will store our documents. \n",
    "There are more than 30 vector stores (DBs) supported by LangChain. \n",
    "For this example we will use [Chroma](https://python.langchain.com/docs/integrations/vectorstores/chroma) which is light-weight and in memory so it's easy to get started with.\n",
    "For other vector stores especially if you need to store a large amount of data - see https://python.langchain.com/docs/integrations/vectorstores\n",
    "\n",
    "We will also import the `HuggingFaceEmbeddings` and `RecursiveCharacterTextSplitter` to assist in storing the documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f94f6f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from langchain.vectorstores import Chroma\n",
    "\n",
    "# embeddings are numerical representations of the question and answer text\n",
    "from langchain.embeddings import HuggingFaceEmbeddings\n",
    "\n",
    "# use a common text splitter to split text into chunks\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bfdacf7",
   "metadata": {},
   "source": [
    "\n",
    "To store the documents, we will need to split them into chunks using [`RecursiveCharacterTextSplitter`](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter) and create vector representations of these chunks using [`HuggingFaceEmbeddings`](https://www.google.com/search?q=langchain+hugging+face+embeddings&sca_esv=572890011&ei=ARUoZaH4LuumptQP48ah2Ac&oq=langchian+hugg&gs_lp=Egxnd3Mtd2l6LXNlcnAiDmxhbmdjaGlhbiBodWdnKgIIADIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCjIHEAAYgAQYCkjeHlC5Cli5D3ABeAGQAQCYAV6gAb4CqgEBNLgBAcgBAPgBAcICChAAGEcY1gQYsAPiAwQYACBBiAYBkAYI&sclient=gws-wiz-serp) on them before storing them into our vector database. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b101485",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split the loaded documents into chunks \n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=20)\n",
    "all_splits = text_splitter.split_documents(documents)\n",
    "\n",
    "# create the vector db to store all the split chunks as embeddings\n",
    "embeddings = HuggingFaceEmbeddings()\n",
    "vectordb = Chroma.from_documents(\n",
    "    documents=all_splits,\n",
    "    embedding=embeddings,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bddc38e8",
   "metadata": {},
   "source": [
    "\n",
    "We then use ` RetrievalQA` to retrieve the documents from the vector database and give the model more context on Llama 2, thereby increasing its knowledge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a2472c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# use another LangChain's chain, RetrievalQA, to associate Llama with the loaded documents stored in the vector db\n",
    "from langchain.chains import RetrievalQA\n",
    "\n",
    "qa_chain = RetrievalQA.from_chain_type(\n",
    "    llm,\n",
    "    retriever=vectordb.as_retriever()\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db71e5d7",
   "metadata": {},
   "source": [
    "For each question, LangChain performs a semantic similarity search of it in the vector db, then passes the search results as the context to the model to answer the question.\n",
    "\n",
    "It takes close to 2 minutes to return the result (but using other vector stores other than Chroma such as FAISS can take longer) because Llama2 is running on a local Mac. \n",
    "To get much faster results, you can use a cloud service with GPU used for inference - see HelloLlamaCloud for a demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd2e62a4-6ea2-4ea7-b7ae-800185177e6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "question = \"What is llama2?\"\n",
    "result = qa_chain({\"query\": question})\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}