wip: pgvector

Justfile

@@ -265,3 +265,10 @@ regenerate-cassettes:
 	fd -td cassettes -X rm -ri
 	rye run unittests-vcr-record-final
 	rye run unittests-debug
+
+brew-deps:
+	brew install libmagic poppler tesseract pandoc qpdf tesseract-lang
+	brew install --cask libreoffice
+
+db-create:
+	rye run psql -d langchain -c 'CREATE EXTENSION vector'

REFERENCES.md

@@ -163,3 +163,16 @@ source: <https://www.youtube.com/watch?v=AeASAsPp9LE>
 - https://github.com/ionelmc/python-manhole/ - `Debugging manhole for python applications.`
 - https://github.com/langchain-ai/langchain/blob/master/cookbook/Multi_modal_RAG.ipynb
 - https://github.com/SAMAD101/Chino/blob/e38f3d9d38702beaed37229f66d79e86a7acab26/src/chino/query.py (write a query module maybe)
+
+
+--------------------------
+
+# Advanced rag suggestions
+
+> https://www.reddit.com/r/LangChain/comments/1cyjfap/best_stack_for_rag/
+
+### Quotes
+
+- If I had to do it over again, I'd just put everything in Postgres with pgvector turned on.
+- 100%. It's extremely powerful and it's nice when you have it mixed in with conventional database tables. You can do joins across relational and vector data. The performance of the vector indexing database engine will never be a significant performance bottleneck. Performance is mostly affected by the embedding model, LLM, and how many tokens the agent library (e.g. langchain) uses. Besides, Postgres is no slouch when it comes to performance and is easy to scale.
+-

docker-compose.yml

@@ -9,8 +9,36 @@ x-default-logging:
 networks:
   net:
     driver: bridge
-
 services:
+# docker run --name pgvector-container -e POSTGRES_USER=langchain -e POSTGRES_PASSWORD=langchain -e POSTGRES_DB=langchain -p 6024:5432 -d pgvector/pgvector:pg16
+  db:
+    image: pgvector/pgvector:pg14
+    container_name: pgvector-container
+    ports:
+      - 5432:5432
+    volumes:
+      # This script initialises the DB for integration tests
+      - ./docker/pgvector/scripts:/docker-entrypoint-initdb.d
+      - db:/var/lib/postgresql/data
+    environment:
+      - POSTGRES_PASSWORD=langchain
+      - POSTGRES_USER=langchain
+      - POSTGRES_DB=langchain
+
+    restart: unless-stopped
+    # logging: *logging
+    networks:
+      - net
+    command: |
+      postgres -c log_statement=all
+    # healthcheck:
+    #   test:
+    #     [
+    #       "CMD-SHELL",
+    #       "psql postgresql://langchain:langchain@localhost/langchain --command 'SELECT 1;' || exit 1",
+    #     ]
+    #   interval: 5s
+    #   retries: 60
 
   # postgres:
   #   image: postgres:14-alpine
@@ -281,3 +309,4 @@ services:
 volumes:
   goob_redis_data:
     driver: local
+  db:

docker/pgvector/scripts/01.sql

@@ -0,0 +1,10 @@
+CREATE EXTENSION vector;
+
+CREATE TABLE IF NOT EXISTS test_place (
+                                          id SERIAL PRIMARY KEY,
+                                          content text,
+                                          type text,
+                                          sourcetype text,
+                                          sourcename text,
+                                          embedding vector
+);

docs/prompt_engineering.md

@@ -0,0 +1,113 @@
+Since the AI Boom, demand for prompt engineers has skyrocketed, with companies like Anthropic offering up to $400,000 for prompt engineers. This article will explain the technical details of prompt engineering and why it's important in the age of [artificial intelligence](https://www.voiceflow.com/articles/ai-model).
+
+## What Is Prompt Engineering?
+
+**A "prompt" is an instruction or query given to an AI system, typically in** [**natural language**](http://www.voiceflow.com/articles/natural-language-processing)**, to generate a specific output or perform a task.**
+
+Prompt engineering is the process of developing and optimizing such prompts to effectively use and guide generative AI (gen AI) models—particularly [large language models (LLMs)](http://www.voiceflow.com/articles/large-language-models)—to produce desired outputs.
+
+**Note that prompt engineering is primarily focused on** [**natural language processing**](http://www.voiceflow.com/articles/natural-language-processing) **(NLP) and communication rather than traditional maths or engineering.** The core skills involve understanding language, context, and how to effectively communicate with AI models.
+
+## Prompt Engineering Example
+
+Here's an example of an instruction-based prompt:
+
+You: "Play the role of an experienced Python developer and help me write code."
+
+Then, AI assumes the role of a senior developer and provides the code. In this case, the prompt engineer has given a specific instruction to the AI, asking it to take on a particular role (experienced Python developer) and perform a task (help write code).
+
+## Different Types of Prompt Engineering Approaches
+
+**There are 9 types of prompt engineering approaches.** Here's a quick table explaining each approach with an example:
+
+| **Approach**              | **Explanation**                                              | **Example**                                                  |
+| ------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |
+| Zero-shot prompts         | The AI is given a task without any examples or specific instructions. | "Translate this sentence into Spanish"                       |
+| Few-shot prompts          | Provide the AI with a few examples to guide its responses.   | "Translate the following sentences into French: 'Hello, how are you?' becomes 'Bonjour, comment ça va?'. Now translate: 'What time is it?'" |
+| Chain-of-thought prompts  | Guide the AI to think step-by-step through a problem.        | "To solve this math problem, first find the value of x, then calculate y using the value of x." |
+| Tree-of-thought prompts   | Encourage the AI to explore multiple branches of reasoning.  | "List the potential impacts of implementing AI in healthcare, considering both positive and negative effects." |
+| Instruction-based prompts | Provide the AI with specific instructions or questions for precise responses. | "Write a summary of the latest AI trends."                   |
+| Example-based prompts     | Offer examples for the AI to follow.                         | "Here is a summary: [Insert example]. Now write a similar summary about AI in healthcare." |
+| Context-based prompts     | Give the AI context or background information to generate relevant responses. | "Considering the advancements in AI for autonomous driving, describe the potential impact on urban planning." |
+| Persona-based prompts     | Instruct the AI to respond as a specific persona or character. | "As a tech-savvy business consultant, explain the benefits of AI in customer service." |
+| Sequential prompts        | Break down complex tasks into smaller, manageable steps for the AI to follow. | "First, outline the key features of GPT-4. Next, explain how it differs from GPT-3." |
+
+## Prompt Tuning vs. Prompt Engineering vs. Fine Tuning
+
+**Prompt tuning, prompt engineering, and fine-tuning are all ways to make AI models work better.** Prompt tuning is about tweaking the questions or instructions given to the AI to get better answers. Prompt engineering involves creating and refining these questions or instructions in different ways to achieve specific goals. Fine-tuning is more involved, requiring retraining the AI on new data to make it perform better for specific tasks. All these methods help improve the AI's ability to provide accurate and relevant responses.
+
+## RAG vs. Prompt Engineering
+
+[**RAG**](http://www.voiceflow.com/articles/retrieval-augmented-generation) **(Retrieval-Augmented Generation) combines information retrieval and text generation.** RAG is not just "glorified prompt engineering" because it adds complexity through the retrieval and integration of external information, such as a [knowledge base](http://www.voiceflow.com/articles/knowledge-base) (KB), whereas prompt engineering focuses on optimizing how we interact with the AI model's existing knowledge.
+
+Create a Custom AI Chatbot In Less Than 10 Minutes
+
+Join Now—It's Free
+
+
+
+![img](https://cdn.prod.website-files.com/656f60dc2d85b496beec7c35/6642224fbd43152bc0bda067_Frame%2048096240.webp)
+
+## What's Reverse Prompt Engineering?
+
+**Reverse prompt engineering is the process of figuring out the specific input or prompt that would produce a given output from an AI model.**
+
+For example, if you have an AI-generated piece of text that describes what a chatbot is, you would work backward to identify the likely prompt.
+
+## Chatbot Prompt Engineering Best Practices
+
+To prompt engineer chatbots like ChatGPT, follow these best tips:
+
+- **Use role prompting**: Assign a role or persona to the chatbot, such as "You are a creative writing coach". This helps frame the conversation and guide the AI model to respond with the correct tone.
+- **Provide examples**: Using a "few-shot learning" approach can help the chatbot better understand your request.
+- **Use delimiters**: Include triple quotes or brackets to highlight specific parts of your input. This can help the chatbot understand the important sections of your prompt.
+
+## Build Gen AI-Powered Chatbots Using Prompt Engineering with Voiceflow In 5 Minutes
+
+**You can build a generative AI agent with Voiceflow quickly, easily, and effortlessly!**
+
+1. Sign up for a free Voiceflow account and create a new project.
+2. Start with a blank canvas and add a "Talk" block to greet the user, then add an "AI" block and configure it to use a large language model of your choice. Voiceflow supports GPT, Claude, and many more!
+3. In the AI block, craft a prompt that defines the chatbot's persona, knowledge, and capabilities. For example, you can tell the chatbot: "You are a helpful AI assistant for Voiceflow. Your role is to answer questions about our products and services. Be friendly."
+4. Add few-shot examples to guide the AI's responses.
+5. When you're satisfied with the chatbot, you can deploy it to the platform of your choice (website, WhatsApp, social media apps, and more) using Voiceflow's integration options.
+
+That's it! You can design, prototype, and launch your AI agent in 5 minutes without writing a single line of code. Get started today—it's free!
+
+Create an AI Chatbot Today
+
+
+
+## Frequently Asked Questions
+
+### How does prompt engineering work?
+
+Prompt engineering involves designing specific inputs or questions to guide an AI model's responses. By crafting precise prompts, you can improve the relevance and accuracy of the AI's output.
+
+### How does prompt engineering impact the output of AI models?
+
+Prompt engineering directly affects the quality of the AI's responses by providing clear and specific instructions. Better prompts lead to more accurate, relevant, and useful outputs from the AI model.
+
+### What are some real-world examples of prompt engineering?
+
+In customer service, prompt engineering helps chatbots provide accurate answers to common questions. In education, it guides AI to offer detailed explanations and personalized tutoring.
+
+### What are the ethical considerations in prompt engineering?
+
+Ethical considerations include avoiding biased or harmful prompts that could lead to unfair or offensive responses. It's important to ensure prompts encourage safe, inclusive, and truthful outputs.
+
+### Why is prompt engineering important in AI?
+
+Prompt engineering is crucial because it optimizes the interaction between humans and AI, making the AI's responses more useful and relevant. It enhances the effectiveness and reliability of AI applications.
+
+### What are chain-of-thought and tree-of-thought prompting?
+
+Chain-of-thought prompting guides the AI to think step-by-step through a problem. Tree-of-thought prompting encourages the AI to explore multiple possibilities and outcomes.
+
+### How does few-shot prompting differ from zero-shot prompting?
+
+Few-shot prompting provides the AI with a few examples to guide its responses. Zero-shot prompting asks the AI to perform a task without any prior examples or instructions.
+
+### How do you evaluate the effectiveness of a prompt?
+
+You evaluate a prompt's effectiveness by checking if the AI's response is accurate, relevant, and useful. Testing with different prompts and comparing the quality of the outputs helps determine the best prompts.

docs/pyenv.md

@@ -0,0 +1,33 @@
+## pyenv - :coffee: Getting Started
+
+> https://raw.githubusercontent.com/Unstructured-IO/community/main/README.md
+>
+Goob_ai's open-source packages currently target Python 3.10. If you are using or contributing
+to Goob_ai code, we encourage you to work with Python 3.10 in a virtual environment. You can
+use the following instructions to get up and running with a Python 3.10 virtual environment
+with `pyenv-virtualenv`:
+
+#### Mac / Homebrew
+
+1. Install `pyenv` with `brew install pyenv`.
+2. Install `pyenv-virtualenv` with `brew install pyenv-virtualenv`
+3. Follow the instructions [here](https://github.com/pyenv/pyenv#user-content-set-up-your-shell-environment-for-pyenv)
+   to add the `pyenv-virtualenv` startup code to your terminal profile.
+4. Install Python 3.10 by running `pyenv install 3.10.15`.
+5. Create and activate a virtual environment by running:
+
+```
+pyenv virtualenv 3.10.15 unstructured
+pyenv activate unstructured
+```
+
+You can changed the name of the virtual environment from `unstructured` to another name if you're
+creating a virtual environment for a pipeline. For example, if you're a creating a virtual
+environment for the SEC preprocessing, you can run `pyenv virtualenv 3.10.15 sec`.
+
+#### Linux
+
+1. Run `git clone https://github.com/pyenv/pyenv.git ~/.pyenv` to install `pyenv`
+2. Run `git clone https://github.com/pyenv/pyenv-virtualenv.git ~/.pyenv/plugins/pyenv-virtualenv`
+   to install `pyenv-virtualenv` as a `pyenv` plugin.
+4. Follow steps 3-5 from the Mac/Homebrew instructions.

pyproject.toml

@@ -139,9 +139,10 @@ dependencies = [
     "jedi-language-server>=0.41.4",
     "requests-toolbelt>=1.0.0",
     "pdf2image>=1.17.0",
-    "unstructured[all-docs]==0.10.19",
+    "unstructured[all-docs]==0.15.8",
     "jq>=1.8.0",
     "pyinstrument>=4.7.2",
+    "langchain-postgres>=0.0.9",
 ]
 
 readme = "README.md"
@@ -1206,6 +1207,7 @@ select = [
 # SOURCE: https://github.com/ansible-collections/cloud-content-handbook/blob/9be137d78af4d1cc140b210f3058977164021c9d/proposals/ruff_transition.md - end
 # Conflicts with the formatter
 ignore = [
+    "N815",
     "PLW0603",
     "D",
     "COM812",