Recipe for each HAPI endpoint + refactor of DB schema

.gitignore

@@ -23,4 +23,6 @@ recipes-management/checked_out/*
 !recipes-management/checked_out/skills.py
 recipes-management/files/
 database.sqlite
+.cli_config
+recipes-management/work/
 

.pre-commit-config.yaml

@@ -7,6 +7,7 @@ repos:
     rev: 24.4.2
     hooks:
     - id: black
+      exclude: 'skills.py'
       language_version: python3.11
 -   repo: https://github.com/pycqa/flake8.git
     rev: 7.0.0

README.md

@@ -163,41 +163,50 @@ As the robocorp actions might differ slightly, this can lead to differing requir
 
 The management of recipes is part of the human in the loop approach of this repo. New recipes are created in status pending and only get marked as approved, once they have been verified by a recipe manager. Recipe managers can 'check out' recipes from the database into their local development environment such as VS Code to run, debug, and edit the recipes, before checking them back in. To make this process platform independent, recipes are checked out into a docker container, which can be used as the runtime environment to run the recipes via VSCode. 
 
-1. To check out recipes:
+Recipes are managed using the recipes Command Line Interface (CLI), which allows you to check out recipes, run and refine, the commit them back to the recipes database for use in data recipes AI.
 
-`cd recipes-management`
-`docker exec haa-recipe-manager python recipe_sync.py --check_out <YOUR NAME>`
+To run the cli, you will need to install some packages ...
+
+`pip3 install typer`
 
-Note: This will lock the recipes in the database so others cannot edit them.
+Once this is done, and you have your docker environment running as  described above, you start the recipes CLI with ...
+
+`cd recipes-management`
+`python cli.py`
 
-2. The checked_out folder in the recipes-management directory now shows all the recipes that were checked out from the database including the recipe code as a .py file. Note that after this step, the recipes in the database are marked as locked with your name and the timestamp you checked them out. If someone else tries to check them out, they are notified accordingly and cannot proceed until you've unlocked the records (more on that below).
+When you first log in, you will be asked for your name. This is used when checking in recipes. Once in, you will be presented with a menu like this ...
 
-This step checks out three files:
+```
 
-   -  Recipe.py - Contains the recipe code (re-assembled from the corresponding sections in the metadata json file)
-   -  metadata.json - Contains the content of the cmetadata column in the recipe database.
-   -  record_info.json - Contains additional information about the record such as its custom_id, and output.
+Welcome to the recipes management CLI, matt!
 
-   You can edit all files according to your needs. Please makes sure to not change the custom_id anywehere because it's needed for the check in process.
+    Here are the commands you can run:
+
+    'checkout': Check out recipes for you to work on
+    'list': List all recipes that are checked out
+    'run': Run a recipe, you will be prompted to choose which one
+    'add': Add a new recipe
+    'delete': Delete a recipe, you will be prompted to choose which one
+    'checkin': Check in recipes you have completed
+    'makemem': Create a memory using recipe sample output
+    'help': Show a list of commands
+    'quit': Exit this recipes CLI
 
-3. Run the scripts and edit them as you deem fit. Please note: Do not delete the #Functions Code and #Calling Code comments as they're mandatory to reassemble the metadata json for the check in process.
-4. Once you've checked and edited the recipes, run 
+    Type one of the commands above to do some stuff.
 
-   `docker exec haa-recipe-manager python recipe_sync.py --check_in <YOUR NAME>`
 
-   To check the records back into the database and unlock them. All recipes that you've checked in in this fashion are automatically set to status 'approved' with your name as the approver and the timestamp of when you checked them back in.
+>> 
+```
 
-## Testing a Recipe
+The first thing you will want to do is run 'checkout' to get all the recipe code from the database onto your computer so you can run them. Once you have them locally, you can edit them in tools like Visual Studio code. 
 
-You can run a specific recipe like this ...
+To run recipes locally you can use the CLI 'run' command. This will run the recipe in the same environment, and will save the results like sample outputs, for you so they can be published back to the database.
 
-`docker exec haa-recipe-manager python checked_out/retrieve_the_total_population_of_a_specified_country/recipe.py`
+You can create new recipes by entering 'add', where you'll be prompted for an intent. This will call an LLM to generate a first pass at your recipe, using the data that's in the data recipes environment.
 
-You can also exec into the container to do it ...
+When ready, you can check in your new and edited recipes with 'checkin'.
 
-1. `docker exec -it haa-recipe-manager /bin/bash`
-2. `cd ./checked_out`, then `cd <RECIPE_DIR>`
-3. `python recipe.py`
+### Other approaches
 
 You can also configure VS Code to connect to the recipe-manage container for running recipes ...
 

actions/actions_plugins/recipe-server/actions.py

@@ -17,6 +17,7 @@
 )
 from PIL import Image
 from robocorp.actions import action
+from sqlalchemy import create_engine, text
 
 # Load environment variables from .env file
 load_dotenv()
@@ -278,6 +279,54 @@ def get_recipe_memory(intent) -> str:
     return memory_found, recipe_response
 
 
+def connect_to_db(instance="recipe"):
+    """
+    Connects to the specified database instance (RECIPE or DATA) DB and returns a connection object.
+
+    Args:
+        instance (str): The name of the database instance to connect to. Defaults to "RECIPE".
+
+    Returns:
+        sqlalchemy.engine.base.Engine: The connection object for the specified database instance.
+    """
+
+    instance = instance.upper()
+
+    host = os.getenv(f"POSTGRES_{instance}_HOST")
+    port = os.getenv(f"POSTGRES_{instance}_PORT")
+    database = os.getenv(f"POSTGRES_{instance}_DB")
+    user = os.getenv(f"POSTGRES_{instance}_USER")
+    password = os.getenv(f"POSTGRES_{instance}_PASSWORD")
+    conn_str = f"postgresql://{user}:{password}@{host}:{port}/{database}"
+    # add an echo=True to see the SQL queries
+    conn = create_engine(conn_str)
+    return conn
+
+
+def get_associated_data(metadata):
+
+    conn = connect_to_db("recipe")
+
+    with conn.connect() as connection:
+
+        if metadata["mem_type"] == "memory":
+            query = f"""
+                SELECT
+                    result,
+                    result_type
+                FROM
+                    memory
+                WHERE
+                    custom_id = '{metadata['custom_id']}'
+            """
+            result = connection.execute(text(query))
+            result = result.fetchall()
+            metadata["result"] = result[0][0]
+            metadata["result_type"] = result[0][1]
+
+    return metadata
+
+
 def check_memory(intent, mem_type, db, chat):
     """
     Check the memory for a given intent.
@@ -306,10 +355,7 @@ def check_memory(intent, mem_type, db, chat):
         score = m["score"]
         content = m["content"]
         metadata = m["metadata"]
-        if (
-            metadata["calling_code_run_status"] != "ERROR"
-            and metadata["mem_type"] == mem_type
-        ):
+        if metadata["mem_type"] == mem_type:
             print(f"\n Reranking candidate: Score: {score} ===> {content} \n")
             # Here ask LLM to confirm our match
             prompt = f"""
@@ -326,21 +372,17 @@ def check_memory(intent, mem_type, db, chat):
             response = call_llm(prompt_map[mem_type], prompt, chat)
             print(response)
 
-            if "user_intent_output_format" in response:
-                if (
-                    response["user_intent_output_format"]
-                    != response["generic_db_output_format"]
-                ):
-                    response["answer"] = "no"
-                    response["reason"] = "output formats do not match"
-
             print("AI Judge of match: ", response)
 
             if response["answer"].lower() == "yes":
                 print("We have a match!!!")
                 r["score"] = score
                 r["content"] = content
                 r["metadata"] = metadata
+
+                # Here we query the DB to get supporting data from other tables
+                metadata = get_associated_data(metadata)
+
                 return True, r
 
     return False, r
@@ -374,10 +416,7 @@ def get_matching_candidates(intent, mem_type, db, cutoff=None):
         content = d[0].page_content
         metadata = d[0].metadata
         print("\n", f"\n\nMatches: Score: {score} ===> {content}\n\n")
-        if (
-            metadata["calling_code_run_status"] != "ERROR"
-            and metadata["mem_type"] == mem_type
-        ):
+        if metadata["mem_type"] == mem_type:
             if d[1] < cutoff:
                 print("\n", " << MATCHED >>")
                 r["score"] = score
@@ -458,8 +497,12 @@ def get_memory(user_input, chat_history, generate_intent=True) -> str:
         user_input = json.dumps(user_input)
     memory_found, result = get_recipe_memory(user_input)
     if memory_found is True:
-        response_text = result["metadata"]["response_text"]
-        response_image = result["metadata"]["response_image"]
+        if result["metadata"]["result_type"] == "image":
+            response_image = result["metadata"]["result"]
+            response_text = ""
+        else:
+            response_text = result["metadata"]["result"]
+            response_image = ""
         recipe_id = result["metadata"]["custom_id"]
         print("Recipe ID: ", recipe_id)
         if response_image is not None and response_image != "":
@@ -474,3 +517,7 @@ def get_memory(user_input, chat_history, generate_intent=True) -> str:
     else:
         result = "No memory found"
     return result
+
+
+if __name__ == "__main__":
+    get_memory("What is the population of Mali?", [], False)