Assessing LLMs suitability for KGC.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aijz1aGDUYeU"
      },
      "source": [
        "# Assessing LLMs Suitability for Knowledge Graph Completion\n",
        "\n",
        "!!! You need your own API Keys for Hugging Face and OpenAI. The notebook is intended to work on Google Colab, therefore save them there as Secrets. Save the HF token as \"HF_TOKEN\", and the OpenAI token as \"GPT_TOKEN\". The code will load them automatically, if instructions are followed.\n",
        "\n",
        "Large Language Models (LLMs) have demonstrated incredible abilities of solving diverse tasks formulated in natural language. Recent work has demonstrated their capacity to solve tasks related to Knowledge Graphs (KGs), such as KG Completion (KGC), even in Zero- or Few-Shot paradigms. However, they are known to hallucinate answers, or output results in a non-deterministic manner, thus leading to wrongly reasoned responses, even if they satisfy the user’s demands. To highlight opportunities and challenges in KG-related tasks, we experiment with three distinguished LLMs, namely Mixtral 8x7B-instruct-v0.1, GPT-3.5-turbo-0125, and GPT-4o on KGC for Static KGs, using prompts constructed following the TELeR taxonomy, in Zero- and One-Shot contexts, on a Task-Oriented Dialogue System use case. When evaluated using both strict and flexible metrics measurement manners, our results show that LLMs may be fit for such a task if prompts encapsulate sufficient information and relevant examples.\n",
        "\n",
        "All the necessary resources to reproduce the experiments are at -> https://github.com/IonutIga/LLMs-for-KGC\n",
        "The associated paper reports results on four different experiments: prompting the models using hand-written or model-rephrased system prompts on both datasets.\n",
        "\n",
        "To generate the datasets from scratch, simply load either *templates_easy.txt* or *templates_hard.txt*, in the \"Generate dataset\" sub-section.\n",
        "\n",
        "To load the datasets, simply load either *templates_easy_final.txt* or *templates_hard_final.txt*, in the \"Load dataset\" sub-section.\n",
        "\n",
        "To generate the prompts from scratch, you have to first generate/load a dataset, then enter the \"Generate prompts\" sub-section, where you have to choose the type of prompts to be attached to the input (either system_prompts or final_model_prompts).\n",
        "\n",
        "To load the prompts, simply load the desired file in the \"Load prompts\" sub-section.\n",
        "\n",
        "To run the experiments on a given prompt, simply run the \"General pipeline\" sub-section. Enter it to set variables according to your needs.\n",
        "\n",
        "To measure already existing runs (even the ones provided by the paper), enter the \"Calculate metrics from existing predictions\" sub-section, load the desired predictions either from the hub (files with \"run_...\" as name) or generated b yourself and set at which levels you wish to generate metrics."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AlCW25IgglpT"
      },
      "source": [
        "## Install and import libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "74w0CHkv74eo"
      },
      "outputs": [],
      "source": [
        "!pip install --upgrade openai # GPT library\n",
        "!pip install --upgrade tiktoken # count tokens in an input for openai\n",
        "!pip install transformers # for all other models available on HuggingFace\n",
        "!pip install pydantic # data validation\n",
        "\n",
        "# !pip install python-dotenv # load env variables\n",
        "# !pip install --upgrade accelerate bitsandbytes # GPU optimization and quantization"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VpQYbmWVYCtd"
      },
      "outputs": [],
      "source": [
        "from openai import OpenAI\n",
        "import tiktoken\n",
        "import os\n",
        "from transformers import  AutoTokenizer # to load the correct tokenizers of specific models\n",
        "import requests # to send requests to HG's serveless API\n",
        "import torch # multi purposes\n",
        "from google.colab import userdata # to load the secret keys\n",
        "import gc\n",
        "from datetime import datetime # for the current date and time\n",
        "import json\n",
        "from tqdm import tqdm # to display the loading time\n",
        "from google.colab import files # for files workloads\n",
        "import re # for regex\n",
        "from jinja2.exceptions import TemplateError # for the apply chat_templates function\n",
        "\n",
        "# might be useful for alternatives in code\n",
        "#from itertools import combinations\n",
        "# from dotenv import load_dotenv\n",
        "# from transformers import pipeline\n",
        "# from transformers import AutoModelForCausalLM\n",
        "# from huggingface_hub import InferenceClient"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "to9GnwT3hpLX"
      },
      "source": [
        "## Define functions and classes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZkCJRYDljAJx"
      },
      "outputs": [],
      "source": [
        "# class to hold information regarding each prompt fed to a model\n",
        "\n",
        "\n",
        "class Prompt:\n",
        "\n",
        "  '''\n",
        "  Parameters\n",
        "  ----------\n",
        "    prompt : str\n",
        "            The system prompt\n",
        "    text : str\n",
        "          The input text\n",
        "    golden_labels : list\n",
        "                    The list of correct triples to be predicted\n",
        "    alternative_labels: list\n",
        "                        The list of possbile alternative triples to some golden ones; Under flexible metrics, it will be counted as correct;\n",
        "                        eg. For :Project1 :hasManager :Employee1; :Employee1 rdf:type :Employee; :Employee1 :hasName \"X\"; the triple :Project1 :hasManager \"X\" is accepted;\n",
        "                        The final metrics won't subsitute whole three triples with one, but count it as a correct alternative with a 1/3 ratio\n",
        "    fp_ok_labels : list\n",
        "                  \"false positive okay labels\", the list of predictions that may be accepted as okay, but are not part of the golden labels;\n",
        "                    False positives that might have been counted as corrected if additional background knowledge was provided;\n",
        "                    eg. When the triple :Employee1 :hasRole \"manager\" is generated, as the model inferred it from the :Project1 :hasManager :Employee1;\n",
        "                    The final metrics will not count them as correct, but reduce the number of false positives\n",
        "    version : float\n",
        "              The version of the system prompt\n",
        "    level  : float\n",
        "            The level of the system prompt, on a scale of 1, 2, 3, 4.1, and 4.2\n",
        "    text_type : str\n",
        "                The type of the input text, as mentioned in the provided templates; the templates file come with a legend\n",
        "    class_type : str\n",
        "                The type of the instance mentione in the input text, from the ontology; as for now, Project, Employee, Status, or None if the type is not in the ontology\n",
        "    sys_mesg_ord : int\n",
        "                  The position of the system prompt; 0 - at the beginning of the final prompt, 1 - at the end of the final prompt\n",
        "  '''\n",
        "\n",
        "\n",
        "  def __init__(self, prompt, text, golden_labels, alternative_labels, fp_ok_labels, version, level, text_type, class_type, sys_mesg_ord = 0):\n",
        "\n",
        "    self.prompt = prompt\n",
        "    self.text = text\n",
        "    self.metadata = []\n",
        "    self.golden_labels = golden_labels\n",
        "    self.alternative_labels = alternative_labels\n",
        "    self.fp_ok_labels = fp_ok_labels\n",
        "    self.version = version\n",
        "    self.level = level\n",
        "    self.text_type = text_type\n",
        "    self.class_type = class_type\n",
        "    self.data_index = -1 # the index of the data from the dataset for later references\n",
        "\n",
        "    if sys_mesg_ord == 0:\n",
        "\n",
        "      self.messages = [{'role': 'system', 'content' : self.prompt}, {'role': 'user', 'content' : f' Input text: {self.text}'}]\n",
        "    else:\n",
        "     self.messages = [{'role': 'user', 'content' : f' Input text: {self.text}'}, {'role': 'system', 'content' : f'\\n {self.prompt}'}]\n",
        "\n",
        "  def append_metadata(self, metadata):\n",
        "    self.metadata.append(metadata)\n",
        "\n",
        "  # function to convert the Prompt object to a dictionary, for serialization purposes\n",
        "\n",
        "  '''\n",
        "  Return\n",
        "  ------\n",
        "    type : dict\n",
        "          The Prompt object converted in a dictionary\n",
        "  '''\n",
        "\n",
        "  def to_json(self):\n",
        "\n",
        "    json_prompt = {\n",
        "                    \"prompt\" : self.prompt,\n",
        "                    \"text\" : self.text,\n",
        "                    \"metadata\" : self.metadata,\n",
        "                    \"golden_labels\" : self.golden_labels,\n",
        "                    \"alternative_labels\" : self.alternative_labels,\n",
        "                    \"fp_ok_labels\" : self.fp_ok_labels,\n",
        "                    \"version\" : self.version,\n",
        "                    \"level\" : self.level,\n",
        "                    \"text_type\" : self.text_type,\n",
        "                    \"class_type\" :self.class_type,\n",
        "                    \"data_index\" : self.data_index\n",
        "                    }\n",
        "\n",
        "    return json_prompt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uKbD1kvbDHb_"
      },
      "outputs": [],
      "source": [
        "# function to read lines from a file\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  file : str\n",
        "        The name of the file to be read\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : str\n",
        "         The contained text of the file\n",
        "'''\n",
        "\n",
        "def read_lines(file):\n",
        "  f = open(file)\n",
        "  lines = f.read().splitlines()\n",
        "  f.close()\n",
        "  return lines"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ePOx2qtjQON5"
      },
      "outputs": [],
      "source": [
        "# function to count the tokens from the input message of a OpenAI's model\n",
        "# copied from their documentation\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "  ----------\n",
        "  messages : list\n",
        "            The list containing dictionaries of the role (user/system) and associated message\n",
        "  model : str\n",
        "          The name of the OpenAI model\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : int\n",
        "         The number of tokens\n",
        "'''\n",
        "\n",
        "def num_tokens_from_messages_openai(messages, model=\"gpt-3.5-turbo-0125\"):\n",
        "\n",
        "    \"\"\"Return the number of tokens used by a list of messages.\"\"\"\n",
        "    try:\n",
        "        encoding = tiktoken.encoding_for_model(model)\n",
        "    except KeyError:\n",
        "        print(\"Warning: model not found. Using cl100k_base encoding.\")\n",
        "        encoding = tiktoken.get_encoding(\"cl100k_base\")\n",
        "    if model in {\n",
        "        \"gpt-3.5-turbo-0125\",\n",
        "        \"gpt-3.5-turbo-0613\",\n",
        "        \"gpt-3.5-turbo-16k-0613\",\n",
        "        \"gpt-4-0314\",\n",
        "        \"gpt-4-32k-0314\",\n",
        "        \"gpt-4-0613\",\n",
        "        \"gpt-4-32k-0613\",\n",
        "        }:\n",
        "        tokens_per_message = 3\n",
        "        tokens_per_name = 1\n",
        "    elif model == \"gpt-3.5-turbo-0301\":\n",
        "        tokens_per_message = 4  # every message follows <|start|>{role/name}\\n{content}<|end|>\\n\n",
        "        tokens_per_name = -1  # if there's a name, the role is omitted\n",
        "    elif \"gpt-3.5-turbo\" in model:\n",
        "        print(\"Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0125.\")\n",
        "        return num_tokens_from_messages_openai(messages, model=\"gpt-3.5-turbo-0613\")\n",
        "    elif \"gpt-4\" in model:\n",
        "        print(\"Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.\")\n",
        "        return num_tokens_from_messages_openai(messages, model=\"gpt-4-0613\")\n",
        "    else:\n",
        "        raise NotImplementedError(\n",
        "            f\"\"\"num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens.\"\"\"\n",
        "        )\n",
        "    num_tokens = 0\n",
        "    for message in messages:\n",
        "        num_tokens += tokens_per_message\n",
        "        for key, value in message.items():\n",
        "            num_tokens += len(encoding.encode(value))\n",
        "            if key == \"name\":\n",
        "                num_tokens += tokens_per_name\n",
        "    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>\n",
        "    return num_tokens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X7ct3mnz-hsl"
      },
      "outputs": [],
      "source": [
        "# function to count the tokens from the input message of HG models\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "  ----------\n",
        "  tokenized_chat : str\n",
        "            The text fed into the model, after applying its template design\n",
        "  tokenizer : Tokenizer\n",
        "          The tokenizer specific to the model\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : int\n",
        "         The number of tokens\n",
        "'''\n",
        "def num_tokens_from_messages(tokenized_chat, tokenizer):\n",
        "  return len(tokenizer.encode(tokenized_chat))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c8wprKW6GAib"
      },
      "outputs": [],
      "source": [
        "# function to view the predictions of the models from the checkpoint\n",
        "# it takes as input a list of dictionaries that contain other lists of Prompt objects\n",
        "# it loops through each prompt while prompting useful information from it\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "  ----------\n",
        "  predictions : list\n",
        "                The list of dictionaries that contain other lists of Prompt objects which hold the predictions of each model\n",
        "  all_prompts : list\n",
        "                The list of dictionaries that contain other lists of Prompt objects in their initial state; here used only to provide the initial number of Prompts to ensure that all predictions are generated\n",
        "  show_system_prompt : bool\n",
        "                Show the system prompt, which contains the directives to the model, without the input to work on\n",
        "  generate_model_prompts : bool\n",
        "                If set to true, the function will interpret each prediction as a new system prompt specific to a model and output a new set of prompts; eg. when we ask the models to reformulate the standard prompts\n",
        "  model_prompts: dict\n",
        "                If generate_model_prompts = True, and a dict is provided, the model will append to it the generated model-specific system prompts\n",
        "'''\n",
        "\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  If generate_model_prompts = True\n",
        "    type : dict\n",
        "           The dictionary containing the output prompts\n",
        "  Else\n",
        "    Prints the content of each Prompt object\n",
        "'''\n",
        "\n",
        "def view_predictions(predictions, all_prompts, show_system_prompt = True, generate_model_prompts = False, model_prompts = {}):\n",
        "\n",
        "  for i, pred_per_model in enumerate(predictions):\n",
        "\n",
        "    for model, pred_list in pred_per_model.items():\n",
        "      print(f'Predictions: {len(pred_list)}/{len(all_prompts[i][model])}')\n",
        "\n",
        "      for pred in pred_list:\n",
        "\n",
        "        print('-' * 25, f'Data instance index {pred.data_index}, prompt version {pred.version} and level {pred.level}, class type {pred.class_type}, text type {pred.text_type}', '-' * 25, '\\n')\n",
        "\n",
        "        try:\n",
        "\n",
        "          for metadata in pred.metadata: # each prediction of a model is hold as a dictionary in the metadata of a prompt\n",
        "\n",
        "            version_dict = {}\n",
        "            version_list = []\n",
        "            level_text = {}\n",
        "            flag = False\n",
        "\n",
        "            print('*' * 25, f'Model: {metadata[\"model\"]}', '*' * 25) if 'model' in metadata.keys() else print('*' * 25, f'Model: unknown', '*' * 25)\n",
        "\n",
        "            if generate_model_prompts: # if the goal of the predictions was to reformulate/generate model-specific system prompts\n",
        "\n",
        "              if metadata[\"model\"] in model_prompts.keys():\n",
        "                    version_dict = model_prompts[metadata[\"model\"]]\n",
        "\n",
        "              if pred.version in version_dict.keys():\n",
        "                version_list = version_dict[pred.version] # override the existing system propmts, if at the same version\n",
        "\n",
        "              for i,lt in enumerate(version_list):\n",
        "                if pred.level in lt.keys():\n",
        "                  flag = i\n",
        "              if flag != False: # override the existing system prompts, if at same level\n",
        "                version_list[flag][pred.level] = metadata['generated_text']\n",
        "              else:\n",
        "                version_list.append({pred.level : metadata['generated_text']})\n",
        "\n",
        "              version_dict[pred.version] = version_list\n",
        "              model_prompts[metadata[\"model\"]] = version_dict\n",
        "\n",
        "\n",
        "            print(f'\\nTime to generate (s): {int(metadata[\"processing-ms\"]) / 1000}')\n",
        "            print(f'Number of tokens of the prompt: {metadata[\"tokens\"]}')\n",
        "            if show_system_prompt:\n",
        "              print(f'Prompt: {metadata[\"prompt\"]}')\n",
        "            print(f'Generated text: {metadata[\"generated_text\"]}')\n",
        "            print(f'Alternative labels: {pred.alternative_labels}')\n",
        "            print(f'FP accepted labels: {pred.fp_ok_labels}')\n",
        "            print(f'Golden labels: {pred.golden_labels}\\n')\n",
        "\n",
        "        except:\n",
        "          print(\"Error in parsing the Prompt object.\")\n",
        "\n",
        "  if generate_model_prompts:\n",
        "    return model_prompts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ExbV6UGEzFq5"
      },
      "outputs": [],
      "source": [
        "# function to create prompts that indicate a reprhasing task; to obtain new, model-specific system prompts\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  system_prompts : list\n",
        "                   List containing Prompt objects; should only be objects that have different levels and/or versions of the system prompt\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : list\n",
        "         The list of prompts with the task of rephrasing the input text; the list has a dictionary structure similar to the one obtained via generating the standard prompts\n",
        "'''\n",
        "\n",
        "def rephrase_system_prompts(system_prompts):\n",
        "  rephrase_prompt = 'You are given a test prompt delimited by double quotes. Your task is to rephrase the prompt in order for you to better understand it, without changing the specified output format and any provided examples. Output only the rephrased prompt in double quotes. \\n Input prompt: '\n",
        "  rr_prompts = [{'all' : []}]\n",
        "  for prompt in system_prompts:\n",
        "    p = Prompt(rephrase_prompt,f' \"{prompt.prompt}\"',['None'],['None'],['None'], prompt.version, prompt.level, 'Rephrase', 'None', 0) # no golden/alternative/fp_ok labels needed\n",
        "    rr_prompts[0]['all'].append(p)\n",
        "  return rr_prompts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2SOZiBIxGiON"
      },
      "outputs": [],
      "source": [
        "# function to generate the Prompt objects\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  system_prompts : dict\n",
        "                   The dictionary containing the system prompts of a model/all models, on versions and levels\n",
        "  dataset : list\n",
        "            The list containing the dictionaries with the input text, golden/alternative/fp_ok labels, text and class type\n",
        "  versions : list\n",
        "             The list with the versions of the system prompts to be included; 'all' is default for all versions\n",
        "  levels : list\n",
        "             The list with the levels of the system prompts to be included; 'all' is default for all levels\n",
        "  sys_mesg_ord : int\n",
        "                  The position of the system prompt; 0 - at the beginning of the final prompt, 1 - at the end of the final prompt\n",
        "  file : str\n",
        "         The name of the file to place the serialized Prompt objects for later reuse\n",
        "  download : bool\n",
        "             Option to automatically download the file\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : list\n",
        "         The list of all generated prompts; alternatively, it generates a text file of the serialized Prompt objects\n",
        "'''\n",
        "\n",
        "def generate_prompts(system_prompts, dataset, versions = ['all'], levels = ['all'], sys_mesg_ord = 0, file = 'prompts.txt', download = True):\n",
        "\n",
        "  all_prompts = []\n",
        "\n",
        "  i = 0\n",
        "\n",
        "  for model, versionss in system_prompts.items(): # loop through all system prompts specific to a models/all models' version(s)\n",
        "\n",
        "    prompts_per_model = {}\n",
        "    prompts_per_model[model] = []\n",
        "\n",
        "    for version, prompts in versionss.items(): # loop through all prompts at a specific version\n",
        "\n",
        "      if version in versions or versions[0] == 'all':\n",
        "        for prompt in tqdm(prompts):\n",
        "          for level, text in prompt.items(): # loop through prompts at certain levels\n",
        "            for j, instance in enumerate(dataset): # loop through each dataset instance; the final Prompt objects will contain each instance at a certain level, followed by the next level and so on\n",
        "              if level in levels or levels[0] == 'all':\n",
        "                if level in [4.1, 4.2]: # this labels are few-shot ones, therefore they need an example to be attached to them\n",
        "                  if instance['golden_labels'] == 'None': # attach appropiate example to the golden labels\n",
        "                    p = Prompt(text + example_none if level == 4.1 else text + example_none_cot, instance['text'], instance['golden_labels'],instance['alternative_labels'],instance['fp_ok_labels'], version, level, instance['text_type'], instance['class_type'], sys_mesg_ord = sys_mesg_ord)\n",
        "                  else:\n",
        "                    p = Prompt(text + example_labels if level == 4.1 else text + example_labels_cot, instance['text'], instance['golden_labels'], instance['alternative_labels'],instance['fp_ok_labels'], version, level, instance['text_type'], instance['class_type'], sys_mesg_ord = sys_mesg_ord)\n",
        "                else:\n",
        "                  p = Prompt(text, instance['text'], instance['golden_labels'], instance['alternative_labels'],instance['fp_ok_labels'],  version, level, instance['text_type'], instance['class_type'],sys_mesg_ord = sys_mesg_ord)\n",
        "\n",
        "              p.data_index = j\n",
        "              prompts_per_model[model].append(p)\n",
        "              write_prompt(p, file)\n",
        "              i += 1\n",
        "\n",
        "    all_prompts.append(prompts_per_model)\n",
        "\n",
        "  if download:\n",
        "    files.download(file)\n",
        "\n",
        "  return all_prompts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_3kMElfUbCce"
      },
      "outputs": [],
      "source": [
        "# function to write a Prompt object as a text serialization\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  prompt : Prompt\n",
        "           The Prompt object\n",
        "  file : str\n",
        "         The name of the final where to write the serialization\n",
        "'''\n",
        "\n",
        "def write_prompt(prompt, file = 'prompts.txt'):\n",
        "  f = open(file, 'a')\n",
        "  f.write(str(prompt.to_json()))\n",
        "  f.write('\\n')\n",
        "  f.close()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QyBujBBQiLZN"
      },
      "outputs": [],
      "source": [
        "# function to deserialize the text version of a Prompt object\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  file : str\n",
        "         The file from which to read the text\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : list\n",
        "         The list with the deserialized Prompt objects; similar format to the generate_prompts function\n",
        "'''\n",
        "\n",
        "def from_text_to_Prompt(file = 'prompts.txt'):\n",
        "  all_prompts = []\n",
        "  prompts = {'all' : []} # put them under the label of 'all', even if they were originally under a model's name; after all, it doesn't really matter as the Prompt object metadata contains the model's name under each prediction\n",
        "  lines = read_lines(file)\n",
        "  for line in tqdm(lines):\n",
        "    prompt = eval(line) # eval it, generate a dictionary back\n",
        "    new_prompt = Prompt(prompt['prompt'], prompt['text'], prompt['golden_labels'], prompt['alternative_labels'], prompt['fp_ok_labels'], prompt['version'], prompt['level'], prompt['text_type'], prompt['class_type'])\n",
        "    new_prompt.metadata = prompt['metadata']\n",
        "    new_prompt.data_index = prompt['data_index']\n",
        "    prompts['all'].append(new_prompt)\n",
        "\n",
        "  all_prompts.append(prompts)\n",
        "  return all_prompts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TA-NOrcqqVog"
      },
      "outputs": [],
      "source": [
        "# function to generate a dataset from a text file where serialized instances were output by generate_dataset function\n",
        "# useful to not regenerate the dataset each time you run the experiments\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  file : str\n",
        "         Name of the file from where to read the text\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : list\n",
        "         The dataset, similar to generate_dataset function\n",
        "'''\n",
        "\n",
        "def from_text_to_dataset(file):\n",
        "  dataset = []\n",
        "  lines = read_lines(file)\n",
        "  for line in tqdm(lines):\n",
        "    instance = eval(line)\n",
        "    dataset.append(instance)\n",
        "  return dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jjaejY8nQGjw"
      },
      "outputs": [],
      "source": [
        "# function to generate a dataset from a template text file (specific template style)\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  file : str\n",
        "         Path of the file that stores the templates to be read\n",
        "  dataset : list\n",
        "            If a dataset is already generated and one wishes to add to it, they can provide an already existing list\n",
        "  download : bool\n",
        "             Option to download the text file with the generated dataset\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : list\n",
        "         The list with the instances containing the input text, golden/alternative/fp_ok labels, text and class type\n",
        "'''\n",
        "\n",
        "def generate_dataset(file = \"\", dataset = [], download = True):\n",
        "\n",
        "  lines = read_lines(file)\n",
        "  dataset_file_path = f\"{file[:-4]}_final.txt\"\n",
        "\n",
        "  for line in tqdm(lines):\n",
        "\n",
        "    instance = {}\n",
        "    part_lines = line.split(\"*\") # each line has input_text*relationships*objects*intent*text_type\n",
        "\n",
        "    instance[\"text\"] = f\"{part_lines[0]}.\"\n",
        "    golden_labels = []\n",
        "    alternative_labels = []\n",
        "    fp_ok_labels = []\n",
        "    instance_type = \"\"\n",
        "\n",
        "    if part_lines[3] == \"insert\": # for the moment, only the texts with the intent type of 'insert' are accepted\n",
        "      for k, v in zip(part_lines[1].split(\"|\"), part_lines[2].split(\"|\")): # go in parallel over relationship-object pairs\n",
        "\n",
        "        if k == \"entity\": # if entity, create the appropiate id as 'Class1'\n",
        "          instance_type = v.capitalize()\n",
        "          id = instance_type + \"1\" # a simple way to create an id not to confuse the models, as for now\n",
        "\n",
        "          if instance_type in [\"Project\", \"Status\", \"Employee\"]: # only accept the supported classes from the ontology, else is None\n",
        "            golden_labels.append({\"subject\" : f\"{id}\", \"relationship\" : \"rdf:type\", \"object\" : instance_type})\n",
        "          else:\n",
        "            golden_labels = \"None\"\n",
        "            break\n",
        "\n",
        "        elif instance_type == \"Project\" and k in [\"hasManager\",\"hasStatus\"]: # these relationships might refer to other instances\n",
        "\n",
        "          sub_instance_type = \"Employee\" if k == \"hasManager\" else \"Status\"\n",
        "          sub_id = v.split(\".\")[1] if v.startswith(\"id.\") else sub_instance_type + \"1\" # id follows the same format as above, if not provided with the template\n",
        "\n",
        "          if v.startswith(\"role.\") == False:\n",
        "\n",
        "            if v.startswith(\"id.\") == False: # if no id is provided by the template, we can assume the text refers to the employee's name\n",
        "              golden_labels.append({\"subject\" : f\"{sub_id}\", \"relationship\" : \"hasName\", \"object\" : v})\n",
        "              alternative_labels.append({\"subject\" : f\"{id}\", \"relationship\" : k, \"object\" : v}) # add as alternative label :Project1 :hasManager \"name\"\n",
        "\n",
        "            if k == \"hasManager\": # then triples referring to their role as manager are fp_ok accepted\n",
        "              fp_ok_labels.append({\"subject\" : f\"{sub_id}\", \"relationship\" : \"hasRole\", \"object\" : \"Manager\"})\n",
        "              fp_ok_labels.append({\"subject\" : f\"{sub_id}\", \"relationship\" : \"hasRole\", \"object\" : \"manager\"})\n",
        "\n",
        "          else: # when role is provided, it is appended to the golden labels\n",
        "            golden_labels.append({\"subject\" : f\"{sub_id}\", \"relationship\" : \"hasRole\", \"object\" : v.split(\".\")[1]})\n",
        "\n",
        "          golden_labels.append({\"subject\" : f\"{id}\", \"relationship\" : k, \"object\" : f\"{sub_id}\"}) # the direct relationship between instances\n",
        "          golden_labels.append({\"subject\" : f\"{sub_id}\", \"relationship\" : \"rdf:type\", \"object\" : f\"{sub_instance_type}\"}) # the rdf:type relationship of the sub instance\n",
        "\n",
        "        else: # any other relationship\n",
        "          golden_labels.append({\"subject\" : f\"{id}\", \"relationship\" : k, \"object\" : v})\n",
        "\n",
        "      instance[\"golden_labels\"] = f'{golden_labels}' if type(golden_labels) is list else golden_labels # if None is the only golden label provided, then there is no list, just \"None\"\n",
        "      instance[\"alternative_labels\"] = f'{alternative_labels}'\n",
        "      instance[\"fp_ok_labels\"] = f'{fp_ok_labels}'\n",
        "      instance[\"class_type\"] = instance_type\n",
        "      instance[\"text_type\"] = part_lines[4]\n",
        "\n",
        "      dataset.append(instance)\n",
        "\n",
        "      f = open(dataset_file_path, \"a\") # write it in the text file\n",
        "      f.write(str(instance))\n",
        "      f.write(\"\\n\")\n",
        "      f.close()\n",
        "\n",
        "  if download:\n",
        "    files.download(dataset_file_path)\n",
        "\n",
        "\n",
        "  return dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Us6zAzKUvisJ"
      },
      "outputs": [],
      "source": [
        "# function to find the required json objects in the output of the models and to serialize provided golden/alternative/fp_ok labels to the same format with the output\n",
        "# it allows two metrics measurements styles: strict and flexible;\n",
        "# strict metrics needs the output to exactly follow the instructions given in the system prompts\n",
        "# flexible metrics allows non-conforming outputs to still be allowed, with a penalization that can be applied to the accuracy metric;\n",
        "# it assumes '1' is the max possible accuracy value, therefore: 0.1 max pen for not outputing in a list (only one) format; 0.01 for providing the full IRI of a value; 0.01 for adding ':' in front of a value; 0.1 if a triple doesn't follow the provided format, while also not adding it\n",
        "# the strict paradigm allows for full verification of models respecting the input prompt, while flexible allows for some room where post-processing steps can convert the output in a correct format\n",
        "# !!! GPT models have a tendency to surround their output with a JSON format tag (i.e. \"```json ... ```\"), due to their ability to output JSON-formatted output when requested; while on strict metrics, we want the output to be as requested (namely just a list of JSON objects). We have to do an extra processing step that we consider trivial, thus not count it as an error\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  text : str\n",
        "         The text to be searched for json objects\n",
        "  strict : bool\n",
        "           True for the strict paradigm, False for the flexible one\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : (str/list, float)\n",
        "         Return a tuple containing either 'er1ror'/'incorrect format'/list of json objects and a float number represting the penalty value\n",
        "'''\n",
        "\n",
        "def find_json_objects(text, strict = False):\n",
        "\n",
        "  pattern = r'\\{(?:[^{}]|)*\\}' # Regular expression pattern to match JSON objects\n",
        "  penalize = 0\n",
        "  flag = False # flag to check whether the output json object respects the provided format\n",
        "\n",
        "  text = text.strip()\n",
        "\n",
        "  if text.startswith('```json'): # GPT models surround their output with a JSON format tag, as they can generate such text when requested; we process it out\n",
        "    text = text.replace('```json','')\n",
        "\n",
        "  text = text.strip()\n",
        "\n",
        "  if text.endswith('```'): # as well as the end tag\n",
        "    text = text.replace('```','')\n",
        "\n",
        "  text = text.strip() # remove any trailing whitespaces\n",
        "\n",
        "  if text.startswith('er1ror'): # custom text to mark an error from the server; written so it is not missunderstood as an actual correct text\n",
        "    print(text)\n",
        "    return 'error', penalize\n",
        "\n",
        "  json_objects = re.findall(pattern, text) # find json objects; all of them from the output, even if the model rewrites some examples as in COT type prompts\n",
        "\n",
        "  if not json_objects:\n",
        "\n",
        "    if strict: # on strict, only \"None\" has to be the output\n",
        "\n",
        "      if text == 'None':\n",
        "\n",
        "        parsed_objects = [\"None\"]\n",
        "\n",
        "      else:\n",
        "        return 'incorrect format', penalize # else it is an incorrect format output\n",
        "\n",
        "    else:\n",
        "      parsed_objects = [\"None\"] # flexible allows for any output that does not contain json objects to be considered as \"None\"\n",
        "\n",
        "\n",
        "  else: # Parse each JSON object and store them in a list\n",
        "\n",
        "    list_starts = re.findall('\\[',text)\n",
        "    if len(list_starts) > 1: # the format requires only one list to be output\n",
        "      if strict == False:\n",
        "        penalize += 0.025\n",
        "      else:\n",
        "        return 'incorrect format', 0\n",
        "\n",
        "    if text.startswith('[') == False: # the output must contain only the list\n",
        "      if strict == False:\n",
        "        penalize += 0.075\n",
        "      else:\n",
        "        return 'incorrect format', 0\n",
        "\n",
        "    elif text.endswith(']') == False: # and finish as a list\n",
        "      if strict == False:\n",
        "        penalize += 0.075\n",
        "      else:\n",
        "        return 'incorrect format', 0\n",
        "\n",
        "    parsed_objects = []\n",
        "\n",
        "    for obj in json_objects:\n",
        "        try:\n",
        "\n",
        "            if strict: # no post-processing to be done\n",
        "              try:\n",
        "                eval_obj = eval(obj)\n",
        "                parsed_objects.append(eval_obj)\n",
        "              except:\n",
        "                pass\n",
        "\n",
        "            else:\n",
        "\n",
        "              transformed_text = re.sub(r'(\\S+)#(\\S+)', '\"' + r'\\2', obj) # remove IRI with #, eg. http://www.example.org/john#Employee\n",
        "              if transformed_text != obj:\n",
        "                penalize += 0.01 # penalize each IRI removal\n",
        "\n",
        "              transformed_text2 = re.sub(r'(\\S+)/(\\S+)', '\"' + r'\\2', transformed_text) # remove IRI with /, eg. http://www.example.org/john/Employee\n",
        "\n",
        "              if transformed_text2 != transformed_text:\n",
        "                penalize += 0.01 # penalize each IRI removal\n",
        "              try:\n",
        "                triple = eval(transformed_text2)\n",
        "              except:\n",
        "                triple = ''\n",
        "\n",
        "              if type(triple) is dict: # allow only dict/json objects triples\n",
        "\n",
        "                if len(triple.keys()) == 3: # only three keys are needed as for the format\n",
        "\n",
        "                  for k, v in triple.items():\n",
        "                    flag = False\n",
        "                    if k.lower() in ['subject', 'relationship', 'object']: # only this keys are allowed\n",
        "\n",
        "                      if type(v) is not list:\n",
        "\n",
        "                        if type(v) == str and v.startswith(':'):\n",
        "                          triple[k] = v[1:] # remove : from value texts, eg. 'subject' : ':Employee1'\n",
        "                          penalize += 0.01 # penalize the removal\n",
        "                    else:\n",
        "                      flag = True\n",
        "                else:\n",
        "                  flag = True\n",
        "\n",
        "                if flag != True: # allow correctly formatted triples\n",
        "\n",
        "                  parsed_objects.append(triple)\n",
        "\n",
        "                elif strict != True: # penalize each wrong triple under the flexible paradigm\n",
        "\n",
        "                  penalize += 0.1\n",
        "\n",
        "\n",
        "\n",
        "        except json.JSONDecodeError:\n",
        "            pass  # Ignore any invalid JSON objects\n",
        "\n",
        "  return parsed_objects, penalize"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b_YoCbenvkHt"
      },
      "outputs": [],
      "source": [
        "# function to check the validity of a triple against a provided list\n",
        "# it also allows the flexible paradigm, by changing the ID (last character changed to 1) to check if the model's mistake was only related to wrong number concatenated to the class name\n",
        "# it makes sure that repetition of triples (the model outputs the same triples more times) are not allowed, while also keeping track of the least penalized triple for a golden one\n",
        "# it also allows only the verification of the validity of a single ID\n",
        "# we assume 1 is the highest metric value for a triple, therefore: 0.33 penalty is given for each wrong ID\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  triple : dict\n",
        "           The dictionary containing the json object/triple\n",
        "  only_IDs : bool\n",
        "             Flag for only checking for the ID correctness, rather of the whole triple\n",
        "  valid_triples : list\n",
        "                  The list of valid triples against which to check the triple\n",
        "  checked_index : list[list,list]\n",
        "                  A list containing other two lists, one containing the index of the triple in the valid_triples list, while the second stores the associated penalty\n",
        "  valid_ids : list\n",
        "              List of valid IDs format, regarding the provided ontology and format\n",
        "  class_range_rels : list\n",
        "                     List containing those relationships that require other instance, as suggested by the rdfs:range relationship from the ontology\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : (bool, list[list, list])\n",
        "         A tuple with the boolean value of the validity evaluation, with the list of checked_index\n",
        "'''\n",
        "\n",
        "def check_triple(triple, only_IDs = False, valid_triples = [], checked_index = [[],[]], valid_ids = ['Project1', 'Employee1', 'Status1'], class_range_rels = ['hasStatus', 'hasManager']):\n",
        "\n",
        "        penalty = 0\n",
        "\n",
        "        if triple['subject'] in valid_ids:\n",
        "\n",
        "          if only_IDs:\n",
        "            return True, checked_index # if only the validity of the ID is asked for\n",
        "\n",
        "        else:\n",
        "          triple['subject'] = triple['subject'][:len(triple['subject']) - 1] + '1' # change the last char of the ID to '1', to check for small format errors\n",
        "\n",
        "          if triple['subject'] in valid_ids:\n",
        "            penalty += 0.33 # penalize the wrong ID\n",
        "\n",
        "            if only_IDs:\n",
        "              return True, checked_index\n",
        "\n",
        "          elif only_IDs:\n",
        "            return False, checked_index # if the ID is not valid and we only check its validity, return\n",
        "\n",
        "        if triple in valid_triples:\n",
        "\n",
        "          index = valid_triples.index(triple) # store the referred golden triple's index\n",
        "\n",
        "          if index not in checked_index[0]:\n",
        "            checked_index[0].append(index) # add it to the list\n",
        "            checked_index[1].append(penalty) # with its related penalty\n",
        "\n",
        "            return True, checked_index\n",
        "\n",
        "          elif checked_index[1][checked_index[0].index(index)] < penalty: # if already in the list, check if this triple has a lower penalty value and replace it\n",
        "            checked_index[0][checked_index[0].index(index)] = index\n",
        "            checked_index[1][checked_index[0].index(index)] = penalty\n",
        "\n",
        "          else:\n",
        "\n",
        "            return False, checked_index # it means the found triple is not better than an already existing variant for the golden one\n",
        "\n",
        "        elif triple['relationship'] in class_range_rels: # if not yet valid, check the object's ID, if the relationship requires such object\n",
        "\n",
        "          triple['object'] = triple['object'][:len(triple['object']) - 1] + '1' # same ID modification as for the subject\n",
        "\n",
        "          if triple in valid_triples:\n",
        "            penalty += 0.33 # penalize the wrong object ID format\n",
        "\n",
        "            # same as above\n",
        "            index = valid_triples.index(triple)\n",
        "            if index not in checked_index[0]:\n",
        "              checked_index[0].append(index)\n",
        "              checked_index[1].append(penalty)\n",
        "              return True, checked_index\n",
        "\n",
        "            elif checked_index[1][checked_index[0].index(index)] < penalty:\n",
        "              checked_index[0][checked_index[0].index(index)] = index\n",
        "              checked_index[1][checked_index[0].index(index)] = penalty\n",
        "\n",
        "            else:\n",
        "\n",
        "              return False, checked_index\n",
        "\n",
        "          else:\n",
        "            return False, checked_index # no valid triple, even after modifications\n",
        "\n",
        "        else: # no valid triple, even after modifications\n",
        "          return False, checked_index"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TWJCQcvTUZ8h"
      },
      "outputs": [],
      "source": [
        "# function to calculate the metrics (accuracy, precision, recall) and existence of error or incorrect format triples\n",
        "# allows both types of paradigms, strict and flexible\n",
        "# we assume 1 as the maximum metric value for a triple, therefore: we deduct the penalty out of 1 for each triple, where it exists; for the fp_ok triples, we deduct the total number of triples, to allow a small bonus for the model\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  predicted_triples : list\n",
        "                      The list of predicted triples\n",
        "  golden_triples : list\n",
        "                      The list of golden triples\n",
        "  alternative_triples : list\n",
        "                      The list of alternative triples\n",
        "  fp_ok_triples : list\n",
        "                      The list of fp_ok triples\n",
        "  strict : bool\n",
        "           True means strict measurement of the triples vailidity, False means flexible measurement\n",
        "  class_range_rels : list\n",
        "                     List containing those relationships that require other instance, as suggested by the rdfs:range relationship from the ontology\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : (float, float, float, int, int)\n",
        "         A tuple of accuracy, precision, recall, error, incorrect_format\n",
        "'''\n",
        "def calculate_acc_precision_recall(predicted_triples, golden_triples, alternative_triples, fp_ok__triples, strict = True, class_range_rels = ['hasStatus', 'hasManager']):\n",
        "\n",
        "    correct_count = 0 # number of correct triples\n",
        "    precision_pass = 0 # number of false positive triples that are allowed under flexible paradigm\n",
        "    checked_index_strict = {'golden_triples' : [], 'alternative_triples' : [], 'fp_ok__triples' : []} #indexes of verified triples, to avoid duplicate triples allowance\n",
        "    checked_index_loose = {'golden_triples' : [[],[]], 'alternative_triples' : [[],[]], 'fp_ok__triples' : [[],[]]} # indexes of verified golden triples, along with their associated penalty value, to redeem only the highest scoring triples and avoiding triples' duplicates\n",
        "\n",
        "    if predicted_triples == 'error':\n",
        "      return 0, 0, 0, 1, 0 # increase only the error counter by one\n",
        "    if predicted_triples == 'incorrect format':\n",
        "      return 0, 0, 0, 0, 1 # increase only the incorrect format counter by one\n",
        "\n",
        "    for i, triple1 in enumerate(predicted_triples):\n",
        "\n",
        "      if triple1 in golden_triples:\n",
        "        index = golden_triples.index(triple1)\n",
        "        if index not in checked_index_strict['golden_triples']:\n",
        "          checked_index_strict['golden_triples'].append(index)\n",
        "          correct_count += 1\n",
        "\n",
        "\n",
        "\n",
        "      elif strict == False and golden_triples[0] != \"None\" and triple1 != \"None\": # enable felxible metrics check\n",
        "\n",
        "        status, checked_index_loose['golden_triples'] = check_triple(triple1.copy(), valid_triples = golden_triples, checked_index = checked_index_loose['golden_triples'])\n",
        "\n",
        "        if status == False and alternative_triples: # check if the triple is in the alternative triples\n",
        "\n",
        "          # a bug (unknown at the moment, does not let the function run properly, unless we print the results of the function first, then use it)\n",
        "          print(check_triple(triple1.copy(), valid_triples = alternative_triples, checked_index = checked_index_loose['alternative_triples']))\n",
        "          status, checked_index_loose['alternative_triples'] = check_triple(triple1.copy(), valid_triples = alternative_triples, checked_index = checked_index_loose['alternative_triples'])\n",
        "\n",
        "          if status == False and fp_ok__triples: # check if the triple is in the fp_ok triples\n",
        "\n",
        "            valid_ids = ['Employee1', fp_ok__triples[0]['subject']] if fp_ok__triples[0] != 'None' else ['Employee1'] # only allow Employee1 and the provided employee's ID as valid\n",
        "            status, checked_index_loose['fp_ok__triples'] = check_triple(triple1.copy(), valid_triples = fp_ok__triples, valid_ids = valid_ids, checked_index = checked_index_loose['fp_ok__triples'])\n",
        "\n",
        "    for k, v in checked_index_loose.items():\n",
        "\n",
        "      for i, p in zip(v[0], v[1]):\n",
        "\n",
        "        if i not in checked_index_strict[k]:\n",
        "\n",
        "          if k in ['golden_triples', 'alternative_triples']:\n",
        "            correct_count += 1 - p # add as correct only the part of the triple without the penalty\n",
        "\n",
        "          elif k == 'fp_ok__triples':\n",
        "            precision_pass += 1 - p # pass only the part of the triple that is valid, without the penalty\n",
        "\n",
        "\n",
        "    accuracy = correct_count / len(golden_triples) # accuracy is the total number of correct triples over the total number of golden triples\n",
        "    recall = accuracy # as we measure here the metrics, recall is always equal with accuracy, as we have no other labels apart from being correct\n",
        "    try: # avoid divison by zero\n",
        "      precision = correct_count / len(predicted_triples) if strict else correct_count / (len(predicted_triples) - precision_pass) # allow the flexible paradigm, by not counting the fp_ok triples\n",
        "    except:\n",
        "      precision = 0\n",
        "\n",
        "    return accuracy,precision, recall, 0, 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OJp8N2JjUduq"
      },
      "outputs": [],
      "source": [
        "# function to place the metrics in the relevant list\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  metrics : list\n",
        "            The list of metrics\n",
        "  accuracy : float\n",
        "             The accuracy metric score\n",
        "  precision : float\n",
        "              The precision metric score\n",
        "  recall : float\n",
        "           The recall metric score\n",
        "  penalize : float\n",
        "             The penalty value\n",
        "  error : int\n",
        "          The total number of errors generated by the server\n",
        "  incorrect_format : int\n",
        "                     The total number of incorrect formatted model outputs\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : list\n",
        "         The metrics' list, with the aformentioned metrics placed in it\n",
        "'''\n",
        "def place_metrics(metrics, accuracy, precision, recall, penalize, error, incorrect_format):\n",
        "\n",
        "  metrics[0] += 1 # increase the number of encountered instances\n",
        "  metrics[1] += accuracy\n",
        "  metrics[2] += precision\n",
        "  metrics[3] += recall\n",
        "  metrics[4] += penalize\n",
        "  metrics[5] += error\n",
        "  metrics[6] += incorrect_format\n",
        "\n",
        "  return metrics"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0BAlpOvzUhA9"
      },
      "outputs": [],
      "source": [
        "# function to calculate target metrics, such as accuracy, precision, recall, f_penalize, f1, error, incorrect_format\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  metrics : list\n",
        "            The list of metrics\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : (float, float, float, float, float, int, int)\n",
        "  The tuple containing the values for accuracy, precision, recall, f_penalize, f1, error, incorrect_format\n",
        "'''\n",
        "def calculate_metrics(metrics):\n",
        "\n",
        "  nr_instances = metrics[0] - metrics[5] # when errors were generated, those cannot be counted, therefore deduct them from the total number\n",
        "\n",
        "  if nr_instances:\n",
        "    accuracy = round(metrics[1]/nr_instances, 2)\n",
        "    precision = round(metrics[2]/nr_instances, 2)\n",
        "    recall = round(metrics[3]/nr_instances, 2)\n",
        "    f_penalize = round(metrics[4]/nr_instances ,2)\n",
        "    error = metrics[5]\n",
        "    incorrect_format = metrics[6]\n",
        "    try:\n",
        "      f1 = round(2 * (precision * recall) / (precision + recall),2)\n",
        "    except:\n",
        "      f1 = 0\n",
        "\n",
        "  else: # if all instances generated errors\n",
        "    return 0,0,0,0,0, metrics[5], metrics[6]\n",
        "\n",
        "  return accuracy, precision, recall, f_penalize, f1, error, incorrect_format"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zGP0boowv0Pd"
      },
      "outputs": [],
      "source": [
        "# function to generate the metrics (accuracy, precision, recall, error absolute rate, incorrect format absolute rate) at each combination of levels\n",
        "# as input, the user has to specify what combination of levels they want their metrics to be calculated at, from:\n",
        "# 0 - at model level, 1 - at prompt's level, 2 - at prompt's version, 3 - at the class type (Project, Employee, Status), 4 - at input text's type\n",
        "# eg. providing [0], will have the metrics for each model's performance, [1] gives the performance of all models at a specific prompt level, while [0, 1] will give the permformance of each model at a specific prompt's level\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  predictions : list\n",
        "                The list containing the Prompt objects with the predicted outputs\n",
        "  file : str\n",
        "         Path of the file where to write the metrics\n",
        "  levels : list\n",
        "           The list of coresponding combined levels at which each metrics to be generated; 'all' generates for the combination of all levels\n",
        "  template_strict : bool\n",
        "                    True means strict paradigm at find_json_objects(), False means flexible paradigm\n",
        "  triple_strict : bool\n",
        "                    True means strict paradigm at calculate_acc_precision_recall(), False means flexible paradigm\n",
        "  download : bool\n",
        "             Option to download the text file with the generated metrics\n",
        "'''\n",
        "\n",
        "def view_metrics(predictions, file, levels = ['all'], template_strict = False, triple_strict = False, download = True):\n",
        "\n",
        "  model_acc = {} # store the results at each desired level\n",
        "  levels_names = []\n",
        "\n",
        "  for prompts in predictions:\n",
        "\n",
        "    for kk, vv in prompts.items():\n",
        "\n",
        "      for i in tqdm(range(len(vv))):\n",
        "\n",
        "        pred = vv[i]\n",
        "\n",
        "        for j,p in enumerate(pred.metadata): # check the Prompt's metadata, where the predictions of each model are stored\n",
        "\n",
        "          model = p['model']\n",
        "          levels_names = [model, pred.level, pred.version, pred.class_type, pred.text_type]\n",
        "\n",
        "          metrics_category = ''\n",
        "\n",
        "          if levels[0] != 'all':\n",
        "\n",
        "            for I in levels:\n",
        "              metrics_category += f'_{levels_names[I]}' if I <= len(levels_names) - 1 else ''\n",
        "          else:\n",
        "            for l in levels_names:\n",
        "              metrics_category += f'_{l}'\n",
        "\n",
        "          metrics_category = metrics_category[1:]\n",
        "\n",
        "          if metrics_category not in model_acc.keys():\n",
        "            model_acc[metrics_category] = [0,0,0,0,0,0,0] # nr_instance, acc, precision, recall, penalize, error, incorrect format\n",
        "\n",
        "\n",
        "          json_objects, penalize = find_json_objects(p['generated_text'], template_strict)\n",
        "\n",
        "          golden_labels = find_json_objects(pred.golden_labels)\n",
        "\n",
        "          alternative_labels = find_json_objects(pred.alternative_labels) if find_json_objects(pred.alternative_labels)[0] != \"None\" else []\n",
        "\n",
        "          fp_ok_labels = find_json_objects(pred.fp_ok_labels) if find_json_objects(pred.fp_ok_labels)[0] != \"None\" else []\n",
        "\n",
        "          accuracy, precision, recall, error, incorrect_format = calculate_acc_precision_recall(json_objects,golden_labels[0], alternative_labels[0], fp_ok_labels[0], triple_strict)\n",
        "          model_acc[metrics_category] = place_metrics(model_acc[metrics_category], accuracy, precision, recall, penalize, error, incorrect_format)\n",
        "\n",
        "  f = open(file, 'a')\n",
        "  header = f'\\n---template strict {template_strict}, triple strict {triple_strict}---'\n",
        "  f.write(header)\n",
        "  f.write('\\n\\n')\n",
        "\n",
        "  print(header)\n",
        "\n",
        "\n",
        "  for k,v in model_acc.items():\n",
        "\n",
        "    print(v)\n",
        "    nr_instances = v[0]\n",
        "    accuracy, precision, recall, f_penalize, f1, error, incorrect_format = calculate_metrics(v)\n",
        "\n",
        "    model_acc[k] = [nr_instances, accuracy, precision, recall, f_penalize, error, incorrect_format]\n",
        "\n",
        "    results = f'Model: {k} \\n accuracy: {accuracy};\\n precision: {precision};\\n recall: {recall};\\n f1 score: {f1};\\n penalize: {f_penalize}; \\n nr of instances: {nr_instances}; \\n incorrect output format: {incorrect_format}; \\n error: {error}\\n'\n",
        "    print(results)\n",
        "\n",
        "    f.write(results)\n",
        "    f.write('\\n')\n",
        "\n",
        "  f.close()\n",
        "  if download:\n",
        "    files.download(file)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2WAa8q9jOn-r"
      },
      "outputs": [],
      "source": [
        "# function to call each model from the checkpoint list via API, store their results and metadata, and generate the list of predicted Prompt objects\n",
        "# the input checkpoint list must be [{'model' : 'exact api model name', 'max_context_window' : int (optional)}, ... ]\n",
        "\n",
        "'''\n",
        "Parameters\n",
        "----------\n",
        "  checkpoint : list\n",
        "               A list containing dictionaries of models to be queried\n",
        "  all_prompts : list\n",
        "               The list containing all the Prompts objects\n",
        "  file : str\n",
        "         Path to the file where to write the serialized Prompt objects\n",
        "  output_template : str\n",
        "                    A string that is added to the final prompt for marking the beginning of the output from the model\n",
        "  download : bool\n",
        "             Option to download the text file with the serialized Prompt objects\n",
        "'''\n",
        "'''\n",
        "Return\n",
        "------\n",
        "  type : list\n",
        "         The list of all generated prompts\n",
        "'''\n",
        "\n",
        "def query(checkpoint, all_prompts, file, output_template = \" Output:\", download = True):\n",
        "\n",
        "  predictions = []\n",
        "  gpt_counter = 0 # to count how many calls were made to gpt models\n",
        "\n",
        "  try:\n",
        "    for d in all_prompts:\n",
        "\n",
        "      for model, prompts in d.items():\n",
        "\n",
        "        prompts_per_model = {}\n",
        "        prompts_per_model[model] = []\n",
        "\n",
        "        for prompt in tqdm(prompts):\n",
        "\n",
        "          messages = prompt.messages\n",
        "\n",
        "          messages[1]['content'] = messages[1]['content'] + output_template # output format to be always placed at the second role message content\n",
        "\n",
        "          for cp in checkpoint: # for each model\n",
        "\n",
        "            model_name = cp['model']\n",
        "\n",
        "            if model == model_name or model == 'all':\n",
        "\n",
        "              if model_name.startswith('gpt'):\n",
        "\n",
        "                client = OpenAI(api_key = userdata.get('GPT_TOKEN')) # load YOUR OWN KEY; here stored as a secret in the google colab\n",
        "\n",
        "                try:\n",
        "                  completion = client.chat.completions.with_raw_response.create(\n",
        "                    model=model_name,\n",
        "                    messages=messages\n",
        "                  )\n",
        "                except:\n",
        "                  metadata['generated_text'] = f\"er1ror:\" # if errors are generated from the server\n",
        "\n",
        "                req_headers = ['openai-model', 'openai-processing-ms', 'x-ratelimit-remaining-requests', 'x-ratelimit-remaining-tokens', 'x-ratelimit-reset-requests']\n",
        "\n",
        "                metadata = {k.replace('openai-',''): v for k, v in completion.headers.items() if k in req_headers}\n",
        "                metadata['date'] = str(datetime.now()) + ' GMT'\n",
        "                metadata['model'] = cp['model'] # seems like OpenAI removed the model parameter, therefore we take it from the input list\n",
        "\n",
        "                generated_response = completion.parse()\n",
        "                metadata['system-fingerprint'] = generated_response.system_fingerprint\n",
        "                metadata['prompt'] = messages[0]['content'] + messages[1]['content']\n",
        "\n",
        "                gpt_response = generated_response.choices[0].message.content # get the object that `chat.completions.create()` would have returned\n",
        "\n",
        "                metadata['generated_text'] = gpt_response\n",
        "                metadata['tokens'] = num_tokens_from_messages_openai(messages)\n",
        "\n",
        "                prompt.append_metadata(metadata)\n",
        "\n",
        "                gpt_counter += 1\n",
        "\n",
        "                if metadata['x-ratelimit-remaining-requests'] == '1': # if we reach the tire's limit, prompt and stop the call\n",
        "                  print('gpt counter: ', gpt_counter)\n",
        "                  print('remaining tokens: ', metadata['x-ratelimit-remaining-tokens'])\n",
        "                  return 'GPT exhausted'\n",
        "\n",
        "              else: # it's not an OpenAI model, but some model from HG\n",
        "\n",
        "                tokenizer = AutoTokenizer.from_pretrained(cp[\"model\"])\n",
        "\n",
        "                try:\n",
        "                  tokenized_chat = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)\n",
        "\n",
        "                except TemplateError:\n",
        "                  final_messages = [{'role' : 'user', 'content' : ''}]\n",
        "                  for m in messages:\n",
        "                    final_messages[0]['content'] += f\" {m['content']}\"\n",
        "                  tokenized_chat = tokenizer.apply_chat_template(final_messages, tokenize=False, add_generation_prompt=True)\n",
        "\n",
        "                API_URL = f\"https://api-inference.huggingface.co/models/{cp['model']}\"\n",
        "                # load YOUR OWN TOKEN from HG\n",
        "                headers = {\"Authorization\": f\"Bearer {userdata.get('HF_TOKEN')}\", \"x-compute-type\" : \"cpu+optimized\"} # to make sure we get the optimized compute option, if available\n",
        "\n",
        "                payload = {'inputs' : tokenized_chat, 'parameters' : {'return_full_text' : False, 'max_new_tokens' : None}, 'options' : {'use_cache' : False}}\n",
        "\n",
        "                metadata = {'date' : str(datetime.now()) + ' GMT'}\n",
        "                metadata['prompt'] = tokenized_chat\n",
        "                metadata['model'] = cp['model']\n",
        "\n",
        "                response = requests.post(API_URL, headers=headers, json=payload)\n",
        "\n",
        "                metadata['processing-ms'] = round(response.elapsed.microseconds / 1000)\n",
        "\n",
        "                try:\n",
        "                  metadata['generated_text'] = response.json()[0]['generated_text']\n",
        "                except KeyError:\n",
        "                  metadata['generated_text'] = f\"er1ror: {response.json()['error']}\" # to avoid any generated text that starts with error from being counted as wrong\n",
        "\n",
        "                metadata['tokens'] = num_tokens_from_messages(tokenized_chat, tokenizer)\n",
        "\n",
        "                prompt.append_metadata(metadata)\n",
        "\n",
        "          prompts_per_model[model].append(prompt)\n",
        "          write_prompt(prompt, file)\n",
        "\n",
        "        predictions.append(prompts_per_model)\n",
        "  except: # in case any unknown error occurs, to be able to download and return the predictions until that point\n",
        "    print('Error')\n",
        "\n",
        "    if download:\n",
        "      files.download(file)\n",
        "\n",
        "    return predictions\n",
        "\n",
        "  if download:\n",
        "    files.download(file)\n",
        "\n",
        "  return predictions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AUd_GWp-nv5S"
      },
      "source": [
        "## Generate dataset (enter to set variables)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YdatoOagn5O4"
      },
      "outputs": [],
      "source": [
        "# set your own preferences\n",
        "\n",
        "download_files = False # if you want to download the serialized dataset\n",
        "file_to_generate_from = 'templates_easy.txt' # file path to the templates"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s6lLSVceKetx"
      },
      "outputs": [],
      "source": [
        "dataset = []\n",
        "dataset = generate_dataset(file_to_generate_from, dataset = dataset, download = download_files)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SVib9spHojks"
      },
      "source": [
        "## Load dataset (enter to set variables)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zYqnrpT3oopt"
      },
      "outputs": [],
      "source": [
        "# set your own preferences\n",
        "\n",
        "file_to_load_from = 'templates_easy_final.txt' # file path to the serialized dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uQVVgvYPrLvp"
      },
      "outputs": [],
      "source": [
        "dataset = from_text_to_dataset(file_to_load_from)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "te-rlOB8o8vx"
      },
      "source": [
        "## Generate prompts (enter to set variables)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Grfzl4AMJU2A"
      },
      "outputs": [],
      "source": [
        "# provide the ontology\n",
        "ontology = '''\"\n",
        "    @prefix : <http://john.com/project> .\n",
        "    @prefix owl: <http://www.w3.org/2002/07/owl#> .\n",
        "    @prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .\n",
        "    @prefix xsd: <http://www.w3.org/2001/XMLSchema#> .\n",
        "\n",
        "    <http://john.com/project> rdf:type owl:Ontology .\n",
        "\n",
        "    :hasClass rdf:type owl:ObjectProperty ;\n",
        "        rdfs:domain :Project ;\n",
        "        rdfs:range xsd:string .\n",
        "\n",
        "    :hasCode rdf:type owl:ObjectProperty ;\n",
        "        rdfs:domain :Project ;\n",
        "        rdfs:range xsd:string .\n",
        "\n",
        "    :hasManager rdf:type owl:ObjectProperty ;\n",
        "        rdfs:domain :Project ;\n",
        "        rdfs:range  :Employee .\n",
        "\n",
        "    :hasName rdf:type owl:ObjectProperty ;\n",
        "        rdfs:domain owl:Thing ;\n",
        "        rdfs:range xsd:string .\n",
        "\n",
        "    :hasRole rdf:type owl:ObjectProperty ;\n",
        "        rdfs:domain :Employee ;\n",
        "        rdfs:range xsd:string .\n",
        "\n",
        "    :hasStatus rdf:type owl:ObjectProperty ;\n",
        "        rdfs:domain :Project ;\n",
        "        rdfs:range :Status .\n",
        "\n",
        "\n",
        "    :Employee rdf:type owl:Class ;\n",
        "        rdfs:subClassOf owl:Thing.\n",
        "\n",
        "    :Status rdf:type owl:Class;\n",
        "        rdfs:subClassOf owl:Thing.\n",
        "\n",
        "    :Project rdf:type owl:Class;\n",
        "        rdfs:subClassOf owl:Thing.\"'''\n",
        "\n",
        "# provide examples for each level;\n",
        "\n",
        "# here for level 4.1, for ICL style\n",
        "example_labels = f'Example: \\n Input text: Insert a project that has code 0A0, name it ColourRun, its class is Python and manager is Maia. \\n Output: [{{\"subject\" : \"Project1\", \"relationship\" : \"rdf:type\" , \"object\" : \"Project\" }}, {{\"subject\" : \"Project1\", \"relationship\" : \"hasCode\" , \"object\" : \"0A0\" }}, {{\"subject\" : \"Project1\", \"relationship\" : \"hasClass\" , \"object\" : \"Python\" }}, {{\"subject\" : \"Project1\", \"relationship\" : \"hasManager\", \"object\" : \"Employee1\" }}, {{\"subject\" : \"Employee1\", \"relationship\" : \"hasName\", \"object\" : \"Maia\" }}, {{\"subject\" : \"Employee1\", \"relationship\" : \"rdf:type\", \"object\" : \"Employee\" }}, {{\"subject\" : \"Project1\", \"relationship\" : \"hasName\" , \"object\" : \"ColourRun\" }}] \\n'\n",
        "\n",
        "# here for level 4.2, for COT style\n",
        "example_labels_cot = f'Example: \\n Input text: Insert a project that has code 0A0, name it ColourRun, its class is Python and manager is Maia. \\n 1. In the ontology, three classes are found: Project, Employee, and Status. In the input text, the word \"project\" may refer to the class Project. \\n 2. The ID of the instance is the name of the class concatenated with \"1\", therefore the final ID is Project1. \\n 3. The type relationship creates the triple :Project1 rdf:type :Project. \\n 4. Given triples where the rdfs:domain is :Project or :Thing, specific relationships are hasName, hasCode, hasClass, hasManager, and hasStatus. Within the input text, relationship keywords like \"name\", \"class\", \"code\", and \"manager\" may lead to the following relationships: hasName, hasCode, hasClass, and hasManager. \\n 5. The possible object for each relationship may be words that semantically and/or syntactically relate to the relationship keywords. Therefore, we can extract: for hasName the word \"ColourRun\", for hasCode the word \"0A0\", for hasClass the word \"Python\", and for hasManager the word \"Maia\". \\n 6. For hasManager, the ontology specifies an Employee instance, therefore a new instance of type Employee with ID Employee1 is created, and the object \"Maia\" may be the name of it. Each triple has to be put in a JSON object respecting the provided template. All JSON objects must be put in a list. \\n Output: [{{\"subject\" : \"Project1\", \"relationship\" : \"rdf:type\" , \"object\" : \"Project\" }}, {{\"subject\" : \"Project1\", \"relationship\" : \"hasCode\" , \"object\" : \"0A0\" }}, {{\"subject\" : \"Project1\", \"relationship\" : \"hasClass\" , \"object\" : \"Python\" }}, {{\"subject\" : \"Project1\", \"relationship\" : \"hasManager\", \"object\" : \"Employee1\" }}, {{\"subject\" : \"Employee1\", \"relationship\" : \"hasName\", \"object\" : \"Maia\" }}, {{\"subject\" : \"Employee1\", \"relationship\" : \"rdf:type\", \"object\" : \"Employee\" }}, {{\"subject\" : \"Project1\", \"relationship\" : \"hasName\" , \"object\" : \"ColourRun\" }}] \\n'\n",
        "\n",
        "# here for level 4.1, for ICL style of no golden triples\n",
        "example_none = f'Example: \\n Input text: Insert a program with code as A43, manager as Michael and class as Java. \\n Output: \"None\" \\n'\n",
        "\n",
        "# here for level 4.2, for COT style of no golden triples\n",
        "example_none_cot = f'Example: \\n Input text: Insert a program with code as A43, manager as Michael and class as Java. \\n 1. In the ontology, three classes are found: Project, Employee, and Status. In the input text, there are no words to suggest these classes, therefore the output is \"None\". \\n Output: \"None\" \\n'"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UreF3t-Nqoty"
      },
      "outputs": [],
      "source": [
        "# provide the system prompts, following the same format as below;\n",
        "\n",
        "system_prompts = {\n",
        "    \"all\" : # all type of models\n",
        "     {\n",
        "      0.0: # version\n",
        "         [  # level\n",
        "          { 1 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. It may or may not contain references to an instance of a class provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instance from the input text. Each instance should be identified by an ID, using the format :Class + \"1\", where class is the name of the detected class and + is concatenation. Respond only with the triples. The output format for a triple is :Subject :Relationship \"Object\". If no triple is detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "          { 2 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. It may or may not contain references to an instance of a class provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instance from the input text. Each instance should be identified by an ID, using the format :Class + \"1\", where class is the name of the detected class and + is concatenation. Each extracted triple that is related to the detected instance should use the ID as the subject. The first triple you always create is between the ID and the class type, using the rdf:type relationship. Respond only with the triples. The output format for a triple is :Subject :Relationship \"Object\". If no triple is detected, output \"None\". \\n Provided ontology: {ontology} \\n'},\n",
        "          { 3 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. It may or may not contain references to an instance of a class provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instance from the input text, as follows: \\n 1.Check if a class from the provided ontology is mentioned in the input text. Search in the provided ontology for triples with the format :ClassName rdf:type owl:Class, where ClassName is the name of the class. The input text should contain the name of a class from the ontology, even if misspelled. If the class is not known, stop the process and output \"None\". \\n 2. Each instance should be identified by an ID, using the format :Class + \"1\", where class is the name of the detected class and + is concatenation. Each extracted triple that is related to the detected instance should use the ID as the subject. \\n 3. The first triple you always create is between the ID and the class type, using the rdf:type relationship. \\n 4. Check whether any relationships were mentioned in the input text, regarding the detected class. For this, search in the ontology for the triples of form :RelationshipName rdfs:domain :ClassName, and take the relationship name as a known relationship regarding the detected class. Also, search for the triples of form :Relationship rdfs:domain owl:Thing, as owl:Thing can be used as the superclass of all classes, even if it is not directly mentioned. \\n 5. For each detected relationship, extract the possible mentioned object in the input text, regarding the detected instance. \\n 6. Respond only with the triples. The output format for a triple is :Subject :Relationship \"Object\". If no triple is detected, output \"None\". \\n Provided ontology: {ontology} \\n'},\n",
        "          { 4.1 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. It may or may not contain references to an instance of a class provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instance from the input text, as follows: \\n 1.Check if a class from the provided ontology is mentioned in the input text. Search in the provided ontology for triples with the format :ClassName rdf:type owl:Class, where ClassName is the name of the class. The input text should contain the name of a class from the ontology, even if misspelled. If the class is not known, stop the process and output \"None\". \\n 2. Each instance should be identified by an ID, using the format :Class + \"1\", where class is the name of the detected class and + is concatenation. Each extracted triple that is related to the detected instance should use the ID as the subject. \\n 3. The first triple you always create is between the ID and the class type, using the rdf:type relationship. \\n 4. Check whether any relationships were mentioned in the input text, regarding the detected class. For this, search in the ontology for the triples of form :RelationshipName rdfs:domain :ClassName, and take the relationship name as a known relationship regarding the detected class. Also, search for the triples of form :Relationship rdfs:domain owl:Thing, as owl:Thing can be used as the superclass of all classes, even if it is not directly mentioned. \\n 5. For each detected relationship, extract the possible mentioned object in the input text, regarding the detected instance. \\n 6. Respond only with the triples. The output format for a triple is :Subject :Relationship \"Object\". If no triple is detected, output \"None\". \\n Provided ontology: {ontology} \\n Example: \\n Input text: Insert a project that has code 0A0, name it ColourRun, its class is Python and manager is Maia. \\n Triples: :Project1 rdf:type :Project; \\n :Project1 :hasCode \"0A0; \\n :Project1 :hasName \"ColourRun\"; \\n :Project1 :hasClass \"Python\"; \\n :Project1 :hasManager :Employee1; \\n :Employee1 :hasName \"Maia\".\\n'},\n",
        "          { 4.2 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. It may or may not contain references to an instance of a class provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instance from the input text, as follows: \\n 1.Check if a class from the provided ontology is mentioned in the input text. Search in the provided ontology for triples with the format :ClassName rdf:type owl:Class, where ClassName is the name of the class. The input text should contain the name of a class from the ontology, even if misspelled. If the class is not known, stop the process and output \"None\". \\n 2. Each instance should be identified by an ID, using the format :Class + \"1\", where class is the name of the detected class and + is concatenation. Each extracted triple that is related to the detected instance should use the ID as the subject. \\n 3. The first triple you always create is between the ID and the class type, using the rdf:type relationship. \\n 4. Check whether any relationships were mentioned in the input text, regarding the detected class. For this, search in the ontology for the triples of form :RelationshipName rdfs:domain :ClassName, and take the relationship name as a known relationship regarding the detected class. Also, search for the triples of form :Relationship rdfs:domain owl:Thing, as owl:Thing can be used as the superclass of all classes, even if it is not directly mentioned. \\n 5. For each detected relationship, extract the possible mentioned object in the input text, regarding the detected instance. \\n 6. Respond only with the triples. The output format for a triple is :Subject :Relationship \"Object\". If no triple is detected, output \"None\".  \\n Provided ontology: {ontology} \\n Example: \\n Input text: Insert a project that has code 0A0, name it ColourRun, its class is Python and manager is Maia. \\n 1. In the ontology, three class triples are found: Project, Employee, and Status. In the input text, the word \"project\" may refer to the class Project. \\n 2. The ID of the instance is the name of the class concatenated with \"1\", therefore the final ID is :Project1. \\n 3. The type relationship creates the triple :Project1 rdf:type :Project. \\n 4. Given triples where the rdfs:domain is :Project or :Thing, specific relationships are hasName, hasCode, hasClass, hasManager, and hasStatus. Within the input text, relationship keywords like \"name\", \"class\", \"code\", and \"manager\" may lead to the following relationships: hasName, hasCode, hasClass, and hasManager. \\n 5. The possible object for each relationship may be words that semantically and/or syntactically relate to the relationship keywords. Therefore, we can extract: for hasName the word \"ColourRun\", for hasCode the word \"0A0\", for hasClass the word \"Python\", and for hasManager the word \"Maia\". \\n 6. For hasManager, the ontology specifies an Employee instance, therefore a new instance of type Employee with ID :Employee1 is created, and the object \"Maia\" may be the name of it. Triples: :Project1 rdf:type :Project; \\n :Project1 :hasCode \"0A0; \\n :Project1 :hasName \"ColourRun\"; \\n :Project1 :hasClass \"Python\"; \\n :Project1 :hasManager :Employee1; \\n :Employee1 :hasName \"Maia\".\\n'},\n",
        "          ],\n",
        "      1.0: # version\n",
        "         [  # level\n",
        "          { 1 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. The input text may or may not contain references to instances of classes provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instances from the input text. Each instance should be identified by an ID, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class and + is concatenation. Put each triple in a JSON object, as follows: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, add all triples you assumed of that instance too. Respond only with the JSON object(s) in a list. If no triple is detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "          { 2 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. The input text may or may not contain references to instances of classes provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instances from the input text. Each instance should be identified by an ID, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class and + is concatenation. A triple you always create is between the ID and the class type, using the rdf:type relationship. Put each triple in a JSON object, as follows: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, add all triples you assumed of that instance too. Respond only with the JSON object(s) in a list. If no triple is detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "          { 3 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. The input text may or may not contain references to instances of classes provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instances from the input text, as follows: \\n 1. Check if a class from the provided ontology is mentioned in the input text. Search in the provided ontology for triples with the format :ClassName rdf:type owl:Class, where ClassName is the name of the class. The input text should contain the name of a class from the ontology, even if misspelled. If the class is not known, stop the process and output \"None\". \\n 2. Each instance should be identified by an ID, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class and + is concatenation. Each extracted triple that is related to an instance should use the instance\\'s ID as the subject. \\n 3. A triple you always create is between the ID and the class type, using the rdf:type relationship. \\n 4. Check whether any relationships were mentioned in the input text, regarding the detected class. For this, search in the ontology for the triples of form :RelationshipName rdfs:domain :ClassName, and take the relationship name as a known relationship regarding the detected class. Also, search for the triples of form :Relationship rdfs:domain owl:Thing, as owl:Thing can be used as the superclass of all classes, even if it is not directly mentioned. \\n 5. For each detected relationship, extract the possible mentioned object in the input text, regarding the detected instance. \\n 6. Put each triple in a JSON object, as follows: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, add all triples you assumed of that instance too. Respond only with the JSON object(s) in a list. If no triple is detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "          { 4.1 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. The input text may or may not contain references to instances of classes provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instances from the input text, as follows: \\n 1. Check if a class from the provided ontology is mentioned in the input text. Search in the provided ontology for triples with the format :ClassName rdf:type owl:Class, where ClassName is the name of the class. The input text should contain the name of a class from the ontology, even if misspelled. If the class is not known, stop the process and output \"None\". \\n 2. Each instance should be identified by an ID, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class and + is concatenation. Each extracted triple that is related to an instance should use the instance\\'s ID as the subject. \\n 3. A triple you always create is between the ID and the class type, using the rdf:type relationship. \\n 4. Check whether any relationships were mentioned in the input text, regarding the detected class. For this, search in the ontology for the triples of form :RelationshipName rdfs:domain :ClassName, and take the relationship name as a known relationship regarding the detected class. Also, search for the triples of form :Relationship rdfs:domain owl:Thing, as owl:Thing can be used as the superclass of all classes, even if it is not directly mentioned. \\n 5. For each detected relationship, extract the possible mentioned object in the input text, regarding the detected instance. \\n 6. Put each triple in a JSON object, as follows: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, add all triples you assumed of that instance too. Respond only with the JSON object(s) in a list. If no triple is detected, output \"None\".\\n Provided ontology: {ontology} \\n '},\n",
        "          { 4.2 : f'You are a Knowledge Graph Expert. A domain ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. The input text may or may not contain references to instances of classes provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instances from the input text, as follows: \\n 1. Check if a class from the provided ontology is mentioned in the input text. Search in the provided ontology for triples with the format :ClassName rdf:type owl:Class, where ClassName is the name of the class. The input text should contain the name of a class from the ontology, even if misspelled. If the class is not known, stop the process and output \"None\". \\n 2. Each instance should be identified by an ID, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class and + is concatenation. Each extracted triple that is related to an instance should use the instance\\'s ID as the subject. \\n 3. A triple you always create is between the ID and the class type, using the rdf:type relationship. \\n 4. Check whether any relationships were mentioned in the input text, regarding the detected class. For this, search in the ontology for the triples of form :RelationshipName rdfs:domain :ClassName, and take the relationship name as a known relationship regarding the detected class. Also, search for the triples of form :Relationship rdfs:domain owl:Thing, as owl:Thing can be used as the superclass of all classes, even if it is not directly mentioned. \\n 5. For each detected relationship, extract the possible mentioned object in the input text, regarding the detected instance. \\n 6. Put each triple in a JSON object, as follows: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, add all triples you assumed of that instance too. Respond only with the JSON object(s) in a list. If no triple is detected, output \"None\". \\n Provided ontology: {ontology} \\n '},\n",
        "          ],\n",
        "      }\n",
        "    }\n",
        "\n",
        "# mistralai/Mistral-7B-Instruct-v0.1 has been removed because it did not reprhase anything in three runs\n",
        "final_model_prompts = {\n",
        "    'mistralai/Mixtral-8x7B-Instruct-v0.1':\n",
        "     {1.0:\n",
        "          [\n",
        "              {1: f'You are a specialist in Knowledge Graphs and will be given a domain ontology in the Turtle syntax, enclosed by double quotes. The task involves extracting triples about instances mentioned in a provided natural language text. These instances may belong to classes in the given ontology, with possible relationships between them. Each instance should be identified by a unique ID, constructed by concatenating the class name and the number 1 (e.g., \"Class1\"). Create a JSON object for each triple, following the format: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If a triple references another instance, include all assumed triples of that instance as well. The output should be a list of JSON objects. If no triples are detected, respond with \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "              {2: f'You are a specialist in Knowledge Graphs and will be given a domain ontology in the Turtle syntax, enclosed in double quotes. The task requires analyzing a provided natural language text, which may or may not refer to instances of classes in the given ontology along with specific relationships. Your goal is to identify triples related to the mentioned instances from the input text. Each identified instance should be assigned a unique ID, constructed by concatenating the class name and the number 1 (e.g., \"Class1\"). Create a triple connecting the ID and the class type using the rdf:type relationship. Present each triple in a JSON object, following the format: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}, without the whole IRI term. If any triple refers to another instance, include all inferred triples related to that instance as well. Your response should be a list containing the JSON object(s), or \"None\" if no triples are detected.\\n Provided ontology: {ontology} \\n'},\n",
        "              {3: f'You are a Knowledge Graph Expert. You are given a domain ontology in the form of a Turtle syntax string, enclosed by double quotes. Your task is to extract triples from a provided natural language text, which may or may not contain references to instances of classes in the ontology along with specific relationships. Follow this process:\\n1. Identify a class from the provided ontology in the input text. Look for triples in the ontology with the format :ClassName rdf:type owl:Class, where ClassName is the name of the class. If the class is not found in the text, even if misspelled, output \"None\".\\n2. Assign an ID to each instance in the format \"Class\" + \"1\", where \"Class\" is the identified class name. Use this ID as the subject for any triples related to the instance.\\n3. Create a triple between the ID and the class type using the rdf:type relationship.\\n4. Find relationships related to the detected class by searching for triples of the form :RelationshipName rdfs:domain :ClassName and :Relationship rdfs:domain owl:Thing in the ontology.\\n5. Extract possible objects from the input text for each detected relationship.\\n6. Format each triple as a JSON object with the structure {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, include all assumed triples of that instance as well. Output only the list. If no triples are detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "              {4.1: f'You are a Knowledge Graph Expert. You are given a domain ontology in the form of a Turtle syntax string, enclosed by double quotes. Your task is to extract triples from a provided natural language text, which may or may not contain references to instances of classes in the ontology along with specific relationships. Follow this process:\\n1. Identify a class from the provided ontology in the input text. Look for triples in the ontology with the format :ClassName rdf:type owl:Class, where ClassName is the name of the class. If the class is not found in the text, even if misspelled, output \"None\".\\n2. Assign an ID to each instance in the format \"Class\" + \"1\", where \"Class\" is the identified class name. Use this ID as the subject for any triples related to the instance.\\n3. Create a triple between the ID and the class type using the rdf:type relationship.\\n4. Find relationships related to the detected class by searching for triples of the form :RelationshipName rdfs:domain :ClassName and :Relationship rdfs:domain owl:Thing in the ontology.\\n5. Extract possible objects from the input text for each detected relationship.\\n6. Format each triple as a JSON object with the structure {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, include all assumed triples of that instance as well. Output only the list. If no triples are detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "              {4.2: f'You are a Knowledge Graph Expert. You are given a domain ontology in the form of a Turtle syntax string, enclosed by double quotes. Your task is to extract triples from a provided natural language text, which may or may not contain references to instances of classes in the ontology along with specific relationships. Follow this process:\\n1. Identify a class from the provided ontology in the input text. Look for triples in the ontology with the format :ClassName rdf:type owl:Class, where ClassName is the name of the class. If the class is not found in the text, even if misspelled, output \"None\".\\n2. Assign an ID to each instance in the format \"Class\" + \"1\", where \"Class\" is the identified class name. Use this ID as the subject for any triples related to the instance.\\n3. Create a triple between the ID and the class type using the rdf:type relationship.\\n4. Find relationships related to the detected class by searching for triples of the form :RelationshipName rdfs:domain :ClassName and :Relationship rdfs:domain owl:Thing in the ontology.\\n5. Extract possible objects from the input text for each detected relationship.\\n6. Format each triple as a JSON object with the structure {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, include all assumed triples of that instance as well. Output only the list. If no triples are detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "            ]\n",
        "        },\n",
        "  #  'openchat/openchat-3.5-0106':\n",
        "  #    {1.0:\n",
        "  #        [\n",
        "  #            {1: 'You are a Knowledge Graph Expert. An ontology is provided to you, delimited by double quotes. The syntax used to describe the ontology is Turtle. Your input is a natural language text. The input text may or may not contain references to instances of classes provided in the ontology, together with specific relationships. Given the provided ontology, your task is to extract triples about the mentioned instances from the input text. Each instance should be identified by an ID, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class and + is concatenation. Put each triple in a JSON object, as follows: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, add all triples you assumed of that instance too. Respond only with the JSON object(s) in a list. If no triple is detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "  #            {2: 'You are a Knowledge Graph Expert. You are given a domain ontology, which is delimited by double quotes and described using the Turtle syntax. Your input is a natural language text that may or may not contain references to instances of classes provided in the ontology, along with specific relationships. Your task is to extract triples about the mentioned instances from the input text. Each instance should be identified by an ID, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class and + is concatenation. You must always create a triple between the ID and the class type, using the rdf:type relationship. \\n\\nPut each triple in a JSON object, with the format: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, include all triples you assumed of that instance too. Respond only with the JSON object(s) in a list. If no triple is detected, output \"None\".\\nProvided ontology: {ontology} \\n'},\n",
        "  #            {3: f'You are a Knowledge Graph Expert. You are given a domain ontology in Turtle syntax, delimited by double quotes. Your task is to process a natural language text input, extracting triples about instances mentioned in the text based on the provided ontology. The input text may contain references to instances of classes in the ontology, along with specific relationships. Your task is to:\\n1. Identify if a class from the provided ontology is mentioned in the input text. If the class is not known, output \"None\".\\n2. Assign an ID to each instance, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class  + is concatenation. Use the instance\\'s ID as the subject in each extracted triple.\\n3. Create a triple between the ID and the class type using the rdf:type relationship.\\n4. Identify any relationships mentioned in the input text regarding the detected class. Search the ontology for triples of the form :RelationshipName rdfs:domain :ClassName, and take the relationship name as a known relationship for the detected class. Also, search for triples of the form :Relationship rdfs:domain owl:Thing, as owl:Thing can be used as the superclass of all classes.\\n5. Extract the possible mentioned object in the input text, regarding the detected instance, for each detected relationship.\\n6. Place each triple in a JSON object, in the format: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, add all triples you assumed of that instance too. Respond with the JSON object(s) in a list. If no triple is detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "  #            {4.1: f'You are a Knowledge Graph Expert. You are given a domain ontology in Turtle syntax, delimited by double quotes. Your task is to process a natural language text input, extracting triples about instances mentioned in the text based on the provided ontology. The input text may contain references to instances of classes in the ontology, along with specific relationships. Your task is to:\\n\\n1. Identify if a class from the provided ontology is mentioned in the input text. If the class is not known, output \"None\".\\n2. Assign an ID to each instance, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class + is concatenation. Use the instance\\'s ID as the subject in each extracted triple.\\n3. Create a triple between the ID and the class type using the rdf:type relationship.\\n4. Identify any relationships mentioned in the input text regarding the detected class. Search the ontology for triples of the form :RelationshipName rdfs:domain :ClassName, and take the relationship name as a known relationship for the detected class. Also, search for triples of the form :Relationship rdfs:domain owl:Thing, as owl:Thing can be used as the superclass of all classes.\\n5. Extract the possible mentioned object in the input text, regarding the detected instance, for each detected relationship.\\n6. Place each triple in a JSON object, in the format: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, add all triples you assumed of that instance too. Respond with the JSON object(s) in a list. If no triple is detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "  #            {4.2: f'You are a Knowledge Graph Expert. You are given a domain ontology in Turtle syntax, delimited by double quotes. Your task is to process a natural language text input, extracting triples about instances mentioned in the text based on the provided ontology. The input text may contain references to instances of classes in the ontology, along with specific relationships. Your task is to:\\n1. Identify if a class from the provided ontology is mentioned in the input text. If the class is not known, output \"None\".\\n2. Assign an ID to each instance, using the format \"Class\" + \"1\", where \"Class\" is the name of the detected class + is concatenation. Use the instance\\'s ID as the subject in each extracted triple.\\n3. Create a triple between the ID and the class type using the rdf:type relationship.\\n4. Identify any relationships mentioned in the input text regarding the detected class. Search the ontology for triples of the form :RelationshipName rdfs:domain :ClassName, and take the relationship name as a known relationship for the detected class. Also, search for triples of the form :Relationship rdfs:domain owl:Thing, as owl:Thing can be used as the superclass of all classes.\\n5. Extract the possible mentioned object in the input text, regarding the detected instance, for each detected relationship.\\n6. Place each triple in a JSON object, in the format: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If any triple refers to another instance, add all triples you assumed of that instance too. Respond with the JSON object(s) in a list. If no triple is detected, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "  #          ]\n",
        "  #      },\n",
        "    'gpt-3.5-turbo':\n",
        "     {1.0:\n",
        "        [\n",
        "            {1: f'Given a domain ontology represented in Turtle syntax and enclosed with double quotes, along with a natural language input text, as a Knowledge Graph Expert, your task is to extract triples related to instances mentioned in the input. Each instance should be represented with an ID in the format \"Class\" + \"1\", where \"Class\" is the name of the detected class type and + is concatenation. Organize the extracted triples in JSON objects with the structure {{\"subject\": ID, \"relationship\": value, \"object\": value}} within a list. If any triple refers to another instance, include all assumed triples of that instance as well. Output only the list. In case no triples are found, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "            {2: f'Given a domain ontology in Turtle syntax, enclosed with double quotes, and a natural language text input, as a Knowledge Graph Expert, your task is to identify instances of classes mentioned in the text and extract triples about them. Each instance should be represented by an ID in the format \"Class\" + \"1\", where \"Class\" is the name of the detected class type and + is concatenation, and linked to its class type using the rdf:type relationship. Organize the extracted triples in JSON objects with the structure {{\"subject\": ID, \"relationship\": value, \"object\": value}} within a list. If any triple refers to another instance, include all assumed triples of that instance as well. Output only the list. If no triples are found, output \"None\".\\n Provided ontology: {ontology} \\n'},\n",
        "            {3: f'Given a domain ontology presented in Turtle syntax and enclosed with double quotes, your task as a Knowledge Graph Expert is to analyze a natural language text. Extract triples related to instances of classes mentioned in the input text by following these steps: \\n1. Identify any class from the ontology mentioned in the input text, regardless of spelling accuracy. \\n2. Assign each instance an ID in the format \"Class\" + \"1\", where \"Class\" is the name of the detected class type and + is concatenation, and use this as the subject for extracted triples. \\n3. Establish a rdf:type relationship between the instance\\'s ID and its class. \\n4. Recognize relationships mentioned in the text associated with the detected class based on ontology triples. \\n5. Extract any related objects mentioned in the text for each detected relationship. \\n6. Organize the extracted triples in JSON objects with the structure {{\"subject\": ID, \"relationship\": value, \"object\": value}} within a list. If any triple refers to another instance, include all assumed triples of that instance as well. Output only the JSON object(s) in a list format. If no triples are found, output \"None\".\"\\n Provided ontology: {ontology} \\n'},\n",
        "            {4.1: f'Given a domain ontology presented in Turtle syntax and enclosed with double quotes, your task as a Knowledge Graph Expert is to analyze a natural language text. Extract triples related to instances of classes mentioned in the input text by following these steps: \\n1. Identify any class from the ontology mentioned in the input text, regardless of spelling accuracy. \\n2. Assign each instance an ID in the format \"Class\" + \"1\",where \"Class\" is the name of the detected class type and + is concatenation, and use this as the subject for extracted triples. \\n3. Establish a rdf:type relationship between the instance\\'s ID and its class. \\n4. Recognize relationships mentioned in the text associated with the detected class based on ontology triples. \\n5. Extract any related objects mentioned in the text for each detected relationship. \\n6. Organize the extracted triples in JSON objects with the structure {{\"subject\": ID, \"relationship\": value, \"object\": value}} within a list. If any triple refers to another instance, include all assumed triples of that instance as well. Output only the JSON object(s) in a list format. If no triples are found, output \"None\".\"\\n Provided ontology: {ontology} \\n'},\n",
        "            {4.2: f'Given a domain ontology presented in Turtle syntax and enclosed with double quotes, your task as a Knowledge Graph Expert is to analyze a natural language text. Extract triples related to instances of classes mentioned in the input text by following these steps: \\n1. Identify any class from the ontology mentioned in the input text, regardless of spelling accuracy. \\n2. Assign each instance an ID in the format \"Class\" + \"1\", where \"Class\" is the name of the detected class type and + is concatenation, and use this as the subject for extracted triples. \\n3. Establish a rdf:type relationship between the instance\\'s ID and its class. \\n4. Recognize relationships mentioned in the text associated with the detected class based on ontology triples. \\n5. Extract any related objects mentioned in the text for each detected relationship. \\n6. Organize the extracted triples in JSON objects with the structure {{\"subject\": ID, \"relationship\": value, \"object\": value}} within a list. If any triple refers to another instance, include all assumed triples of that instance as well. Output only the JSON object(s) in a list format. If no triples are found, output \"None\".\"\\n Provided ontology: {ontology} \\n'},\n",
        "            ]\n",
        "      },\n",
        "        'gpt-4o':\n",
        "      {1.0:\n",
        "        [\n",
        "            {1: f'Imagine you are an expert in Knowledge Graphs. You are given a specific ontology in Turtle syntax, and it is enclosed in double quotes. The task involves processing a natural language text to extract data in the form of triples, which refer to the given classes and relationships in the ontology. Each identified instance should be denoted by a unique ID in the format \"Class\" + \"1\", where \"Class\" is the class name and + denotes concatenation. The extracted triples must be formatted as JSON objects like this: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If a triple involves another instance, include all related triples. Only provide the JSON objects in a list as the output. If no triples are found, respond with \"None\".\\n Do not add your JSON tags at the beginning (ex. \"```json\") and end of the output text (ex. \"```\"), leave only your answer.\\n Here is the provided ontology: {ontology} \\n'},\n",
        "            {2: f'Your role is a Knowledge Graph Expert working with a domain ontology described using Turtle syntax. The ontology is provided within double quotes. You will receive a natural language text, which may or may not mention instances of the classes defined in the ontology along with specific relationships. Your task is to use the provided ontology to extract RDF triples about the mentioned instances from the input text. Each detected instance should be given an ID in the format: \"Class\" + \"1\", where \"Class\" is the name of the detected class. You must always create a triple that specifies the ID and its class type using the rdf:type relationship. Present each triple as a JSON object in the following structure: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If a triple mentions another instance, include all triples related to that instance too. Provide your response as a list of JSON objects. If no triples are detected, output \"None\".\\n Do not add your JSON tags at the beginning (ex. \"```json\") and end of the output text (ex. \"```\"), leave only your answer.\\n Provided ontology:  {ontology} \\n'},\n",
        "            {3: f'You are an expert in Knowledge Graphs. You have been given a domain ontology within double quotes, described using the Turtle syntax. Your task is to analyze a natural language input text, which may or may not refer to instances of classes and relationships described in the ontology. Using the ontology provided, your job is to extract triples about the instances mentioned in the input text as follows: \\n1. Determine if any class from the ontology is mentioned in the input text. Look in the ontology for triples like :ClassName rdf:type owl:Class, where ClassName is the class name. The class name in the input text could be misspelled. If no known class is mentioned, output \"None\". \\n2. Assign an ID to each identified instance using \"Class\" + \"1\", where \"Class\" is the detected class name. \\n3. Always create a triple between the ID and the class type using the rdf:type relationship. \\n4. Identify any mentioned relationships from the input text regarding the identified class. Search for triples like :RelationshipName rdfs:domain :ClassName or :Relationship rdfs:domain owl:Thing in the ontology, where owl:Thing represents the superclass of all classes. \\n5. For each detected relationship, extract the mentioned object in the input text with respect to the identified instance. \\n6. Present each triple as a JSON object with the format: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If a triple refers to another instance, include all triples for that instance as well. If no triples are detected, output \"None\".\\n Do not add your JSON tags at the beginning (ex. \"```json\") and end of the output text (ex. \"```\"), leave only your answer.\\n Provided ontology: {ontology} \\n'},\n",
        "            {4.1: f'You are an expert in Knowledge Graphs. You have been given a domain ontology within double quotes, described using the Turtle syntax. Your task is to analyze a natural language input text, which may or may not refer to instances of classes and relationships described in the ontology. Using the ontology provided, your job is to extract triples about the instances mentioned in the input text as follows: \\n1. Determine if any class from the ontology is mentioned in the input text. Look in the ontology for triples like :ClassName rdf:type owl:Class, where ClassName is the class name. The class name in the input text could be misspelled. If no known class is mentioned, output \"None\". \\n2. Assign an ID to each identified instance using \"Class\" + \"1\", where \"Class\" is the detected class name. \\n3. Always create a triple between the ID and the class type using the rdf:type relationship. \\n4. Identify any mentioned relationships from the input text regarding the identified class. Search for triples like :RelationshipName rdfs:domain :ClassName or :Relationship rdfs:domain owl:Thing in the ontology, where owl:Thing represents the superclass of all classes. \\n5. For each detected relationship, extract the mentioned object in the input text with respect to the identified instance. \\n6. Present each triple as a JSON object with the format: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If a triple refers to another instance, include all triples for that instance as well. If no triples are detected, output \"None\".\\n Do not add your JSON tags at the beginning (ex. \"```json\") and end of the output text (ex. \"```\"), leave only your answer.\\n Provided ontology: {ontology} \\n'},\n",
        "            {4.2: f'You are an expert in Knowledge Graphs. You have been given a domain ontology within double quotes, described using the Turtle syntax. Your task is to analyze a natural language input text, which may or may not refer to instances of classes and relationships described in the ontology. Using the ontology provided, your job is to extract triples about the instances mentioned in the input text as follows: \\n1. Determine if any class from the ontology is mentioned in the input text. Look in the ontology for triples like :ClassName rdf:type owl:Class, where ClassName is the class name. The class name in the input text could be misspelled. If no known class is mentioned, output \"None\". \\n2. Assign an ID to each identified instance using \"Class\" + \"1\", where \"Class\" is the detected class name. \\n3. Always create a triple between the ID and the class type using the rdf:type relationship. \\n4. Identify any mentioned relationships from the input text regarding the identified class. Search for triples like :RelationshipName rdfs:domain :ClassName or :Relationship rdfs:domain owl:Thing in the ontology, where owl:Thing represents the superclass of all classes. \\n5. For each detected relationship, extract the mentioned object in the input text with respect to the identified instance. \\n6. Present each triple as a JSON object with the format: {{\"subject\" : ID, \"relationship\" : value, \"object\" : value}}. If a triple refers to another instance, include all triples for that instance as well. If no triples are detected, output \"None\".\\n Do not add your JSON tags at the beginning (ex. \"```json\") and end of the output text (ex. \"```\"), leave only your answer.\\n Provided ontology: {ontology} \\n'}\n",
        "            ]\n",
        "      }\n",
        "    }\n",
        "\n",
        "# dummy dataset\n",
        "'''dataset = [\n",
        "            {'text':'insert a project with code as something like 4A7 manager is someone with anything assistant class is Java status is something with anything finished and name is BestRobot.', 'golden_labels' : '[{\"subject\" : \"Project1\", \"relationship\" : \"rdf:type\" , \"object\" : \"Project\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasCode\" , \"object\" : \"4A7\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasClass\" , \"object\" : \"Java\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasManager\", \"object\" : \"Employee1\" }, {\"subject\" : \"Employee1\", \"relationship\" : \"hasRole\", \"object\" : \"assistant\" }, {\"subject\" : \"Employee1\", \"relationship\" : \"rdf:type\", \"object\" : \"Employee\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasStatus\" , \"object\" : \"Status1\" }, {\"subject\" : \"Status1\", \"relationship\" : \"rdf:type\" , \"object\" : \"Status\" },{\"subject\" : \"Status1\", \"relationship\" : \"hasName\" , \"object\" : \"finished\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasName\" , \"object\" : \"BestRobot\" }]','alternative_labels' : '[{\"subject\" : \"Project1\", \"relationship\" : \"hasStatus\" , \"object\" : \"finished\" }]', 'fp_ok_labels':'[]', 'text_type' : 'Project', 'class_type' : 'S'},\n",
        "            {'text':'insert a project with code WF6, name as BestRobot, class is java and Employee123 as manager.', 'golden_labels' : '[{\"subject\" : \"Project1\", \"relationship\" : \"rdf:type\" , \"object\" : \"Project\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasCode\" , \"object\" : \"WF6\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasClass\" , \"object\" : \"java\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasManager\", \"object\" : \"Employee123\" }, {\"subject\" : \"Employee123\", \"relationship\" : \"rdf:type\", \"object\" : \"Employee\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasName\" , \"object\" : \"BestRobot\" }]', 'alternative_labels' : '[]', 'fp_ok_labels':'[{\"subject\" : \"Employee123\", \"relationship\" : \"hasRole\", \"object\" : \"Manager\"}, {\"subject\" : \"Employee123\", \"relationship\" : \"hasRole\", \"object\" : \"manager\"}]', 'text_type' : 'Project', 'class_type' : 'S'},\n",
        "            {'text':'Tell me more about the reports we have in the database.','golden_labels':'None', 'alternative_labels' : '[]', 'fp_ok_labels' : '[]', 'text_type' : 'None', 'class_type' : 'SN'}, # None output\n",
        "            {'text':'Please add a prject with code as 123 and manager as John','golden_labels':'[{\"subject\" : \"Project1\", \"relationship\" : \"rdf:type\" , \"object\" : \"Project\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasCode\" , \"object\" : \"123\" }, {\"subject\" : \"Project1\", \"relationship\" : \"hasManager\", \"object\" : \"Employee1\" }, {\"subject\" : \"Employee1\", \"relationship\" : \"hasName\", \"object\" : \"John\" }]', 'alternative_labels' : '[{\"subject\" : \"Project1\", \"relationship\" : \"hasManager\", \"object\" : \"John\"}]', 'fp_ok_labels':'[{\"subject\" : \"Employee1\", \"relationship\" : \"hasRole\", \"object\" : \"Manager\"}, {\"subject\" : \"Employee1\", \"relationship\" : \"hasRole\", \"object\" : \"manager\"}]'}, # misspelled type\n",
        "            {'text':'i want you to insert an app instance with code being something like A-9I its class is Pascal named BestApp and put Robert as the manager','golden_labels':'None', 'alternative_labels' : '[]', 'fp_ok_labels' : '[]'} # not known type\n",
        "           ]'''"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Q3b00Mk2p-4K"
      },
      "outputs": [],
      "source": [
        "# set your own preferences\n",
        "\n",
        "sys_mesg_ord =  0 # the position of the system message regarding the prompt; 0 - beginning, 1 - end\n",
        "file = 'model_prompts_easy_sys_mesg_ord_0.txt' # file path to write the serialized prompts\n",
        "download_files = False # option to download the files"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "s8gQCAj2qtQn"
      },
      "outputs": [],
      "source": [
        "all_prompts = generate_prompts(final_model_prompts, dataset, versions = [1], sys_mesg_ord =  sys_mesg_ord, file = file, download = download_files)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "di5bgNrJqVIu"
      },
      "source": [
        "## Load prompts (enter to set variables)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jq-vb1n0qdux"
      },
      "outputs": [],
      "source": [
        "# set your own preferences\n",
        "\n",
        "file = 'all_prompts_easy_sys_mesg_ord_0.txt' # file path to load from the serialized prompts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_jJLipKkXtiK"
      },
      "outputs": [],
      "source": [
        "all_prompts = from_text_to_Prompt(file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dgzCc1yFA4cD"
      },
      "source": [
        "## General Pipeline (enter to set variables)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vr-7OXGqq-X4"
      },
      "outputs": [],
      "source": [
        "# set your own preferences\n",
        "\n",
        "file = 'run_1_model_prompts_easy_sys_mesg_ord_0.txt' # path where to save the serialized predicted Prompt objects\n",
        "download_files = False # option to download the serialized predicted Prompt objects"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Try9VPa_RXpN"
      },
      "outputs": [],
      "source": [
        "%%time\n",
        "\n",
        "# provide the models' checkpoints, optional their max context window\n",
        "\n",
        "checkpoint = [\n",
        "              {\n",
        "                \"model\" : \"mistralai/Mixtral-8x7B-Instruct-v0.1\",  # doesn't have system prompt\n",
        "                \"max_context_window\" : 32768\n",
        "              },\n",
        "              #{\n",
        "              #  \"model\" : \"mistralai/Mistral-7B-Instruct-v0.1\", # doesn't have system prompt\n",
        "              #  \"max_context_window\" : 32768\n",
        "              #},\n",
        "              #{\n",
        "              #  \"model\" : \"mistralai/Mistral-7B-Instruct-v0.2\", # doesn't have system prompt\n",
        "              #  \"max_context_window\" : 32768\n",
        "              #},\n",
        "              #{\n",
        "              #  \"model\" : \"openchat/openchat-3.5-0106\", # doesn't have system prompt\n",
        "              #  \"max_context_window\" : 8192\n",
        "              #},\n",
        "              #{\n",
        "              #  \"model\" : \"HuggingFaceH4/zephyr-7b-beta\", # does have system prompt\n",
        "              #  \"max_context_window\"  : 32768\n",
        "              #},\n",
        "              {\n",
        "                \"model\" : 'gpt-3.5-turbo-0125', # does have system prompt\n",
        "                \"max_context_window\" : 16385\n",
        "              },\n",
        "              {\n",
        "                \"model\" : 'gpt-4o', # does have system prompt\n",
        "                \"max_context_window\" : 128000\n",
        "              }\n",
        "              ]\n",
        "\n",
        "predictions = query(checkpoint, all_prompts, file, download = download_files)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3P_xDyaxC98p"
      },
      "outputs": [],
      "source": [
        "# view the predictions\n",
        "\n",
        "view_predictions(predictions, all_prompts, False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A6QvwSKFjul5"
      },
      "outputs": [],
      "source": [
        "# generate the metrics reports\n",
        "\n",
        "# set your own preferences\n",
        "\n",
        "# as input, the user has to specify what combination of levels they want their metrics to be calculated at, from:\n",
        "# 0 - at model level, 1 - at prompt's level, 2 - at prompt's version, 3 - at the class type (Project, Employee, Status, None), 4 - at input text's type\n",
        "# eg. providing [0], will have the metrics for each model's performance, [1] gives the performance of all models at a specific prompt level, while [0, 1] will give the permformance of each model at a specific prompt's level\n",
        "\n",
        "\n",
        "levels = [0]\n",
        "file = \"results_run_1_model_prompts_easy_sys_mesg_ord_0.txt\"\n",
        "template_strict = False\n",
        "triple_strict = False\n",
        "download_files = False\n",
        "\n",
        "view_metrics(predictions, file, levels, template_strict = template_strict, triple_strict = triple_strict, download = download_files)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wMOzhd-rr4nP"
      },
      "source": [
        "## Calculate metrics from existing predictions (enter to set variables)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J5QGiN5nz1-e"
      },
      "outputs": [],
      "source": [
        "# set your own preferences\n",
        "from google.colab import files\n",
        "\n",
        "\n",
        "file = 'run_1_2_3_all_prompts_hard_sys_mesg_ord_0.txt'\n",
        "\n",
        "uploaded = files.upload()\n",
        "\n",
        "predictions2 = from_text_to_Prompt(file)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g2dLFhgVsDX1"
      },
      "outputs": [],
      "source": [
        "# view the predictions\n",
        "\n",
        "view_predictions(predictions2, all_prompts, False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2VryJR6EsDX_"
      },
      "outputs": [],
      "source": [
        "# generate the metrics reports\n",
        "\n",
        "# set your own preferences\n",
        "\n",
        "# as input, the user has to specify what combination of levels they want their metrics to be calculated at, from:\n",
        "# 0 - at model level, 1 - at prompt's level, 2 - at prompt's version, 3 - at the class type (Project, Employee, Status, None), 4 - at input text's type\n",
        "# eg. providing [0], will have the metrics for each model's performance, [1] gives the performance of all models at a specific prompt level, while [0, 1] will give the permformance of each model at a specific prompt's level\n",
        "\n",
        "levels = [0]\n",
        "file = \"results_run_1_2_3_all_prompts_easy_sys_mesg_ord_0_wo_json_tag.txt\"\n",
        "template_strict = True\n",
        "triple_strict = True\n",
        "download_files = False\n",
        "\n",
        "view_metrics(predictions2, file, levels, template_strict = template_strict, triple_strict = triple_strict, download = download_files)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "AlCW25IgglpT",
        "to9GnwT3hpLX",
        "AUd_GWp-nv5S",
        "SVib9spHojks",
        "te-rlOB8o8vx",
        "di5bgNrJqVIu",
        "dgzCc1yFA4cD",
        "wMOzhd-rr4nP"
      ],
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}