ARCE_experiments01_converge_entropy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "df0dc59f",
   "metadata": {},
   "source": [
    "# ARCE experiments-01: observe $H(P_{lmw}(h))$\n",
    "\n",
    "The most straigtforward setting, where both $h$ and $P(h)$ are observable.\n",
    "\n",
    "Experiments in mind:\n",
    "\n",
    "a.) Iterated learning without interaction phase. Show that the system will finally converge to $h$ with the highest prior. We can use *the name of objects*, or *specific prompt* to introduce bias, as illustrated in \"Embers\".\n",
    "\n",
    "## 0. Prepare the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "727cda73",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import random\n",
    "import pandas as pd\n",
    "import json\n",
    "import time\n",
    "from tqdm import tqdm\n",
    "from matplotlib import pyplot as plt\n",
    "from itertools import permutations\n",
    "import os\n",
    "import openai\n",
    "import copy\n",
    "\n",
    "from utils.gpt_api import multi_turn_chatgpt\n",
    "from utils.text_logger import text_logger\n",
    "from utils.h_and_d import data_generator\n",
    "from utils.h_and_d import h_x, cnt_of_status, gen_hstar_rnd\n",
    "from utils.standard_prompts import gen_hd_prompt, gen_dh_prompt, gen_data_prompt\n",
    "from utils.evaluations import eval_feedback_h, convert_chatcompl_to_json\n",
    "import toml\n",
    "\n",
    "def rnd_seed(seed):\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    \n",
    "\n",
    "GLOBAL_TMP = 1 # [1, 0.5, 0.1]\n",
    "SEED = 10086  # [10086, 14843, 42, 1314, 916, 1024]\n",
    "GEN = 6\n",
    "M_generate = 4   # How many example generated by agent for the next generation\n",
    "LOOK_BACK = 2 # [0, 2, 4]\n",
    "MODEL_NAME = \"gpt-3.5-turbo-0125\"  #  [\"gpt-3.5-turbo-0125\", \"gpt-4-0125-preview\", \"claude-3-haiku-20240307\", \"claude-instant-1.2\"]\n",
    "##### Note that Claude cannot report logits\n",
    "\n",
    "EXP_PATH = './exp_logs_' + MODEL_NAME + '/entropy'\n",
    "\n",
    "rnd_seed(SEED)\n",
    "EXP_NAME = \"tmp%.1f_M%d_LB%d_seed%d\"%(GLOBAL_TMP, M_generate, LOOK_BACK, SEED)\n",
    "exp_path = os.path.join(EXP_PATH, EXP_NAME)\n",
    "LOG = text_logger(file_name='chat_log', exp_path=exp_path, silence = True)\n",
    "LOG.write_to_file('This is an experiment trying to see the convergence of H(P(h))')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1573aef6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'A': 'on', 'B': 'on', 'C': 'off', 'D': 'off', 'E': 'on'}\n"
     ]
    }
   ],
   "source": [
    "# ------------- Step 1: generate h_star, generate d0\n",
    "N = 5\n",
    "N_SMP = 2**N-1\n",
    "N_Train = N_SMP - 10\n",
    "PROBS = [0.5, 0.3, 0.2]\n",
    "OBJECTS = ['A', 'B', 'C', 'D', 'E']\n",
    "STATES = [\"on\",\"off\",\"und\"]\n",
    "\n",
    "h_star = gen_hstar_rnd(OBJECTS, PROBS)\n",
    "print(h_star)\n",
    "# --------- Get d0 given h_star\n",
    "D_GENERATOR = data_generator(h_star, N_test=10)\n",
    "d0 = D_GENERATOR.sample_d0(M=4)\n",
    "\n",
    "LOG.write_to_file('In this experiment,h* is %s. Global temperature is %f\\n'%(h_star, GLOBAL_TMP))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ad3fe95",
   "metadata": {},
   "source": [
    "## 1. Iterated Learning Without Interaction Phase\n",
    "\n",
    "### Experiments explanation\n",
    "We wish to see the bias in the prior will be amplified, no matter what $d_0$ we give. \n",
    "\n",
    "#### Exp.1-1: observable $h$\n",
    "There are three steps for this part:\n",
    "\n",
    "- a.) Select $d_0$, can be randomly, can also select the most correct/incorrect $d_0$\n",
    "\n",
    "- b.) Repeat the following two phases:\n",
    "    - $h^t\\sim P(h\\mid d^{t-1})$\n",
    "    - $d^t\\sim P(d\\mid h^t)$\n",
    "\n",
    "- c.) Observe how $d$ changes, $P(h),\\forall h$ if possible.\n",
    "\n",
    "#### Exp.1-2: hidden $h$\n",
    "Similar to 1.1, but we no longer let the model output the rule explicitly. This is more similar to the \"sample $h$\" case.\n",
    "So we only need to repeat $d^t\\sim P(d\\mid d^{t-1})$.\n",
    "\n",
    "\n",
    "### Prompt examples\n",
    "\n",
    "\n",
    "- For $h\\sim P(h\\mid d)$\n",
    "\n",
    "  Let us play a rule finding game. You need to generate a rule that maps the given inputs to their corresponding outputs. Each example is an input-output pair. The input is a list of objects. The presence of certain objects will trigger the light to turn on. The output is either \"on\" or \"off\", indicating the state of the light. For each object, determine whether it triggers the light to turn on, does not trigger it, or if it is undetermined. We only have four objects in this game, i.e., A, B, C, D.\n",
    "\n",
    "  Input: A, B, C\n",
    "\n",
    "  Output: on\n",
    "\n",
    "  Please format your rule as follows:\n",
    "\n",
    "  Rule: {\"object 1\": <\"on\"/\"off\"/\"undetermined\">, \"object 2\": <\"on\"/\"off\"/\"undetermined\">, ...}\n",
    "  \n",
    "  [Rule: {object 1 on/off/undetermined; object 2 on/off/undetermined; ...}]\n",
    "\n",
    "\n",
    "- For $d\\sim P(d\\mid h)$\n",
    "\n",
    "  The rule you provided is {\"A\": \"on\", \"B\": \"off\", \"C\": \"off\", \"D\": \"undetermined\"}. Based on this rule, can you determine the output of the following examples?\n",
    "\n",
    "  Input: D\n",
    "\n",
    "  Output: \n",
    "\n",
    "\n",
    "\n",
    "  The rule you provided is {\"A\": \"on\", \"B\": \"off\", \"C\": \"off\", \"D\": \"undetermined\"}. Remember that the presence of certain objects will trigger the light to turn on. Based on this rule, can you give 4 more examples? (Only give the input-output pairs)\n",
    "\n",
    "  Input: D\n",
    "\n",
    "  Output: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c46fb9cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "game_prompt = 'Let us play a rule following game. You need to generate a rule that maps the following inputs to their corresponding outputs, or generate examples following a given rule. Each example is an input-output pair. The input is a list of objects. If any objects with status on in the input, the output should be on. If all objects in the input are off, the output is off. If only objects with off and und in the list, the output should be undetermined (und for short). The output is either \"on\", \"off\", or \"und\", indicating the state of the light. For each object, determine whether it triggers the light to turn on, does not trigger it, or if it is undetermined. We only have %d objects in this game: %s.\\n'%(N, OBJECTS)\n",
    "LOG.msg_to_gpt(game_prompt)\n",
    "GPT_AGENT = multi_turn_chatgpt(model=MODEL_NAME, temperature=GLOBAL_TMP, top_p=1, logger=LOG, game_description=game_prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "895d0cf3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|          | 0/6 [00:00<?, ?it/s]\n"
     ]
    },
    {
     "ename": "NotFoundError",
     "evalue": "Error code: 404 - {'error': {'message': 'This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?', 'type': 'invalid_request_error', 'param': 'model', 'code': None}}",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNotFoundError\u001b[0m                             Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[4], line 13\u001b[0m\n\u001b[0;32m     11\u001b[0m hd_prompt \u001b[38;5;241m=\u001b[39m gen_hd_prompt(data\u001b[38;5;241m=\u001b[39mdata_str, ask_rule\u001b[38;5;241m=\u001b[39mrule_format)\n\u001b[0;32m     12\u001b[0m     \u001b[38;5;66;03m# --------- Get feedback\u001b[39;00m\n\u001b[1;32m---> 13\u001b[0m hd_feedback, hd_fb_probs, cnt_tokens \u001b[38;5;241m=\u001b[39m \u001b[43mGPT_AGENT\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall_chatgpt\u001b[49m\u001b[43m(\u001b[49m\u001b[43mhd_prompt\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mfake_response\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mNone\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\n\u001b[0;32m     14\u001b[0m \u001b[43m                                          \u001b[49m\u001b[43mlogprobs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtop_logprobs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m5\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlookback_round\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mLOOK_BACK\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m     15\u001b[0m results[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrules\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;241m.\u001b[39mappend(hd_feedback)\n\u001b[0;32m     16\u001b[0m results_prob_list\u001b[38;5;241m.\u001b[39mappend(hd_fb_probs)\n",
      "File \u001b[1;32mE:\\P5_iICL\\iterated_learning_exp\\utils\\gpt_api.py:63\u001b[0m, in \u001b[0;36mmulti_turn_chatgpt.call_chatgpt\u001b[1;34m(self, msg_to_api, fake_response, logprobs, top_logprobs, lookback_round, update_hist)\u001b[0m\n\u001b[0;32m     60\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m     61\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmodel\u001b[38;5;241m.\u001b[39mstartswith(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mgpt\u001b[39m\u001b[38;5;124m'\u001b[39m):\n\u001b[0;32m     62\u001b[0m         \u001b[38;5;66;03m# ------------- API for gpt-xxx\u001b[39;00m\n\u001b[1;32m---> 63\u001b[0m         response \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mclient\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mchat\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcompletions\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcreate\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m     64\u001b[0m \u001b[43m            \u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmodel\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m     65\u001b[0m \u001b[43m            \u001b[49m\u001b[43mmessages\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mmsg\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m     66\u001b[0m \u001b[43m            \u001b[49m\u001b[43mtemperature\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtemperature\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m     67\u001b[0m \u001b[43m            \u001b[49m\u001b[43mtop_p\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtop_p\u001b[49m\u001b[43m,\u001b[49m\u001b[43m         \u001b[49m\n\u001b[0;32m     68\u001b[0m \u001b[43m            \u001b[49m\u001b[43mlogprobs\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[43mlogprobs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m     69\u001b[0m \u001b[43m            \u001b[49m\u001b[43mtop_logprobs\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m \u001b[49m\u001b[43mtop_logprobs\u001b[49m\n\u001b[0;32m     70\u001b[0m \u001b[43m            \u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m     71\u001b[0m         receive_msg \u001b[38;5;241m=\u001b[39m response\u001b[38;5;241m.\u001b[39mchoices[\u001b[38;5;241m0\u001b[39m]\u001b[38;5;241m.\u001b[39mmessage\u001b[38;5;241m.\u001b[39mcontent\n\u001b[0;32m     72\u001b[0m         cnt_tokens \u001b[38;5;241m=\u001b[39m (response\u001b[38;5;241m.\u001b[39musage\u001b[38;5;241m.\u001b[39mcompletion_tokens, \n\u001b[0;32m     73\u001b[0m                     response\u001b[38;5;241m.\u001b[39musage\u001b[38;5;241m.\u001b[39mprompt_tokens, \n\u001b[0;32m     74\u001b[0m                     response\u001b[38;5;241m.\u001b[39musage\u001b[38;5;241m.\u001b[39mtotal_tokens)\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\openai\\_utils\\_utils.py:275\u001b[0m, in \u001b[0;36mrequired_args.<locals>.inner.<locals>.wrapper\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    273\u001b[0m             msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mMissing required argument: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mquote(missing[\u001b[38;5;241m0\u001b[39m])\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    274\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m(msg)\n\u001b[1;32m--> 275\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m func(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\openai\\resources\\chat\\completions.py:667\u001b[0m, in \u001b[0;36mCompletions.create\u001b[1;34m(self, messages, model, frequency_penalty, function_call, functions, logit_bias, logprobs, max_tokens, n, presence_penalty, response_format, seed, stop, stream, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)\u001b[0m\n\u001b[0;32m    615\u001b[0m \u001b[38;5;129m@required_args\u001b[39m([\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmessages\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m], [\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmessages\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mmodel\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mstream\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[0;32m    616\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcreate\u001b[39m(\n\u001b[0;32m    617\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    665\u001b[0m     timeout: \u001b[38;5;28mfloat\u001b[39m \u001b[38;5;241m|\u001b[39m httpx\u001b[38;5;241m.\u001b[39mTimeout \u001b[38;5;241m|\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;241m|\u001b[39m NotGiven \u001b[38;5;241m=\u001b[39m NOT_GIVEN,\n\u001b[0;32m    666\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m ChatCompletion \u001b[38;5;241m|\u001b[39m Stream[ChatCompletionChunk]:\n\u001b[1;32m--> 667\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_post\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    668\u001b[0m \u001b[43m        \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m/chat/completions\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[0;32m    669\u001b[0m \u001b[43m        \u001b[49m\u001b[43mbody\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmaybe_transform\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    670\u001b[0m \u001b[43m            \u001b[49m\u001b[43m{\u001b[49m\n\u001b[0;32m    671\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmessages\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmessages\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    672\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmodel\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    673\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfrequency_penalty\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfrequency_penalty\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    674\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfunction_call\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfunction_call\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    675\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mfunctions\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mfunctions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    676\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mlogit_bias\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mlogit_bias\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    677\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mlogprobs\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mlogprobs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    678\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmax_tokens\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mmax_tokens\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    679\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mn\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mn\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    680\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mpresence_penalty\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mpresence_penalty\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    681\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mresponse_format\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mresponse_format\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    682\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mseed\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mseed\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    683\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mstop\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mstop\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    684\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mstream\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    685\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtemperature\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtemperature\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    686\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtool_choice\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtool_choice\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    687\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtools\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtools\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    688\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtop_logprobs\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtop_logprobs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    689\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtop_p\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mtop_p\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    690\u001b[0m \u001b[43m                \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43muser\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43muser\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    691\u001b[0m \u001b[43m            \u001b[49m\u001b[43m}\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    692\u001b[0m \u001b[43m            \u001b[49m\u001b[43mcompletion_create_params\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mCompletionCreateParams\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    693\u001b[0m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    694\u001b[0m \u001b[43m        \u001b[49m\u001b[43moptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mmake_request_options\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    695\u001b[0m \u001b[43m            \u001b[49m\u001b[43mextra_headers\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mextra_headers\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mextra_query\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mextra_query\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mextra_body\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mextra_body\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtimeout\u001b[49m\n\u001b[0;32m    696\u001b[0m \u001b[43m        \u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    697\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mChatCompletion\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    698\u001b[0m \u001b[43m        \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m,\u001b[49m\n\u001b[0;32m    699\u001b[0m \u001b[43m        \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mStream\u001b[49m\u001b[43m[\u001b[49m\u001b[43mChatCompletionChunk\u001b[49m\u001b[43m]\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    700\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\openai\\_base_client.py:1208\u001b[0m, in \u001b[0;36mSyncAPIClient.post\u001b[1;34m(self, path, cast_to, body, options, files, stream, stream_cls)\u001b[0m\n\u001b[0;32m   1194\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mpost\u001b[39m(\n\u001b[0;32m   1195\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[0;32m   1196\u001b[0m     path: \u001b[38;5;28mstr\u001b[39m,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   1203\u001b[0m     stream_cls: \u001b[38;5;28mtype\u001b[39m[_StreamT] \u001b[38;5;241m|\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[0;32m   1204\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m ResponseT \u001b[38;5;241m|\u001b[39m _StreamT:\n\u001b[0;32m   1205\u001b[0m     opts \u001b[38;5;241m=\u001b[39m FinalRequestOptions\u001b[38;5;241m.\u001b[39mconstruct(\n\u001b[0;32m   1206\u001b[0m         method\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpost\u001b[39m\u001b[38;5;124m\"\u001b[39m, url\u001b[38;5;241m=\u001b[39mpath, json_data\u001b[38;5;241m=\u001b[39mbody, files\u001b[38;5;241m=\u001b[39mto_httpx_files(files), \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39moptions\n\u001b[0;32m   1207\u001b[0m     )\n\u001b[1;32m-> 1208\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m cast(ResponseT, \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrequest\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mopts\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m)\u001b[49m)\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\openai\\_base_client.py:897\u001b[0m, in \u001b[0;36mSyncAPIClient.request\u001b[1;34m(self, cast_to, options, remaining_retries, stream, stream_cls)\u001b[0m\n\u001b[0;32m    888\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mrequest\u001b[39m(\n\u001b[0;32m    889\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[0;32m    890\u001b[0m     cast_to: Type[ResponseT],\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    895\u001b[0m     stream_cls: \u001b[38;5;28mtype\u001b[39m[_StreamT] \u001b[38;5;241m|\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m,\n\u001b[0;32m    896\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m ResponseT \u001b[38;5;241m|\u001b[39m _StreamT:\n\u001b[1;32m--> 897\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_request\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m    898\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcast_to\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcast_to\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    899\u001b[0m \u001b[43m        \u001b[49m\u001b[43moptions\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43moptions\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    900\u001b[0m \u001b[43m        \u001b[49m\u001b[43mstream\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    901\u001b[0m \u001b[43m        \u001b[49m\u001b[43mstream_cls\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mstream_cls\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    902\u001b[0m \u001b[43m        \u001b[49m\u001b[43mremaining_retries\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mremaining_retries\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m    903\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32m~\\AppData\\Roaming\\Python\\Python310\\site-packages\\openai\\_base_client.py:988\u001b[0m, in \u001b[0;36mSyncAPIClient._request\u001b[1;34m(self, cast_to, options, remaining_retries, stream, stream_cls)\u001b[0m\n\u001b[0;32m    985\u001b[0m         err\u001b[38;5;241m.\u001b[39mresponse\u001b[38;5;241m.\u001b[39mread()\n\u001b[0;32m    987\u001b[0m     log\u001b[38;5;241m.\u001b[39mdebug(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mRe-raising status error\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m--> 988\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_make_status_error_from_response(err\u001b[38;5;241m.\u001b[39mresponse) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m    990\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_process_response(\n\u001b[0;32m    991\u001b[0m     cast_to\u001b[38;5;241m=\u001b[39mcast_to,\n\u001b[0;32m    992\u001b[0m     options\u001b[38;5;241m=\u001b[39moptions,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m    995\u001b[0m     stream_cls\u001b[38;5;241m=\u001b[39mstream_cls,\n\u001b[0;32m    996\u001b[0m )\n",
      "\u001b[1;31mNotFoundError\u001b[0m: Error code: 404 - {'error': {'message': 'This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?', 'type': 'invalid_request_error', 'param': 'model', 'code': None}}"
     ]
    }
   ],
   "source": [
    "# ------------ Step 1: random choose M training data samples as d0\n",
    "rule_format = 'Rule: {\"object 1\": <\"on\"/\"off\"/\"und\">, \"object 2\": <\"on\"/\"off\"/\"und\">, ...}'\n",
    "#rule_format = 'Rule: {object 1 on/off/und; object 2 on/off/und; ...}'\n",
    "\n",
    "data_str = gen_data_prompt(d0, need_stat=True)\n",
    "results = {'nh_corr':[],'nh_perf':[],'d_sampled':[],'prompt_token':[],'rules':[]}\n",
    "results_prob_list = []\n",
    "for g in tqdm(range(GEN)):\n",
    "    LOG.write_to_file('----------- Gen %d -----------'%g)\n",
    "    # ------------ Step 2: h~P(h|d)\n",
    "    hd_prompt = gen_hd_prompt(data=data_str, ask_rule=rule_format)\n",
    "        # --------- Get feedback\n",
    "    hd_feedback, hd_fb_probs, cnt_tokens = GPT_AGENT.call_chatgpt(hd_prompt, fake_response=None, \n",
    "                                              logprobs=True, top_logprobs=5, lookback_round=LOOK_BACK)\n",
    "    results['rules'].append(hd_feedback)\n",
    "    results_prob_list.append(hd_fb_probs)\n",
    "    results['prompt_token'].append(cnt_tokens)\n",
    "    hd_feedback_str = json.loads(hd_feedback.split(\"Rule: \")[-1])\n",
    "    nh_corr, nh_perf = eval_feedback_h(h=h_star, fb_h=hd_feedback_str, N=N)\n",
    "    results['nh_corr'].append(nh_corr/N)\n",
    "    results['nh_perf'].append(nh_perf)\n",
    "    \n",
    "    if g<GEN-1:\n",
    "        # ------------ Step 3: d~P(d|h)\n",
    "        dh_prompt = gen_dh_prompt(M = M_generate, rule = hd_feedback_str)\n",
    "            # --------- Get feedback\n",
    "        dh_feedback, _, cnt_tokens = GPT_AGENT.call_chatgpt(dh_prompt, fake_response=None, lookback_round=LOOK_BACK)\n",
    "        results['prompt_token'].append(cnt_tokens)\n",
    "        results['d_sampled'].append(dh_feedback)\n",
    "        # ----------- Step 4: data_str <-- dh_feedback\n",
    "        data_str = dh_feedback\n",
    "print(results['nh_corr'])\n",
    "print(results['prompt_token'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a1a44a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fa22eb4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "751f9241",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ------------ Save prob_lists\n",
    "if MODEL_NAME.startswith('gpt'):\n",
    "    file_name = 'prob_list_all.json'\n",
    "    save_path = os.path.join(exp_path, file_name)    \n",
    "    json.dump(convert_chatcompl_to_json(results_prob_list), open(save_path, 'w' ))\n",
    "\n",
    "file_name2 = 'other_results_all.json'\n",
    "save_path2 = os.path.join(exp_path, file_name2)\n",
    "json.dump(results, open(save_path2, 'w' ))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e29e8be",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e087124e",
   "metadata": {},
   "source": [
    "### Visualize the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89bf0259-f780-4eb6-ab4d-dc5abc06425f",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_path = exp_path + '//prob_list_all.json'\n",
    "save_path2 = exp_path + '//other_results_all.json'\n",
    "# save_path = \"E://P5_iICL//iterated_learning_exp//exp_logs//entropy//tmp1.0_M4_LB4_seed10086//prob_list_all.json\"\n",
    "# save_path2 = \"E://P5_iICL//iterated_learning_exp//exp_logs//entropy//tmp1.0_M4_LB4_seed10086//other_results_all.json\"\n",
    "prob_list_read = json.load( open( save_path ))\n",
    "results_read = json.load(open(save_path2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b2fa234",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ============= Assign number of correct predictions to each h\n",
    "def dstr_to_pairs(d_str):\n",
    "    tmp_str = d_str.split('\\n')\n",
    "    while \"\" in tmp_str:\n",
    "        tmp_str.remove(\"\")\n",
    "    data_pairs = []\n",
    "    for s in range(int(len(tmp_str)*0.5)):\n",
    "        tmp_input = tmp_str[2*s].split(': ')[1].split(', ')\n",
    "        tmp_output = tmp_str[2*s+1].split(': ')[1]\n",
    "        data_pairs.append((tmp_input, tmp_output))\n",
    "    return data_pairs\n",
    "\n",
    "def dlist_to_pairs(d_list):\n",
    "    data_pairs = []\n",
    "    if not (d_list[0].startswith('Input') or d_list[0].startswith('Output')):\n",
    "        d_list = d_list[1:]\n",
    "    for i in range(int(0.5*len(d_list))):\n",
    "        tmp_input = d_list[2*i].split(': ')[1].split(', ')\n",
    "        tmp_output = d_list[2*i+1].split(': ')[1]\n",
    "        data_pairs.append((tmp_input, tmp_output))\n",
    "    return data_pairs\n",
    "\n",
    "def count_corr_d0_pairs(d_pairs, rule):\n",
    "    corr_cnt, all_cnt = 0, 0\n",
    "    for x,y in d_pairs:\n",
    "        all_cnt += 1\n",
    "        if h_x(x, rule)==y:\n",
    "            corr_cnt += 1\n",
    "    return corr_cnt, all_cnt\n",
    "\n",
    "def count_corr_d0(d, h):\n",
    "    # Calculate how many examples in d can be explained by given h\n",
    "    corr_cnt = 0\n",
    "    for _, row in d.iterrows():\n",
    "        x, y = row['obj'], row['stat']\n",
    "        y_pred = h_x(x, h)\n",
    "        if y==y_pred:\n",
    "            corr_cnt += 1\n",
    "    return corr_cnt\n",
    "\n",
    "def cal_entropy(hd_fb_probs):\n",
    "    token_logprob = extract_probs(hd_fb_probs, OBJECTS, top_n=5)\n",
    "    entropy = 0\n",
    "    for i in range(len(all_possible_statuses)):\n",
    "        h_tmp = {}\n",
    "        for j in range(len(OBJECTS)):\n",
    "            h_tmp[OBJECTS[j]] = all_possible_statuses[i][j]\n",
    "        if h_tmp == h_star:\n",
    "            h_star_idx = i\n",
    "        obj_logprob, obj_prob = cal_prob_of_h(h_tmp, token_logprob)\n",
    "        entropy += -obj_prob*obj_logprob\n",
    "    return entropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98e08e7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ============= Code for generate P(h) ===============\n",
    "def extract_probs(gpt_fb, objects, top_n=5):\n",
    "    token_logprob = {}\n",
    "    for i in range(len(gpt_fb)):\n",
    "        if gpt_fb[i]['token'] in objects:\n",
    "            obj = gpt_fb[i]['token']\n",
    "            obj_toplogs = gpt_fb[i+3]['top_logprobs']\n",
    "            token_logprob[obj]={}\n",
    "            for j in range(top_n):\n",
    "                candi_token = obj_toplogs[j]['token']\n",
    "                candi_prob = obj_toplogs[j]['logprob']\n",
    "                token_logprob[obj][candi_token] = candi_prob #np.exp(candi_prob)\n",
    "    return token_logprob\n",
    "\n",
    "def cal_prob_of_h(h_star, token_logprob):\n",
    "    obj_logprob = 0\n",
    "    for obj in h_star.keys():\n",
    "        status = h_star[obj]\n",
    "        if status in token_logprob[obj].keys():\n",
    "            tmp_logprob = token_logprob[obj][status]\n",
    "        else:\n",
    "            tmp_logprob = -10\n",
    "        obj_logprob += tmp_logprob\n",
    "    return obj_logprob, np.exp(obj_logprob)\n",
    "\n",
    "# ------------- Generate h satisfying \n",
    "def generate_all_statuses(N,M, packed=False):\n",
    "    def generate_all_statuses_(N, M):\n",
    "        def generate_status_helper(current_status):\n",
    "            if len(current_status) == N:\n",
    "                all_statuses.append(current_status.copy())\n",
    "                return\n",
    "            for state in range(M):\n",
    "                current_status.append(STATES[state])\n",
    "                generate_status_helper(current_status)\n",
    "                current_status.pop()\n",
    "        all_statuses = []\n",
    "        generate_status_helper([])\n",
    "        return all_statuses\n",
    "    tmp_all_possible_statuses = generate_all_statuses_(N, M)\n",
    "    if packed:\n",
    "        all_possible_statuses = []\n",
    "        for s in tmp_all_possible_statuses:\n",
    "            all_possible_statuses.append(s[::-1])\n",
    "        return all_possible_statuses\n",
    "    else:\n",
    "        return tmp_all_possible_statuses\n",
    "    \n",
    "all_possible_statuses = generate_all_statuses(N=len(OBJECTS), M=len(STATES), packed=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "107c758d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# ============= Assign number of correct predictions to each h\n",
    "def count_corr_d0(d, h):\n",
    "    # Calculate how many examples in d can be explained by given h\n",
    "    corr_cnt = 0\n",
    "    for _, row in d.iterrows():\n",
    "        x, y = row['obj'], row['stat']\n",
    "        y_pred = h_x(x, h)\n",
    "        if y==y_pred:\n",
    "            corr_cnt += 1\n",
    "    return corr_cnt\n",
    "\n",
    "def cal_entropy(hd_fb_probs):\n",
    "    token_logprob = extract_probs(hd_fb_probs, OBJECTS, top_n=5)\n",
    "    entropy = 0\n",
    "    for i in range(len(all_possible_statuses)):\n",
    "        h_tmp = {}\n",
    "        for j in range(len(OBJECTS)):\n",
    "            h_tmp[OBJECTS[j]] = all_possible_statuses[i][j]\n",
    "        if h_tmp == h_star:\n",
    "            h_star_idx = i\n",
    "        obj_logprob, obj_prob = cal_prob_of_h(h_tmp, token_logprob)\n",
    "        entropy += -obj_prob*obj_logprob\n",
    "    return entropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d21c161",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ============= Code for generate P(h) ===============\n",
    "def extract_probs(gpt_fb, objects, top_n=5):\n",
    "    token_logprob = {}\n",
    "    for i in range(len(gpt_fb)):\n",
    "        if gpt_fb[i]['token'] in objects:\n",
    "            obj = gpt_fb[i]['token']\n",
    "            obj_toplogs = gpt_fb[i+3]['top_logprobs']\n",
    "            token_logprob[obj]={}\n",
    "            for j in range(top_n):\n",
    "                candi_token = obj_toplogs[j]['token']\n",
    "                candi_prob = obj_toplogs[j]['logprob']\n",
    "                token_logprob[obj][candi_token] = candi_prob #np.exp(candi_prob)\n",
    "    return token_logprob\n",
    "\n",
    "def cal_prob_of_h(h_star, token_logprob):\n",
    "    obj_logprob = 0\n",
    "    for obj in h_star.keys():\n",
    "        status = h_star[obj]\n",
    "        if status in token_logprob[obj].keys():\n",
    "            tmp_logprob = token_logprob[obj][status]\n",
    "        else:\n",
    "            tmp_logprob = -10\n",
    "        obj_logprob += tmp_logprob\n",
    "    return obj_logprob, np.exp(obj_logprob)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "672e95f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "uni_entropy, oht_entropy = 0, 0\n",
    "for i in range(243):\n",
    "    if i==3:\n",
    "        prob = 1-242*1e-10\n",
    "        oht_entropy += -prob*np.log(prob)\n",
    "    uni_entropy += -(1/243)*np.log(1/243)\n",
    "    oht_entropy += -1e-10*np.log(1e-10)\n",
    "\n",
    "entropy_list = []\n",
    "for i in range(6):\n",
    "    entropy = cal_entropy(prob_list_read[i])\n",
    "    entropy_list.append(entropy)\n",
    "\n",
    "uni_entropy, oht_entropy = 0, 0\n",
    "for i in range(243):\n",
    "    if i==3:\n",
    "        prob = 1-242*1e-10\n",
    "        oht_entropy += -prob*np.log(prob)\n",
    "    uni_entropy += -(1/243)*np.log(1/243)\n",
    "    oht_entropy += -1e-10*np.log(1e-10)\n",
    "\n",
    "entropy_list = []\n",
    "for i in range(6):\n",
    "    entropy = cal_entropy(prob_list_read[i])\n",
    "    entropy_list.append(entropy)\n",
    "\n",
    "print(entropy_list)\n",
    "plt.plot(entropy_list,label='GPT return')\n",
    "#plt.plot([0,5],[uni_entropy,uni_entropy], label='Uniform')\n",
    "#plt.plot([0,5],[oht_entropy,oht_entropy], label='One-hot')\n",
    "plt.legend(fontsize=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dda7e3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def draw_pic(hd_fb_probs, h_star, d0_str, ax=None, y_log=True, ylim=None, legend=True, x_tickle=True, xlabel=None, star=True, color_type='screen'):\n",
    "    if ax is None:\n",
    "        fig, ax = plt.subplots(1,1,figsize=(15,5))\n",
    "    token_logprob = extract_probs(hd_fb_probs, OBJECTS, top_n=5)\n",
    "\n",
    "    prob_list, corr_list, screen_list = [], [], []\n",
    "    for i in range(len(all_possible_statuses)):\n",
    "        h_tmp = {}\n",
    "        for j in range(len(OBJECTS)):\n",
    "            h_tmp[OBJECTS[j]] = all_possible_statuses[i][j]\n",
    "        if h_tmp == h_star:\n",
    "            h_star_idx = i\n",
    "        #if h_tmp == h_bar:\n",
    "        #    h_bar_idx = i\n",
    "        obj_logprob, obj_prob = cal_prob_of_h(h_tmp, token_logprob)\n",
    "        prob_list.append(obj_prob)\n",
    "\n",
    "        d0_pairs = dlist_to_pairs(d0_str.split('\\n'))      \n",
    "        corr_cnt,_ = count_corr_d0_pairs(d0_pairs, h_tmp)\n",
    "        corr_list.append(corr_cnt)\n",
    "        if all_possible_statuses[i][-1]=='on':\n",
    "            screen_list.append(0)\n",
    "        elif all_possible_statuses[i][-1]=='off':\n",
    "            screen_list.append(1)\n",
    "        else:\n",
    "            screen_list.append(2)\n",
    "    prob_list = np.array(prob_list)\n",
    "    corr_list = np.array(corr_list)\n",
    "    screen_list = np.array(screen_list)\n",
    "    x_axis = np.arange(0,243)\n",
    "\n",
    "    if color_type=='d0':\n",
    "        ALPHA_LIST = [0.05, 0.2, 0.3, 0.45, 0.55, 0.7, 0.8, 0.9 , 1]#[0.05, 0.2, 0.3, 0.5, 1]\n",
    "        for i in range(len(ALPHA_LIST)):\n",
    "            mask = corr_list==i\n",
    "            if i>0:\n",
    "                label = \"%d corr\"%i\n",
    "            else:\n",
    "                label = None\n",
    "            ax.bar(x_axis[mask],prob_list[mask],width=1, color='royalblue',alpha=ALPHA_LIST[i],label=label)\n",
    "    elif color_type=='screen':\n",
    "        COLOR_LIST = ['#f8ac8c', 'royalblue','#9e9e9e']#'#2878b5',\n",
    "        LABELS = ['on', 'off', 'und']\n",
    "        for i in range(len(COLOR_LIST)):\n",
    "            mask = screen_list==i\n",
    "            ax.bar(x_axis[mask],prob_list[mask],width=1, color=COLOR_LIST[i],alpha=0.7,label=LABELS[i])\n",
    "        \n",
    "    if y_log:\n",
    "        ax.set_yscale('log')\n",
    "    ax.set_xlim(-3, 245)\n",
    "    if ylim is not None:\n",
    "        ax.set_ylim(ylim)\n",
    "    if star:\n",
    "        ax.plot((h_star_idx), (prob_list[h_star_idx]), color='red',alpha=1,linestyle=' ',marker='*',markersize=10, label=r'$h^*$')\n",
    "        #ax.plot((h_bar_idx), (prob_list[h_bar_idx]), color='red',alpha=1,linestyle=' ',marker='+',markersize=10, label=r'$\\hat{h}$')\n",
    "    if legend:\n",
    "        ax.legend(fontsize=10, ncol=1, loc='upper left')\n",
    "    if not x_tickle:\n",
    "        ax.set_xticks([])\n",
    "    if xlabel is not None:\n",
    "        ax.set_xlabel(xlabel,fontsize=16)\n",
    "    return prob_list, corr_list"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5091b921-08fb-47b6-8e9e-463f4fbc48da",
   "metadata": {},
   "source": [
    "## Observe results in another folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6031cec-7674-4f88-ba67-1a711c5c9881",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cal_entropy(hd_fb_probs):\n",
    "    token_logprob = extract_probs(hd_fb_probs, OBJECTS, top_n=5)\n",
    "    entropy = 0\n",
    "    for i in range(len(all_possible_statuses)):\n",
    "        h_tmp = {}\n",
    "        for j in range(len(OBJECTS)):\n",
    "            h_tmp[OBJECTS[j]] = all_possible_statuses[i][j]\n",
    "        if h_tmp == h_star:\n",
    "            h_star_idx = i\n",
    "        obj_logprob, obj_prob = cal_prob_of_h(h_tmp, token_logprob)\n",
    "        entropy += -obj_prob*obj_logprob\n",
    "    return entropy\n",
    "\n",
    "def get_entropy(prob_list_read):\n",
    "    entropy_list = []\n",
    "    for i in range(GEN):\n",
    "        entropy = cal_entropy(prob_list_read[i])\n",
    "        entropy_list.append(entropy)\n",
    "    return entropy_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6653c51-ff3d-4e24-8039-277fa50e881c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ============== Extract all results, obj is ABCDE\n",
    "OBJECTS = ['A', 'B', 'C', 'D', 'E']\n",
    "all_possible_statuses = generate_all_statuses(N=len(OBJECTS), M=len(STATES), packed=False)\n",
    "GEN = 6\n",
    "TMP = [1]\n",
    "SEED = [10086, 42, 1314, 1024, 14843, 916] # 1024\n",
    "exp_path_load = \"E://P5_iICL//iterated_learning_exp//exp_logs_gpt-4-0125-preview//entropy//\" #\"E://P5_iICL//iterated_learning_exp//exp_logs//entropy//\" #\n",
    "np_entropy = np.zeros((len(TMP), len(SEED), GEN))\n",
    "\n",
    "for i in range(len(TMP)):\n",
    "    t = TMP[i]\n",
    "    for j in range(len(SEED)):\n",
    "        s = SEED[j]\n",
    "        file=\"tmp%.1f_M%d_LB%d_seed%d\"%(t, 4, 2, s)\n",
    "        save_path = os.path.join(exp_path_load, file, 'prob_list_all.json') \n",
    "        save_path2 = os.path.join(exp_path_load, file, 'other_results_all.json')   \n",
    "        prob_list_read = json.load( open( save_path ))\n",
    "        results_read = json.load(open(save_path2))\n",
    "        h_star = results_read['rules'][0]\n",
    "        \n",
    "        entropy = get_entropy(prob_list_read)\n",
    "        np_entropy[i][j][:] = entropy\n",
    "np_entropy_mean = np_entropy.mean(1)\n",
    "np_entropy_std = np_entropy.var(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75c6336b-9d2d-4490-a1f4-2a45fe9520d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_axis = np.arange(1,7,1)\n",
    "plt.plot(x_axis, np_entropy_mean[0], label='$\\\\tau$=1.0')\n",
    "plt.fill_between(x_axis, np_entropy_mean[0]-np_entropy_std[0], np_entropy_mean[0]+np_entropy_std[0], alpha=0.15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fcdbdfcb-105d-41f5-b250-383621d91ffd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7fb94c4-a45e-42dd-b4f9-7ffb00152a06",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25bcfafd-76f5-4f64-8a21-3cefdc7c8d0f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "778b70c6-de05-4c83-979f-2b44748b3ca0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5943704c",
   "metadata": {},
   "outputs": [],
   "source": [
    "fold_path = 'E://P5_iICL//iterated_learning_exp//exp_logs_gpt-4-0125-preview//entropy//tmp1.0_M4_LB0_seed10086'\n",
    "save_path = os.path.join(fold_path, \"prob_list_all.json\")\n",
    "save_path2 = os.path.join(fold_path, \"other_results_all.json\")\n",
    "prob_list_read = json.load( open( save_path ))\n",
    "results_read = json.load(open(save_path2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f2be8ab-3f26-4c66-aa6b-24be798d4192",
   "metadata": {},
   "outputs": [],
   "source": [
    "uni_entropy, oht_entropy = 0, 0\n",
    "for i in range(243):\n",
    "    if i==3:\n",
    "        prob = 1-242*1e-10\n",
    "        oht_entropy += -prob*np.log(prob)\n",
    "    uni_entropy += -(1/243)*np.log(1/243)\n",
    "    oht_entropy += -1e-10*np.log(1e-10)\n",
    "\n",
    "entropy_list = []\n",
    "for i in range(6):\n",
    "    entropy = cal_entropy(prob_list_read[i])\n",
    "    entropy_list.append(entropy)\n",
    "\n",
    "uni_entropy, oht_entropy = 0, 0\n",
    "for i in range(243):\n",
    "    if i==3:\n",
    "        prob = 1-242*1e-10\n",
    "        oht_entropy += -prob*np.log(prob)\n",
    "    uni_entropy += -(1/243)*np.log(1/243)\n",
    "    oht_entropy += -1e-10*np.log(1e-10)\n",
    "\n",
    "entropy_list = []\n",
    "for i in range(6):\n",
    "    entropy = cal_entropy(prob_list_read[i])\n",
    "    entropy_list.append(entropy)\n",
    "    \n",
    "plt.plot(entropy_list,label='GPT return')\n",
    "#plt.yscale('log')\n",
    "#plt.plot([0,5],[uni_entropy,uni_entropy], label='Uniform')\n",
    "#plt.plot([0,5],[oht_entropy,oht_entropy], label='One-hot')\n",
    "plt.legend(fontsize=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c4f5cfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# h_star = json.loads(results_read['rules'][0].split(\"Rule: \")[1])\n",
    "# d0_str = results_read['d_sampled'][0].replace(\"\\n\\n\",\"\\n\")\n",
    "# OBJECTS = ['A', 'B', 'C', 'D', 'E']\n",
    "# all_possible_statuses = generate_all_statuses(N=len(OBJECTS), M=len(STATES), packed=True) # Control the fashion of h-243\n",
    "# fig, ax = plt.subplots(6,1,figsize=(15,35))\n",
    "# for i in range(4):\n",
    "#     if i>0:\n",
    "#         legend=False\n",
    "#     else:\n",
    "#         legend=True\n",
    "#     prob_list, corr_list = draw_pic(prob_list_read[i],h_star, d0_str, ax[i], True, ylim=[1e-12,1],legend=legend,star=False, color_type='screen')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cabeb1ff",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3845b427",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "174dfc6e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e30daa26",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4addf4aa",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33a50563",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5413c00c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f8d7cf3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdc46824",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}