maximum_entropy_irl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Maximum Entropy Inverse Reinforcement Learning\n",
    "\n",
    "Based on the paper of the same name by Brian D. Ziebart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "The Inverse Reinforcement Learning Problem is defined as follows:\n",
    "\n",
    "Given \n",
    "* Markov Decision Process (S, A, T, r, $\\gamma$)\n",
    "* Dataset $\\mathcal{D} = \\{ \\tau_1, \\tau_2, ...\\}$\n",
    "* and Expert Trajectories $\\tau = ((s_1, a_1), (s_2, a_2), ...)\\in \\mathcal{D}$\n",
    "\n",
    "find the reward function $r(s)$ that results in an optimal policy $\\pi$ that generates the Expert Trajectories $\\mathcal{D}$.\n",
    "\n",
    "However, there exist multiple reward functions that solve this problem.\n",
    "This ambiguity needs to be resolved to make IRL a well-defined optimization problem.\n",
    "The approach presented by Ziebart strives to eliminate the ambiguity by framing it as a Constrained Optimization Problem. The Entropy of the resulting Trajectory Distribution is maximized, while enforcing matching feature expectations to Expert Trajectories. By maximizing the Entropy, bias towards certain trajectories that is not backed up by the feature matching constraint is eliminated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import gym\n",
    "import os\n",
    "import time\n",
    "from itertools import product\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import clear_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Utility Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_onehot(state, num_states):\n",
    "    \"\"\"\n",
    "    represent a state as a vector\n",
    "    can be interpreted as a mapping from state to feature\n",
    "    \n",
    "    example: f(1,4) -> (0,1,0,0)\n",
    "             f(3,4) -> (0,0,0,1)\n",
    "    \n",
    "    state:       integer representing the current state\n",
    "    num_state:   total number of states\n",
    "    \"\"\"\n",
    "    result = np.zeros(num_states)\n",
    "    result[state] = 1\n",
    "    return result\n",
    "\n",
    "def softmax1D(vec: np.ndarray):\n",
    "    \"\"\"\n",
    "    compute the softmax of vec\n",
    "    \"\"\"\n",
    "    transformed_vec = vec - np.max(vec)\n",
    "    numerator = np.exp(transformed_vec)\n",
    "    denominator = np.sum(numerator)\n",
    "    return numerator/denominator\n",
    "\n",
    "def visualize_policy(policy, other_policy=None):\n",
    "    left = \"\\u2190\"\n",
    "    down = \"\\u2193\"\n",
    "    right = \"\\u2192\"\n",
    "    up = \"\\u2191\"\n",
    "    action_map = [left,down,right,up]\n",
    "    for i in range(policy.shape[0]):\n",
    "        for j in range(policy.shape[1]):\n",
    "            arrow = action_map[policy[i,j]]\n",
    "            if other_policy is not None:\n",
    "                if policy[i,j]!=other_policy[i,j]: arrow = '\\x1b[6;30;42m'+arrow+'\\x1b[0m'\n",
    "            print(arrow, end=\" \")\n",
    "                \n",
    "        print(\"\")\n",
    "\n",
    "def plot_reward(reward, text=True, use_hsv=False, vrange=None):\n",
    "    \"\"\" \n",
    "    Function from https://github.com/humans-to-robots-motion/rl-course/blob/master/ex05-td/ex05-td.py\n",
    "    Adapted to plot reward and visualize environment\n",
    "    \n",
    "    reward:   reward for each state\n",
    "    env:      gym environment\n",
    "    text:     whether to add values as text\n",
    "    use_hsv:  whether to use hsv color map\n",
    "    \"\"\"\n",
    "    fig = plt.figure()\n",
    "    \n",
    "    dims = (8,8)\n",
    "    reward_reshaped = np.zeros(dims)\n",
    "    for s in range(len(reward)):\n",
    "        idx = np.unravel_index(s, dims)\n",
    "        reward_reshaped[idx] = np.max(reward[s])\n",
    "    \n",
    "    min_r = np.min(reward_reshaped)\n",
    "    max_r = np.max(reward_reshaped)\n",
    "    \n",
    "    if vrange is not None:\n",
    "        min_r, max_r = vrange\n",
    "    \n",
    "    colormap = plt.cm.hsv if use_hsv else plt.cm.RdYlGn\n",
    "    \n",
    "    plt.imshow(reward_reshaped, origin='upper', \n",
    "               extent=[0,dims[0],0,dims[1]], vmin=min_r, vmax=max_r, \n",
    "               cmap=colormap, interpolation='none')\n",
    "    \n",
    "    if text:\n",
    "        for x, y in product(range(dims[0]), range(dims[1])):\n",
    "            plt.text(y+0.5, dims[0]-x-0.5, '{:.2f}'.format(reward_reshaped[x,y]),\n",
    "                    horizontalalignment='center', \n",
    "                    verticalalignment='center')\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "\n",
    "def plot_frozenlake_8x8():\n",
    "    \"\"\"\n",
    "    Plot a visual representation of the FrozenLake8x8 environment\n",
    "    \"\"\"\n",
    "    env_map = np.arange(env.num_states).reshape((8,8))\n",
    "    env_map[[1,3,5,7]] += 1\n",
    "    env_map %= 2\n",
    "    env_map = np.array(env_map, dtype=float)\n",
    "    env_map *= 0.01\n",
    "    env_map += 0.35\n",
    "    \n",
    "    env_map = env_map.flatten()\n",
    "    env_map[env.terminal_states] = 0.8\n",
    "    env_map[env.num_states-1] = 0.2\n",
    "    env_map[0] = 0.1\n",
    "    \n",
    "    plot_reward(env_map, text=False, use_hsv=True, vrange=(0.0,0.6))\n",
    "\n",
    "def value_iteration(env, reward_fn=None, gamma = 0.99, delta_threshold = 0.001):\n",
    "    \"\"\"\n",
    "    env:              environment implementing gym interface, with added attributes\n",
    "                      (num_states, num_actions, P, terminal_states)\n",
    "    reward_fn:        function(state,action) -> reward\n",
    "                      if None: use reward function of environment\n",
    "    gamma:            discounting factor\n",
    "    delta_threshold:  stopping criterion based on update magnitude\n",
    "    \"\"\"\n",
    "    num_states = env.num_states\n",
    "    num_actions = env.num_actions\n",
    "    P = env.P\n",
    "    terminal_states = env.terminal_states\n",
    "    \n",
    "    delta = np.inf\n",
    "    \n",
    "    max_iter = 10000\n",
    "    count = 0\n",
    "    V = np.zeros(num_states)\n",
    "    while not delta < delta_threshold and count < max_iter:\n",
    "        delta = 0\n",
    "        for state in range(num_states):\n",
    "            if state in terminal_states:\n",
    "                continue\n",
    "            \n",
    "            old_v = V[state]\n",
    "            \n",
    "            # for each action calculate value\n",
    "            action_vals = np.zeros(num_actions)\n",
    "            for action in range(num_actions):\n",
    "                action_val = 0\n",
    "                for prob, next_state, reward, _ in P[state][action]:\n",
    "                    if reward_fn is not None:\n",
    "                        reward = reward_fn(next_state, action)\n",
    "                    action_val += prob*(reward+gamma*V[next_state])\n",
    "                action_vals[action] = action_val\n",
    "            \n",
    "            V[state] = np.max(action_vals)\n",
    "            delta = max((delta, np.abs(old_v-V[state])))\n",
    "            \n",
    "        count += 1\n",
    "\n",
    "    return V\n",
    "\n",
    "def stochastic_policy_from_values(V, P, gamma, num_states, num_actions, reward_fn=None):\n",
    "    \"\"\"\n",
    "    compute a stochastic policy given the value function\n",
    "    this is done by computing the softmax on the q-values to get a probability distribution over actions\n",
    "    \n",
    "    V:          Value Function\n",
    "    P:          State-Action Transition Probabilities\n",
    "    gamma:      discounting factor\n",
    "    num_states: total number of states\n",
    "    reward_fn:  f(state,action)->reward\n",
    "                if None: use reward function of environment\n",
    "    \"\"\"\n",
    "    Q = np.zeros((num_states,num_actions))\n",
    "    policy = np.zeros((num_states,num_actions))\n",
    "    for state in range(num_states):\n",
    "        for action in range(num_actions):\n",
    "            for prob, next_state, reward, done in P[state][action]:\n",
    "                if reward_fn is not None:\n",
    "                    reward = reward_fn(next_state, action)\n",
    "                Q[state][action] += prob*(reward+gamma*V[next_state])\n",
    "        \n",
    "        policy[state] = softmax1D(Q[state])\n",
    "    return policy\n",
    "    \n",
    "\n",
    "def deterministic_policy_from_values(V, P, gamma, num_states, num_actions, reward_fn=None):\n",
    "    \"\"\"\n",
    "    compute a deterministic, greedy policy given the value function\n",
    "    always choses the stochastic action with the highest probability\n",
    "    \n",
    "    V:          Value Function\n",
    "    P:          State-Action Transition Probabilities\n",
    "    gamma:      discounting factor\n",
    "    num_states: total number of states\n",
    "    reward_fn:  f(state,action)->reward\n",
    "                if None: use reward function of environment\n",
    "    \"\"\"\n",
    "    stochastic_policy = stochastic_policy_from_values(V, P, gamma, num_states, num_actions, reward_fn=reward_fn)\n",
    "    return np.argmax(stochastic_policy, axis=1)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing State Visitation Frequencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "State Occupancy Frequencies for the current reward function are required to compute the gradient for Maximizing the Log Likelihood of the trajectories conditioned on the reward weights.  \n",
    "\n",
    "The gradient is defined as follows:\n",
    "\n",
    "$ \\displaystyle L_\\theta = \\tilde{f} - \\sum_\\tau {P(\\tau | \\theta) f_\\tau}$\n",
    "\n",
    "However, it is not feasible to sum over all trajectories. By summing over the substantially smaller set of states the problem can be avoided:\n",
    "\n",
    "$ \\displaystyle L_\\theta = \\tilde{f} - \\sum_s {D_s f_s}$\n",
    "\n",
    "The new Gradient Equation now requires the expected state occupancy frequncies $D_s$.  \n",
    "They can be computed efficiently using a forward-backward algorithm that is similar to Value Iteration.  \n",
    "\n",
    "In the first three steps, the probability distribution over actions, conditioned on the state is computed (stochastic policy) for the current reward function.\n",
    "First, factors for state ocurrences Zs are initialized by assigning one to all terminal states:\n",
    "$Z_{s_{terminal}} = 1$.  \n",
    "The next step is the backward pass:  \n",
    "$ \\displaystyle Z_{a_{i,j}} = \\sum_k P(s_k | s_i, a_{i,j}) e^{r_\\theta(s_i)} Z_{s_k}$  \n",
    "$ \\displaystyle Z_{s_i} = \\sum_{a_{i,j}} Z_{a_{i,j}} + [s_i \\in S_{terminal}]$  \n",
    "Intuitively, the weight $e^{r_\\theta(s_i)}$ makes coming from a higher reward state $s_i$ into $s_k$ more likely compared to lower reward states, as $s_i$ has a higher weight.\n",
    "\n",
    "Due to summing over exponentiated reward values for multiple iterations, overflows can occurr. To mend this problem to a certain degree, the factors are normalized after every step.\n",
    "\n",
    "Finally, one can derive a stochastic policy by normalizing $Z_{a_{i,j}}$ using $Z_{s_i}$ for each state-action pair:  \n",
    "$ \\pi(a_{i,j}|s_i) = \\frac{Z_{a_{i,j}}}{Z_{s_i}}$  \n",
    "\n",
    "As this method does not perform well on FrozenLake, it was replaced by value iteration. Taking the softmax over the q-values of each state produces the stochastic policy.\n",
    "\n",
    "During the last three steps, the state occupancy frequencies $D_{s,t}$ are calculated in a forward pass, making use of the previously calculated policy. \n",
    "$D_{s_i,t=0}=P(s_i = s_{initial})$  \n",
    "$ D_{s_k,t+1} = \\sum_{s_i}\\sum_{a_{i,j}} D_{s_i,t} \\pi(a_{i,j}|s_i) P(s_k | s_i, a_{i,j}) $  \n",
    "This results in the state occupancy frequencies for each timestep, where timestep zero corresponds to the initial state distribution. Summing the frequencies over all timsteps yields the overall state occupancy frequncies.  \n",
    "$\\displaystyle D_{s_i} = \\sum_t D_{s_i,t}$\n",
    "\n",
    "These are normalized by the sum of all vector entries (multiplying with $\\frac{1}{num\\_timesteps})$ to get a vector of length one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_state_visitation_frequencies(env, reward_fn, iterations = 200, use_paper_backward_pass=False):\n",
    "    \"\"\"\n",
    "    compute state visitation frequencies based on the transition probabilities P of environment env and terminal states\n",
    "    \"\"\"\n",
    "    \n",
    "    P = env.P\n",
    "    num_states = env.num_states\n",
    "    num_actions = env.num_actions\n",
    "    terminal_states = env.terminal_states\n",
    "    initial_Ps = env.initial_Ps  \n",
    "\n",
    "    # use backward pass from paper\n",
    "    if use_paper_backward_pass:\n",
    "        \n",
    "        # step 1: initialize factors\n",
    "\n",
    "        # factor Zs initially corresponds to terminal state distribution\n",
    "        Zs_final = np.zeros(num_states)\n",
    "        Zs_final[terminal_states] = 1 \n",
    "        Zs = np.zeros(num_states)\n",
    "        Zs[terminal_states] = 1\n",
    "\n",
    "        # factor Za is initialized by the backward pass from Zs\n",
    "        Za = np.zeros((num_states, num_actions))\n",
    "\n",
    "        # step 2: backward pass\n",
    "        for _ in range(iterations):\n",
    "\n",
    "            # update Za based on state-factors Zs\n",
    "            for state in range(num_states):\n",
    "                for action in range(num_actions):\n",
    "\n",
    "                    Za[state, action] = sum(\n",
    "                            [prob*(np.e**(reward_fn(state,action)))*Zs[next_state]\n",
    "                                for prob, next_state, _, done in P[state][action]]\n",
    "                        )\n",
    "\n",
    "            # normalize factor to prevent overflow\n",
    "            Za /= np.sum(Za)\n",
    "\n",
    "            # update Zs by summing over state-action factors Za\n",
    "            Zs = np.zeros(num_states)\n",
    "            for state in range(num_states):\n",
    "                for action in range(num_actions):\n",
    "                    Zs[state] += Za[state, action]\n",
    "\n",
    "                Zs[state] += Zs_final[state]\n",
    "\n",
    "            # normalize factor to prevent overflow\n",
    "            Zs /= np.sum(Zs)\n",
    "\n",
    "\n",
    "        # step 3\n",
    "        # Local action probability computation\n",
    "        P_sa = np.array([\n",
    "                [Za[state, action]/Zs[state] for action in range(num_actions)]\n",
    "                for state in range(num_states)\n",
    "            ])\n",
    "        P_sa = P_sa/np.sum(P_sa, axis=1)[:,None]\n",
    "\n",
    "        policy = np.argmax(P_sa, axis=1)\n",
    "    \n",
    "    # use value iteration for backward pass\n",
    "    else:\n",
    "        gamma = 0.999\n",
    "        V = value_iteration(env, reward_fn=reward_fn, gamma = gamma, delta_threshold = 0.01)\n",
    "        P_sa = stochastic_policy_from_values(V, P, gamma, num_states, num_actions, reward_fn=reward_fn)\n",
    "        # when using deterministic policy:\n",
    "        # policy = deterministic_policy_from_values(V, P, gamma, num_states, num_actions, reward_fn=reward_fn)\n",
    "\n",
    "    # step 4\n",
    "    # set D at timestep 0 initial state distribution\n",
    "    timesteps = 200\n",
    "    D = np.zeros((timesteps, num_states))\n",
    "    D[0] = initial_Ps\n",
    "        \n",
    "    # step 5: forward pass\n",
    "    # calculate visitation frequences at each timestep\n",
    "    for t in range(timesteps-1):\n",
    "        \n",
    "        for state_i in range(num_states):\n",
    "\n",
    "            for action in range(num_actions):\n",
    "                \n",
    "                for p_sas, state_k, _, done in env.P[state_i][action]:\n",
    "                    \n",
    "                    # from state_i to state_k\n",
    "                    # old paper suggests D[t+1, state_i] += ( D[t, state_k] * P_sa[state_i, action] * p_sas )\n",
    "                    # correct version of paper on bziebarts homepage:\n",
    "                    D[t+1, state_k] += ( D[t, state_i] * P_sa[state_i, action] * p_sas )\n",
    "                    # Version using deterministic policy:\n",
    "                    # D[t+1, state_k] += ( D[t, state_i] * (policy[state_i] == action) * p_sas )\n",
    "                    \n",
    "        \n",
    "    # step 6: marginalize over time\n",
    "    Ds = np.sum(D, axis=0)\n",
    "\n",
    "    # normalize state visitation frequencies\n",
    "    Ds = Ds / np.sum(Ds)\n",
    "\n",
    "    return Ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Expert Feature Expectation\n",
    "\n",
    "Revisiting the computation of the Gradient of the Trajectory Log-Likelihood, one component is still missing:  \n",
    "$ \\displaystyle L_\\theta = \\tilde{f} - \\sum_s {D_s f_s}$  \n",
    "\n",
    "The Expert Empirical Feature Expectations $\\tilde{f}$ are computed by summing over the accumulated features $f_\\tau$ of all trajectories $\\tau$:  \n",
    "$f_\\tau = \\sum_{s_t \\in \\tau} f_{s_t}$  \n",
    "$ \\tilde{f} = E_{P(\\tau)}[f_\\tau] = \\sum_{\\tau} P(\\tau) f_\\tau \\approx \\frac{1}{|\\mathcal{D}|} \\sum_{\\tau \\in \\mathcal{D}} f_\\tau$\n",
    "\n",
    "As not all trajectories have the same length, the length of the longest trajectory is set as the duration of the outer loop (in case of FrozenLake the limits is 200 timesteps). For shorter trajectories, the last state of the trajectory is a terminal state that the agent is staying at for the remaining amount of time. This is in line with the forward pass for state occupancy computation, where the probability mass is shifting to terminal states over the same number of timesteps.    \n",
    "\n",
    "Finally, the feature expectations are normalized over the number of expert trajectories and the number of timesteps to be comparable with the results of the state occupancy calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_expert_feature_expectation(expert_trajectories, feature_fn, feature_len):\n",
    "    \"\"\"\n",
    "    expert_trajectories: [(action_1, reward_1, obs_1),...,(action_n, reward_n, obs_n)]\n",
    "    feature_fn           function(state)->feature\n",
    "    feature_len:         length of the feature vector\n",
    "    \"\"\"\n",
    "    T = len(max(expert_trajectories,key=lambda x: len(x)))\n",
    "    expert_feature_expectations = np.zeros(feature_len)\n",
    "    \n",
    "    for trajectory in expert_trajectories:\n",
    "        trajectory_feature_expectation = np.zeros(feature_len)\n",
    "        for t in range(T):\n",
    "            expert_feature_expectations += feature_fn(trajectory[t if t<len(trajectory) else -1][2])\n",
    "    \n",
    "    return expert_feature_expectations / (len(expert_trajectories)*T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Maximum Entropy IRL Algorithm\n",
    "\n",
    "Having seen the components of the gradient update, the Gradient Ascend update of the reward weights can be assembled.  \n",
    "\n",
    "The following steps are taken to update the reward weights:\n",
    "\n",
    "1. Initialize reward function weights $\\theta$  \n",
    "2. Calculate the Empirical Feature Expectation $\\tilde{f}$ of the Expert Trajectories  \n",
    "Repeat:  \n",
    "3. Calculate a (stochastic) policy $\\pi$ according to the current approximation of the reward function\n",
    "4. Calculate the State Visitation Frequencies $P(s|\\theta,T)$ for $\\pi$\n",
    "5. Update the rewards weights $\\theta$ by Gradient Ascend on the Log Likelihood of the Expert Trajectories:  \n",
    "  $\\theta_{t+1} = \\theta_t +  \\alpha \\frac{1}{|\\mathcal{D}|} (\\tilde{f} - \\sum_{s \\in S}  P(s | \\theta, T) f_s)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def max_ent_irl(env, reward_weights, reward_fn, feature_fn, expert_demonstrations, num_gradient_steps = 100):\n",
    "    \"\"\"\n",
    "    env:    environment implementing the OpenAI gym interface\n",
    "            needs to have:\n",
    "                - fixed set of discrete states\n",
    "                - fixed set of discrete actions\n",
    "                - env.P: fixed discrete transition probability distribution over states and actions\n",
    "    \n",
    "    reward_weights:        weights used in the reward function, len(reward_weights) = len(feature)\n",
    "    reward_fn:             function(state, action) -> reward\n",
    "    feature_fn:             \n",
    "    expert_demonstrations: [(action_1, reward_1, obs_1),...,(action_n, reward_n, obs_n)]\n",
    "\n",
    "    return: reward_weights of linear reward function\n",
    "    \"\"\"\n",
    "    \n",
    "    # environment properties\n",
    "    num_states = env.num_states\n",
    "    num_actions = env.num_actions\n",
    "    terminal_states = env.terminal_states\n",
    "    initial_Ps = env.initial_Ps\n",
    "    \n",
    "    # verbose\n",
    "    plot_progress = False\n",
    "    track_progress = True\n",
    "    gradient_norms = []\n",
    "    \n",
    "    feature_len = len(reward_weights)\n",
    "\n",
    "    expert_feature_expectation = calculate_expert_feature_expectation(\n",
    "                                    expert_demonstrations, feature_fn, feature_len)\n",
    "\n",
    "    \n",
    "    def step_size(step):\n",
    "        return 1\n",
    "    \n",
    "    def plot_progress_fn(step, gradient, expert_feature_expectation, current_feature_expectation):\n",
    "        print(\"expert feature expectation\")\n",
    "        plot_reward(expert_feature_expectation, env, vrange=(0,0.6))\n",
    "        plt.show()\n",
    "        print(\"learner feature expectation\")\n",
    "        plot_reward(current_feature_expectation, env, vrange=(0,0.6))\n",
    "        plt.show()\n",
    "        irl_reward = [reward_fn(state,0) for state in range(env.num_states)]\n",
    "        print(\"reward\")\n",
    "        plot_reward(irl_reward, env)\n",
    "        plt.show()\n",
    "    \n",
    "    for step in range(num_gradient_steps):\n",
    "        \n",
    "        # computing the expert feature expectations\n",
    "        D = compute_state_visitation_frequencies(env, reward_fn)\n",
    "        \n",
    "        # computing the current feature expectations\n",
    "        current_feature_expectation = sum([D[state]*feature_fn(state) for state in range(num_states)])\n",
    "        \n",
    "        # the gradient is the difference of expert feature expectations and current feature expectations\n",
    "        gradient = expert_feature_expectation - current_feature_expectation\n",
    "        \n",
    "        # gradient ascend step\n",
    "        reward_weights[:] += step_size(step) * gradient\n",
    "        \n",
    "        if track_progress:\n",
    "            clear_output(wait=True)\n",
    "            print(f\"Progress: {step/num_gradient_steps : 6.5f}, Gradient L1 Norm: {np.sum(np.abs(gradient)) : 6.5f}\")\n",
    "            gradient_norms.append(np.sum(np.abs(gradient)))\n",
    "            \n",
    "        if plot_progress:\n",
    "            plot_progress_fn(step, gradient, expert_feature_expectation, current_feature_expectation)\n",
    "            \n",
    "    if track_progress:\n",
    "        clear_output(wait=True)\n",
    "        plot_progress_fn(step, gradient, expert_feature_expectation, current_feature_expectation)\n",
    "        print(\"L1 norm of the gradient (difference of expert and learner feature expectations)\")\n",
    "        plt.plot(gradient_norms)\n",
    "        plt.show()\n",
    "    \n",
    "    return reward_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Rollouts using (Optimal) Policy\n",
    "\n",
    "In order to generate the expert trajectories, a policy is calculated using Value Iteration.\n",
    "The expert policy is calculated from the True Reward Function, making it optimal with respect to the true reward function and the given discounting factor $\\gamma=0.999$.  \n",
    "After computing the expert policy, it can be rolled out on the environment, producing the dataset of expert trajectories.  \n",
    "The dataset $\\mathcal{D}$ is a list of expert trajectories of the following form:  \n",
    "$((-, -, s_0), (a_1, r_1, s_1), \\ldots, (a_n, r_n, s_n)) \\in \\mathcal{D}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_rollouts(env, policy, num_rollouts=10, render=False, render_fps=2):\n",
    "    \"\"\"\n",
    "    generate rollouts on env with specified policy \n",
    "    \n",
    "    env:           gym environment\n",
    "    policy:        policy of the agent, list containing actions for each state\n",
    "    num_rollouts:  number of rollouts to generate\n",
    "    render:        if set True, animate the rollout\n",
    "    render_fps:    delay between steps is 1/render_fps\n",
    "    \n",
    "    return:        rollouts following the specified policy of follwing format:\n",
    "                   [(action_1, reward_1, obs_1),...,(action_n, reward_n, obs_n)]\n",
    "    \"\"\"\n",
    "    expert_demonstrations = []\n",
    "    for _ in range(num_rollouts):\n",
    "        done = False\n",
    "        obs = env.reset()\n",
    "        trajectory = [(None, None, obs)]\n",
    "        while not done:\n",
    "            action = policy[obs]\n",
    "            obs, reward, done, info = env.step(action)\n",
    "            if render:\n",
    "                clear_output(wait=True)\n",
    "                env.render()\n",
    "                time.sleep(1/render_fps)\n",
    "                \n",
    "            trajectory.append((action, reward, obs))\n",
    "        expert_demonstrations.append(trajectory)\n",
    "    \n",
    "    return expert_demonstrations\n",
    "\n",
    "def get_expert_demos_valiter(env):\n",
    "    \"\"\"\n",
    "        finds the optimal policy for env using value iteration\n",
    "        \n",
    "        env:    gym environment\n",
    "        \n",
    "        return: rollouts using the optimal policy of following format:\n",
    "                [(action_1, reward_1, obs_1),...,(action_n, reward_n, obs_n)]\n",
    "    \"\"\"\n",
    "    num_actions = env.num_actions\n",
    "    num_states = env.num_states\n",
    "    gamma = 0.999\n",
    "    V = value_iteration(env,gamma=gamma)\n",
    "    policy = deterministic_policy_from_values(V, env.P, gamma, num_states, num_actions)\n",
    "    return generate_rollouts(env, policy, num_rollouts=100), policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the FrozenLake environment\n",
    "\n",
    "The FrozenLake8x8-v0 environment was chosen as a challenging stochastic environment.  \n",
    "Additional helper variables are added to simplify the code and avoid redefinition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run maximum entropy inverse reinforcement learning on FrozenLake8x8\n",
    "env = gym.make(\"FrozenLake8x8-v0\")\n",
    "env.num_states = 64\n",
    "env.num_actions = 4\n",
    "env.terminal_states = [19, 29, 35, 41, 42, 46, 49, 52, 54, 59, 63]\n",
    "env.initial_Ps = np.zeros(env.num_states)\n",
    "env.initial_Ps[0] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining Feature Functions\n",
    "\n",
    "Feature functions are used to map integer-valued state to vector-valued features.\n",
    "Two different feature functions are defined:  \n",
    "\n",
    "The ```onehot_feature_fn``` encodes each state as onehot vector with the size of the features.  \n",
    "Example:  \n",
    "\n",
    "    num_states = 5\n",
    "    state = 3\n",
    "    onehot_feature_fn(state, num_states) == [0,0,0,1,0]\n",
    "    \n",
    "The ```neighbor_feature_fn``` encodes the currently occupied state and the four neighboring states as a onehot-representation of their types.\n",
    "Example assuming only 3 types H, F and G:  \n",
    "\n",
    "    neighbors:\n",
    "     \n",
    "        F\n",
    "     F *H* H\n",
    "        G\n",
    "    \n",
    "    H corresponds to [1,0,0]\n",
    "    G corresponds to [0,1,0]\n",
    "    F corresponds to [0,0,1]\n",
    "    \n",
    "                                    center, H  left, F  right, H  left, F  bottom, G\n",
    "    neighbor_feature_fn(state) = [  1,0,0,     0,0,1,   1,0,0,    0,0,1,   0,1,0      ]\n",
    "    \n",
    "However, this feature function produced low agent success rates around $30\\%$ hence it is not evaluated here.\n",
    "The following evaluation is entirely based on the ```onehot_feature_fn```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_len = 64\n",
    "# for neighbor feature function:\n",
    "# feature_len = 25\n",
    "\n",
    "reward_weights = np.zeros(feature_len)\n",
    "    \n",
    "def onehot_feature_fn(state):\n",
    "    return to_onehot(state, feature_len)\n",
    "\n",
    "def neighbor_feature_fn(state):\n",
    "    row = int(state % 8)\n",
    "    col = int(state // 8)\n",
    "    \n",
    "    mapping = {b\"S\": 0, b\"F\": 1, b\"H\":2, b\"G\":3}\n",
    "    num_types = 5\n",
    "    \n",
    "    center = mapping[env.unwrapped.desc[row,col]]\n",
    "    if col < 1:\n",
    "        left = 4\n",
    "    else:\n",
    "        left = mapping[env.unwrapped.desc[row,col-1]]\n",
    "    \n",
    "    if col > 6:\n",
    "        right = 4\n",
    "    else:\n",
    "        right = mapping[env.unwrapped.desc[row,col+1]]\n",
    "    \n",
    "    if row < 1:\n",
    "        top = 4\n",
    "    else:\n",
    "        top = mapping[env.unwrapped.desc[row-1,col]]\n",
    "        \n",
    "    if row > 6:\n",
    "        bottom = 4\n",
    "    else:\n",
    "        bottom = mapping[env.unwrapped.desc[row+1,col]]\n",
    "    \n",
    "    feature_vec = np.concatenate((to_onehot(center, num_types), \n",
    "                                  to_onehot(left, num_types),\n",
    "                                  to_onehot(right, num_types),\n",
    "                                  to_onehot(top, num_types),\n",
    "                                  to_onehot(bottom, num_types)))\n",
    "    \n",
    "    return feature_vec\n",
    "\n",
    "                          \n",
    "feature_fn = onehot_feature_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining the Linear Reward Function\n",
    "The reward function is defined as linear combination of reward weights and state-features:  \n",
    "$r(s) = w^T f_s$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reward_fn(state,action):\n",
    "        return reward_weights.T@feature_fn(state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Expert Demonstrations\n",
    "This cell gathers the expert trajectories and returns them, alongside the expert policy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "expert_demos, expert_policy = get_expert_demos_valiter(env)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning the Reward Weights using Maximum Entropy IRL\n",
    "The following cell executes the maximum entropy algorithm implementation to fit the reward weights.  \n",
    "\n",
    "Warning:  \n",
    "This takes quite some time to converge. \n",
    "The L1 Norm of the gradient is decreased to less than 0.2 after 300 gradient steps.\n",
    "This process took 12 minutes on an Intel i5 2400 Desktop Processor from 2011.\n",
    "\n",
    "After the computation, expert and learner feature expectations are plotted, which match quite well.\n",
    "The largest difference can be observed at the goal state. Adding more optimization steps decreases the difference.  \n",
    "The decrease in the L1 norm of the difference in expert and learner feature expectations is displayed in the plot at the bottom of the output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "expert feature expectation\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAdbklEQVR4nO2deZwcZZ3/33X19Mx098yk58zkmEnAcAgi9yUhgLqKgiKHInitrroei7DqvjxWXDx+ymWUw58g1yqKsO4ix4pyhCMk3MgZIIQJCclkzkxPd093V1c9+0d16GPuGaZj9Xzff03Sz/S7n+r5dHVXVz0fTSmFIAh//+i7+wEIgjA1JKyC4BMkrILgEySsguATJKyC4BMkrILgE8zpDK7TDdWsW3P1WIop4zdKZf/yqkK/LSvrtMq8DWsjRlk8W0fSDGSy2li3TSuszbrFz0JL35pHNQmuWxZN2V27w1custnyucq9DY88KlQWz4lrN4x7m7wNFgSfIGEVBJ8gYRUEnyBhFQSfIGEVBJ8gYRUEnyBhFQSfMK3vWUt5wk7wq1QPLvAeq47TgguKbreVyyUj3Wx00oQ1g2/WtNGiWzxlJ7gu1UcWhYnGZ6qbeIdZM7Erm+DqVA9OznVq1WjXpSnPFdEMvl6dc2UT3JDOuz5VNbnryWyCqzPevN5t1vGRwGjXz9LdvOp68/rXKs8VUw4/TW1jo5viODPCP1W1TLoNn3QSXGN7rhOMOk6xRrtW291sctOEMTgv0EazbjGsHC7MeK5VRoTPBSZ3ldv3tJvgWsdzHa/X8SFjtOsyp5tNytuO5xhtNGue6xJnGxtVimP1CP9oTMGlElzveq7jtDpO1ke7LlfdvKbShDD4Fz3vutTdxqukWKlF+Iw+uWtN7xDnv7gVR8FHF0X50vLWotvTjsvXnuni2dgIDZbB5Qd0srimCttVfOO5zTw3lMRRilPao3y55HcnYsZ7Vkcprkz18P3adq4IdXC/HeN1J1005i+ZGLWawVXhTk4O1HNdqheAiG7w77XtXB7u4Gs1rVyc3D6p6/+nevheTTuX13bwQHa06692jJBm8KtQJycF6rk+nXNpBt+pbucXtR2cE2zl0tQUXJke/j3Yzi+qO3jQibHFLXFlPdcvazo5yarnhoznCqBxZqCRTwWaJt+AOddVdg/fCbSzumps191OjBAGVwQ7+aBZzw1Zz2Wh8TGzkU9aU3OV2+cqxa+dHr5ltnOp2cFaN8ZWVey61/X+Pn5hdXKiXs9vnbzrDL2Rs/Wpu65xe/g3vZ2L9Q7WqtGu+5Q3r9VGJydq9dyo8q7T9UbO0qb+nH3n+S1cf/Ae3POuvfnT9kFeHh4pGnPT1n7qLJMHV+7LZzua+fFLbwBwR/cgGVfx13ftwx1H7c2NW/rYkkyPpRmTGYf1ZSdFm27RqgewNI1jrAjr7UTRmPXZOMdbEQCOtsL8LZtEKcVyI0hU93bqS/UAGRS2Gv+UlFfcYte7zAiPZItdj2TjHJdzHWWG+Zsz2rVED5BR03MdbYx2PerEWWV6riONMM/kXEFNZx+jGosxzxYbxUY3RZtW7HrUKXY95sRZZXiuI4wwzxa49p6Gq9y+jSpFq2bRogUwNY0j9QiPucWux1WcYzXPdbgW5jmVd+2lVxPQpugiRSsFLi3C42q065ic6zAtzPOFLm3q83p6Z4KO2iqW1lQR0HU+2NbAX3qGisb8pWcnp7Z7e/b3tzawtn8YpRQakMw6ZF1FynGxNI2wOfXTGGcc1n6VpUnLv4tu1E36lV08xs3SlAuKoWnUaAaxkqCszcZZrgextPEfSr+bpVGfxKWyNGp5Vy0GwyWuh7NxlhsTuwYK7gcgqpkMlLgG3GJXjWYwzPTPf+snS3QSV7/KvvliMxtXuX0DZIlS4qJkO6r84zE0jRpm4SqY1wLGcBU8HkPTqJ6hqztlszAYePPfbUGLHSl73DGm7gVy0HZ4f2sDNabBwfc+y+FrnuOfOluoD0z9k+huPcC02UlzXaqPL1c3z7nrdSfN9ek+/jk49y5BGIunhxIYwGPH7cfalftyVdcONpfjbXBUM+lV+TO3+9wsUa34ipyobtLremMcpUgqh0hur9bn2vwwuY1zq1tpMwJMRFQ36XMncWkmfSrvSuAQLnD9aGQb5wRbadMndi0ouB/w9jQLSlwL9GJXUjmEZ7Apo5j0T+KKaib97uxd5fYtwKSfEhcl21HLPx5HKZLMwlUwrwHGcBU8HkcpRmboag1abEtl3vz39pRNS9Aad0zWVQxnHRosg1u3DbCyKYKlazRWWRxcH+KZoeSU3TMO69uMINscm27XxlaKB+wYh1m1RWMOM0PcY8cAeMgeZn+zBk3TiCuH8xNv8KlgI/uY1ZO69tSDbHPzrgezMQ4zi12HmiHuzbnWZofZ38i7/mPkDT5RNXXXdtdmR871kBPj0FKXEeK+rOd62Blmv5xruuyhB9muil2HGMWuQ4wQ9zmea90sXOX2Ldc8V4+yySrFw26Mg/Vi10FaiDXKc61Xw+yrzdBFkG4KXCrGQdpo1wM51yOzcL2jrpbXEmleT6bJuC63bR/k3c11RWPe3VzPLW8MAHBn9yBHRsNomsbCYICH+4cB77PrkzsT7FFbNWW3Np2lSPc0g6rwErnH7DhXpXq9rzisCGcEo/wm1ceeRpDDrBAZ5XJx0vsaIKTpfLOmjVY9wO9T/dycHmBhwV7ugtp26gs+l5ZeAvV4Ns7VOdcJVoTTq6L8Nt3HHkaQw0zPdUmqm01OmrCm8/Vqz3VTup9bMsWu71dP7rom04sDnGBGOC0Q5cZMH3voQQ7NuX6Wzn29oemcV+W5AD6X3MSIcsmiqNUMzg+2s1gvfkIKfU84ca6xvXkdb0Q41YryO7uP5XqQQw3PtTqz6ysHnXMDedfnU3lXDQbfqxrtKmUufaWXyD3pxrne8Vyr9AinGFFucvpYrgU5WPdclzk5l6ZzjtFGi+a5vmRvIkluO2LwHbOdRVreVfqcPaXiXO/mXFqED+tR/uD2sUwLcrDmuS53u+nCm9dX9bzry84mRgpc39KLXQBHvid/idy9PUN8/8WtOEpxxqIoX9mjjYtf3sZ+dTW8p6WelONyzjNdPB8bod4yuOyATpbWVJHIOpz37GZeiadQCk5fFOULy4q/Kjpx7QaeGUqM+Soyq7DOJXI9q/+o6OtZ31O+61nHC6ucwSQIPkHCKgg+QcIqCD5BwioIPkHCKgg+QcIqCD5BwioIPkHCKgg+YVoXn7sOxONz9VCKCUx8Cu9bil7BL1mVenJJuU+KsOP25IPeApQ7/klKFfxnKgiVhYRVEHyChFUQfIKEVRB8goRVEHyChFUQfIKEVRB8goRVEHyChFUQfMKs6jPKWVlQztqHSq7PeMpNcG22B1fB8UYdHzZH+36R3bVulsG5Vv45u8jexqu5SovPWn9flRZ/UwluwHOtoo6TtNGuK+nmNTzXV2mjKbey461qgDUMoQOfoJl3lCy2Vsr9A8NcsGkbjoIzWhv4wuLi5W3Trsu/vrSV5+JefcbP91rComCAW3sGuWpr35vjNiRS/Omde7BPaPKF/GAWe9ZyVxaUq/ah0uszrrZ7+LbVzqWBDh5yR/vucWPUYnBZVScfMOr5TYHvo2YjZ5t/f5UWrlJcSw/foJ0L6eBhRrvW4M3rUq2T91HP7/BcW1WadcT4KUv5Jou4lh7cCdYlc5Ti/Fe3cc2+ndx10J7c1jvEK4lU0ZibuwepMw3uO2QFn17YyE9e6wbg5OYGbj9wT24/cE8uXrGYxcHAlIMKswhrOSsLyln7UNH1GQWVFpamcdQYlRaPOXGO3eXTwzzrFvj0agJTnVs5/z5I0VLgOoIIT1DiIs67yLkI8xye6wkSHEEES9Np1ixasNhIaiwNAH8bTrI0GGBJdYCArvOBpjruHogVjbm7P8YpLfUAvK+pjnU745QuTHhb705ObCpewnQyZhzWclYWlLX2oYLrM6Y0N0rm5oNKi8GSqo6xXIMlrl3zGsAurvnAZJDxl2nckc7SVpVf1Ls1YLEjXVKfkbFpq8rVZ+T6bAazTtGYO3qH+GBT/bTmKQeYBKHMPB1LEtQ1VtQGp/V7Mw5rOSsLylr7UMH1GVOaGyVz80GlRUNJVcdYroYS1655LcAqrvkgS8MEx11bqky2F+xJuzM2LVUl9RkBi+3pXH2GytVnFLTF3d67c9p7VZhFWMtZWVDO2oeKrs/IVVrsUJ5vrRvjkJJKi4ONEGt2+dxh3q7//VdalLrWEeMgSlyEeJCci2H2xXMdRC3riGErlx5l043NHoy/x9s/XENXKs2WVIaM63J77xDHL4gUjTk+GuGPO3YC8L+9QxxRH3pzXq5S3Nk3xAdmENZprci/XAuqHxn5FfnnsrKg9OLzuax9KL34vJLqM0ov0n7SiXOt04ur4DgjwkfMKL/PepUWh+R8P89205WrPPmalX/OvpjOP2c1GHzXmrg+Yy7/Psaqz/hPPNexRPiQFuVm1ccyghyUc11BN5tJU4vOV8i7/kf1s4YYBnA2zRwwxlc3hx2dd983EOMHm7bjKji1pYEvLWnm0q4d7Beu5oRohLTrct5LW3g+nqLeNFi91xKWVHuu9TvjXNjVzX8dsMcoB8DJT23k2eHk7OszSsM6l1TyShGVuqJCJddnFIZ1LpkorHKASRB8goRVEHyChFUQfIKEVRB8goRVEHyChFUQfIKEVRB8goRVEHzCtC4+V4DjTDrsLWFkpDweQZgKmXimLB7ljH+2h+xZBcEnSFgFwSdIWAXBJ0hYBcEnSFgFwSdIWAXBJ0hYBcEnSFgFwSdIWAXBJ8yqPuNvJPjPXGXBsdRxEiWVBXiVBV25yoKv0EZTbtW5Wxng/oLKgv2ZuLJAXLN3VfLcyul6MJbgR1t3eGswRev4XGu06PaM6/LNzd28kPTWYLqkYyHtuRUQXxpJ8b3XdxB3XXTg5hVLqZriukIzr89AcV2usuCndLCOGFsZu7LgEkoqC0iznhg/YSnf2FVZwPhrQYlr9q5Knls5XY5SXLBlB79avojb9u7kjsFhNo4Uu27pH6LO0Llr32V8ormBi7Z5rqxSfKNrO+cvbuH2vTu5fs8lmNNYzXHGYX01V1nQTAATjcPHqCx4gjjH5CoLDiXM8yRReJUFhxPBQqcZr7Lg1QkqC8Q1e1clz62crmeSKZZUWSyuChDQNd7fEObeoXjRmHuH4pwc9aox3lsfZv2wV9WxNpZgRXUVe9V4S502mAZGOcI6MEZlweAYlQULdlUW4FUWxHEZLKks8OoOxl8aT1yzd1Xy3Mrp6slkaQ3kF/VuCZjssIvH77CztFnefZqaRtjQ2ek4dKUzgMZnN27hlA1dXL2jf1zPWMzqM6sgCFPHUfBkYoSbVywhqOt8+pUt7FsT5Ijw5McaYLb1GSWVBQ1jVBbsepVy8CoLQug0lFQWDBS86olrblyVPLdyupoDJt2Z/F57RyZLi1U8vsUy2Z7b22aVYthxqTcMWgImB4eqaTBNqnWdY+pqeSFZ/Hl3ImYc1mW7KguwyaJYP0ZlwYGEeCBXWfDorsoCvMqC9cSwcenBqyxYPkFlgbhm76rkuZXTtV9NkM1pm63pDBlXcefgMKvqQkVjVtWFuLV/CIC7dg5zeNir6jg6XMvLI2lGXJesUjw2PMLy4NRXs5/WivzLtKD6AfkV+Z8mX1mwkggfIsot9NFJkIMIkckdLi+sLGgmV1lAP/fnKgvOopkDJjlcLq7Zuyp5bnPtOvCd+QNB9w/F+fEbXnv8KdE6vtAa5efb+3h7TZDj6kKkXZdvbt7Oi8k0dabBxR1tLM5VQP5pYIhf7RhAA46J1PL19uLW9FM3dPFcMjX7+ozSsArCfKEwrHPJRGGVM5gEwSdIWAXBJ0hYBcEnSFgFwSdIWAXBJ0hYBcEnSFgFwSdIWAXBJ0hYBcEnyFU3gjAFRpJTP9NvNrjjV93InlUQ/IKEVRB8goRVEHyChFUQfIKEVRB8goRVEHyChFUQfIKEVRB8gtRnzCNXJc+tnK61iQQX7ejBAT5cV8eno8WujOvy3e5uXkylqTcM/t/CNhZaFttsm4+81sXSgLce037BIN9ubZnQVYjUZ8wTVyXPrdz1GT/Z0cMvFrXzX50d/Hk4xqZ0set/hmJEdIM/Levk4w31rO7tffO2RZbF7zuW8vuOpdMKKkh9xrxxVfLcyul6LpVikWWxKBDA0jTeG46wJl7sWhOP84E6z3V8OMxjSa8+Y7ZIfcY8cVXy3Mrp6s1maS1Y1LvZNOnJ2qPHmPn6jJBusNPxTvp9w7b5WNdmPvv6Fp5MJsf1jIWcyC8IZaLRMLhz+TLqDYMXUinOe2MbN3csJWQYU/p9qc+YJ65Knls5XU2mSXdBEVVPNkuzaY0ek83XZ8Rdh3pDJ6Dr1OeCuU8wyCLL4nW7eK88EVKfMU9clTy3crr2DQbZYtu8kbGxleKu4RgrQ8WulaEQtw95rnuGhzmkxqvPGMxmcXKfXbdmMrxuZ2i3rFGO8ZD6jHnkquS5zbVr7xX5nx+Kx7mox3OdVBfhs9EoV/b1sU8wyMqQV5/x3e3dbEinqTN0ftzWxqJAgHuGh7myrx9T09CBzzdGWRkq7sn5eNdmXkhJfYYgzJjCsM4lE4VVzmASBJ8gYRUEnyBhFQSfIGEVBJ8gYRUEnyBhFQSfIGEVBJ8gYRUEnyBhFQSfIFfdCMIUSI6Ux+NOcEKh7FkFwSdIWAXBJ0hYBcEnSFgFwSdIWAXBJ0hYBcEnSFgFwSdIWAXBJ0hYBcEnSNfNPHJV8tzK6Vo/kmD1oOf6QG0dZ9eVdN0olx/0d/NSJk1EN/iPxjbaCpYr7c7anL29i0/XRTkzsoCpIl0388RVyXMrd9fNJYM9XNTczm/aOrg7GeM1u9h1ezxGWDe4aWEnZ4TruXJnb9Htlw32clhw8hfWUqTrZp64Knlu5XS9mEmxyLRoN72umxNqIjyULHY9NBLnfbWe69iaME+k8l03DyTjtJkWnVZgXMd4SNfNPHFV8tzK2nXjZGk28uObTJNexx53jKlp1OoGQ65L0nX5bWyAT9dFx73/iZADTIJQJq4Z6uf0cAM1+sxiN+MDTNPpF4lila3LRFzjU6lzK6eryTDpcfLje7NZmgxrzDHNpkVWKRKuQ52u80ImxZrkMFfu7CXuumgaVGkaHwk3jOsrRLpu5omrkudWTtdeAa/rZlvW67q5OxnjqOpi11HVIf434bnWJIc5MOh13VzRsphb2pdxS/syTgvXc3ZkwZSDCtJ1M69clTy3uXYtXZL/ed1InNWDnuvE2gifrIty9c4+9goEObomRFq5XNDXzSt2moiuc35jG+1m8QGlX+/so1rXR31184/dm9mQlq4bQZgxhWGdSyYKqxxgEgSfIGEVBJ8gYRUEnyBhFQSfIGEVBJ8gYRUEnyBhFQSfIGEVBJ8g9RmCMAWSyfJ4XHf822TPKgg+QcIqCD5BwioIPkHCKgg+QcIqCD5BwioIPkHCKgg+QcIqCD5BwioIPkHqM+aRq5LnVk7XY5kEVyZ6cBX8Q7COj9aMrs+4cLibV7JpwrrBt8NttBoWG+wRfhbvyY1SnFUT5eiq8ISuQqQ+Y564Knlu5a7PuCzeww8j7VzV0MGadIzN2WLXn1MxQrrBdQs6OaW6nl8nPFeHWcXl9Uv4ZcNSfli3iNXxHpxprIEm9RnzxFXJcyun66VsioWGRZvh1WesrIrwcKbYtS4T591VnuuYQJinbK8+I6jpGJq3FlpGKcZcFW0CpD5jnrgqeW7ldPW5WZr0gvoM3aTftccdY2gatZpBTHln6L9oj/C5wS4+P9jFV0PNb4Z3KshVN4JQRva2qrmqoYPXs2kujHdzaKCWgDa1feaM96zTqSwAylaPIK7xqdS5ldPVqJv0ugX1GW6WqG6NO8ZRioRyiJQEcolZRVDT6cpmxnWVIvUZ88RVyXMrp2uFGeQNx2a749Vn3J+OcUSg2HVEIMRf057rgcwwB1hefcZ2x37zgNIOx2aLk6GlpCdnIqQ+Yx65Knluc+1qasz//GgmzpVxz/XeYIQza6Jcn+jjbWaQI6pCZJTLT4a7eTWbJqzrfCvcRpsR4O5UjJtGBjDQ0IGP10Q5qipU5PnSzs28bEt9hiDMmMKwziUThVXOYBIEnyBhFQSfIGEVBJ8gYRUEnyBhFQSfIGEVBJ8gYRUEnyBhFQSfICfyC8IUiCcmH/NW4Drj3yZ7VkHwCRJWQfAJElZB8AkSVkHwCRJWQfAJElZB8AkSVkHwCRJWQfAJElZB8AlSnzGPXJU8t3K6nnQSXGN7rhOMOk6xSlzKZbXdzSY3TRiD8wJtNOsWw8rhwsw2NropVhkRPhdomdBTitRnzBNXJc+t3PUZV9k9fCfQzuqqDh50Ymxxi113OzFCGFwR7OSDZj03ZD2XhcbHzEY+aTWNe/8TIfUZ88RVyXMrp2ujm6JNs2jVvfqMo40IjzrFrsecOKsMz3WEEeZZJ1+fsbdRjTXt4gwPqc+YJ65Knls5Xf1kiWr58VHNZEAVu/pVlmhBfUaNZjCMO+59ThU5wCQIPkHqM+aJq5LnVk5XFJN+lR/fr7Is0IpdUc2kv6A+I6kcwm/BflHqM+aJq5LnVk7XHnqQ7cpmh+vVZzzkxDjEKHYdYoS4z/Fc65xh9jO8+ozZIvUZ88hVyXOba1d1df7nJ5w419ie63gjwqlWlN/ZfSzXgxxqePUZqzPdvKbShNA5N9BGq+65Pp/axIhyyaKoweB7Ve0s1qvevO+vpzaz0ZX6DEGYMYVhnUsmCqscYBIEnyBhFQSfIGEVBJ8gYRUEnyBhFQSfIGEVBJ8gYRUEnyBhFQSfIGEVBJ8gXTe7AcMon8uZoDtFmDp/vv1dZfEMfXFg3NtkzyoIPkHCKgg+QcIqCD5BwioIPkHCKgg+QcIqCD5BwioIPkHCKgg+QeozdrPraZXgetdzHafVcbI+uorhcrVrPR+Df9HbaNa8KoZL3W28SoqVWoTP6FOrYqjU7VhO15ZHt/Lw5etRrste71/BAR97x5jjNj3wGnd//14+fMVJNK1ows263H/Rg/Rt7Ec5Lnu+e0/eeebYvzsWUp+xO11KcY3bw7/p7Vysd7BWxdiqil33Ka+KYbXRyYlaPTeqfBXD6XojZ2lTr2Ko2O1YTpfj8tDPH+Z9P34Pp13zETbeu4nBrsFR4zLJDM/98Xma984/P5vufw3Hdjjt6lM45coP8eLtGxjuHh7XVYrUZ+xG10ZStGLRogUwNY0jtQiPq2LX4yrOMZrnOkwL87zKVzHspU2viqFSt2M5Xb0beqlrjxBZGMGwDJavWkbXw6+PGvf4tU9ywEf3xwgUnFuqQTaVxXVcsukshqlj1QTGdZUi9Rm726WVjrdHjyFfxVDNzKsYKno7lsmV6EtS25R/m1zbVEOir/iFoe/lPuK9CZYcvqTo/5cd04kZNPnNab/jxjNvYv/T9yMYqWKqyIn8gvAWolzFul8+wrHfOGbUbT0betENnbP+8DHSw2n+dM4dtB+4kMjCyJTuW+ozdrdLlY63Ro8hX8UwwsyrGCp6O5bJVdtYQ6I3vydN9Capbczvae2kzcBrg9x27p3ceOZN9LzQy13fvZvel3rZeM+rLDqkHd3UqW6opuXtzfS+3DeuqxSpz9iNruW7XMomqxQPqxgHacWug7QQDyjP9YgaZl9t5lUMlbody+lq2quJoTdixLYP49gOr963iaVH5t/uBkIBPvnfZ3HmjWdw5o1n0LxPE++94ASaVjQRaq5l21PbAbBHbHpe6KV+cf24rlKkPmM3uAqvZ31Kxbne9VyrtAgf1qP8we1jmRbkYM2rYrjc3fWVg85X9TZaNM/1ZWcTI3hVDLUYfEtvZ5FW/Bmo9HrWStqO5XTdf0/+etbXH9nCusvX47qKFe97Gwd+/AAev/YJGlc00nFkcWPFbefeweGfP5SmFU3YIzZrfvoAOzfvRClY8Q978o4z9i8a/8cv3krvS71Sn/H3glx87j8KwzqXTBRWOYNJEHyChFUQfIKEVRB8goRVEHyChFUQfIKEVRB8goRVEHyChFUQfIKEVRB8wrTOYNI0rRfYPHcPRxDmPUuVUmOuKDCtsAqCsPuQt8GC4BMkrILgEySsguATJKyC4BMkrILgEySsguATJKyC4BMkrILgEySsguAT/g/svxXEJMaWsAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "learner feature expectation\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAfZElEQVR4nO2deXxcZb3/3+ecOZNJMjNZJk0yTWnTBdmXa1soa9nUqyKCRZSrKC6IytWruF0UBdefClpZFBFoBRUU8XUV0StXkLK1yFJAytqFrtmTkslkMts5z++PM2lmJnvSnPZMv++/CPN03vPMOZ+cyZlzno+mlEIQhP0ffV+/AEEQJoaEVRA8goRVEDyChFUQPIKEVRA8goRVEDyCbzKDq3RD1evmTL2WQkr4GyVXp+airIQ3GZUhd45rOwfS9KSz2kiPTSqs9brJ9aF5e+dVjUPWckWzT7BtcXmNpcsqXfGc/cQroz4mH4MFwSNIWAXBI0hYBcEjSFgFwSNIWAXBI0hYBcEjSFgFwSNM6nvWYp7O9HPzQAc28DZ/FRcEagsezyibaxNtbLJShDSDKyqiNBgm6zP9/DLZRUYpTE3jo4FZHGtWjOl6JtvPrckOLOCtZhXnlw13rUw6rrBm8KXyKA26ybPZfu5IdZFF4UPj4rJZHOPbf1zrs/3cmnbew7f4qljhH+76SaqNzbbzHn6xzHE9Z/VzR7qLrFL4NI2L/bM42hjbBbDe6mdVxvGdZVTxHnO477pMG1vsFCEMvuCPUq+b9CmLa9ItbLKTnG6EucTfMK7rWbuf1dkObAVnGlWc5xvuuiHruIKaweVmlHrNcV2baWGzSnKaHubj5v7lWtMV45uv7MRSivfPifDp+Y0Fj6dsm8tf2MYLsQQ1po8bj2nmoPIyMrbiKy9uZ0NfgqxSrIjWctmCxlEsw5nykdVSip8NdPCtyiZ+Hmrm4XSM7VaqYMz96RhBzeC28HzOK6tmVbITgCrN4KrKJm4KN3N5RSM/SrSO67o52cFVFU38tLKZR7LDXX/POK5fBOdzjr+a21OOK6wZXFnexA2VzXwu0MjK5H7mSnfwjUATN5Q386gVY4dd5Mo6rp9XzOccs5o70jkXBleWNXF9RTP/VdbIT1JjuwZ9t2Q6uNLfxHVlI/sesGIEMfhZYD7v8lVzR9bxmWhc6Kvjw+ascT2DrlszHXzNbGKlv5nH7OGuB+0YlRjcWDafs41qfp3ner+vjot8+6fr6y/v4PY3L+SBkw7j3tbdvBYfKBjzu53dVJkGj5xyBB+bV8/3X2sB4C/tu0krm/878TD+suxQ7tzZzY6B1EiaEZlyWF+zkszWTaKGH1PTONUfZl2mv2DME5k4Z/nDAJxshng+m0ApxUJfgIjuHNTn6X5SKDJq9MtfNtpJorpJo+64TvGF+We20PXPbJwzTMd1ki/E81bOZQy55up+0mr/dZ1sDHc9acU53ee4TjRC/CvnWmAEqB10aeO7ADbZSaJaoe9Jq9D3lBXndMPxnWCEeCHnC2g6hxnlmIx4Jdxwl0rSqJk0aI7rJD3MU/Zw12mDLj3EC3aeSy/Hvx+6nutN0FxRxtyKMvy6zrsaa/h7R2/BmL939rJidgSAdzRU83hPH0opNCCRtcnaiqRlY+oaIZ8xIS9MI6zddpY6fehTdJ3uo9vODBszKzfG0DQqNINY0Q71eCbOIiOAqY3+UkZ0qSKXylKnDbkqMegrcq3Nxlm4H7l68p4HIKL56Cly9diFrgrNoI9C1zorzgJ9bBdAN1ki4/i6VXbPL5zRfBNhQnOjaG7s/662ZJpowL/n52jAT1sqUzQmw+yAcw29LxfI3RmLdzTUUOHTWfrwBk545EU+0VxPtTnxv0T36QmmbVaKVckuPlNeP+Ou7VaK21NdfDpQYi47xe3pLj5VNvMuYXo819uPjsaTy4/ksVMO55atHWxPuPAxOKL76LKze37usrNEiu7Iieg+OnNjLKVIKItw7rd/l53h2/0tfKGikajhZyxGdGlFLs1Hlxpy9WMRynN9b6CFzwUaier7j6s273nAOarVFrlq9UJXQlmEGHJ9P9nC58rGdwFE8NE9ji+i+ei2R/ZNhgnNjaK5sf+7GgN+WpPpPT+3JtM0lplFY0xaks7RNmsr+rIWNabBn9p2c1pdGFPXqCszWVxdyb9iiQm7pxzWNxkBWuwMbVaGjFI8ko6xzCy8M+F4M8gD6RgAj2X6ONpXgaZpxG2Lq+K7+EigjiN85eO6DtZzLttxPZqNcbyv0HWcL8g/Mo7r8WwfRxs5l7L41sAuPlRWx+H7oavVztCecz1mxTiu2GUEeSjruNZafRyV5/pOahcX+es4zBjfBbBID9CqCn1LjULfUiPIQ5bjW5fnmyyLtJxLOa7H7RhL9ULXEiPImkGX3ceR+v7vOiZcweuJFNsTKdK2zZ/bdvOW+qqCMWfNquIPLd0A/LX9DU6sDaFpGk0BP2t7+gBIZC2e7U2wsDIwYbc2maVID/YFVP4tck9l4tw80IkNvNUf5v2BCL8a6OJgX4BlZpB07qubzVaKkKbzlYooUcPPXclu7k720JR3NPhOsInqvL8Vi2+Rezob59ak4zrLDHNBWYTfpLpYZAQ43ue4fpxsY0vO9aXyKI26n9+lurkn3cPsPNc3ywtdxcy0K/9WsqezcValO7GAs3xh3uuPcGe6i0V6gONyrp+kcl+laDpfKHNcd6e7+UOmp+CIenWgiWptdBfAM1acVRlnbmcaYc43I9yV6WKhHuA4w/Fdl27jdZUiiM7lfscHcGlyCwPKJouiAoOrypo4SC8b1bXeirPa6sRWcIYRZoUvwm+zXSzUAizNua7PtrHVThHUdD5vRmnQHNenUlsYYMj1dbPQVcxMu5YuHwr/Pzp7+daru7CU4oKmCJ9Z0MiPNrVydLiCt9RXkbRsPr9hGy/GElSbPm48upm5FWX0Zy2++OJ2NsaTKOC9s2v55PzCr4rOfuIV/tWbGPG3yLTCOpPI/azi2p/ID+tMMlZY5QomQfAIElZB8AgSVkHwCBJWQfAIElZB8AgSVkHwCBJWQfAIElZB8AiTuvnctiExMP64vYEuv0b2Ctns+GP2Fm5eFOH2/mGl3LlKZ6y7HCUSguARJKyC4BEkrILgESSsguARJKyC4BEkrILgESSsguARJKyC4BEkrILgEaZVn+FmZYGbtQ+l6gJ4zu5nteX4ztSrONcY7rvRamOLcuo6PmcMbbMfWy1sym2zjxnj+55T/dxuO64ztCrerQ93/VQNrvdk8F/6kGul3cJmkizXwnxUn4DLxXk93NPHtzbtwlZwQbSWT80tXAY2Zdt88ZUdbOgboNo0uOHwecwJ+MnYiite28mG+ACWUpzXUMOn5058Cdlp1We4WVngZu1DKboAbKW4zergq74mVvqaedyOsVMV+v5hx6jUDG4w5/NOvZrfWEO+9+l1XKRPzGcrxSq7g//Wm/iR3szjarjrIeXM7TpjPu/UqrlTDbku0Ov4oDZxl1vzspTiqo27WH3UfO5f+ib+3PEGG/uTBWPubu0h7DN46PhD+eicWfxgi1Nt8tfON0jbNn9b8ibuffPB3NXSzc68ZU3HY8phdbUewc3ahxJ1QeE282kaJ46wzZ5WcU7THN8yLcQGNeQ7VC/HP8HlOzeRpJE8lxbmaTXcdWrOdbwW4sV8lza1qo6ZntfzsQTzyv3MLXfqM86ur+bv3bGCMQ90x1jRUAPA22dVsXZ33KnP0CBh2WSVImk79RlBY+IRnHJY3awscLP2oVRd4GyPCEU+iraZGnpN06q0KJpbLSO48l6PoWmUT8fl0rza0hmieYt6R8tM2ovqM9pTGaKD9Rlarj4ja/H2umoqDJ1l617i5Cde5pI5syZVnzGtv1kFQZg4z/cl0DWNdcsOpzdr8b7nNnFSTZC55aOvh5zPlI+sblYWuFn7UKoucLZHN0U+iraZNvSaplVpUTS3HkZw5b0eSykGpuNyaV6NfpPWvCNpaypDQ1F9RkOZSetgfYbK1Wf4DO7teIPltSGnPsPvY3FVJS/0Tfye0ymH1dV6BDdrH0rUBbAwt806VIasUqy1Yywp2maLtSBrlON7QvVxhDY130ICtJHnUjEWa8Ndj+Rc/5yOy8V5HR2uYOtAmh0DadK2zX0db3BWJFww5sxImD+07wbgfzt7OaEmiKZpzC4zWbs7Djh/uz4XS7CgYmJHVZjkivwL9YD6oX9oRf6ZrCwovrl4JmsfiiklV/HN5+vtOLdbju90Pcx7jAi/s5xttkR3fDdaOZ+m8zljaJtdltlCIrfNKjG40tfEHG30+oxnVZzb7ZxLC3OeHuFuu4sFWoAlmuP6qd3GVpy5fVYfcv2nNbR/VGLwVb3QVbx/zOS8AJacONRJ81B3jG9vbsFW8N7GGi6b18DK19s4KlTOWXVVTvP5yzt4KT5AlWlw/WFzmVteRr9l8eVXdrIp4dRnnN9YwycOKvzq5pxnNvJC316ozygO60wiK0XsHWSliL1DflhnkrHCKpEQBI8gYRUEjyBhFQSPIGEVBI8gYRUEjyBhFQSPIGEVBI8gYRUEjzCpC/mVgvTEb7/zDJO4LmSvUDbxK8ymTSYz/hhhfNJxd3Z8ZY++M8qRVRA8goRVEDyChFUQPIKEVRA8goRVEDyChFUQPIKEVRA8goRVEDyChFUQPMK0liJ9XvVzB05lwelUcY42vLLgJtp4Hace4bNEmZVbve9Pqoc19KIDH6KeY4oW09qnLvr5Vc51GlWcQ5ELx7U15/oMUWblVtP7Ez08nOc6mrFdblaQuD03N11TreoA+KPdw0PKcV2sj79/PNob57s7OrBRnF9XzScaIwWPp22br2xt5cVEkmrD4McLZjOnzFnv6dVEkm9sb6PfstHQuOeweZRNcI2aKR9ZbaVYTQdfpolraGYtwysL1uDUZ6zU5vN2qrkLp7Jgp0qxjhg/ZB5fYQ6r6cAe45o/V10ofplz/ZBm1hFjJyO7fkyRixRPEOMHzOPLgy5Gd7lZQeL23Fx1TaOqY6dKsVbFuFafxxX6HG6zx94/LKX41vZ2bjl4DvcdvoC/9MTYNFDouqerl7Bh8H9HLuTDDbX8aJfjyirFl7a28s25jdx3xALuOGQuvkmssDj1+gySNOTVI5xAmGcoqiwgzink6hEIsQGnsuAZ+jmBMKamU6+ZNGCyieRIGtddm3Ouevz40Fg2gusZ4pyacx1HiBdJoHBcywhjolOP49o81rxcrCBxe26uvo/TqOp4WvVzoja0fzSOs3/8qz/J3ICfg8r8+HWNd9SEefCNeMGYB3vjnBupAuBtNSHWxRzX47F+Dikv49AKZ/G1Gp+B4UZYdxdVFoxUj7C7qB5hsLKgh0xh3QE+djP6MnxuuoqrGGrxsXsEV+2gC8cVx2Z3kct5nWO4XKwgcX1ubrumWNVRvH/UamO72jMZonmVF41+H+1Fd0t0pDNE/c4Yn6YRMnTesCy2JtNowMc27uA9L73OrW3do3pGQuozBMElskrxTHyAew6bR0DXufi17RxREeCE8Nh/Iw8y5SNrTVFlwUj1CPlj8isLajEL6w7IUjPG7w03XcVVDD1kqRnBNfjb18JxBdGpKXL15B05RnS5WEHi+tzcdk2xqqN4/+hRY7saTJPWzND4tnSWBrPQVe83aU07Y7JK0WfZVBsGjX6TJcFyanw+ynWd5VVBXkqM/pG7mCmHtbgeYR0xFhedsVtMkEfJ1SPQxxE4lQWLqWQdMTLKpkNlaCPDIkZfRNlN14JBFxmyKJ4YwfVmgjyScz056MJxPUGMDDYdOK6FY7jcrCBxe25uuqZT1bFYq2Stmvj+cVRlgG3JNDtTadK24q+7Y5xRHSwYc0ZVkD929wJw/+4+loUd18nhSjYOpBiwndrHp/oSLJxgKRVMckX+BVpAfVcbWpH/WRXnV3TmTs2HOVeL8HvVxQICLM7VI/yMNraRohKdzzBUWfBH1c0aYhjARdRz7Diny2fSVfwWPMeQazlhziXCPXQxnwCLCZLOfeWQ76on56Kbh3OuD1LPsSN85ZB/8/lMVpAApApPVM743NxyGUahazpVHf9jd/OQclwf0uv5txH2xWOPHTquPdwb53s72rEVrKir4pPROq5v6eTIigBnVIdI2TZffr2VlweSVOW+ujko99XNvd29/KKtGw04tSrIl+YU1meseHkrG/oHpl+fURzWUqGUV4ooDmupUBzWmSY/rDPJWGGVK5gEwSNIWAXBI0hYBcEjSFgFwSNIWAXBI0hYBcEjSFgFwSNIWAXBI0hYBcEjTPquG7ev9ilFSvWqIjexp3aX4JRJxN0R2pZ03QiC55GwCoJHkLAKgkeQsAqCR5CwCoJHkLAKgkeQsAqCR5CwCoJHmF59RolWMZSqq5Tn5ma9yuP9/Vzb3oEFnFdVxUciha60bfP1tjZeTqaoNgy+PzvKbNOkJZNhxetbmed31mM6KhDga40Tqz2B6dRnlGoVQ4m6SnlubtarWErxg/YObpjTxB/mN/O3vhhbii5J+2NvjLBucO+C+XygpprrOjv3PDbHNPlt8zx+2zxvUkGFaYS1VKsYStVVynNzs15lQzLJHNNkjt+pPHlbKMyaeKFrTTzO2VWO68xQiKcSjmu6TDmsJV3FUIKuUp6bm/UqndksjXn1GfU+Hx3ZzPAxvqH6jKBu8IblXFu8K5Phwq3b+Pj2HaxPJEb1jITUZwiCS9QZBn9duIBqw+ClZJIv7Grh983zCE5wXdUpH1lLuoqhBF2lPDc361Vm+Xy05dVndGSz1PvM4WOyQ/UZcdui2tDx6zrVuWAeHggwxzTZXlRqNRZTDmupVjGUqquU5+ZmvcoRgQA7Mhl2pZ3Kk/v7YiwPFrqWB4Pc1+u4HuzrY2mF49qdzWLl/nbdmU6zPZOmqagnZywmvSL/dxhakb9UqhgOFFcpza247memq1wOfdPQfz8Wj3Nth+M6pyrMxyMRburq4vBAgOXBICnb5uutbbySSlFl6Py/aJQ5fj8P9vVxU1c3Pk1DBy6ti7A8WNiT84Gt23gpmdw79Rn5YRWEfcUUu7mmTH5YZ5KxwipXMAmCR5CwCoJHkLAKgkeQsAqCR5CwCoJHkLAKgkeQsAqCR5CwCoJHkLAKgkeQu272AWVl7rlKtarD7RqXoltWZ4yxakHkyCoIHkHCKggeQcIqCB5BwioIHkHCKggeQcIqCB5BwioIHkHCKggeQcIqCB5Bum72setZu5/V2Q5sBWcaVZznG97RckO2jS12iqBmcLkZpV4z6VMW12Za2KySnKaH+bg5sSqGUn0f3XT9M9nP9W842+ydlVV8MFzUdaNsvtvTxmvpFGHd4OpIlGjecqXt2Qwfat/KxeEIF4Zqi59+VKTrZh+6LKW4NdPB18wmVvqbecyOscMudD1oO64by+ZztlHNr7OOy0Tj/b46LvLNGvX5D5T30e1ttnJ3B9fUNXFHYzMPDsTYmil0/aU/Rkg3uCs6nwtC1fy8t7Pg8Rt7Ozk+MP5KlMVI180+dG1SSRo1p6PF1DRO0sM8ZRe6nrLinGY4rhP0EC/YTm9KQNM5TC/Hz8SX+SvV99FN18vpJE0+k9k+Z5udWR7msYFC12MDcf69wnEtLw+xPjXUdfPoQJyoYdLs84/qGA3putmXLpWlTsvrWdF89KhCVw9DY/I7WqZCyb6PLrq6rCz1xtD4WYaPTisz6hifplGpGfTaNgnb5s6+Hi4OR0Z9/rGQE0yC4BKrY928N1hDhT612E35BNNkukwimK71pnjKpfnoUnk9KypLrVboqsUZE9HMgo6WqVCy76OLrjrDR4c1NL7TyjLLMEccU+8zySpFv7Ko0nVeTid5eKCPn/d2ErdtNMCvaawI1ozqy0e6bvaha5EWoFVlaFdOb8rjdoyleqFriRFkjeW41tl9HKk7vSlToVTfRzddh/oD7MxmaMk62+zBgRgnlRe6TioP8reE43p4oI83lznb7Mb6g7g7uoC7ows4P1jNB8O1Ew4qSNfNPnHl33y+3oqz2urEVnCGEWaFL8Jvs10s1AIsNZyOluuzbWy1UwQ1nc+bQx0tn0ptYQCbLIoKDL5uNnGQXnhne/HN56X0PrrpOmjO0H+vG4hzQ6+zzd5RGeZD4Qi39XZxiD/AyeVBUrmvbjamU4R0nasjUWYXnVBa1dtFua4P++rmkvZtvJKWrpv9Blkpwnvkh3UmGSuscoJJEDyChFUQPIKEVRA8goRVEDyChFUQPIKEVRA8goRVEDyChFUQPILUZ+DuRQogFyp4kUTCHY/UZwhCCSBhFQSPIGEVBI8gYRUEjyBhFQSPIGEVBI8gYRUEjyBhFQSPIGEVBI8g9Rkj4GalhZvzcttXqq6n0/3clHBc/15WxfvKh9dnXNvfxsZsirBmcEUwSqNhsj7Tz6pEF1kUPjQ+XjGLY82KMV35SH1GEW5WWrg5L7d9peqylOKniQ6+E2riF1XNrEnH2GYVuu5PxQhqBqur53NeoJpVA44rrBl8M9TEz6ua+WJlI9fEW0f1jITUZxThZqWFm/Ny21eqrlezSaK6SdRw9o/l/jDr0oWudek4Z/kd1yn+EM9lnP1jkS9ARHc+zM4z/KRQpNXE2xWkPqPY5WKlhZvzcttXqq5ulWVWXn1Gne6j286MOsbI1WfEikL5WCbOIiOAX5t4BOUEkyC4zNZsilWJLj5bWT+pfzflsE6msgBwrR5h2q5JVFoA06q0cHNebvtK1RXRfHTm1Wd02VkiujnqGCtXnxHOHUE77QzfjrfwxcpGZhuTa5KT+owi3Ky0cHNebvtK1XWIL0CLnaHNcvaPh9MxlpmFrmX+IA+kHdej6T6OMZ39I25bfKNvFx+pqOMIs3xUx2hIfQbDbz53s9LCzfewlLaZ265I3rczT6bj3JxwXG8tC3NheYQ7El0c7Atwgt/ZP34Yb2OzlSKk6VwRjBI1/Nw50M3vBnpoyjuifi/URLU+dCT/TO82XstKfcaoyEoRwnhEascfszcYK6xygkkQPIKEVRA8goRVEDyChFUQPIKEVRA8goRVEDyChFUQPIKEVRA8gtRnUNoXKUzhKsgpM4nra6aNm/MCSAy445H6DEEoASSsguARJKyC4BEkrILgESSsguARJKyC4BEkrILgESSsguARJKyC4BGkPuMAcgE8r/q5I+c7nSrO0YZXg9xEG6/nfJ8lyqzc6o5/Uj2syfMdo+0/c3NzXuutflZlHNdZRhXvMYe7rss49SohDL7gj1KvO/Uq16Rb2GQnOd0Ic4l//HqVfKQ+4wBxAdhKsTrnu4Zm1hJjpxrZt1Ir8qkU64jxQ+bxlUHfGNcXuvo+ujgvSyluyXRwpb+J68qaedQaXq/ygBUjiMHPAvN5l6+aO/LqVS701fFhc2L1KsVIfcYB4gLYlPM1aH58msYJI/ieJs4pOd/xhNiAU/3wDP2cQBhT06nXHN+m/WRubs5rk50kqpk06k59xslGmCet4fUqpw/WqxghXrDy6lWMcswJ1qsUI/UZB4hr8LmG/5vhvsEx+dUgPUW+CD527ydzc3Ne3WSJjFOv0q2yezptDE2jQptavUoxcoJJEDyC1GccIK7B5xr+b4b7BsfkV4PUFvm6yVKzn8zNzXlF8NE9Tr1KRPPRbee51NTqVYqR+owDxAWwcNCnMmSVYt0IvsUEeTTn++egT3N864iRUTYdyvEt2k/m5ua8Fum5ehXbqc94zIqx1Ch0LTWCPDRYr2L1cZQxtXqVYqQ+o8RdxfvIs2rIdxphztUi/F51sYAAizWn+uFnRb7BapA/qm7W5HwXUc+xRV9xFO9KMzk3N+cFEMjL7zNWnFUZx3WmEeZ8M8JdmS4W6gGOy9WrXJdu43WVIojO5f4ojbrjujS5hQE1VK9yVVlhvcqXktvYZEt9xgGJrBSxdwiM/aFlrzFWWOUEkyB4BAmrIHgECasgeAQJqyB4BAmrIHgECasgeAQJqyB4BAmrIHgECasgeATpuilx3LyqyE3cnpex4Z3uiN79m1EfkiOrIHgECasgeAQJqyB4BAmrIHgECasgeAQJqyB4BAmrIHgECasgeASpzziAXKU8Nzdd6x/eyi3fXoNt2bzlfUdy/iePG3Hc2r9t5AeX3ce1/3MhBx/dSGz3AD+47D42vdDOGSsO59KrzxjTU4zUZxwgrlKem5suy7K5+ep/cNWqc7nx/g/z6J9fZfvG7mHjEvE0f/7ls7zp2MY9/89f5uMDl5/IxVecMurzj4XUZxwgrlKem5uujc+30Tivmsa51Zh+g1POPoQnH9g8bNydK9ey4tIl+MuGPrwGKkwOX9KE3z+1D7RSn3GAuEp5bm66utvj1EVDe36ONAbpbo8XjNm8oZ2u1j6WnL5g1OeZCnKCSRD2IratWPW9R/jIV0/d688t9RkHiKuU5+amK9IQpKu1b8/P3W1xIg3BPT8P9KfZ9loXV/7HPVxy6m28+mwr3730Xjb+q23U55woUp9xgLhKeW5uug4+upHWrbtp39FLJm3x6H2vctyZQx93K0Nl/PrpT3HLIx/jlkc+xiH/FuVrN5/DwUc3jvqcE0XqMw4gVynPbaZdwc1D97M+/dDr3PadNdi24szzj+CCy47nNyvXsuioBo4/a2HBv/vaf/yei//7lD1hveTU20jEU2QzNpXhMq7+5XuYe3Bkz/jL3/0bNr3QLvUZgjBV8sM6k4wVVjnBJAgeQcIqCB5BwioIHkHCKggeQcIqCB5BwioIHkHCKggeQcIqCB5BwioIHmFSVzBpmtYJbJu5lyMIBzzzlFKzRnpgUmEVBGHfIR+DBcEjSFgFwSNIWAXBI0hYBcEjSFgFwSNIWAXBI0hYBcEjSFgFwSNIWAXBI/x/OBUAhoQDy3gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reward\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABJTElEQVR4nO2dd3wb5f3436ctWZLlKXnEM5NMVoGuUAqhbaBAB6WMDqDfTuigpYyyyt4FSim0pbQJtGUUKBBKCCNQVstIQiAkTryHbMuSrGWt0/3+OHlLtoyNqfp73q8Xr2DppM89d8/n7rlHd5+3pCgKAoHgvx/Nh70CAoEgN0SyCgR5gkhWgSBPEMkqEOQJIlkFgjxBJKtAkCfoZrJwkU6rVBn0H9S6fGj8L/94Jc1jrPn8FVBOzV8sAINxfs5rndE43ngy426bUbJWGfQ8sLh2btbqv4jUPO/4+USrm790TcTnL1sDwXkLBUBVo3le4nz+9T1Z3xPDYIEgTxDJKhDkCSJZBYI8QSSrQJAniGQVCPIEkawCQZ4gklUgyBNm9DvrRJqjcS7scPPuUIwfuko4vbx4yuWv7Orj795B3li5CIBruvp4LTQEQDSVwpuUeW3lwlnF+kWHm3ciURSgzmjgygUuCrTqMelJf5Db3QNIwFKzketrK7LGuqjTzbvRGGc7S/hmWeZYF3a4eT0cwarVqu2rdrLUbOLufi9P+NUfAmVFoTkW58VljRTqtHMeKyjLnNfhpieRQFbgG6VFnFBcmPE71HgxLmhTt+OPKko53Zk5XmcszjmtPfiTMvtZTFxbW4FBI9EVT/CLNjfeZJJCnZbraitwZblRpiUW56IuN7uiMc4qL+EbpZljvRaKcGNvPwlFYT+TicuqnOgkiSf8Ae72eFGAAo2GX1Q6WWIyZvyOFyMhfj/oQUJCK8HZjnJWmyb/NvpMOMifAwPIwEdNBXyvqAyAvwZ8PB4aRCuBQ6Pl/BIXLl3mdm0dCHBZUzcpFL5SUcx3a52Tlnm8z88tLW4kYJnVzC3L1fsTGp/bzhKrCYBKo4Hfr6rPGCMTs0rWQq2GC6rKeWYwNO2yOyNRArI87rXzqspH/n9jv49dQ7FZxzqvsmykQ1/b1cd9Hj/fchbTGovzu14v9y5cQKFOy0AimT2WTsN5leU8G5i+XedUlLGu0DbutdPLijk9nXTPB0L82ePLmKhzEesvA34ajQZur6vCm0xyzJ5WjnHY0Wsy3wxRqNVyYfX02/HGbg9fKy9ifZGdS9vdPDTg56tlRVzf1cdxxXaOLynk1WCYm7o9XFeX+aBn12o4r2LqtqUUhV90ufldXTV1RgO393n4hz/AF4oKqTLo+WP9AuxaLS8Gw1zW3ct9DTUZv+dAk4WPm2uRJIm98RgXe7q5r3J8IgzKMrf7+/mDq4YirY4rBty8Ho1wkMnCYoOR37tqMGk0PBz08xt/P78srZwUR1YULt7TxYY1DbiMeo57vYkjSwtZVGAaWaYlEuOOtl4ePGAhhXodnnhi5D2TVsOmg5dMue2zMathcIlex0qLielukpEVhRu6+/lpRVnWZTb5g6wvsmV9P9dYw4mqKApRRUFKL//gwCAnlzpGkqZEn/04VaLLLVYubPIH+ZxjinbNMpaERDiVQlEUIimFQq0W7RTfVaLXsbLAjE7KvpCiKLwajHB0er2PKykcSe690TiH2CwAHGK18OwUSV+i07HCPHXb/LKMXpKoMxoAOLTAwpb07UlrLGbs6f252mKiL5HI+j0WjQYp3aaokkLKcKNldzLBAp2eIq267w8yWXg+osY6wGTBpFHTYbnRRH8y88F8eyBCrdlAjdmIQaPhWKeDpz2D45b5a/cAp1WVUpjuY6VzdIvuvFyz3ufx86lCK2VZEqQrnqAznuAQq2VO4l3Q7uaT7zbTEo1zSqkDgNZYnNZYnFOa2jmpqZ0XA+E5iXWr28MJTa1c291HfMJ9i0OpFP8KhTnKnj1ZZxvr5BIHzbE4n3qvmROaWjmvogzNFImYC35Zxq7VjCS0S6+jNz0SWWo28rRfTdCnB0OEUyl8STnrd01HkVaLjMI7Q1H1OwMh3BlGPX/3DfIxa8GU37U1EuTk7hZ+1t/F+SWTh6ZVej3tyQQ9yQRJReHFSIi+DEn5eGiQQ8yZY7ljCSpMhpG/XUY97tj4g0jLUIyWSIwvvdHECW80sXUgMPJeLJXi86/v4YQ3mtjcPz7Jp2NWw+Bc6Eskecof5J6FC7Iu86Q/yLpCG9pZdrJhrqpxISsKV3b18aQ/yBeKC5EVaIvFuWfhAnrjSb62r4NHltSOHLnfDz9ylVKq05JQFC7t6uMP/T6+6ywZef/5QJj9LeasQ+C5iPVSKMxSk5G766vpiCf4VksnBxaYR0YYc825VeVc3tHLI95BDrKacep1zCaSJElcV13Bde5+EkqKwwoKJvWDf4cjPOwL8Kf67H0IYK3FxlqLjW3RCL/zD3CLs3rc+3aNlnOKyrnY04MGWGE0052Mj1vmqXCA9+Ixfu3MPgqcDllRaB2K8Zf9F+KOxfnKW/v458FLsOu1/Ouw/XAZ9bQPxTh52z6WWE3UmjNfh09kxsl6n8fPAwPqEeHOhirKpxhOAuwaitIWT/CZXS0ARFMKR+9q4allo9cTm3xBLqoun/TZmcYai1aS+JzDxh/6fHyhuBCnQccqiwm9JFFt1FNr1NMWS7DSona1vwz4edCrxrqjLrdYwyMFgyRxfJGdezy+ce8/ORjIOASey1gP+wKcWVaEJEnUGA1UGfS0xOKstIxOrtzb7+PB4e3YWD1tPIdWS0BOkVQUdJKEO5HEmf5MuV7HbQ1VAITlFJv9IexjDkZ/HfDzkF+NdXtNbm1bbTGPJOLLoTBt8dEE2hONcWlXL7+prcIx4aD3UNDPYyE11g1lVZTq1FhrTBa6B3rxyzKOCQetj1usfNxiBeDRkH/cgeY/0TB/HvTya2c1BinzoNNl1NMTHV0/dyyBy6ifsIyBNXYLeo3EArORerORlqEYq/WWkWVrzEYOdVh5Jzj0wSXryaUOTk4PLXNhrd3Ki8utI38f+HbTuERtjsYJyDJrLKZJn51pLEVRaI8nqDUaUBSFZwNh6tNDlk/brWxKn2V9SZm2WIIFY64lvlri4KsluccC6E8kKdPr0rFCLBozPArKMq+Hh7hmweTJl7mMVaHX8WoowoEFFjyJJK2xONUGw7jPnlJWxCllRTnHkiSJQ2xmnvIHWV9k59GBQY4oVPehL5mkUKtFI0n8rneAL5SMn3k+qcTBSTNs20AySYlORzyV4m6Pl2+VqqOTnniCH3d0c1W1a+SadixftDn4ok2N1ZmIoygKkiSxOx4lQYpCzeSE88lJirQ6AimZh4OD/LJU3T974lGu9/ZxY1nVyDVtJlbZLLQOxekYiuE06nms1z8y0zvMulI7/+jz8+WKYrzxJC1DMWrMBgYTSUxaDUaNBm88yRuDYb5dM/kklY1ZDYP7E0lObGonJKfQABs8fh5bUotVq+XbzZ1cvsA17ZF1k189+0jTDIFziVWq03JBu5tQKoUCLDEZuSR9xv64zcLLwTDHvNeKVoKfVpROOlIP40kk+credkIpNdZGj59HF6uxvtvSyWXVart+3tGDLymrscxGLnGOXic9EwjxUWsBlgwdZi5jfae8hAs73ZywpxUF+LGrjKIpht39iSRf3t2mbkcJ/tzv4/FldVi1Wv5vXydX1Kjxzqks45zWHm7t9rDMYuRL6aT8d3CIm3r6kYCDrBYuzjAiGtu2k5rbCQ+3bcDPIwvVtn2vrZNLK9VY93h8vBAMk0LhxGLHyNzFb/sH8CdlruzpA0AL/LUx8yOazw+F+Gc4gA4JoyRxWUnlSJ/6Rk8b91Son/uVr599cfVXh28UFlOjVw8Ct/s8DKVSXOTpAcCp03FtWdWkODqNxGWLq/ja9mZSCny5opjFBSZuanaz0m7mqNJCPlls40VvkKNeew+tJHF+YwVFeh1vDIa5cHcnEuoz1N+pKR83izwd0kzqBq+wmBTxPGt+IZ5nnRvm83nWHYFIxp0m7mASCPIEkawCQZ4gklUgyBNEsgoEeYJIVoEgTxDJKhDkCSJZBYI8QSSrQJAnzOgOplQKIpEPalXGU2Cdvx/z9Yb5PWZp9fMXL5WavxsVlFT2Z4TzHTk2P21Tpthf4swqEOQJIlkFgjxBJKtAkCeIZBUI8gSRrAJBniCSVSDIE0SyCgR5gkhWgSBPEMkqEOQJs6rB9GQgwJ98o3qD88qdLDZOrtT270iEWzz9KAqYNRKXOl0sMBiIp1Jc0utmVyxGoUbL1RUVVOozF0TOVfvws9Zudkai6CSJVRYTl9a40EsSj3kD/L53QF1XrYZLqp0szVCkDdTCa1e297J1MIRJo+Ga+gqWF2Qv6/Gdpg46YwkeX9EAwHuRKJe0uomkUlQZ9NzQWJm1NKiiKFze4uZ5fwizRuLahVWssE6OtTM0xLl7u4imFA53WLmo3oUkSdzS3sf9fT6K05X9zqkt5/ApiqUrisIVbb1s9QUxazVc01g5qW1Dcoqzmzppj8bRShKfKrLys5rR+lKbBga5rbMfCYmlFiM3LaqeGAaYvV5lmM3+ID9q6+H+RTWsyLLPctFnRFIpvtfbMfJ3v5xgXYGdHxaVz0yf4QtyRYsbGTix3MF3qseXLY2lUvysqYud4ShFOi23LK6m2mSgMxrn6G17aUgrQNbYzFzeOLnqfzZmlayVej13Vat6g5fCYa7s7eVPNZP1Btf09XJjZSX1BiMP+P38wevlUpeLRwMBbBotj9TV81QwwG2efq6uyLzyuWofjimyc13aYfPT1h4e9Kjah2qDnj8vqqFQp+WFwRCXdPTytyWZ60m9MBimNRZn88pGtoejXNrm5oH9MjtJNnsDFEwoinZhSw8/X1DOR+wFPNjv5/c9A/woS2Gxrf4QrdE4z+y/kG2hIS5p7uGhVQ2Tlru4uYcrGytZYzVzxq52XvCHWJtOym9WlHBmVemU22VcvKEYT69ZyPZ0vAdXTo53RkUJhxYWEE8pfH1XK1t9QdYW2WgdinFn1wB/XV4/vYZklnoVUMudbvD4WZUlSYfJRZ9h0WhGCqcBnN7TxlqzWrVxJvqMS5t7+NPyOlwGHV/Y0cyni20sGrN+D/T6KdRpefaARTzuGeS6tl5uXaKWWq0xGnhsTeO02yMTsxoGrzaP6g1Wmkz0JbPrDcLpqmShVIqy9FlgazjEMXY7AJ+22vh3JEK2Am65aB8A1hZakSQJSZJYWWAaqSS/v3W02PbqAnPGqu/DPOMPcnxJIZIkscZqJiCn6ItPbltYTvHHXi/frRyfKK2xOAenFRMfsxew2Ze9utcWb5ATyhxIksT+NguBpDwpVl88QUiW2d9mQZIkTihz8LT3/VUMe8Y3Gm+NzUIwQ9vMWg2HFqoV6Q0aieUFZtxxdXvd3+fnFFdRbhqSOdCr3Or2cGZ5EcZp9nsu+oyxtCfi+FMyq43q2TdnfUZoSNVnmAwYNBrWlxayZcK+2OILcEK5A4DPlNh5ZTCctV/PhDm7Zn00MMhHCzIrBy5yOvlhVxefa2lmUzDA14vUGrZ9ySTOdOLqJAmrVsvgHJUaTCgK//AG+Lh98jo9NDDIJzK8PkxvPDnOjDZWHzGWW7r6Od1VjGmCBGqRycgzacXEP30BeuLZDwy98QQVxtEO7zLq6Z2w/KT1MeroHZNgG9xe1m/by3l7uxicRmUx8bucBt2keGMJJGWe9QU5LJ28LdE4rUNxTtrZwpd3tvCCf/qz5nRk06u8G4niTiRZa7dm+eR4ptNnjOWZSJAjLJlL4E6lz+iNJagYuy8MGfZXLDmyjNqvNSOKkc5YnGO37+OrO1v4zwwVLnOSrK9HIjw6GOCs0szKgft8fm6pqmJTfQPH2u3c7Omfi7BT8suOXg6yWjhogj/ntWCEhwYGOafy/esRAHZForTH4hxVZJ/03pX1FdzX5+ML77QQllMY5kgLkolTXMU8e8AiHlvdSJlex9Wt7jn77qSi8OOmTr7mKqYmXVRcVhRao3E27FfHTQur+EVzN4FZuG6G9SqnTCjmnlIUru3u59wZ7Ke1Fhv3VdZzdWklv/MPTLnsM+EgRxZMvrYf1mecbM+9KHqulBl0vHDgYh5b3ciFdS5+vKeT4Ay23YyvWe/3+3lkUFUW3FJVhV+Wubyvl1srqyapCkCt4L4nHmNF+mJ/ndXGWd1dAJTrdPQmkzj1epKKQkiWx1VRn6n2YZjbezz4kjKX1Y8/uu4einJRu5s7G6snFcK+t9fL/f1+AFYWmHGPOXON1UcM81ZoiJ3hKEds30tSUfAmk5z2XhsbltbSaDZy9xL12r0lGuP5CddsG3oGuL83HctqomfM41fuWAKnYXwsp0E3fn1iSZzpI3fpmGW/4iziW7vaJ22PjW4v9/f50vHGt603npwUb5iLmnuoMxv5RsWov8dl0LPaalbVECYDdSYDrdE4q9KTYnOlV3lwcQ1N0Rhf36tOCHmSMt9v6eL2+qqRSab3o88AaIrHSKKw1DD+OjgXfYbTqKdn7L6IZ9hfRh098QQVxuF+naJIp0WSJIzp/r3CaqYmve1WZphQzMSMk/VEh4MTHQ51RRMJftbTzS+dLmoNk/UGADatlpAs0xaPU2sw8GokQl26CvonC6w8HgiwymzmmVCQgy2WccOSmWofAB7w+PlXMMwfFy4YZ1Prjic4u7mba2srRpQaYznFWcwp6Rnm5/1BNvb5WF9sZ3s4ik2roXyCtu/k8iJOLlfXrTMW5ztNnWxYqk5eDCSSlOh1pBSFO7oHOGlCG06rKOG0dAI85w2ywe3lmFI720JD2HTaSbHKDXqsWi1vBSOssZp5uN/P1yrUde2LJ0aW3+wNsNgyeTb+VFcxp7rU5Z/zBdno9rK+xM720BDWDG0DuLmjj6Asc2XDeP3HkcU2HvcM8sVyB95EktZonAVjXC9zqVd5ecWoWPvrezv4WWXZuNng96PPANgSCXCUZfxZNWd9htVM21Ccjmgcp0HHE55Bblo8fjb800U2Hu7zc4DNwj8HAhxaWIAkSQwkkjh0WrSSRHs0TtuEbTcds5oN/p13gEFZ5tq+tN5Agg01aoc9u6uTi5wuynQ6fuF0cm5PNxrU5L04rX44zm7n4l43x7e2YNdouKois5QXctc+XNbRS6VBz1f3qGeYIwutfL+ilN+4B/DLMr/s7FXXFXhwaV3GWGsLrWwdDHPU2/swazRcVT+6XsftbObRFZNnT8fy+ECA+9JnsqOKbHyxNLuJ/PAiK8/7gxzxZhNmrYZrF44qG47dtm9k5vCyhgrObeoimkqxtsjGWofawa9t62VXOIoEVBkNXNGYfRsCHO6wstUf4shtezFrNFw95qeDz+/Yxz9WNeKOJbijy0ODycDxbzcDasKfWF7EJwoL+Jc/xGe370WLxLk1ToqynD3nQq+SK7nqMwCeDYe4oXy8GiNnfYYkcUlDBd98tw1ZUfiys4jFFhO/au9jhdXEkcV2TnQWcU5TF0e82YRDp+VX6WT+TyDCr9r70EsSkgS/bKjEMYP2z0ifsZ/JpAwn4wfNfFaK0OhEpYi5IDk0f5UivL7pl5lLXNVzI0SejuO37+Pt0JDQZwgE+YxIVoEgTxDJKhDkCSJZBYI8QSSrQJAniGQVCPIEkawCQZ4gklUgyBNmdPuIJEGWuwrnHDk5fz/mS5r5iwXw7tvZHyWca5YszfzQ+weBpJm/G1lMxvndZ5J2ns5rUzz0Ic6sAkGeIJJVIMgTRLIKBHmCSFaBIE8QySoQ5AkiWQWCPEEkq0CQJ4hkFQjyBJGsAkGeMKsCOC2xOBd1udkVjXFWeQnfKM2sR3g1FOGm3n4UwKKRuLzSRY3RQHc8wcXdbnxJmUKtlquqXbiy6jPiXNTp5t1ojLOdJXyzLHOsCzvcvB6OjOgqrqx2stRsYlCWuajTTUc8gVGSuLzaxSLT5OJikNZndPTxQkDVZ1xdV8HyCRXhQ7LMqbtHKwm640k+X2LnggVqfaknvQF+3eNBApaYTdzYkF2T8HYywu9j/SRRsEtarrYsmLTM43Ef/4j7cSsJNhY0Yteo7Xs1EeLeuAcNElrgTGM5++myV8vLVQ1y2ntt9CWSmNJ31Ny9pIYSvY6r2nt5LV3vNppSGEgmef2AJRlj5ao86YzFOae1B39SZj+LiWtrKzBoJP4TinB1Zx97hmLcWFfJ0VNoQeZT5bLVG+Ty5m5kBb7iKuI7C8bbFu7rGWBD9wBaScKi1XDlwioWFZjYHoxwYZNa2VMBzq4p5+gp6nNNZFbJatdqOK+inGcDUxd6vrKnl1tqKmkwGvmr189dHi9XVLm4sbefYx12jnMU8loowq29Hq6qzlzwq1Cn4bzK6WMBnFNRxrrC8Tv2d31elppM3FpbRXM0zpXdvfyhYXJSALwQCNMWi/PU8ga2h6Nc1ubm/mV145axarU8Mkap8YVdLRzlUGO2RuPc5R7gviW10yomQorMb2N9XGquokyjx5/KvOwyrZmDLVYujHSMe321zsIhOlUb0SLHuC7azR26zKoPmJka5IaGSlZOSOQLxjhvNvR6eTcSzRorV+XJjd0evlZexPoiO5e2u3loQFWeVOr1XF3r4u7e6QsuzZfKRVYULt3XzZ9W1OMy6jhh2z4+XWxnUcHowfzYMgcnp6tXbhkIcGVLD/esqGexxcQj+y9EJ0n0xROsf7OJT5fYp7VMDDOrYXCJTscK8/R6BICQnBr5d1if0RyLc0iBWoT7IwVmngtmr1BeostNxZCNfbE4h6TrszaYDHQlkniyJNEz/hDHTdRnTJFwLdE43oTMQenvf8Dj5+Ty3BQTLySCHKazUqZRj+IOTeZlG7UmnJrJR3qzNKqNiDG9NiJXNUguPDEQ4JjiyUXOh8lFeaIoCq8GIxydPtAdV1I4ktxVRj1LzCZyueV4vlQu24MRak0GasyqPuOYskK2eAPjlrGNqUkdkVMje8Ss1Yxsi1hKmXZfTWRu6kBOw6WVTr7f3oVR0mDVathYr57RFpuMbAmEOLWkiGeCIcKpFP6kjEM3u5vPb3V7uKNvgEMLLPzYVYpBo2GJyciWwRAHFlh4OzJETzxBbzJJaYZE6k0kqBhTuNllUHUV2cpmbvIG+GyxfSRpWqNxAL76XhspFH5QUconCjMrILpScWQULoh0MKSkONZQxBH67AmQiVcSQf4c9zCYkrnYMrl85ri2ZVGDZKodfEFLDxpgXbGN71WUjqvp3BVL0BmPc+gUGpJc8Msy9jGdOJuqZCbkonIxajQUaDT8sVrti9lULhOLhPfGklQYx+sztgcjk+Js6B7g7i4P8ZTCxlWjI5dtgQjnNXXSFU1ww5LqnM+qME8TTBsH/NxeU8WWJQ0c57BzvVvVZ5zjLOON8BAn7mvj9fAQ5TpdTkfRqfiRq5THFtfxt8YaBuUUf+hXh1BnlhURTKX4YlMb9w74WWo2MlfPo2zyBVg/RqORRKEtFufPS2q4sb6Si9rcWRUTMgp75RgXm6u4zFzN32IDdKXiM4p/mN7GHQX1XGCu5N7Y1NqIXLmhoZLHVjRw77Ja3ggO8Wi6yv4wT3gHObrIjvYDVIO8H/5bVC6nVZbw3MFL+Hm9i9vb+0ZeX2O38M8DF/Pw/o38tqOf2AzcTjM+s/51wM9DfnXH3V4zvR7Bm0yyOxpjlUUdIn7GbuO77Wl9hl7HzTXqdUFETrElEBoZygD8ZcDPg1411h1108cCRuRGBkni+CI793jUZLVqtVxR7QLUodfRu1uoHnM2ubfPxwMePwArC0zjZFLu+KiuYiLvRaIkFYUVY65ZXHo9qwpM6CWJaqOqmGiLxVmZnvh5Iu5nc0Jt18d0Vg7QaTFJGkwSLNeZaZFjVGlm/iziCp2FW6K9BFLyyAQUzFwNAoy016rVckyJnR3hKMePkeVt8ga4uMY16XMzVZ44tFoCcoqkoqCTpKzrk4n5VLkM4zTq6IlN0GdMUVX/mLJCLtrbNen1hRYTFq2G3eEoq2yWDJ+czIyT9aQSByeVOHJe3q7VEkrJtMbi1BkNvBKOUJ9+KFadBdagkSR+7/FywgTJ01dLHHx1BrFArQJfptehKArPBkIsSqsyArKMWdKg10g85BvkwALzOMHxKeVFnJLWYTw/GOLePh/ri2yj+owsHegJb4D1E67bjnRYedwb4IulDnxJVTFRbRxNvvUGB+sNars65Bh3xvqQFYUkCnvkKMfpc1eGdKfiVEh6JElin6xqI2wTPC0zVYMkFYVAUqZYryORUnjeH+KwMcPdfUMxAskU+2dwtMxUeSJJEofYzDzlD7K+yM6jA4MckeWSYSLzqXIZZpXNQms0NqLPeLx/kJuXjJ+obBmKUW9WZ6Kf8wapS/9/RzROhVGPTpLoisZpHopRnUHlko1ZXbN6EklOam4nnFL1CBsH/DyyUNUjfK+tk0srVT3CJZVOftKh6jPsWi2/rFJnFP8TjnBrn/rzxgEWMxdWZBYOD8f6yt52QsOxPH4eXazG+m5LJ5dVq7F+3tGDLymjAEvMRi5Jqzqao3Eu7HQjAY0mI7+szq4EXGsv4IXBEOt2NmPSaLiqbvQMcvy7LeNmgZ/0Bblrgvn74/YC/hUIs/6dZjRI/Ky6fJIIa5gFWiMH6Ao4O9KGBBylL6RWq+7cyyKd/MDkokSj47G4j7/HffiUJGdHWjlQV8BZJhevJEI8m1S1EQYkzjVVZuxkI23LQQ0STymcuaeDhKKQUhQOsxdwYpljZLlN3gCfG3ONno1clSfnVJZxTmsPt3Z7WGYx8qUS9eeMt8NDnNXSTUCWeW4wxG1uD48vyzxzPV8qF50kcUljJd/Y2UJKgS85i1hcYOLm1l5W2swcWWJnQ/cAL/tD6CQJu07L9Wl9xuuDYe7s7EcnSWiAyxorKf6g9BnLzSblr43zo8/I4hT6QNAZ5vfekD175sZBmwvzWSkilZy/doVD81sposiV+Tf5uea4t/bydjAi9BkCQT4jklUgyBNEsgoEeYJIVoEgTxDJKhDkCSJZBYI8QSSrQJAniGQVCPIEkawCQZ4wM9eNRsJgmp/8nk9vynyzdPm8PJk47+hncOvcbDG8z2dw3y96a/ab9eeSqfq9OLMKBHmCSFaBIE8QySoQ5AkiWQWCPEEkq0CQJ4hkFQjyBJGsAkGeIJJVIMgTZvUrdi6aCYDHvQHu7BlAktSKhtfXV1CkGw19d+8A13X288rqheNenxRrFtoHUGsH/bqrHwlYajFxY2PmGruzjRVPpTi3uZt3IlEcOi03N1aNK5g2MdYVbb1s9QUxazVc01g5KdaQnOLspk7ao3G0ksSniqz8LF0Z/+6eAR7o86GTJIp0Wq5urKQqS6xc443lO7vb6YgmeGJ147jX/9A9wLXtvbx64OKsdYQUReHyFjfP+0OYNRLXLqxiRYYiaztDQ5y7t4toSuFwh5WL6l1IksSucJSL9nUTSaWoMuq5aVH1uALaY5mtymWYpwNBzuno4S8NNSw3T+7LP93RxjP9g5QYdGz5xH4ZY7wyEOSyXZ0kFIVivY4HDl0MwEef30mBVoNWktBKEk98bGnGz2djVsmai2YiqShc1dHLE8vrKdLpuL6zj419Ps6qVOu69sQTvBSIUGmYelVmq31ojca5q8fDX5bVTau0mG2sBzx+7DotT69ayBMDg9zQ0cevFlZn/PxWf4jWoRhPr1nI9tAQlzT38ODKhknLnVFRwqGFBcRTCl/f1cpWX5C1RTb2s5j4+4oGzFoN97m9XNfWxy2LM8eaSTyAp7wBLBmKYfXEErw0GKIyS3nWcbGicZ7ZfyHb0rEeWjU51sXNPVzZWMkaq5kzdrXzgj/E2iIbF+zt4rw6F4cUFvBAr4/fd3v4cU3mQnezVbkAhOUU9w74WZkhSYf5cnUxX68t48c7WjO+P5hIcuE7HWw4eCFVZgOe2Pg7rf52yGKKp+nr2ZjVMDgXzYSS/i8iKyiKQkhOUT5G+HN1Rx8/q8pckHl8rNlpH+7v93FKjkqL2cZ61hfihLRw6OhiO68EM6sYAJ7xBTmhzKHGslkIZohl1mo4tFAtBWrQSCwvMONO1zU+tLAAs1bdjWtsZnqnWc9c4oHacf/YM8D3Muybq9rc/KzGOa38YYt3NNb+NguBpDwpVl88QUiW2d+mlv48oczB094goGpJPmJXa+p+zGHlnwPBrLFmq3IB+HWfh9NLizBOUbXxkGIbDn32InSPdvv4rMtBlVk9W5dOUVN4pswqWbNpJsailyQuqXHy+Xdb+OSOveyLxvhSuiM/4w/iNOhYmmHoPClWFu1DJi5o6eG4nc3c3t0/kiSt0Tgt0Tgn7WrlxHdbeGEKWdJsY/UmklSkP6+TJGxaDb4sFfknxnIadPTGs5/1A0mZZ31BDiucrId4oM/PJx1T19zNNd4tHX2cXlGCacK9qlu8QZwGPcsKctlnCSqMY/qHUT8p1qRtbRztQ4ssRrakE/fJgUHcsdnfDzyscjlydzOPDwY4o1StcfzuUBR3IsknbbnVLM5GczjKYCLJia/t4XMv7eLBrlFDggSc+p8mPvfSLu5t98z4uz/wCaaEovDXfj8P71fHC6sWsths5C73AEOpFHf2DHB2Zen0XzIDsmkfZAXaonE2LKnlxsYqLmrpyaq0mG2sD4qkovDjpk6+5iqmZkJx6Ef7/ewMRzmzsmTWcd4NR2mPxVk3oXj5kJzit139/LB6+pHQXHBNYxUb3V6O276PsJxCPwcPd2RSuaQUhRvc/fzUNft2yQq8PRjhngMb2XjwIm7d66Y5rJr2Hjp0MZs+tow/H7SQP7f385o3+0ghEzMePM9UM/FeWgk4fBH/2SI7v3MPcIQjTmc8wXHvtgDqEfYL77Zy/7K6EQXGXGofnAYdqwvM6DUSC9JKi9ZonFXpCY85jaXX0RNP4DKoKoagnBpX5Huj28v9farWY6V1fKzeeBJnlmuai5p7qDMb+UbF+IR8aTDEHV0e7l1ehyHDNeZM420LRtgZivKpN5tIouBNJDn1nVYuqnfRGUvw+R3N6naJJzjh7WYeXNFAWfo7NvQMcH9vejtaTfTExvSPWGJSLKdBN35bx0b7UKPFyJ+W1wFqlfvnfeNHQ3OlcgmnUuyNxTijVdVpepIyZ7d3cWtNVcZJpqlwmfQ4DHYsOi0W4JBiK+8GhmgoMOEyjQ6Nj3YWss0f4ZDi7M7Zicw4WWeqmSjX69g3FMObSFKs1/FyIEyDycASs4mXVy8aWe6It/fy0LK6cbPBc6l9OLLIxhMDAb5Y5sCbUJUWC0z6DyTWEQ4rD3sG2d9q4SlvgENt41UMp7qKOdWlxnrOF2Sj28v6EjvbQ0NYM8QCuLmjj6Asc2XD+Erx74aHuLi5hz8srcl6HT7TeCe7ijk5vXxnNM63d3ewMZ00rx40Kk7+1JtNPLSyftxs8GkVJZyWPpg85w2ywe3lmFI720JD2HTaSbHKDXqsWi1vBSOssZp5uN/P1yrU2APxJCUGHSlF4fbOfr7qHK/lmCuVi02r5YWlC0eWO72lg3NcZTNOVIB1zkIuereTZEohoSi85Q9zZl05kaRMCrDqtESSMi96gvxw4WRX0FTMajY4F82E06Dn+5WlnLqnHZ0kUWnQcXVddgt41liz1D58wl7AS4NhPvf2PrSSxLkLyrP+TDTbWF8qc/Cz5m6O2rGXQp2WmxuyaxgPd1jZ6g9x5La9mDUarm4c3Taf37GPf6xqxB1LcEeXhwaTgePfVs9qp7qKObG8iGvb+oik1J92ACqNen67ZLJEeCbx5orDi6w87w9yxJtNmLUarl04uh2O3baPx9aosS5rqODcpi6iqRRri2ysTV93P+YZZKPbC8C6EjtfKndkjTVblUuu/GBbC694g/jiST7y7Nv8ZFEFifRcxWk1ZSyymjm81M66f+1CI8FJ1aUssZlpi8T4vzfVfZdUFI6vKOLwstyt5zBDfcaKArPy0ISfZj4o/pcfPpe0/5tt08zjPosMzu/D546a2U085cr6l95jx2BY6DMEgnxGJKtAkCeIZBUI8gSRrAJBniCSVSDIE0SyCgR5gkhWgSBPEMkqEOQJIlkFgjxhxrcbztedRVr9/+5x5LWX5+/um4MOzv7s5VyTTKTmLVaWO0U/MAzW7NU35pKp7m77380IgeB/DJGsAkGeIJJVIMgTRLIKBHmCSFaBIE8QySoQ5AkiWQWCPEEkq0CQJ4hkFQjyhNm7bnJwwgzznaYOOmMJHl+hKhR2RaJc0tpDLKWglSQurXWNlAbNFCsXb8qNbb083O8nkEyx49BlI68/1OfjmtbekYLSp1YU85UJ1fLmKtYw/xwI8IPdHTy8qoGVWdoVQeY3uBkggQysp4i1jC+kFSPFrfTQSxwNEgdQwEmoNW634Odp/GiQMCFxBk6qMWaMNdy22Xh8/uge4IF+P1pJolin5ar6SqqyVJ3PxYUUkmVO3d0+8rc7nuTzJXYuWOCkK5bgwrYevEmZQq2G6+srxxUEH0tzNM6FHW7eHYrxQ1cJp5dndt0Mc2VXH3/3DvLGSrXCZjyV4rx2N+8MxXDotNxUW0FVhlhdkRhn/Xsf/dEEkgSnNZTzrUXjK07640l+/J99tIZjGDUSNx/cyLJC1SwwGE/yk9eb2R2IIAE3H9zIQSW5lSOdnetmBk6Yzd4ABRNq2l7f0cf3K8tYm664d31nHxuW1mb8fK7elCOKbZxWUcyRb+6d9N760kIunVDK84OKFZJl/tQzwOosSTrM0/ipwsBPqSJAkp/Sysewo5sgp/gcRSzHQhKFq+hkG2HWUMBHsXEkDgDeIMS99PNzsrtuZuvxWWYx8dB+9apbp8/H9R29WT0+ubiQrFotj4yJ/4VdLRzlUDvvdZ19HFdSyAklhbwaCHNTVz/X1WeujFmo1XBBVTnPTGFaGGZnJEpAHl/g/SFvALtOy1PL6tnkC3Bjdz83ZajCqZMkLl1dy6qiAkIJmXVb3uaTzkKWpDUfALfs6mK5o4A/fmwJTYEhzn+rhQfXqhKrX2xr5QiXgz98dDHxVIqhZO63aM7SdZObEyYsp/hjr5fvTqi+L6XfAwjK8pRFmnPxpgDsb7NkrLs7E+Yi1q/a+/i/ylKM095LLRElhYJCFAUr2kk7xYiG5aidQYdEHUa8qOtjYfTe3xjT7/jZenwOtY9x6xSYcU8h+MrFhTSWlmgcb0LmoPQBbl80xqE2td2H2Cw848+eiCV6HSst07tuZEXhhu5+floxvvr+s4Mhji9SDQTrHDZeDWX2EznNBlYVqfWhrXoti+xm3EPxccvsCQzx8XL1uxbZzXSEY/RH4wQSSV7tD3JyvRrboNFQOANJ1excNzk6YW7p6ud0V/Ekb8oFNU6u6+xl7bYmru3o4yfV5VPEmt6bMh1PDQRYv20v33+vg+4pvCmzjbUzNERPLMGncqi2vg4HXcT5Ac2cRyunUYZmCuVTGJk3CbOC0SP5Zvz8mBb+goevk30bwuw9PmN50OPnk4XZS3Tm4kIayyZvgM8W20cKoi8xm3japyomnvaHCKdSWZ1BuXKfx8+nCq0j1oeRdU0mcaVfU/1EWvzy1Ae/9nCUnb4wBxSP3wbLHRY2dan1jt/0huiMxOgeitMejlFi1PHD/+zjyKd38JPX9xGeQXs+8AmmXRHVm3JUkX3Se3/p83H+Aidb1yzi/BonF7Z2f2DrcUSRjecPXMQTaxbycUcB5zZ1fSBxUorCVa1uzq/Lrdr6DsLUYuTXNHAVtfyJPiJk3oEyCr+mh6NxUM7oUyDrcHAz9ZxEKY/gnZN2TOfxedQzyM7wEGe6pr42nAmbfAHWj+kn51aX8Z9QhBPebeE/oQhOvY7ZPEPUl0jylD/IKaWOWa9rOClz5stN/HJNHbYJiX/W0koGEzKf3ryDu5vcrHAUoJUkkimFt/1hvtHoZMtRq7Botfz6vdz7/MxdNzN0wrwVGmJnOMoR2/eSVBS8ySSnvdfGhqW1PDwwyIVp3+Zni2z8oqVn3Gdn6k2ZiqIx63Wis4hr23o/kFhhOUVTJMYp77QC0B9P8u1d7dy5rGZkkmkzfp5D7fwFaPgSJUhIuDBQhp4e4jQy+Vr3D/TiwsBnyTwxdhg2/kjfpNfn0uMD8PJgmN/2eNi4tHaSW2emLqRh3otESSoKK8bY6ZwGPbc1qtfDYTnFZl8Q+xhn0H0ePw+kDyJ3Nkzvutk1FKUtnuAzu1S/UjSlcPSuFp5aVo9Tp8OdSI7xE8k4tJnPZYlUijNe3sMXaktZXz35YGXT67jlYNU2oCgKB296i9oCI0NyigqzgQPSE0rHVBdz2weZrDN1wpxcXsTJaTdOZyzOd5o6RyaRyvU6/h2McIi9gFeDEeommNFm6k2Zir54YmT5Z7xBGs3jZ0znKpZNp+U/Hxk1Wp+8s4Xz61zjZoPX4WBdelLobnp5hwhLsTBIkh7i486aw9yPhwgpzmS87sFNHFd6+W2EcTF5PefS4/NuOMrFbT38fnFmt85MXUjDPOENsH6Ctc6XTFKo1aKRJO5yD/DF0vGz5CeXOjh5BmfJtXYrLy4fHbIe+HYTTy1TJ7c+VWjlEV+ANQVmNvuDHGId7ycaRlEUfvx6M4vsZr6zOPNk5WA8iVmnwaDRcG9LH4eW2bHpddj0UGUxsjc4xEKbmRf7Bllsn3oCciyzc93k4ISZisvrKriqvZekomDUSPxyiqFjrt6Ua1vd/MMzyFAqxcde382J5UX8sKacP/V4ecYbRCdBoU7LdQun8M/MMtZMOIESfoubn9MKwEmUYUsP9s6njaupZYAEj+KlEgMXov7MsQ4Hn6KQzfjZSQQtEgVo+A5TD79n6/G5rrOXiJzih3tVt06FUc9vFy3IHCsHF9IwT/qC3LVo/Kzya8EIN3f1A3Cw1cLFWaznAP2JJCc2tROSVdfNBo+fx5aorptvN3dy+QLXlGfeLxbb+Xm7m6N3teDQarihNnMi/nsgyINtHpYVWvj05h0AnL9yAV0RdZLp641OmoJDnP3vfUjAkkIzNx006g+6cv86vvfaXhIphdoCI786OHe30IxdN39fnnmaf64RlSLmhvmsFKGkcu9LsyU1g5885oLixXN3bT4V67a8zXZvSLhuBIJ8RiSrQJAniGQVCPIEkawCQZ4gklUgyBNEsgoEeYJIVoEgTxDJKhDkCTO6g0mSpi7vn6/seGv+blIAOOSjs3uEbyak5vFGhfnsGzrj/N3sAfOnjZkqijizCgR5gkhWgSBPEMkqEOQJIlkFgjxBJKtAkCeIZBUI8gSRrAJBniCSVSDIE0SyCgR5wqz1GVe09bLVF8Ss1XBNY+XU+ozd7XREEzyxWq078+RAgNs6+9k3FOPBFfVZFRPDseZLaQGwU45wd6IfGQWbpOUK4+Q6Q4qicF9ygJflIBokPqMrZL1utPJgUyrK+bF2fmKo4KPazDWEZ9uuP3R7uL/Xj06CYr2OaxorqTJNLrg2Nl4u++yMXao+Q1bgIJuFS+pdaNMFxP7s9nKv24tWkjjcYeXc2sy1kWbbP27t6OP+Pj/FevVupZ8sKOfwog93O0blFMc9u5O4rCArCscsKOHcFeP7RkxO8YPX9rLDF6LIoOeujy6ipsDEVrefK3a0E0+lMGg0XLy6lk84CyfFyMasknWrP0TrUIyn1yxke1oz8eDKzEXSnvIGsEwoW7nIYuTXi6u5uLkn42cmxZonpUVYkbkr0cdFhirKNHr8SuYi2M/KATxKgtuMdWgkadxysqKwIdHPGo0l42fnql37FZh5ZFUxZq2Ge91erm3r5dYlmQuYjcTLYZ/dsqgaq06Loiic1dTJkwMBjikt5NXBMM94gzy2qgGDRsPAFBX2Z9s/AL5ZUcwZE0wOWWPNw3Y0aiT+fvhyCvRaEqkUxz7zDke4HBxUOnoQua+5D4dBx2vrD+Dhdg+Xb2/ndx9dTLFRz4ZPLMVlNrDLH+GkF95l++cPmrZtw8xOn+Eb1UyssVkITqXP6Bnge1XjlQULzUYazNklSmOZT6XFC3KQQ7VWyjTq9zikzMe0p5J+TtSVoEmfccYut0n2c5jWRmGWz85Vuw4rHKOzsI6vCZyJXPeZNV2fN6lAIqWM3LP6l14f/1dVMlIvOFM50pnGytY/ZsJ8bUdJkihIn+kTKYVkSmFixdJ/dns5sU5ty7HVJfyrdxBFUVhZVIDLrJ6tlxaaicopYtNU/R/LnOoznAZdRs3ELR19nF5RMkmfMbNY86e06E7FCSkyF8U6+Gm0jeeSgYzLuZUEL8lBfhZt4/JYJ90ptRzlgJLgNTnE0drphzhzoQUZ5oE+P2sdU7cv130GcPquNg57YzcFWg2fKVFr+rZEY7weiPClt5s55Z1WdoSGZh1rqv6x0e3j2B37OH9fN4NTqCbmczvKKYUjntrO8kdfZ62rkAMnWOB6InGqLGpS6jQSNr0W74R1ebzTy8oiK8YshcQz8YFPML0bVvUZ64on6zPmi5kqLVIo7EvFuNBQxcXGah5MDowk4liSKOiRuN5Uy1G6Qm5PqFX+7473c5q+dOSMOx880u/n7dAQZ1aVzNl33r2slpcOXExcUXh1MAyArMBgUuaBFfWcW+vkR02dGT04uTJV/zjZWcyW/Rfy6MoGyvQ6rplgUfggyGU7ajUSzx69mm3HHsib3hC7/JEZxXhvMMLl29u44aCp62pPZMbXrBvdXu7v8wGwcsJwoTeenKSZ2BaMsDMU5VNvNpFEwZtIcuo7rWxcXjdtrPlUWjyZ9PN0UlUxfFRrZX+tFpOkwQTspzHTqsSonFApv0TScWh64ugQjZVfp9TOtE+JclNcvQ4PKjJvyGG0BolDtNY5bdcwL/lD3NHZz33L6zFmuO6b6T4bi1Gj4dNFNrb4gnzMYcVl0LEuLY9abTUjAb509f73E2uq/lE6ZtkTyx18e3fHuM/O93acSKFBx8fL7Tzn9rPMMTo3UWEx0BWJU2kxkkwpBBMyxel16Y7E+Oa/dvPrQxZSZzVl++qMzDhZT3UVc2paRvScL8hGt5f1JXa2h4awZtJnuIo5Ob18ZzTOt3d35JSoML9Ki8/qHHxW51DXMxXjd4k+ZEUhicKeVJRjdJP9Mh/RWtmZiuDUFPJOaogKSV2f35pGj5i3xd0cqC0YSdS5bBfAO6EhfrGvmz/uV0tJls45030WllOEZZnytPfleV+Ig9L+0SOLbbwWCHNoYQEtQzESikLRGP/MXPaPscqTp31BFlk+GOVJrtsRwBNNoNdIFBp0DCVltroH+cGy8XaHoyuLub+1n4NLbTzWOcDHnar2cjCe5JQX3uMXq2v4SNnMR5qzmg0+PC1BPnLbXswaDVc3jspnP79jH/9YNbUaYLM3wOWtbrwJmf/b3c4yi4m7l2WWKc+n0qJaY2R/TQE/jrUhAUfqCqnVqB3lilgn3zO4KJZ0fEFXzM1xN48lfZjQ8D1DbsPsuWzXtW29RFIpzto9qrO4a1lN9ng57LMhOcV3dg/rM+AQu4Wvpi3xXywr4oLmbtZv34dekri2sSqjEybXWFNxXXsf74WjSBJUGfX8sj67CHu+tmNvNM7Zr+1FVtTLq+NqSlhXWcS1b7ezutjKZ6qKObmhnB+82sQhT7yJw6DjzsMWA/CHJjctoSg3vtPJje+ocf62dj/KTLkdVGakz1hpNSt/zzL1Ptdo5unJfJj/ShGr9v/frBQxn8xn/wCwLZifOZd1m3ewTegzBIL8RiSrQJAniGQVCPIEkawCQZ4gklUgyBNEsgoEeYJIVoEgTxDJKhDkCbO6g+mDRE7k/ujQbJnPmxTmm77u9/fkyfvBVT1/29HTM783suit0XmJk5rikTlxZhUI8gSRrAJBniCSVSDIE0SyCgR5gkhWgSBPEMkqEOQJIlkFgjxBJKtAkCeIZBUI8oQPVZ/hT8r8qKmTrliCKqOeWxZVUzim+NbEWFe297J1MIRJo+Ga+oqMsU57T9U+mNJ1ge5eUkOJXkd3LMHPW7oJyjKyAj+tLmetwzrp88OxclEx7AwNce7eLqIphcMdVi6qdyFJEte0unnWF0QvSdSYDFy7sAp7lnbNRbxhft/l4Zq2Xv598JKRaoMTeTES4veDHiQktBKc7ShntWl8rEgqxfd6RysJ9ssJ1hXY+WFROe5kgisH3IRSKVIofMdRymHm2W3HbEqLrmic8/Z1400kKdRpuXFRNRXGzHdJvRAOcafPg0aS0AI/KilnjWlyrDu8Hp4MBQimZJ6rWzTyek8iwZUeNz5Zxq7VclmZi3Ld5Fg/3dnGs/0BSgw6nv7YZD3Lwz1eftvSiwIUaLVcuV81+9ks7AtH+cGO1pHl2iMxfrKwgjNqc68PNqsz61g9wuX1FVwyhQYjkx7hri4Ph9kLeHrNQg6zF3BXlyfr518YDNMai7N5ZSOX11VwaZs767I3NFTy6IoGHl3RMFIx/o4eD58ttvPI8gZubqzisik+P1bFcEVjZdZ2Xdzcw5WNlTyz/0Jao3Fe8IcA+JjDyqY1C3lizULqTQZ+29mfNdZcxAPojiX412CIymkq+h1osnCPq5Z7Kmo5v9jFtd7J28Gi0XBPRe3If06tnrXphPzToJcjLDb+WFHLpaUV3Ojtm3W7jii28fcMqour23o5oczBE2sWctaCMm5oz143+CCzhY1VtWyoquXCMhdXezLv309YCri7cnIhtNu8/XzWaufe6jrOcJTwG2/mvvjlyhL+dGD2Qm8LzAbuP3gRmz+6jLMbnJz/jnrQayww8eRhS3nysKU8fugSzFoNR5c7sn5PJj5UfYb6ebVq/QllhWzxBbPH8gc5vkQt6bjGaiaQJVY2JFTXDUBQlimfQvuQi4qhL54gJMvsb7MgSRInlDl42quu/yccVnTpM94amwX3NJXhZxsP4MoWNz+vdU1SOUzEotGMnI2jSgqJqT/QnojjT8msNqpnKUmCsKLevxpOpSjVzm47Qnalxd5IjEMLCwA41F7AFm/2/jGuXakUZGnXCpOZUt3kdW5JxDnIrJZbPdBk5oVIOOPnDym24tBnHyUd5LBSmO5bBzgK6IlNbu9LA0FqLEaqzdkFYpmY1TA4mx5h4obPpkfwJEaXLdPr8EwhOZoYy6XX0ZuYHAvggpYeNMC6YhvfqyhFkiR+UFnKGXs62NjrYyiV4o9LspfrzKZiGBtr0voYdfRm6IgP9PlYXzq1RmO28Z72BnAZdSwryK1o9NZIkDv9HnwpmevLqqZc9plIkCMstpFEOL2whJ/0dfJQ0M9QKsWvnNWzatdULCswsXkgwDcqS9jsDRKWU/gSSYqyHGifDwe5w+fBJ8vc6Jy6XRNZZDDyfDjEVwqLeD4SIqKkGJRlCrXZE3M6/to1wOGlk6si/sPt4/OuyXWop+O/Rp8hSdMd43PjhoZKHlvRwL3LankjOMSjA2qV/Se8AU4oLeSFNYu4a/ECzm3uJjUL7UMu/KazH50kcdw0yTobhuQUv+308KMFuV/7rLXYuK+ynqtLK/mdf2DKZZ8JBzmyYNTlsiUc5LMFdh6uauCG8iqu8Lg/sO14Xp2TfwfCHLt9H/8OhHEadCPayUwcXmDjb9X1XFteyZ2+qds1kbOKy3gzOsTXutp4KzpEmVY3q+R42Rvkb10DnL+octzr8VSKLf2DrHc6ZvydH6o+o1SvG6m63hdPTDKS3dvr5f5+vxqrYHwsdyKJM8MR1pk+alu1Wo4psbMjHOX4Uniw38/vF6tn0/2tFmKKgi8pj8ScqYrBadCNX59YciQ2wEN9Pp71BtmwvC5jEey5itcejdMRjXPM9n0jnz1uezN/X1VPWXp9Hgr6eSykHrRuKKsaGQauMVnoHujFL8s4MpxBmuIxkigsNYyesR8PD3Jj+my8wmgmpigMpmSKtO9vO06F06DnN0vVfRaWZf45EBg3UfdgwM+jQbVdNzmrKEu3a3+zhW5P9nZlokyn41qnmliRVIrnwiFs7/Osuis4xM/faedPBzRSNKG9z3sCrLBbKMsyUTYVH6o+44giGw/3D/LtqlIe7h/k0xNEuac4iznFqX72eX+QjX0+1hfb2R6OYssQK6koBNLelURK4Xl/iMPs6jVPhVHPK8EwXzA62DcUI5ZSKB6z42eqYig36LFqtbwVjLDGaubhfj9fq1DXdasvyF1dA9y3om5EIziRuYq3pMDEv8eoQda+sYeHVzWMmw3+os3BF20OdR8k4iiKgiRJ7I5HSZCiMIvXZUskwFGW8fvEqdXxRjTC56yFtCZixEnh0Lz/7TgV3kQSh06LRpL4baeHL5ePHzp+ye7gS3a1XR1j2vVeLEpCyd6uTPhlGbtGg0aS+JPfy7G291fUu2sozre3NXPzyloaMlyWvN8hMHzI+oz/qyzhh02dPNjvp9Kg55bF2a9/1hZa2ToY5qi392HWaLhqjErhuJ3NPLqigXhK4cw9w9oHhcPsBZxY5gDgvAVOftHawz1uLxJwTX1Fdu1DjiqGyxoqOLepi2gqxdoi28hPQZe1uImnUnzjnTYA1tjMXN5YOTnQHMWbCc8PhfhnOIAOCaMkcVlJ5ch2+EZPG/dUjOpLng2HuKF8/LXfD4rKuG6gl78FfUhIXFjsmvV2zKa0eG0wzA3tfUjAwXYLlzZk12c8Fw7xZCiATlLbdXn5aLtO62pjQ5Xartu8/WwOBYkqCse2N/N5m51vFZXyZjTCb7weJGCNyczPSjNfVpy1o4VXvCF8iSSHbN3JjxsrSKYvA05dUMotzW58CZmLdql6DK0Ejx+qHkwjSZkXB4JcNYXeZCr+a/UZijx/2get/n/33hB35/xVVPhfrhRRsahgXuIc8+p77BiMCH2GQJDPiGQVCPIEkawCQZ4gklUgyBNEsgoEeYJIVoEgTxDJKhDkCSJZBYI8QSSrQJAn/Ne6bv6X7yrSGudvs7uy38GZ15RWzK+fqLZhZs+evm/eyt7v/3czQiD4H0Mkq0CQJ4hkFQjyBJGsAkGeIJJVIMgTRLIKBHmCSFaBIE8QySoQ5Akfqj7jyYEAt3X2s28oxoMr6lmZQa0wNtZ8KS3mM9bWgQCXNXWTQuErFcV8t9Y5aZnH+/zc0uJGApZZzdyyvJZXfCEu39s1ssy+SIzb9qtlXdnUZU9n27ZNnkFu7VD32d9XNczJPsumz7jP7WWj24sWCYtWwxWNFSyyZK6NPBcKkj/3DLDR7UWDxKeKrPy8zjX6wVgStjaDd0j9+/AGcJjh6SYIxsBmhHWLYOINL8EYPLUHFCClwAonLE/v49c6YI9H/e4zD866HYf5UPUZiyxGfr24moNtltxizZPSYr5iyYrCxXu6uGd1PZs/soR/9PppCkfHLdMSiXFHWy8PHrCQzYcs5aJ0HdrDiqxsOngJmw5ewn1rGjFrNHyi2JYpzJy2bbHFxG+WLuBg+9zts2z6jGNLC9m0ZiGPrWnkW1UlXNWaXZ8x23a9MhhmizfIY6sb+ef+CzmzsnT8B19qgwUOOGk1fHklFJnhrW6oLoST16j/vtU9OaBFDycsVz/zheXqMuG4+l6dQ30tRz5UfcZCs5EGszGnWPOptJivWNsDEWrNBmrMRgwaDcc6HTztGRy3zF+7BzitqnREyVCaoZTnpv5BDi+xZS17OpdtW2iZ230G2fUZtjGjkSFZmbII/GzbdZ/by7erSjGmTyglY+r9BpMy9ARhabr/ajXqGbTVB4vTSb24FFp8k1dMq1H/A5BT499z2qAg99sYP1R9xsxizZ/SYr5iuWMJKkyjO8tl1LMtEBm3TMtQDIAvvdGEDPyozsnakvE1bR/v9XPGgvEHwvlo21zEmo4NPQPc3T1AQlFG6k2/31hTtat1KM5/AhFuau/DoJE4v9bFKps6jO6IxcGkg+eaYSACZQXwsVoYSowmm0Wv/p2JUAw27YZADA5dMKMEHct/jT5jvpgPpcVcxpIVhdahGH/ZfyG37lfD+bs7CSTkkff7Ygl2h4f4ZA5D4HzktIoSnjtwMefWOrl9GhvfbEgqCoNJmQdX1nNerZOz93QwXKZXVgBPWL3W/PJK0GkmD3mnMoJZjXDiKvjqatjtgcj7K6P6oeozpmM+lRbzrc8A9ejfE42Pi+OaoFVwGQ2ssVvQayQWmI3Um420DMVYrVevGZ/o87OutBD9FKOWuW7bVMylPmMsx5QWcvGE69C5bJfLqGddiR1JklhtsyAB3rRexWXQqWdDZ7qoemMxvNUDZr16/VlgUP81T7ONCgxQbIGeADSWzHgbzPjMeqqrmH+sauQfqxo5ssjGw/1+FEVhWzCSVZ/xrwMX89wBi/jLfnXUmYw5JSqoR9XH1jTy2JpGjiq2j8R6KxiZVjGhKAoP9/s5Mn3GGVZa3LmsJuO13XzGGmaVzULrUJyOoRjxVIrHev0cOeEsvK7UzqvpSRBvPEnLUIyaMarAf/T5+bxzah3DXLZtOmYaaypa05cAAM/5QtSZxg8f57JdRxXbeG1Q1Ty2DMVIKKN6lTKDXj07+tMzwZ0BdYKprkidzQX137oM+yEUg2T6WjWWBHdQnUV+H8yqIr+iKFzW6uZFf2hEnzE8lZ9JnzHsuhn+6WazN8DlrW68CRm7TsMyi4m7l6maA82EM4WiKFza0sMLvtCIimE41lgVw9uhoXGKiUvSU/NHvNlEPJWiaFjKNIXS4oOONfZ51ucGAvyyqYuUAl+uKOYHdU5uanaz0m7mqNJC1fi+t5ut3iBaSeL7teUcm07OzqE4X3qziZc/uh+aLGdvOTZ+cmu2bds8EOCylp7RfVZg4p796ma1HYf1GX3xJOUG3Yg+4/KWHl7yh9FLEnadhksaKlg8xU83s2lXPJXivL3d7ApH0Wskzq9zcljhqJ5kYaUEW1vUSSK7CT7VAIoCT+8d/enmqEXqtW1fCN7tU3/e6RiEV9rUYbKiwAoX7JdWc7zSDns9EE5AgR6WlkO7H6UvlHFn/tfqMyYm6/8S8/nw+cRkFbw/Fq6auVfoffHQzqzJKu5gEgjyBJGsAkGeIJJVIMgTRLIKBHmCSFaBIE8QySoQ5AkiWQWCPEEkq0CQJ4hkFQjyhBndwSRJUj/Q9sGtjkDw/z21iqJkfN5xRskqEAg+PMQwWCDIE0SyCgR5gkhWgSBPEMkqEOQJIlkFgjxBJKtAkCeIZBUI8gSRrAJBniCSVSDIE/4fya+r+pzQ9eMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "L1 norm of the gradient (difference of expert and learner feature expectations)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAjC0lEQVR4nO3deXwV9b3/8dcnOTkJWUlI2AJhk0URFYiIK1jbivZWtGorat2LRe1tf732Xvtrb9ffXWqvvfe2iooWbdWKS2nV1latey0qYd8EAwiENRASErIn398fZ0hDyEY4yWRO3s/HI49zZubLmc8w8M6c73xnxpxziIhI8MX5XYCIiESHAl1EJEYo0EVEYoQCXUQkRijQRURiRMivFWdnZ7uRI0f6tXoRkUBatmzZfudcTmvLfAv0kSNHUlBQ4NfqRUQCycy2tbVMXS4iIjFCgS4iEiMU6CIiMUKBLiISIzoMdDNbaGb7zGxtG8szzOwlM1tlZuvM7ObolykiIh3pzBH648CsdpbfCax3zp0OzATuM7PwiZcmIiLHo8NAd869A5S01wRIMzMDUr229dEpT0REOisafej3AycDu4A1wNedc41R+NxWbdxTzn2vbuRARU13rUJEJJCiEegXAyuBocAZwP1mlt5aQzOba2YFZlZQXFzcpZVtLq7gF28UUqxAFxE5SjQC/WZgsYsoBLYCE1pr6Jxb4JzLd87l5+S0euVqhxJDkZJr6rrtS4CISCBFI9C3AxcBmNkgYDywJQqf26rEUDwAtQ0KdBGR5jq8l4uZPU1k9Eq2mRUB3wcSAJxzDwE/Bh43szWAAf/inNvfXQUnJugIXUSkNR0GunNuTgfLdwGfjVpFHWjqcqlv6KlViogEQuCuFD3S5VJTryN0EZHmAhfoYR2hi4i0KnCBrlEuIiKtC26gq8tFROQowQv0BG/YogJdROQowQt09aGLiLQqcIEeijPiTF0uIiItBS7QzYzEULwCXUSkhcAFOkSuFq2pU5eLiEhzgQz0cHycjtBFRFoIZKAnJijQRURaCmagh+I1ykVEpIWABnqcxqGLiLQQ2EBXl4uIyNECGujxupeLiEgLwQz0hDj1oYuItNBhoJvZQjPbZ2Zr22kz08xWmtk6M3s7uiUeS10uIiLH6swR+uPArLYWmll/YD5wmXNuInB1VCprh64UFRE5VoeB7px7Byhpp8m1wGLn3Hav/b4o1damcEhXioqItBSNPvRxQKaZvWVmy8zshrYamtlcMysws4Li4uIur1BdLiIix4pGoIeAqcDngIuBfzWzca01dM4tcM7lO+fyc3JyurzCxFC8xqGLiLQQisJnFAEHnHOHgcNm9g5wOrApCp/dKl36LyJyrGgcob8AnGdmITNLBs4CNkThc9uUGIqjtqGRxkbXnasREQmUDo/QzexpYCaQbWZFwPeBBADn3EPOuQ1m9mdgNdAIPOqca3OIYzQkhrzH0DU0khQX352rEhEJjA4D3Tk3pxNtfgr8NCoVdULTY+jqGklKUKCLiECArxQFPVdURKS5QAZ6OP5IoOvEqIjIEYEM9ESvm0VH6CIifxfIQE9PinT97z1U43MlIiK9RyADPX9kFqE4492P9/tdiohIrxHIQE9NDJE/MpO3Nnb7bWNERAIjkIEOMHP8QD7aU86esmq/SxER6RUCG+gzxkXuBfPOpq7f5EtEJJYENtAnDE5jcHoSb21St4uICAQ40M2MGeNyePfj/dQ3aDy6iEhgAx1g5vgcyqvr+fCT9p6/ISLSNwQ60GeMzyE5HM+LK3f5XYqIiO8CHejJ4RCzJg7mj2t2U61H0olIHxfoQAeYPTmX8up6jUkXkT4v8IF+7pgBZKeG+f0KdbuISN/WYaCb2UIz22dm7T60wszONLN6M7sqeuV1LBQfx+dPH8obH+2jrLKuJ1ctItKrdOYI/XFgVnsNzCwe+AnwahRqOm5fmDyM2oZGfru8yI/Vi4j0Ch0GunPuHaCjcYFfA34L+NKRPWlYBlNHZPKrJZ/QoOeMikgfdcJ96GaWC1wBPHji5XTdLeeOYtuBSt74SCdHRaRvisZJ0f8B/sU51+HlmmY218wKzKyguDi692C5eOIghmYksfCvW6P6uSIiQRGNQM8HFpnZJ8BVwHwzu7y1hs65Bc65fOdcfk5OThRW/Xeh+DhuPGckS7YcYN2usqh+tohIEJxwoDvnRjnnRjrnRgLPA3c4535/op/bFdecmUdqYogH39rsx+pFRHzVmWGLTwNLgPFmVmRmt5rZV83sq91f3vHJSE7g+ukj+OOa3WwprvC7HBGRHhXqqIFzbk5nP8w5d9MJVRMFt543isfe28pDb2/m3qtO97scEZEeE/grRVvKSUvkmjOHs3j5TnaWVvldjohIj4m5QAeYO2MMAAveVl+6iPQdMRnouf378YUpuSxauoN95XrmqIj0DTEZ6ABfnTGG2oZGnliyze9SRER6RMwG+uicVC6aMIinPtiue6WLSJ8Qs4EOcMt5Iyk5XMsLK3f6XYqISLeL6UA/e/QATh6SzsK/foJzummXiMS2mA50M+OWc0eycW85fy3c73c5IiLdKqYDHeCyM4YyMC1RtwMQkZgX84GeGIrnK+eP5m+bD7B8+0G/yxER6TYxH+gA156VR//kBOa/Weh3KSIi3aZPBHpKYohbzh3FXzbsY8PuQ36XIyLSLfpEoAPcePZI0hJD/Oy1TX6XIiLSLfpMoGckJzD3gtG8tn6v+tJFJCb1mUAHuOW8UWSnhrn3zx9pXLqIxJw+FegpiSHuuvAk3t9SoodJi0jM6cwTixaa2T4zW9vG8uvMbLWZrTGzv5lZr36qxJyz8hg7MJW7n1vFjpJKv8sREYmazhyhPw7Mamf5VmCGc24S8GNgQRTq6jaJoXgW3JBPfaNj7hPLqKyt97skEZGo6DDQnXPvACXtLP+bc+7IWcb3gWFRqq3bjMpO4RdzJvPRnkP88/Or1Z8uIjEh2n3otwJ/amuhmc01swIzKyguLo7yqo/PzPED+eeLJ/CH1bu5/w1dcCQiwdfhQ6I7y8wuJBLo57XVxjm3AK9LJj8/3/fD4q/OGM2mveXc99omBmckcXX+cL9LEhHpsqgEupmdBjwKXOKcOxCNz+wJZsZPrjyN/RU13LN4DdlpiVw4fqDfZYmIdMkJd7mYWR6wGPiycy5wl2GGQ3E8eP1UJgxO444nl7NqR6nfJYmIdElnhi0+DSwBxptZkZndamZfNbOvek2+BwwA5pvZSjMr6MZ6u0VqYojHbj6T7LQwtzy+lE/2H/a7JBGR42Z+jfDIz893BQW9K/u3FFdw1UNLSE0M8dt555CTluh3SSIiRzGzZc65/NaW9akrRTsyOieVX96YT3F5DXMeeZ9dpVV+lyQi0mkK9BYm52Xy+M1nsresmisf/BuF+8r9LklEpFMU6K04a/QAFt0+nboGx1UPLWGF7s4oIgGgQG/DxKEZLJ53Dhn9Erju0Q/4mx4yLSK9nAK9HXkDknnu9rMZnpnMTY8v5dV1e/wuSUSkTQr0DgxMT+KZ26dz8pB05j21nN+tKPK7JBGRVinQO6F/cpinbjuLs0Zl8X+eWcWT72/zuyQRkWMo0DspNTHEwpvO5KIJA/nu79fy/DIdqYtI76JAPw5JCfE8cN0Uzjspm39+fhV/WrPb75JERJoo0I9TUkI8C26YyuS8TP5x0Qre3uTvbYBFRI5QoHdBcjjS/TJ2YBrznlzGht2H/C5JRESB3lUZ/RJ47OYzSU9K4LZfFVBcXuN3SSLSxynQT8Cg9CQevTGfksO13P5EAXUNjX6XJCJ9mAL9BJ2am8G9V53G8u2l3Pvnj/wuR0T6MAV6FHz+9KHccPYIHnl3q64mFRHfdOYBFwvNbJ+ZrW1juZnZz82s0MxWm9mU6JfZ+33ncyczKTeDu59bRdHBSr/LEZE+qDNH6I8Ds9pZfgkw1vuZCzx44mUFT2IonvuvnUyjg28sWkm9+tNFpId1GOjOuXeAknaazAZ+7SLeB/qb2ZBoFRgkIwak8G9XnErBtoP87+sf+12OiPQx0ehDzwV2NJsu8ub1SbPPyOXqqcO4/81C3XJXRHpUj54UNbO5ZlZgZgXFxbF7heUPZ09kVHYKX39mJfvKq/0uR0T6iGgE+k5geLPpYd68YzjnFjjn8p1z+Tk5OVFYde+UHA4x/7oplFfXcddvVmh8uoj0iGgE+ovADd5ol+lAmXOuz9+1asLgdP7jC5P4cGuJxqeLSI8IddTAzJ4GZgLZZlYEfB9IAHDOPQS8DFwKFAKVwM3dVWzQXDF5GCu2l/LIu1uZnJfJpZP65LliEekhHQa6c25OB8sdcGfUKoox3/3cKazZWca3nlvFxKHpjBiQ4ndJIhKjdKVoNwuH4rj/2inExRnfeEbj00Wk+yjQe0Bu/378+xWTWLG9lF+8Ueh3OSISoxToPeTzpw/lyinD+MUbH1PwSXvXaYmIdI0CvQf94LJTGJaZzDeeWUl5dZ3f5YhIjFGg96C0pAT++0tnsKu0iv/3hw1+lyMiMUaB3sOmjshk3swxPFOwg7+s3+t3OSISQxToPvj6ReM4eUg69yxezYEKPbpORKJDge6DcCiO//7S6Ryqque7v19LZCi/iMiJUaD7ZMLgdL752XH8ae0eXli5y+9yRCQGKNB99JXzR5M/IpN/fWEtu8uq/C5HRAJOge6j+Djjvi+eTkOj41vPraaxUV0vItJ1CnSfjRiQwnc+dzJ/LdzPkx9s87scEQkwBXovcO20PGaMy+HfX97A1v2H/S5HRAJKgd4LmBn3XnUaiaF4vvnsSj0QQ0S6RIHeSwxKT+LfrjiVFdtL+ZfnV2soo4gctw7vhy495x9OG8rW4sPc99omxg5KY97MMX6XJCIB0qkjdDObZWYbzazQzO5pZXmemb1pZivMbLWZXRr9UvuGuz51Ev9w2hB++spHvFe43+9yRCRAOgx0M4sHHgAuAU4B5pjZKS2afRd41jk3GbgGmB/tQvsKM+MnV57GmJxUvvb0CnaWany6iHROZ47QpwGFzrktzrlaYBEwu0UbB6R77zMAXfp4AlISQzz05anU1jdyx5PLqK5r8LskEQmAzgR6LrCj2XSRN6+5HwDXew+Rfhn4WmsfZGZzzazAzAqKi4u7UG7fMSYnlf+6+nRWFZXxw5fW+V2OiARAtEa5zAEed84NAy4FnjCzYz7bObfAOZfvnMvPycmJ0qpj16xTBzNv5hie/nAHzyzd7nc5ItLLdSbQdwLDm00P8+Y1dyvwLIBzbgmQBGRHo8C+7u7Pjue8k7L51xfWsbqo1O9yRKQX60ygLwXGmtkoMwsTOen5Yos224GLAMzsZCKBrj6VKIiPM34+ZzI5qYnMe3I5JYdr/S5JRHqpDgPdOVcP3AW8AmwgMpplnZn9yMwu85r9E/AVM1sFPA3c5HRlTNRkpYSZf90Uistr+PqiFTToJl4i0grzK3fz8/NdQUGBL+sOqkUfbueexWu468KTuPvi8X6XIyI+MLNlzrn81pbp0v8AuWZaHtecOZz73yzk1XV7/C5HRHoZBXrA/OCyiZw2LIN/enYVm4sr/C5HRHoRBXrAJCXEM/+6KYRDcdz2qwJKK3WSVEQiFOgBNCwzmYe/PJWdB6uY9+Ryaut1u10RUaAHVv7ILH5y1SSWbDnA915Yq9vtiohunxtkV0wexuZ9h7n/zUJOGpjKbeeP9rskEfGRAj3gvvmZcWzZX8G/vbyBkQNS+PQpg/wuSUR8oi6XgIuLM+67+gxOHZrBPy5awbpdZX6XJCI+UaDHgH7heB69MZ+Mfgnc9NhSth+o9LskEfGBAj1GDEpP4te3TKOuoZEbFn7A/ooav0sSkR6mQI8hYwel8csbz2TPoWpueuxDKmrq/S5JRHqQAj3GTB2RyfzrprBhdzm3P1FATb2ediTSVyjQY9CnJgzi3itP473CA9z1mxW68Eikj1Cgx6grpw7jR7Mn8tr6vdz5G11NKtIXKNBj2A1nj2wK9TueUqiLxLpOBbqZzTKzjWZWaGb3tNHmi2a23szWmdlvolumdNWRUP/Lhr185dcFVNbqRKlIrOow0M0sHngAuAQ4BZhjZqe0aDMW+DZwrnNuIvCN6JcqXXXD2SP5zy9M4t2Pi7n+0Q90h0aRGNWZI/RpQKFzbotzrhZYBMxu0eYrwAPOuYMAzrl90S1TTtQ10/KYf91U1u46xNUPLWF3WZXfJYlIlHUm0HOBHc2mi7x5zY0DxpnZe2b2vpnNau2DzGyumRWYWUFxsZ4h3dNmnTqYX908jd1l1Vz14BIK9+kBGSKxJFonRUPAWGAmMAd4xMz6t2zknFvgnMt3zuXn5OREadVyPM4eM4BFc6dTU9/AF+a/x3uF+/0uSUSipDOBvhMY3mx6mDevuSLgRedcnXNuK7CJSMBLL3Rqbga/u+NcBmckcePCD/nNB9v9LklEoqAzgb4UGGtmo8wsDFwDvNiize+JHJ1jZtlEumC2RK9MibbhWcn8dt45nDc2m//7uzX86KX1NDTqIRkiQdZhoDvn6oG7gFeADcCzzrl1ZvYjM7vMa/YKcMDM1gNvAt9yzh3orqIlOtKSEnj0hnxuPnckC9/bys2PL6XksEbAiASV+fXosvz8fFdQUODLuuVYT3+4ne+/sI7s1DAPXDeFyXmZfpckIq0ws2XOufzWlulKUQFgzrQ8fjvvHOLijC8+vIRf/nWrnlMqEjAKdGkyaVgGf/za+cwYl8OP/7Cemx5byr7yar/LEpFOUqDLUTKSE3jkhnx+fPmpvL/lALP+513+sn6v32WJSCco0OUYZsaXp4/gj/94HoPTk7jt1wV8e/FqDlXX+V2aiLRDgS5tOmlgGr+78xxuv2A0zyzdwWd+9javrtvjd1ki0gYFurQrMRTPty89md/dcS6ZyWHmPrGMO59arr51kV5IgS6dcvrw/rz0tfP41sXjeW3DXj5939ss+nA7jboYSaTXUKBLpyXEx3HnhSfxp6+fz4TB6dyzeA2Xz3+PZdsO+l2aiKBAly4Yk5PKM7dP53++dAZ7D1Vz5YN/4xuLVuiWvCI+U6BLl5gZl0/O5Y1/msldF57Ey2v38Kn/epufv/4xh2v0VCQRPyjQ5YSkJIa4++LxvP7NGcwYl8PPXtvEjJ++yWPvbaWmvsHv8kT6FAW6RMXwrGQe+vJUFt9xDmMHpvHDl9Zz4U/f4tmlO6hv0MOpRXqCbs4l3eK9wv3c+8pGVu0oZVR2CvNmjOHyybmEQzqGEDkR7d2cS4Eu3cY5x6vr9/Lz1z9m3a5DDMlI4ivnj+aaacNJDof8Lk8kkBTo4ivnHG9vKmb+W5v5cGsJWSlhbjpnJNdPH0FWStjv8kQCRYEuvcaybSXMf3Mzr3+0j3AojsvPGMrN547i5CHpfpcmEggnfD90M5tlZhvNrNDM7mmn3ZVm5sys1ZWJTB2RxS9vOpPX/s8FXD11GC+t2s0l//su1yxYwivr9ugEqsgJ6PAI3cziiTz0+TNEHga9FJjjnFvfol0a8EcgDNzlnGv38FtH6AJQVlnHoqXb+fWSbewsrWJgWiJXTh3GF/OHMyo7xe/yRHqdEz1CnwYUOue2OOdqgUXA7Fba/Rj4CaC7NkmnZSQncPuMMbz9rZk8/OWpTMrN4OG3N3Phf73Flx5ewuLlRVTVajy7SGd0ZqhBLrCj2XQRcFbzBmY2BRjunPujmX2rrQ8ys7nAXIC8vLzjr1ZiVig+josnDubiiYPZe6ia55cV8WzBDr757Cq+/8I6LjtjKF+YksuUvEzMzO9yRXqlEx47ZmZxwM+Amzpq65xbACyASJfLia5bYtOg9CTuvPAk5s0YwwdbS3i2YAfPLyviqQ+2k9u/H58/fSifP30IpwxJV7iLNNOZPvSzgR845y72pr8N4Jz7D286A9gMVHh/ZDBQAlzWXj+6+tDleJRX1/Hqur28tHoX7368n4ZGx5icFC47PZdLJw3mpIGpCnfpE05o2KKZhYicFL0I2EnkpOi1zrl1bbR/C7hbJ0Wlu5QcruXlNbt5cdUuln5SgnMwKjuFz54yiM+cMojJeZnExyncJTa1F+gddrk45+rN7C7gFSAeWOicW2dmPwIKnHMvRrdckfZlpYS5fvoIrp8+gj1l1by2YS+vrtvDwve28vA7W8hODXPRhEF8duIgzj0pm6SEeL9LFukRurBIYsah6jre2ljMq+v28NbGYipq6kkMxTFtVBYzxuVwwbgcxqprRgJOV4pKn1NT38D7W0p4a+M+3tlUzObiwwAMyUji/LHZXDAuh3PHZJOpWw9IwJxQl4tIECWG4pkxLocZ43IA2FlaxTubinn342L+vHYPzxYUATBhcBrTRw9g+ugspo0aoHvLSKDpCF36nPqGRlYVlbFk837e31JCwbYSqusitxw4EvBnjcpi2qgsBqQm+lytyNHU5SLSjtr6RtbsLOX9LSW8v+UABZ8cpKoucnXqiAHJTMnLZEpefybnZTJhcBqheN3TXfyjQBc5DkcCfuknB1m+7SDLt5eyv6IGgORwPKcNy2ByXiZT8jKZlJvBoPREnWiVHqM+dJHjEA7FMXVEFlNHZAGR+7kXHaxi+faDrNheyvLtB3nknS3UN0YOhrJTE5mUm86puRmcmpvBpNwMhmQkKeSlxynQRTpgZgzPSmZ4VjKzz8gFoKq2gXW7yli7s4w1Ow+xdmcZb28qxst4BqSEvYBPZ8LgdCYMTmNUdoq6a6RbKdBFuqBfOJ78kVnkj8xqmldV28D63Ye8kI+E/V8LI7cpgMiR/0k5qUwYnMaEIWmM94J+YJq6bCQ6FOgiUdIvHM/UEZlMHZHZNK+6roHNxRV8tLucjXvL+WhPOe9t3s/iFTub2vRPTmD8oDTGDkplTI73MzCVIelJxOkWBnIcFOgi3SgpIZ6JQzOYODTjqPkHD9eycW85G/dEQn7jnkO8uHIXh6rrm9r0S4hnVHYKYwamMiYnpSnsR+ek6HYG0ioFuogPMlPC3gVNA5rmOec4cLiWzfsq2Fx8mM3FFWwurmDljoP8YfUujgxIM4PB6UmMGJDMiKwU8gYkH/U+o1+CT1slflOgi/QSZkZ2aiLZqYmc1SzoIdJ1s3W/F/L7DrPtwGG2lVTy+kf7moZUHtE/OYERWcnkDUhhRFYyw7P6kds/mdzMfgztn0RiSEf3sUqBLhIASQnxnDwknZOHpB+z7HBNPdtLKtl2oJLtJYf55EAl2w9UsnLHQV5es7vppOwROWmJ5PbvR25mP4Z5r7nNXtOSdIQfVAp0kYBLSQy1GfZ1DY3sKaum6GAVO0ur2Hmwip2llewsrWL9rkO8tn4vtfWNR/2Z9KQQQ/v3Y3BGEoPTkxiUnnTM+8zkBI3M6YUU6CIxLCE+rmkMfWsaGx37D9d4QV/V9LqrtJq9h6pZt+sQ+ytqaHlBeTgUx6D0RIak92NQRhKD0xOPCv6ctEjXUUqiIqYndepv28xmAf9L5AEXjzrn/rPF8m8CtwH1QDFwi3NuW5RrFZEoi4szBqYlMTAticl5ma22qWtoZF95DXvKIiHf9Oq9X1NUyqtl1dS0ONKHyK0SslMTyUlLJCc1key0MDmpSd5rItne/Jy0RI3ciYIOA93M4oEHgM8ARcBSM3vRObe+WbMVQL5zrtLM5gH3Al/qjoJFpGclxMdF+tj792uzjXOOsqq6ppDfX1HL/ooaissjP/srathcXMEHW2s4WFnX6mekJYaaAj4rJUxWapis5HDkfbOfzJQwA1LC+gXQis4coU8DCp1zWwDMbBEwG2gKdOfcm83avw9cH80iRaR3MzP6J4fpnxxmwuBj+/Kbq61vpORwbSTsK6rZX15L8ZHw9143F1ew9JNaDlbW0tjG/QP7JcQfG/bJYQakRl4j0wn0Tw6T0S+B/skJMf9LoDOBngvsaDZdBJzVTvtbgT+1tsDM5gJzAfLy8jpZoojEknAoLtLXnpEEZLTbtrExcuRfUlnLwcO1HDh89GtJZS0l3vvNxRUcPFzL4dqGNj8vMRTXFO79+4VJb3qf0DQ/Mi981Ly0pIRAPHg8qmcszOx6IB+Y0dpy59wCYAFEbp8bzXWLSOyJizMyvW4Wcjr3Z6rrGjjYFPR1lFXVUVpVS1lVHWWVdZRW/n1eZLRPGaVVdVS284vALNIl1D85TFpSyPtJIC0pRHpSAunNptOSEkjv13w60iYxFNftI4M6E+g7geHNpod5845iZp8GvgPMcM7VtFwuItITkhLiGZLRjyEZbff5t6a2vjES+l74NwV/ZR2lVXUcqqqjtLKW8up6yqvr2VFSSXl1PYeq66ioqT9mJFBLCfFGuhfy108fwW3njz6BrWxdZwJ9KTDWzEYRCfJrgGubNzCzycDDwCzn3L6oVyki0s3CobjIaJy043/sYGOjo6K23gv7ukjQV9U1TR/yfgkc8pZ1ZR2d0WGgO+fqzewu4BUiwxYXOufWmdmPgALn3IvAT4FU4DnvK8V259xl3VKxiEgvExdnXtdLAnB83wyiqVN96M65l4GXW8z7XrP3n45yXSIicpz0+BQRkRihQBcRiREKdBGRGKFAFxGJEQp0EZEYoUAXEYkRCnQRkRhhrqPrVbtrxWbFQFfvmZ4N7I9iOX7StvRO2pbeSdsCI5xzrd7ZxrdAPxFmVuCcy/e7jmjQtvRO2pbeSdvSPnW5iIjECAW6iEiMCGqgL/C7gCjStvRO2pbeSdvSjkD2oYuIyLGCeoQuIiItKNBFRGJE4ALdzGaZ2UYzKzSze/yu53iZ2SdmtsbMVppZgTcvy8xeM7OPvddMv+tsjZktNLN9Zra22bxWa7eIn3v7abWZTfGv8mO1sS0/MLOd3r5ZaWaXNlv2bW9bNprZxf5UfSwzG25mb5rZejNbZ2Zf9+YHbr+0sy1B3C9JZvahma3ytuWH3vxRZvaBV/MzZhb25id604Xe8pFdWrFzLjA/RJ6YtBkYDYSBVcApftd1nNvwCZDdYt69wD3e+3uAn/hdZxu1XwBMAdZ2VDtwKfAnwIDpwAd+19+JbfkBcHcrbU/x/q0lAqO8f4Pxfm+DV9sQYIr3Pg3Y5NUbuP3SzrYEcb8YkOq9TwA+8P6+nwWu8eY/BMzz3t8BPOS9vwZ4pivrDdoR+jSg0Dm3xTlXCywCZvtcUzTMBn7lvf8VcLl/pbTNOfcOUNJidlu1zwZ+7SLeB/qb2ZAeKbQT2tiWtswGFjnnapxzW4FCIv8Wfeec2+2cW+69Lwc2ALkEcL+0sy1t6c37xTnnKrzJBO/HAZ8Cnvfmt9wvR/bX88BF5j3P83gELdBzgR3Npotof4f3Rg541cyWmdlcb94g59xu7/0eYJA/pXVJW7UHdV/d5XVFLGzW9RWIbfG+pk8mcjQY6P3SYlsggPvFzOLNbCWwD3iNyDeIUudcvdekeb1N2+ItLwMGHO86gxboseA859wU4BLgTjO7oPlCF/nOFcixpEGu3fMgMAY4A9gN3OdrNcfBzFKB3wLfcM4dar4saPullW0J5H5xzjU4584AhhH55jChu9cZtEDfCQxvNj3MmxcYzrmd3us+4HdEdvTeI197vdd9/lV43NqqPXD7yjm31/tP2Ag8wt+/vvfqbTGzBCIB+JRzbrE3O5D7pbVtCep+OcI5Vwq8CZxNpIsr5C1qXm/TtnjLM4ADx7uuoAX6UmCsd6Y4TOTkwYs+19RpZpZiZmlH3gOfBdYS2YYbvWY3Ai/4U2GXtFX7i8AN3qiK6UBZsy6AXqlFX/IVRPYNRLblGm8kwihgLPBhT9fXGq+f9ZfABufcz5otCtx+aWtbArpfcsysv/e+H/AZIucE3gSu8pq13C9H9tdVwBveN6vj4/fZ4C6cPb6UyNnvzcB3/K7nOGsfTeSs/Cpg3ZH6ifSVvQ58DPwFyPK71jbqf5rIV946Iv1/t7ZVO5Gz/A94+2kNkO93/Z3Ylie8Wld7/8GGNGv/HW9bNgKX+F1/s7rOI9KdshpY6f1cGsT90s62BHG/nAas8GpeC3zPmz+ayC+dQuA5INGbn+RNF3rLR3dlvbr0X0QkRgSty0VERNqgQBcRiREKdBGRGKFAFxGJEQp0EZEYoUAXEYkRCnQRkRjx/wF3/P8FT/5EfAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "reward_weights = max_ent_irl(env, reward_weights, reward_fn, feature_fn, expert_demos, num_gradient_steps=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inferring a policy from the Learned Reward Function\n",
    "Value Iteration is used to retrieve the policy from the learned reward function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma = 0.999\n",
    "V = value_iteration(env, reward_fn=reward_fn, gamma=gamma)\n",
    "resulting_policy = deterministic_policy_from_values(\n",
    "                        V, env.P, gamma, env.num_states, env.num_actions, reward_fn=reward_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FrozenLake 8x8 environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell shows the Frozenlake8x8-v0 envrionment.\n",
    "The yellow square represents the starting state and the green one the goal state.  \n",
    "Using the actions up, down, left and right, the agent can move to any of the up to four adjacent states.\n",
    "However, the agent slips with chance of $\\frac{2}{3}$ and executes one of the two actions that are orthogonal to the original one.  \n",
    "For example, if the agent takes the action up, he may end up in states left or right of the current state with a probability of $\\frac{1}{3}$ each. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAEMElEQVR4nO3dMa5bZRRG0WOExQSCSJU3iDC0pGYAySwYSJo0SChjgCYCkYYKCReXGimGWEpO2Gat9lq6/u23dV/l73QcxwD/fV987jcAfBixQoRYIUKsECFWiBArRHx5y4sfPTodD08+1Vv5uzdvn+7caGbmvHermZm5LN5r82z3eq6ZvbP9/tMcf7w7ve/STbE+PJn54fXHeU//5vzdjzs3mpn5Zu9WMzPz6+K9Ns92r+ea2Tvb999eveTfYIgQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBBx0498v3n7dO/Ht+/1x6ln7vds93qumb2z/cPSgCcrRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIm6az5jz7M0I3Ovsw8wczx+v3ev04pe1e93zd7Z2tsv1S56sECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRNy2dXOZvc2PzS2TzY2Wud/9mePl4obPs8XPcGbv7/F8/ZInK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSJum884z96MwOakxeZUx8wczxdnJhanOlYnLZa/s7W/x8v1S56sECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiLhtPuMyazMCx8u9iYltm5MW9zpDsjlBMrP4nZ2vX/JkhQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQsRt8xnnWZ1I2LI6ZzGzOmmxOUNyerb8OW7a+s4u1y95skKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBBx29bNZdY2P1b3Zxa3Z2bueH9mcQdpe1fnqxenlfv8+er6NU9WiBArRIgVIsQKEWKFCLFChFghQqwQIVaIECtEiBUixAoRYoUIsUKEWCFCrBAhVogQK0SIFSLEChFihQixQoRYIUKsECFWiBArRJyO4/jwF59Ov83Mz5/u7cD/3sNxHF+/78JNsQKfj3+DIUKsECFWiBArRIgVIsQKEWKFCLFChFgh4i/3/1pQj866LAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_frozenlake_8x8()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reward Function Comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "feature_matrix = np.eye(env.num_states, env.num_states)\n",
    "\n",
    "irl_reward = [reward_fn(state,0) for state in range(env.num_states)]\n",
    "\n",
    "true_reward = np.zeros(64)\n",
    "true_reward[63] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following two cells show the learned reward function (top) and the true reward function (bottom).  \n",
    "\n",
    "Observations:\n",
    "* The goal state has the highest reward in both cases\n",
    "* States on the right border have smaller positive values\n",
    "* The remaining states have negative rewards\n",
    "* Negative rewards are especially large on some of the holes\n",
    "\n",
    "The reward rises the most during the training process, to match the high feature expectation of the expert for this state. States on the right border are visited often, as the expert policy takes the safe path at these states and takes actions to walk towards the wall, resulting in slow progress and many repeated state visitations. Consequently, the reward function receives positive values at these states to imitate this behaviour. Holes that received high visitation frequencies during the training process and were not visited by the expert have especially high negative values because of the mismatch in feature expectations caused by this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABJTElEQVR4nO2dd3wb5f3436ctWZLlKXnEM5NMVoGuUAqhbaBAB6WMDqDfTuigpYyyyt4FSim0pbQJtGUUKBBKCCNQVstIQiAkTryHbMuSrGWt0/3+OHlLtoyNqfp73q8Xr2DppM89d8/n7rlHd5+3pCgKAoHgvx/Nh70CAoEgN0SyCgR5gkhWgSBPEMkqEOQJIlkFgjxBJKtAkCfoZrJwkU6rVBn0H9S6fGj8L/94Jc1jrPn8FVBOzV8sAINxfs5rndE43ngy426bUbJWGfQ8sLh2btbqv4jUPO/4+USrm790TcTnL1sDwXkLBUBVo3le4nz+9T1Z3xPDYIEgTxDJKhDkCSJZBYI8QSSrQJAniGQVCPIEkawCQZ4gklUgyBNm9DvrRJqjcS7scPPuUIwfuko4vbx4yuWv7Orj795B3li5CIBruvp4LTQEQDSVwpuUeW3lwlnF+kWHm3ciURSgzmjgygUuCrTqMelJf5Db3QNIwFKzketrK7LGuqjTzbvRGGc7S/hmWeZYF3a4eT0cwarVqu2rdrLUbOLufi9P+NUfAmVFoTkW58VljRTqtHMeKyjLnNfhpieRQFbgG6VFnFBcmPE71HgxLmhTt+OPKko53Zk5XmcszjmtPfiTMvtZTFxbW4FBI9EVT/CLNjfeZJJCnZbraitwZblRpiUW56IuN7uiMc4qL+EbpZljvRaKcGNvPwlFYT+TicuqnOgkiSf8Ae72eFGAAo2GX1Q6WWIyZvyOFyMhfj/oQUJCK8HZjnJWmyb/NvpMOMifAwPIwEdNBXyvqAyAvwZ8PB4aRCuBQ6Pl/BIXLl3mdm0dCHBZUzcpFL5SUcx3a52Tlnm8z88tLW4kYJnVzC3L1fsTGp/bzhKrCYBKo4Hfr6rPGCMTs0rWQq2GC6rKeWYwNO2yOyNRArI87rXzqspH/n9jv49dQ7FZxzqvsmykQ1/b1cd9Hj/fchbTGovzu14v9y5cQKFOy0AimT2WTsN5leU8G5i+XedUlLGu0DbutdPLijk9nXTPB0L82ePLmKhzEesvA34ajQZur6vCm0xyzJ5WjnHY0Wsy3wxRqNVyYfX02/HGbg9fKy9ifZGdS9vdPDTg56tlRVzf1cdxxXaOLynk1WCYm7o9XFeX+aBn12o4r2LqtqUUhV90ufldXTV1RgO393n4hz/AF4oKqTLo+WP9AuxaLS8Gw1zW3ct9DTUZv+dAk4WPm2uRJIm98RgXe7q5r3J8IgzKMrf7+/mDq4YirY4rBty8Ho1wkMnCYoOR37tqMGk0PBz08xt/P78srZwUR1YULt7TxYY1DbiMeo57vYkjSwtZVGAaWaYlEuOOtl4ePGAhhXodnnhi5D2TVsOmg5dMue2zMathcIlex0qLielukpEVhRu6+/lpRVnWZTb5g6wvsmV9P9dYw4mqKApRRUFKL//gwCAnlzpGkqZEn/04VaLLLVYubPIH+ZxjinbNMpaERDiVQlEUIimFQq0W7RTfVaLXsbLAjE7KvpCiKLwajHB0er2PKykcSe690TiH2CwAHGK18OwUSV+i07HCPHXb/LKMXpKoMxoAOLTAwpb07UlrLGbs6f252mKiL5HI+j0WjQYp3aaokkLKcKNldzLBAp2eIq267w8yWXg+osY6wGTBpFHTYbnRRH8y88F8eyBCrdlAjdmIQaPhWKeDpz2D45b5a/cAp1WVUpjuY6VzdIvuvFyz3ufx86lCK2VZEqQrnqAznuAQq2VO4l3Q7uaT7zbTEo1zSqkDgNZYnNZYnFOa2jmpqZ0XA+E5iXWr28MJTa1c291HfMJ9i0OpFP8KhTnKnj1ZZxvr5BIHzbE4n3qvmROaWjmvogzNFImYC35Zxq7VjCS0S6+jNz0SWWo28rRfTdCnB0OEUyl8STnrd01HkVaLjMI7Q1H1OwMh3BlGPX/3DfIxa8GU37U1EuTk7hZ+1t/F+SWTh6ZVej3tyQQ9yQRJReHFSIi+DEn5eGiQQ8yZY7ljCSpMhpG/XUY97tj4g0jLUIyWSIwvvdHECW80sXUgMPJeLJXi86/v4YQ3mtjcPz7Jp2NWw+Bc6Eskecof5J6FC7Iu86Q/yLpCG9pZdrJhrqpxISsKV3b18aQ/yBeKC5EVaIvFuWfhAnrjSb62r4NHltSOHLnfDz9ylVKq05JQFC7t6uMP/T6+6ywZef/5QJj9LeasQ+C5iPVSKMxSk5G766vpiCf4VksnBxaYR0YYc825VeVc3tHLI95BDrKacep1zCaSJElcV13Bde5+EkqKwwoKJvWDf4cjPOwL8Kf67H0IYK3FxlqLjW3RCL/zD3CLs3rc+3aNlnOKyrnY04MGWGE0052Mj1vmqXCA9+Ixfu3MPgqcDllRaB2K8Zf9F+KOxfnKW/v458FLsOu1/Ouw/XAZ9bQPxTh52z6WWE3UmjNfh09kxsl6n8fPAwPqEeHOhirKpxhOAuwaitIWT/CZXS0ARFMKR+9q4allo9cTm3xBLqoun/TZmcYai1aS+JzDxh/6fHyhuBCnQccqiwm9JFFt1FNr1NMWS7DSona1vwz4edCrxrqjLrdYwyMFgyRxfJGdezy+ce8/ORjIOASey1gP+wKcWVaEJEnUGA1UGfS0xOKstIxOrtzb7+PB4e3YWD1tPIdWS0BOkVQUdJKEO5HEmf5MuV7HbQ1VAITlFJv9IexjDkZ/HfDzkF+NdXtNbm1bbTGPJOLLoTBt8dEE2hONcWlXL7+prcIx4aD3UNDPYyE11g1lVZTq1FhrTBa6B3rxyzKOCQetj1usfNxiBeDRkH/cgeY/0TB/HvTya2c1BinzoNNl1NMTHV0/dyyBy6ifsIyBNXYLeo3EArORerORlqEYq/WWkWVrzEYOdVh5Jzj0wSXryaUOTk4PLXNhrd3Ki8utI38f+HbTuERtjsYJyDJrLKZJn51pLEVRaI8nqDUaUBSFZwNh6tNDlk/brWxKn2V9SZm2WIIFY64lvlri4KsluccC6E8kKdPr0rFCLBozPArKMq+Hh7hmweTJl7mMVaHX8WoowoEFFjyJJK2xONUGw7jPnlJWxCllRTnHkiSJQ2xmnvIHWV9k59GBQY4oVPehL5mkUKtFI0n8rneAL5SMn3k+qcTBSTNs20AySYlORzyV4m6Pl2+VqqOTnniCH3d0c1W1a+SadixftDn4ok2N1ZmIoygKkiSxOx4lQYpCzeSE88lJirQ6AimZh4OD/LJU3T974lGu9/ZxY1nVyDVtJlbZLLQOxekYiuE06nms1z8y0zvMulI7/+jz8+WKYrzxJC1DMWrMBgYTSUxaDUaNBm88yRuDYb5dM/kklY1ZDYP7E0lObGonJKfQABs8fh5bUotVq+XbzZ1cvsA17ZF1k189+0jTDIFziVWq03JBu5tQKoUCLDEZuSR9xv64zcLLwTDHvNeKVoKfVpROOlIP40kk+credkIpNdZGj59HF6uxvtvSyWXVart+3tGDLymrscxGLnGOXic9EwjxUWsBlgwdZi5jfae8hAs73ZywpxUF+LGrjKIpht39iSRf3t2mbkcJ/tzv4/FldVi1Wv5vXydX1Kjxzqks45zWHm7t9rDMYuRL6aT8d3CIm3r6kYCDrBYuzjAiGtu2k5rbCQ+3bcDPIwvVtn2vrZNLK9VY93h8vBAMk0LhxGLHyNzFb/sH8CdlruzpA0AL/LUx8yOazw+F+Gc4gA4JoyRxWUnlSJ/6Rk8b91Son/uVr599cfVXh28UFlOjVw8Ct/s8DKVSXOTpAcCp03FtWdWkODqNxGWLq/ja9mZSCny5opjFBSZuanaz0m7mqNJCPlls40VvkKNeew+tJHF+YwVFeh1vDIa5cHcnEuoz1N+pKR83izwd0kzqBq+wmBTxPGt+IZ5nnRvm83nWHYFIxp0m7mASCPIEkawCQZ4gklUgyBNEsgoEeYJIVoEgTxDJKhDkCSJZBYI8QSSrQJAnzOgOplQKIpEPalXGU2Cdvx/z9Yb5PWZp9fMXL5WavxsVlFT2Z4TzHTk2P21Tpthf4swqEOQJIlkFgjxBJKtAkCeIZBUI8gSRrAJBniCSVSDIE0SyCgR5gkhWgSBPEMkqEOQJs6rB9GQgwJ98o3qD88qdLDZOrtT270iEWzz9KAqYNRKXOl0sMBiIp1Jc0utmVyxGoUbL1RUVVOozF0TOVfvws9Zudkai6CSJVRYTl9a40EsSj3kD/L53QF1XrYZLqp0szVCkDdTCa1e297J1MIRJo+Ga+gqWF2Qv6/Gdpg46YwkeX9EAwHuRKJe0uomkUlQZ9NzQWJm1NKiiKFze4uZ5fwizRuLahVWssE6OtTM0xLl7u4imFA53WLmo3oUkSdzS3sf9fT6K05X9zqkt5/ApiqUrisIVbb1s9QUxazVc01g5qW1Dcoqzmzppj8bRShKfKrLys5rR+lKbBga5rbMfCYmlFiM3LaqeGAaYvV5lmM3+ID9q6+H+RTWsyLLPctFnRFIpvtfbMfJ3v5xgXYGdHxaVz0yf4QtyRYsbGTix3MF3qseXLY2lUvysqYud4ShFOi23LK6m2mSgMxrn6G17aUgrQNbYzFzeOLnqfzZmlayVej13Vat6g5fCYa7s7eVPNZP1Btf09XJjZSX1BiMP+P38wevlUpeLRwMBbBotj9TV81QwwG2efq6uyLzyuWofjimyc13aYfPT1h4e9Kjah2qDnj8vqqFQp+WFwRCXdPTytyWZ60m9MBimNRZn88pGtoejXNrm5oH9MjtJNnsDFEwoinZhSw8/X1DOR+wFPNjv5/c9A/woS2Gxrf4QrdE4z+y/kG2hIS5p7uGhVQ2Tlru4uYcrGytZYzVzxq52XvCHWJtOym9WlHBmVemU22VcvKEYT69ZyPZ0vAdXTo53RkUJhxYWEE8pfH1XK1t9QdYW2WgdinFn1wB/XV4/vYZklnoVUMudbvD4WZUlSYfJRZ9h0WhGCqcBnN7TxlqzWrVxJvqMS5t7+NPyOlwGHV/Y0cyni20sGrN+D/T6KdRpefaARTzuGeS6tl5uXaKWWq0xGnhsTeO02yMTsxoGrzaP6g1Wmkz0JbPrDcLpqmShVIqy9FlgazjEMXY7AJ+22vh3JEK2Am65aB8A1hZakSQJSZJYWWAaqSS/v3W02PbqAnPGqu/DPOMPcnxJIZIkscZqJiCn6ItPbltYTvHHXi/frRyfKK2xOAenFRMfsxew2Ze9utcWb5ATyhxIksT+NguBpDwpVl88QUiW2d9mQZIkTihz8LT3/VUMe8Y3Gm+NzUIwQ9vMWg2HFqoV6Q0aieUFZtxxdXvd3+fnFFdRbhqSOdCr3Or2cGZ5EcZp9nsu+oyxtCfi+FMyq43q2TdnfUZoSNVnmAwYNBrWlxayZcK+2OILcEK5A4DPlNh5ZTCctV/PhDm7Zn00MMhHCzIrBy5yOvlhVxefa2lmUzDA14vUGrZ9ySTOdOLqJAmrVsvgHJUaTCgK//AG+Lh98jo9NDDIJzK8PkxvPDnOjDZWHzGWW7r6Od1VjGmCBGqRycgzacXEP30BeuLZDwy98QQVxtEO7zLq6Z2w/KT1MeroHZNgG9xe1m/by3l7uxicRmUx8bucBt2keGMJJGWe9QU5LJ28LdE4rUNxTtrZwpd3tvCCf/qz5nRk06u8G4niTiRZa7dm+eR4ptNnjOWZSJAjLJlL4E6lz+iNJagYuy8MGfZXLDmyjNqvNSOKkc5YnGO37+OrO1v4zwwVLnOSrK9HIjw6GOCs0szKgft8fm6pqmJTfQPH2u3c7Omfi7BT8suOXg6yWjhogj/ntWCEhwYGOafy/esRAHZForTH4hxVZJ/03pX1FdzX5+ML77QQllMY5kgLkolTXMU8e8AiHlvdSJlex9Wt7jn77qSi8OOmTr7mKqYmXVRcVhRao3E27FfHTQur+EVzN4FZuG6G9SqnTCjmnlIUru3u59wZ7Ke1Fhv3VdZzdWklv/MPTLnsM+EgRxZMvrYf1mecbM+9KHqulBl0vHDgYh5b3ciFdS5+vKeT4Ay23YyvWe/3+3lkUFUW3FJVhV+Wubyvl1srqyapCkCt4L4nHmNF+mJ/ndXGWd1dAJTrdPQmkzj1epKKQkiWx1VRn6n2YZjbezz4kjKX1Y8/uu4einJRu5s7G6snFcK+t9fL/f1+AFYWmHGPOXON1UcM81ZoiJ3hKEds30tSUfAmk5z2XhsbltbSaDZy9xL12r0lGuP5CddsG3oGuL83HctqomfM41fuWAKnYXwsp0E3fn1iSZzpI3fpmGW/4iziW7vaJ22PjW4v9/f50vHGt603npwUb5iLmnuoMxv5RsWov8dl0LPaalbVECYDdSYDrdE4q9KTYnOlV3lwcQ1N0Rhf36tOCHmSMt9v6eL2+qqRSab3o88AaIrHSKKw1DD+OjgXfYbTqKdn7L6IZ9hfRh098QQVxuF+naJIp0WSJIzp/r3CaqYmve1WZphQzMSMk/VEh4MTHQ51RRMJftbTzS+dLmoNk/UGADatlpAs0xaPU2sw8GokQl26CvonC6w8HgiwymzmmVCQgy2WccOSmWofAB7w+PlXMMwfFy4YZ1Prjic4u7mba2srRpQaYznFWcwp6Rnm5/1BNvb5WF9sZ3s4ik2roXyCtu/k8iJOLlfXrTMW5ztNnWxYqk5eDCSSlOh1pBSFO7oHOGlCG06rKOG0dAI85w2ywe3lmFI720JD2HTaSbHKDXqsWi1vBSOssZp5uN/P1yrUde2LJ0aW3+wNsNgyeTb+VFcxp7rU5Z/zBdno9rK+xM720BDWDG0DuLmjj6Asc2XDeP3HkcU2HvcM8sVyB95EktZonAVjXC9zqVd5ecWoWPvrezv4WWXZuNng96PPANgSCXCUZfxZNWd9htVM21Ccjmgcp0HHE55Bblo8fjb800U2Hu7zc4DNwj8HAhxaWIAkSQwkkjh0WrSSRHs0TtuEbTcds5oN/p13gEFZ5tq+tN5Agg01aoc9u6uTi5wuynQ6fuF0cm5PNxrU5L04rX44zm7n4l43x7e2YNdouKois5QXctc+XNbRS6VBz1f3qGeYIwutfL+ilN+4B/DLMr/s7FXXFXhwaV3GWGsLrWwdDHPU2/swazRcVT+6XsftbObRFZNnT8fy+ECA+9JnsqOKbHyxNLuJ/PAiK8/7gxzxZhNmrYZrF44qG47dtm9k5vCyhgrObeoimkqxtsjGWofawa9t62VXOIoEVBkNXNGYfRsCHO6wstUf4shtezFrNFw95qeDz+/Yxz9WNeKOJbijy0ODycDxbzcDasKfWF7EJwoL+Jc/xGe370WLxLk1ToqynD3nQq+SK7nqMwCeDYe4oXy8GiNnfYYkcUlDBd98tw1ZUfiys4jFFhO/au9jhdXEkcV2TnQWcU5TF0e82YRDp+VX6WT+TyDCr9r70EsSkgS/bKjEMYP2z0ifsZ/JpAwn4wfNfFaK0OhEpYi5IDk0f5UivL7pl5lLXNVzI0SejuO37+Pt0JDQZwgE+YxIVoEgTxDJKhDkCSJZBYI8QSSrQJAniGQVCPIEkawCQZ4gklUgyBNmdPuIJEGWuwrnHDk5fz/mS5r5iwXw7tvZHyWca5YszfzQ+weBpJm/G1lMxvndZ5J2ns5rUzz0Ic6sAkGeIJJVIMgTRLIKBHmCSFaBIE8QySoQ5AkiWQWCPEEkq0CQJ4hkFQjyBJGsAkGeMKsCOC2xOBd1udkVjXFWeQnfKM2sR3g1FOGm3n4UwKKRuLzSRY3RQHc8wcXdbnxJmUKtlquqXbiy6jPiXNTp5t1ojLOdJXyzLHOsCzvcvB6OjOgqrqx2stRsYlCWuajTTUc8gVGSuLzaxSLT5OJikNZndPTxQkDVZ1xdV8HyCRXhQ7LMqbtHKwm640k+X2LnggVqfaknvQF+3eNBApaYTdzYkF2T8HYywu9j/SRRsEtarrYsmLTM43Ef/4j7cSsJNhY0Yteo7Xs1EeLeuAcNElrgTGM5++myV8vLVQ1y2ntt9CWSmNJ31Ny9pIYSvY6r2nt5LV3vNppSGEgmef2AJRlj5ao86YzFOae1B39SZj+LiWtrKzBoJP4TinB1Zx97hmLcWFfJ0VNoQeZT5bLVG+Ty5m5kBb7iKuI7C8bbFu7rGWBD9wBaScKi1XDlwioWFZjYHoxwYZNa2VMBzq4p5+gp6nNNZFbJatdqOK+inGcDUxd6vrKnl1tqKmkwGvmr189dHi9XVLm4sbefYx12jnMU8loowq29Hq6qzlzwq1Cn4bzK6WMBnFNRxrrC8Tv2d31elppM3FpbRXM0zpXdvfyhYXJSALwQCNMWi/PU8ga2h6Nc1ubm/mV145axarU8Mkap8YVdLRzlUGO2RuPc5R7gviW10yomQorMb2N9XGquokyjx5/KvOwyrZmDLVYujHSMe321zsIhOlUb0SLHuC7azR26zKoPmJka5IaGSlZOSOQLxjhvNvR6eTcSzRorV+XJjd0evlZexPoiO5e2u3loQFWeVOr1XF3r4u7e6QsuzZfKRVYULt3XzZ9W1OMy6jhh2z4+XWxnUcHowfzYMgcnp6tXbhkIcGVLD/esqGexxcQj+y9EJ0n0xROsf7OJT5fYp7VMDDOrYXCJTscK8/R6BICQnBr5d1if0RyLc0iBWoT7IwVmngtmr1BeostNxZCNfbE4h6TrszaYDHQlkniyJNEz/hDHTdRnTJFwLdE43oTMQenvf8Dj5+Ty3BQTLySCHKazUqZRj+IOTeZlG7UmnJrJR3qzNKqNiDG9NiJXNUguPDEQ4JjiyUXOh8lFeaIoCq8GIxydPtAdV1I4ktxVRj1LzCZyueV4vlQu24MRak0GasyqPuOYskK2eAPjlrGNqUkdkVMje8Ss1Yxsi1hKmXZfTWRu6kBOw6WVTr7f3oVR0mDVathYr57RFpuMbAmEOLWkiGeCIcKpFP6kjEM3u5vPb3V7uKNvgEMLLPzYVYpBo2GJyciWwRAHFlh4OzJETzxBbzJJaYZE6k0kqBhTuNllUHUV2cpmbvIG+GyxfSRpWqNxAL76XhspFH5QUconCjMrILpScWQULoh0MKSkONZQxBH67AmQiVcSQf4c9zCYkrnYMrl85ri2ZVGDZKodfEFLDxpgXbGN71WUjqvp3BVL0BmPc+gUGpJc8Msy9jGdOJuqZCbkonIxajQUaDT8sVrti9lULhOLhPfGklQYx+sztgcjk+Js6B7g7i4P8ZTCxlWjI5dtgQjnNXXSFU1ww5LqnM+qME8TTBsH/NxeU8WWJQ0c57BzvVvVZ5zjLOON8BAn7mvj9fAQ5TpdTkfRqfiRq5THFtfxt8YaBuUUf+hXh1BnlhURTKX4YlMb9w74WWo2MlfPo2zyBVg/RqORRKEtFufPS2q4sb6Si9rcWRUTMgp75RgXm6u4zFzN32IDdKXiM4p/mN7GHQX1XGCu5N7Y1NqIXLmhoZLHVjRw77Ja3ggO8Wi6yv4wT3gHObrIjvYDVIO8H/5bVC6nVZbw3MFL+Hm9i9vb+0ZeX2O38M8DF/Pw/o38tqOf2AzcTjM+s/51wM9DfnXH3V4zvR7Bm0yyOxpjlUUdIn7GbuO77Wl9hl7HzTXqdUFETrElEBoZygD8ZcDPg1411h1108cCRuRGBkni+CI793jUZLVqtVxR7QLUodfRu1uoHnM2ubfPxwMePwArC0zjZFLu+KiuYiLvRaIkFYUVY65ZXHo9qwpM6CWJaqOqmGiLxVmZnvh5Iu5nc0Jt18d0Vg7QaTFJGkwSLNeZaZFjVGlm/iziCp2FW6K9BFLyyAQUzFwNAoy016rVckyJnR3hKMePkeVt8ga4uMY16XMzVZ44tFoCcoqkoqCTpKzrk4n5VLkM4zTq6IlN0GdMUVX/mLJCLtrbNen1hRYTFq2G3eEoq2yWDJ+czIyT9aQSByeVOHJe3q7VEkrJtMbi1BkNvBKOUJ9+KFadBdagkSR+7/FywgTJ01dLHHx1BrFArQJfptehKArPBkIsSqsyArKMWdKg10g85BvkwALzOMHxKeVFnJLWYTw/GOLePh/ri2yj+owsHegJb4D1E67bjnRYedwb4IulDnxJVTFRbRxNvvUGB+sNars65Bh3xvqQFYUkCnvkKMfpc1eGdKfiVEh6JElin6xqI2wTPC0zVYMkFYVAUqZYryORUnjeH+KwMcPdfUMxAskU+2dwtMxUeSJJEofYzDzlD7K+yM6jA4MckeWSYSLzqXIZZpXNQms0NqLPeLx/kJuXjJ+obBmKUW9WZ6Kf8wapS/9/RzROhVGPTpLoisZpHopRnUHlko1ZXbN6EklOam4nnFL1CBsH/DyyUNUjfK+tk0srVT3CJZVOftKh6jPsWi2/rFJnFP8TjnBrn/rzxgEWMxdWZBYOD8f6yt52QsOxPH4eXazG+m5LJ5dVq7F+3tGDLymjAEvMRi5Jqzqao3Eu7HQjAY0mI7+szq4EXGsv4IXBEOt2NmPSaLiqbvQMcvy7LeNmgZ/0Bblrgvn74/YC/hUIs/6dZjRI/Ky6fJIIa5gFWiMH6Ao4O9KGBBylL6RWq+7cyyKd/MDkokSj47G4j7/HffiUJGdHWjlQV8BZJhevJEI8m1S1EQYkzjVVZuxkI23LQQ0STymcuaeDhKKQUhQOsxdwYpljZLlN3gCfG3ONno1clSfnVJZxTmsPt3Z7WGYx8qUS9eeMt8NDnNXSTUCWeW4wxG1uD48vyzxzPV8qF50kcUljJd/Y2UJKgS85i1hcYOLm1l5W2swcWWJnQ/cAL/tD6CQJu07L9Wl9xuuDYe7s7EcnSWiAyxorKf6g9BnLzSblr43zo8/I4hT6QNAZ5vfekD175sZBmwvzWSkilZy/doVD81sposiV+Tf5uea4t/bydjAi9BkCQT4jklUgyBNEsgoEeYJIVoEgTxDJKhDkCSJZBYI8QSSrQJAniGQVCPIEkawCQZ4wM9eNRsJgmp/8nk9vynyzdPm8PJk47+hncOvcbDG8z2dw3y96a/ab9eeSqfq9OLMKBHmCSFaBIE8QySoQ5AkiWQWCPEEkq0CQJ4hkFQjyBJGsAkGeIJJVIMgTZvUrdi6aCYDHvQHu7BlAktSKhtfXV1CkGw19d+8A13X288rqheNenxRrFtoHUGsH/bqrHwlYajFxY2PmGruzjRVPpTi3uZt3IlEcOi03N1aNK5g2MdYVbb1s9QUxazVc01g5KdaQnOLspk7ao3G0ksSniqz8LF0Z/+6eAR7o86GTJIp0Wq5urKQqS6xc443lO7vb6YgmeGJ147jX/9A9wLXtvbx64OKsdYQUReHyFjfP+0OYNRLXLqxiRYYiaztDQ5y7t4toSuFwh5WL6l1IksSucJSL9nUTSaWoMuq5aVH1uALaY5mtymWYpwNBzuno4S8NNSw3T+7LP93RxjP9g5QYdGz5xH4ZY7wyEOSyXZ0kFIVivY4HDl0MwEef30mBVoNWktBKEk98bGnGz2djVsmai2YiqShc1dHLE8vrKdLpuL6zj419Ps6qVOu69sQTvBSIUGmYelVmq31ojca5q8fDX5bVTau0mG2sBzx+7DotT69ayBMDg9zQ0cevFlZn/PxWf4jWoRhPr1nI9tAQlzT38ODKhknLnVFRwqGFBcRTCl/f1cpWX5C1RTb2s5j4+4oGzFoN97m9XNfWxy2LM8eaSTyAp7wBLBmKYfXEErw0GKIyS3nWcbGicZ7ZfyHb0rEeWjU51sXNPVzZWMkaq5kzdrXzgj/E2iIbF+zt4rw6F4cUFvBAr4/fd3v4cU3mQnezVbkAhOUU9w74WZkhSYf5cnUxX68t48c7WjO+P5hIcuE7HWw4eCFVZgOe2Pg7rf52yGKKp+nr2ZjVMDgXzYSS/i8iKyiKQkhOUT5G+HN1Rx8/q8pckHl8rNlpH+7v93FKjkqL2cZ61hfihLRw6OhiO68EM6sYAJ7xBTmhzKHGslkIZohl1mo4tFAtBWrQSCwvMONO1zU+tLAAs1bdjWtsZnqnWc9c4oHacf/YM8D3Muybq9rc/KzGOa38YYt3NNb+NguBpDwpVl88QUiW2d+mlv48oczB094goGpJPmJXa+p+zGHlnwPBrLFmq3IB+HWfh9NLizBOUbXxkGIbDn32InSPdvv4rMtBlVk9W5dOUVN4pswqWbNpJsailyQuqXHy+Xdb+OSOveyLxvhSuiM/4w/iNOhYmmHoPClWFu1DJi5o6eG4nc3c3t0/kiSt0Tgt0Tgn7WrlxHdbeGEKWdJsY/UmklSkP6+TJGxaDb4sFfknxnIadPTGs5/1A0mZZ31BDiucrId4oM/PJx1T19zNNd4tHX2cXlGCacK9qlu8QZwGPcsKctlnCSqMY/qHUT8p1qRtbRztQ4ssRrakE/fJgUHcsdnfDzyscjlydzOPDwY4o1StcfzuUBR3IsknbbnVLM5GczjKYCLJia/t4XMv7eLBrlFDggSc+p8mPvfSLu5t98z4uz/wCaaEovDXfj8P71fHC6sWsths5C73AEOpFHf2DHB2Zen0XzIDsmkfZAXaonE2LKnlxsYqLmrpyaq0mG2sD4qkovDjpk6+5iqmZkJx6Ef7/ewMRzmzsmTWcd4NR2mPxVk3oXj5kJzit139/LB6+pHQXHBNYxUb3V6O276PsJxCPwcPd2RSuaQUhRvc/fzUNft2yQq8PRjhngMb2XjwIm7d66Y5rJr2Hjp0MZs+tow/H7SQP7f385o3+0ghEzMePM9UM/FeWgk4fBH/2SI7v3MPcIQjTmc8wXHvtgDqEfYL77Zy/7K6EQXGXGofnAYdqwvM6DUSC9JKi9ZonFXpCY85jaXX0RNP4DKoKoagnBpX5Huj28v9farWY6V1fKzeeBJnlmuai5p7qDMb+UbF+IR8aTDEHV0e7l1ehyHDNeZM420LRtgZivKpN5tIouBNJDn1nVYuqnfRGUvw+R3N6naJJzjh7WYeXNFAWfo7NvQMcH9vejtaTfTExvSPWGJSLKdBN35bx0b7UKPFyJ+W1wFqlfvnfeNHQ3OlcgmnUuyNxTijVdVpepIyZ7d3cWtNVcZJpqlwmfQ4DHYsOi0W4JBiK+8GhmgoMOEyjQ6Nj3YWss0f4ZDi7M7Zicw4WWeqmSjX69g3FMObSFKs1/FyIEyDycASs4mXVy8aWe6It/fy0LK6cbPBc6l9OLLIxhMDAb5Y5sCbUJUWC0z6DyTWEQ4rD3sG2d9q4SlvgENt41UMp7qKOdWlxnrOF2Sj28v6EjvbQ0NYM8QCuLmjj6Asc2XD+Erx74aHuLi5hz8srcl6HT7TeCe7ijk5vXxnNM63d3ewMZ00rx40Kk7+1JtNPLSyftxs8GkVJZyWPpg85w2ywe3lmFI720JD2HTaSbHKDXqsWi1vBSOssZp5uN/P1yrU2APxJCUGHSlF4fbOfr7qHK/lmCuVi02r5YWlC0eWO72lg3NcZTNOVIB1zkIuereTZEohoSi85Q9zZl05kaRMCrDqtESSMi96gvxw4WRX0FTMajY4F82E06Dn+5WlnLqnHZ0kUWnQcXVddgt41liz1D58wl7AS4NhPvf2PrSSxLkLyrP+TDTbWF8qc/Cz5m6O2rGXQp2WmxuyaxgPd1jZ6g9x5La9mDUarm4c3Taf37GPf6xqxB1LcEeXhwaTgePfVs9qp7qKObG8iGvb+oik1J92ACqNen67ZLJEeCbx5orDi6w87w9yxJtNmLUarl04uh2O3baPx9aosS5rqODcpi6iqRRri2ysTV93P+YZZKPbC8C6EjtfKndkjTVblUuu/GBbC694g/jiST7y7Nv8ZFEFifRcxWk1ZSyymjm81M66f+1CI8FJ1aUssZlpi8T4vzfVfZdUFI6vKOLwstyt5zBDfcaKArPy0ISfZj4o/pcfPpe0/5tt08zjPosMzu/D546a2U085cr6l95jx2BY6DMEgnxGJKtAkCeIZBUI8gSRrAJBniCSVSDIE0SyCgR5gkhWgSBPEMkqEOQJIlkFgjxhxrcbztedRVr9/+5x5LWX5+/um4MOzv7s5VyTTKTmLVaWO0U/MAzW7NU35pKp7m77380IgeB/DJGsAkGeIJJVIMgTRLIKBHmCSFaBIE8QySoQ5AkiWQWCPEEkq0CQJ4hkFQjyhNm7bnJwwgzznaYOOmMJHl+hKhR2RaJc0tpDLKWglSQurXWNlAbNFCsXb8qNbb083O8nkEyx49BlI68/1OfjmtbekYLSp1YU85UJ1fLmKtYw/xwI8IPdHTy8qoGVWdoVQeY3uBkggQysp4i1jC+kFSPFrfTQSxwNEgdQwEmoNW634Odp/GiQMCFxBk6qMWaMNdy22Xh8/uge4IF+P1pJolin5ar6SqqyVJ3PxYUUkmVO3d0+8rc7nuTzJXYuWOCkK5bgwrYevEmZQq2G6+srxxUEH0tzNM6FHW7eHYrxQ1cJp5dndt0Mc2VXH3/3DvLGSrXCZjyV4rx2N+8MxXDotNxUW0FVhlhdkRhn/Xsf/dEEkgSnNZTzrUXjK07640l+/J99tIZjGDUSNx/cyLJC1SwwGE/yk9eb2R2IIAE3H9zIQSW5lSOdnetmBk6Yzd4ABRNq2l7f0cf3K8tYm664d31nHxuW1mb8fK7elCOKbZxWUcyRb+6d9N760kIunVDK84OKFZJl/tQzwOosSTrM0/ipwsBPqSJAkp/Sysewo5sgp/gcRSzHQhKFq+hkG2HWUMBHsXEkDgDeIMS99PNzsrtuZuvxWWYx8dB+9apbp8/H9R29WT0+ubiQrFotj4yJ/4VdLRzlUDvvdZ19HFdSyAklhbwaCHNTVz/X1WeujFmo1XBBVTnPTGFaGGZnJEpAHl/g/SFvALtOy1PL6tnkC3Bjdz83ZajCqZMkLl1dy6qiAkIJmXVb3uaTzkKWpDUfALfs6mK5o4A/fmwJTYEhzn+rhQfXqhKrX2xr5QiXgz98dDHxVIqhZO63aM7SdZObEyYsp/hjr5fvTqi+L6XfAwjK8pRFmnPxpgDsb7NkrLs7E+Yi1q/a+/i/ylKM095LLRElhYJCFAUr2kk7xYiG5aidQYdEHUa8qOtjYfTe3xjT7/jZenwOtY9x6xSYcU8h+MrFhTSWlmgcb0LmoPQBbl80xqE2td2H2Cw848+eiCV6HSst07tuZEXhhu5+floxvvr+s4Mhji9SDQTrHDZeDWX2EznNBlYVqfWhrXoti+xm3EPxccvsCQzx8XL1uxbZzXSEY/RH4wQSSV7tD3JyvRrboNFQOANJ1excNzk6YW7p6ud0V/Ekb8oFNU6u6+xl7bYmru3o4yfV5VPEmt6bMh1PDQRYv20v33+vg+4pvCmzjbUzNERPLMGncqi2vg4HXcT5Ac2cRyunUYZmCuVTGJk3CbOC0SP5Zvz8mBb+goevk30bwuw9PmN50OPnk4XZS3Tm4kIayyZvgM8W20cKoi8xm3japyomnvaHCKdSWZ1BuXKfx8+nCq0j1oeRdU0mcaVfU/1EWvzy1Ae/9nCUnb4wBxSP3wbLHRY2dan1jt/0huiMxOgeitMejlFi1PHD/+zjyKd38JPX9xGeQXs+8AmmXRHVm3JUkX3Se3/p83H+Aidb1yzi/BonF7Z2f2DrcUSRjecPXMQTaxbycUcB5zZ1fSBxUorCVa1uzq/Lrdr6DsLUYuTXNHAVtfyJPiJk3oEyCr+mh6NxUM7oUyDrcHAz9ZxEKY/gnZN2TOfxedQzyM7wEGe6pr42nAmbfAHWj+kn51aX8Z9QhBPebeE/oQhOvY7ZPEPUl0jylD/IKaWOWa9rOClz5stN/HJNHbYJiX/W0koGEzKf3ryDu5vcrHAUoJUkkimFt/1hvtHoZMtRq7Botfz6vdz7/MxdNzN0wrwVGmJnOMoR2/eSVBS8ySSnvdfGhqW1PDwwyIVp3+Zni2z8oqVn3Gdn6k2ZiqIx63Wis4hr23o/kFhhOUVTJMYp77QC0B9P8u1d7dy5rGZkkmkzfp5D7fwFaPgSJUhIuDBQhp4e4jQy+Vr3D/TiwsBnyTwxdhg2/kjfpNfn0uMD8PJgmN/2eNi4tHaSW2emLqRh3otESSoKK8bY6ZwGPbc1qtfDYTnFZl8Q+xhn0H0ePw+kDyJ3Nkzvutk1FKUtnuAzu1S/UjSlcPSuFp5aVo9Tp8OdSI7xE8k4tJnPZYlUijNe3sMXaktZXz35YGXT67jlYNU2oCgKB296i9oCI0NyigqzgQPSE0rHVBdz2weZrDN1wpxcXsTJaTdOZyzOd5o6RyaRyvU6/h2McIi9gFeDEeommNFm6k2Zir54YmT5Z7xBGs3jZ0znKpZNp+U/Hxk1Wp+8s4Xz61zjZoPX4WBdelLobnp5hwhLsTBIkh7i486aw9yPhwgpzmS87sFNHFd6+W2EcTF5PefS4/NuOMrFbT38fnFmt85MXUjDPOENsH6Ctc6XTFKo1aKRJO5yD/DF0vGz5CeXOjh5BmfJtXYrLy4fHbIe+HYTTy1TJ7c+VWjlEV+ANQVmNvuDHGId7ycaRlEUfvx6M4vsZr6zOPNk5WA8iVmnwaDRcG9LH4eW2bHpddj0UGUxsjc4xEKbmRf7Bllsn3oCciyzc93k4ISZisvrKriqvZekomDUSPxyiqFjrt6Ua1vd/MMzyFAqxcde382J5UX8sKacP/V4ecYbRCdBoU7LdQun8M/MMtZMOIESfoubn9MKwEmUYUsP9s6njaupZYAEj+KlEgMXov7MsQ4Hn6KQzfjZSQQtEgVo+A5TD79n6/G5rrOXiJzih3tVt06FUc9vFy3IHCsHF9IwT/qC3LVo/Kzya8EIN3f1A3Cw1cLFWaznAP2JJCc2tROSVdfNBo+fx5aorptvN3dy+QLXlGfeLxbb+Xm7m6N3teDQarihNnMi/nsgyINtHpYVWvj05h0AnL9yAV0RdZLp641OmoJDnP3vfUjAkkIzNx006g+6cv86vvfaXhIphdoCI786OHe30IxdN39fnnmaf64RlSLmhvmsFKGkcu9LsyU1g5885oLixXN3bT4V67a8zXZvSLhuBIJ8RiSrQJAniGQVCPIEkawCQZ4gklUgyBNEsgoEeYJIVoEgTxDJKhDkCTO6g0mSpi7vn6/seGv+blIAOOSjs3uEbyak5vFGhfnsGzrj/N3sAfOnjZkqijizCgR5gkhWgSBPEMkqEOQJIlkFgjxBJKtAkCeIZBUI8gSRrAJBniCSVSDIE0SyCgR5wqz1GVe09bLVF8Ss1XBNY+XU+ozd7XREEzyxWq078+RAgNs6+9k3FOPBFfVZFRPDseZLaQGwU45wd6IfGQWbpOUK4+Q6Q4qicF9ygJflIBokPqMrZL1utPJgUyrK+bF2fmKo4KPazDWEZ9uuP3R7uL/Xj06CYr2OaxorqTJNLrg2Nl4u++yMXao+Q1bgIJuFS+pdaNMFxP7s9nKv24tWkjjcYeXc2sy1kWbbP27t6OP+Pj/FevVupZ8sKOfwog93O0blFMc9u5O4rCArCscsKOHcFeP7RkxO8YPX9rLDF6LIoOeujy6ipsDEVrefK3a0E0+lMGg0XLy6lk84CyfFyMasknWrP0TrUIyn1yxke1oz8eDKzEXSnvIGsEwoW7nIYuTXi6u5uLkn42cmxZonpUVYkbkr0cdFhirKNHr8SuYi2M/KATxKgtuMdWgkadxysqKwIdHPGo0l42fnql37FZh5ZFUxZq2Ge91erm3r5dYlmQuYjcTLYZ/dsqgaq06Loiic1dTJkwMBjikt5NXBMM94gzy2qgGDRsPAFBX2Z9s/AL5ZUcwZE0wOWWPNw3Y0aiT+fvhyCvRaEqkUxz7zDke4HBxUOnoQua+5D4dBx2vrD+Dhdg+Xb2/ndx9dTLFRz4ZPLMVlNrDLH+GkF95l++cPmrZtw8xOn+Eb1UyssVkITqXP6Bnge1XjlQULzUYazNklSmOZT6XFC3KQQ7VWyjTq9zikzMe0p5J+TtSVoEmfccYut0n2c5jWRmGWz85Vuw4rHKOzsI6vCZyJXPeZNV2fN6lAIqWM3LP6l14f/1dVMlIvOFM50pnGytY/ZsJ8bUdJkihIn+kTKYVkSmFixdJ/dns5sU5ty7HVJfyrdxBFUVhZVIDLrJ6tlxaaicopYtNU/R/LnOoznAZdRs3ELR19nF5RMkmfMbNY86e06E7FCSkyF8U6+Gm0jeeSgYzLuZUEL8lBfhZt4/JYJ90ptRzlgJLgNTnE0drphzhzoQUZ5oE+P2sdU7cv130GcPquNg57YzcFWg2fKVFr+rZEY7weiPClt5s55Z1WdoSGZh1rqv6x0e3j2B37OH9fN4NTqCbmczvKKYUjntrO8kdfZ62rkAMnWOB6InGqLGpS6jQSNr0W74R1ebzTy8oiK8YshcQz8YFPML0bVvUZ64on6zPmi5kqLVIo7EvFuNBQxcXGah5MDowk4liSKOiRuN5Uy1G6Qm5PqFX+7473c5q+dOSMOx880u/n7dAQZ1aVzNl33r2slpcOXExcUXh1MAyArMBgUuaBFfWcW+vkR02dGT04uTJV/zjZWcyW/Rfy6MoGyvQ6rplgUfggyGU7ajUSzx69mm3HHsib3hC7/JEZxXhvMMLl29u44aCp62pPZMbXrBvdXu7v8wGwcsJwoTeenKSZ2BaMsDMU5VNvNpFEwZtIcuo7rWxcXjdtrPlUWjyZ9PN0UlUxfFRrZX+tFpOkwQTspzHTqsSonFApv0TScWh64ugQjZVfp9TOtE+JclNcvQ4PKjJvyGG0BolDtNY5bdcwL/lD3NHZz33L6zFmuO6b6T4bi1Gj4dNFNrb4gnzMYcVl0LEuLY9abTUjAb509f73E2uq/lE6ZtkTyx18e3fHuM/O93acSKFBx8fL7Tzn9rPMMTo3UWEx0BWJU2kxkkwpBBMyxel16Y7E+Oa/dvPrQxZSZzVl++qMzDhZT3UVc2paRvScL8hGt5f1JXa2h4awZtJnuIo5Ob18ZzTOt3d35JSoML9Ki8/qHHxW51DXMxXjd4k+ZEUhicKeVJRjdJP9Mh/RWtmZiuDUFPJOaogKSV2f35pGj5i3xd0cqC0YSdS5bBfAO6EhfrGvmz/uV0tJls45030WllOEZZnytPfleV+Ig9L+0SOLbbwWCHNoYQEtQzESikLRGP/MXPaPscqTp31BFlk+GOVJrtsRwBNNoNdIFBp0DCVltroH+cGy8XaHoyuLub+1n4NLbTzWOcDHnar2cjCe5JQX3uMXq2v4SNnMR5qzmg0+PC1BPnLbXswaDVc3jspnP79jH/9YNbUaYLM3wOWtbrwJmf/b3c4yi4m7l2WWKc+n0qJaY2R/TQE/jrUhAUfqCqnVqB3lilgn3zO4KJZ0fEFXzM1xN48lfZjQ8D1DbsPsuWzXtW29RFIpzto9qrO4a1lN9ng57LMhOcV3dg/rM+AQu4Wvpi3xXywr4oLmbtZv34dekri2sSqjEybXWFNxXXsf74WjSBJUGfX8sj67CHu+tmNvNM7Zr+1FVtTLq+NqSlhXWcS1b7ezutjKZ6qKObmhnB+82sQhT7yJw6DjzsMWA/CHJjctoSg3vtPJje+ocf62dj/KTLkdVGakz1hpNSt/zzL1Ptdo5unJfJj/ShGr9v/frBQxn8xn/wCwLZifOZd1m3ewTegzBIL8RiSrQJAniGQVCPIEkawCQZ4gklUgyBNEsgoEeYJIVoEgTxDJKhDkCbO6g+mDRE7k/ujQbJnPmxTmm77u9/fkyfvBVT1/29HTM783suit0XmJk5rikTlxZhUI8gSRrAJBniCSVSDIE0SyCgR5gkhWgSBPEMkqEOQJIlkFgjxBJKtAkCeIZBUI8oQPVZ/hT8r8qKmTrliCKqOeWxZVUzim+NbEWFe297J1MIRJo+Ga+oqMsU57T9U+mNJ1ge5eUkOJXkd3LMHPW7oJyjKyAj+tLmetwzrp88OxclEx7AwNce7eLqIphcMdVi6qdyFJEte0unnWF0QvSdSYDFy7sAp7lnbNRbxhft/l4Zq2Xv598JKRaoMTeTES4veDHiQktBKc7ShntWl8rEgqxfd6RysJ9ssJ1hXY+WFROe5kgisH3IRSKVIofMdRymHm2W3HbEqLrmic8/Z1400kKdRpuXFRNRXGzHdJvRAOcafPg0aS0AI/KilnjWlyrDu8Hp4MBQimZJ6rWzTyek8iwZUeNz5Zxq7VclmZi3Ld5Fg/3dnGs/0BSgw6nv7YZD3Lwz1eftvSiwIUaLVcuV81+9ks7AtH+cGO1pHl2iMxfrKwgjNqc68PNqsz61g9wuX1FVwyhQYjkx7hri4Ph9kLeHrNQg6zF3BXlyfr518YDNMai7N5ZSOX11VwaZs767I3NFTy6IoGHl3RMFIx/o4eD58ttvPI8gZubqzisik+P1bFcEVjZdZ2Xdzcw5WNlTyz/0Jao3Fe8IcA+JjDyqY1C3lizULqTQZ+29mfNdZcxAPojiX412CIymkq+h1osnCPq5Z7Kmo5v9jFtd7J28Gi0XBPRe3If06tnrXphPzToJcjLDb+WFHLpaUV3Ojtm3W7jii28fcMqour23o5oczBE2sWctaCMm5oz143+CCzhY1VtWyoquXCMhdXezLv309YCri7cnIhtNu8/XzWaufe6jrOcJTwG2/mvvjlyhL+dGD2Qm8LzAbuP3gRmz+6jLMbnJz/jnrQayww8eRhS3nysKU8fugSzFoNR5c7sn5PJj5UfYb6ebVq/QllhWzxBbPH8gc5vkQt6bjGaiaQJVY2JFTXDUBQlimfQvuQi4qhL54gJMvsb7MgSRInlDl42quu/yccVnTpM94amwX3NJXhZxsP4MoWNz+vdU1SOUzEotGMnI2jSgqJqT/QnojjT8msNqpnKUmCsKLevxpOpSjVzm47Qnalxd5IjEMLCwA41F7AFm/2/jGuXakUZGnXCpOZUt3kdW5JxDnIrJZbPdBk5oVIOOPnDym24tBnHyUd5LBSmO5bBzgK6IlNbu9LA0FqLEaqzdkFYpmY1TA4mx5h4obPpkfwJEaXLdPr8EwhOZoYy6XX0ZuYHAvggpYeNMC6YhvfqyhFkiR+UFnKGXs62NjrYyiV4o9LspfrzKZiGBtr0voYdfRm6IgP9PlYXzq1RmO28Z72BnAZdSwryK1o9NZIkDv9HnwpmevLqqZc9plIkCMstpFEOL2whJ/0dfJQ0M9QKsWvnNWzatdULCswsXkgwDcqS9jsDRKWU/gSSYqyHGifDwe5w+fBJ8vc6Jy6XRNZZDDyfDjEVwqLeD4SIqKkGJRlCrXZE3M6/to1wOGlk6si/sPt4/OuyXWop+O/Rp8hSdMd43PjhoZKHlvRwL3LankjOMSjA2qV/Se8AU4oLeSFNYu4a/ECzm3uJjUL7UMu/KazH50kcdw0yTobhuQUv+308KMFuV/7rLXYuK+ynqtLK/mdf2DKZZ8JBzmyYNTlsiUc5LMFdh6uauCG8iqu8Lg/sO14Xp2TfwfCHLt9H/8OhHEadCPayUwcXmDjb9X1XFteyZ2+qds1kbOKy3gzOsTXutp4KzpEmVY3q+R42Rvkb10DnL+octzr8VSKLf2DrHc6ZvydH6o+o1SvG6m63hdPTDKS3dvr5f5+vxqrYHwsdyKJM8MR1pk+alu1Wo4psbMjHOX4Uniw38/vF6tn0/2tFmKKgi8pj8ScqYrBadCNX59YciQ2wEN9Pp71BtmwvC5jEey5itcejdMRjXPM9n0jnz1uezN/X1VPWXp9Hgr6eSykHrRuKKsaGQauMVnoHujFL8s4MpxBmuIxkigsNYyesR8PD3Jj+my8wmgmpigMpmSKtO9vO06F06DnN0vVfRaWZf45EBg3UfdgwM+jQbVdNzmrKEu3a3+zhW5P9nZlokyn41qnmliRVIrnwiFs7/Osuis4xM/faedPBzRSNKG9z3sCrLBbKMsyUTYVH6o+44giGw/3D/LtqlIe7h/k0xNEuac4iznFqX72eX+QjX0+1hfb2R6OYssQK6koBNLelURK4Xl/iMPs6jVPhVHPK8EwXzA62DcUI5ZSKB6z42eqYig36LFqtbwVjLDGaubhfj9fq1DXdasvyF1dA9y3om5EIziRuYq3pMDEv8eoQda+sYeHVzWMmw3+os3BF20OdR8k4iiKgiRJ7I5HSZCiMIvXZUskwFGW8fvEqdXxRjTC56yFtCZixEnh0Lz/7TgV3kQSh06LRpL4baeHL5ePHzp+ye7gS3a1XR1j2vVeLEpCyd6uTPhlGbtGg0aS+JPfy7G291fUu2sozre3NXPzyloaMlyWvN8hMHzI+oz/qyzhh02dPNjvp9Kg55bF2a9/1hZa2ToY5qi392HWaLhqjErhuJ3NPLqigXhK4cw9w9oHhcPsBZxY5gDgvAVOftHawz1uLxJwTX1Fdu1DjiqGyxoqOLepi2gqxdoi28hPQZe1uImnUnzjnTYA1tjMXN5YOTnQHMWbCc8PhfhnOIAOCaMkcVlJ5ch2+EZPG/dUjOpLng2HuKF8/LXfD4rKuG6gl78FfUhIXFjsmvV2zKa0eG0wzA3tfUjAwXYLlzZk12c8Fw7xZCiATlLbdXn5aLtO62pjQ5Xartu8/WwOBYkqCse2N/N5m51vFZXyZjTCb7weJGCNyczPSjNfVpy1o4VXvCF8iSSHbN3JjxsrSKYvA05dUMotzW58CZmLdql6DK0Ejx+qHkwjSZkXB4JcNYXeZCr+a/UZijx/2get/n/33hB35/xVVPhfrhRRsahgXuIc8+p77BiMCH2GQJDPiGQVCPIEkawCQZ4gklUgyBNEsgoEeYJIVoEgTxDJKhDkCSJZBYI8QSSrQJAn/Ne6bv6X7yrSGudvs7uy38GZ15RWzK+fqLZhZs+evm/eyt7v/3czQiD4H0Mkq0CQJ4hkFQjyBJGsAkGeIJJVIMgTRLIKBHmCSFaBIE8QySoQ5Akfqj7jyYEAt3X2s28oxoMr6lmZQa0wNtZ8KS3mM9bWgQCXNXWTQuErFcV8t9Y5aZnH+/zc0uJGApZZzdyyvJZXfCEu39s1ssy+SIzb9qtlXdnUZU9n27ZNnkFu7VD32d9XNczJPsumz7jP7WWj24sWCYtWwxWNFSyyZK6NPBcKkj/3DLDR7UWDxKeKrPy8zjX6wVgStjaDd0j9+/AGcJjh6SYIxsBmhHWLYOINL8EYPLUHFCClwAonLE/v49c6YI9H/e4zD866HYf5UPUZiyxGfr24moNtltxizZPSYr5iyYrCxXu6uGd1PZs/soR/9PppCkfHLdMSiXFHWy8PHrCQzYcs5aJ0HdrDiqxsOngJmw5ewn1rGjFrNHyi2JYpzJy2bbHFxG+WLuBg+9zts2z6jGNLC9m0ZiGPrWnkW1UlXNWaXZ8x23a9MhhmizfIY6sb+ef+CzmzsnT8B19qgwUOOGk1fHklFJnhrW6oLoST16j/vtU9OaBFDycsVz/zheXqMuG4+l6dQ30tRz5UfcZCs5EGszGnWPOptJivWNsDEWrNBmrMRgwaDcc6HTztGRy3zF+7BzitqnREyVCaoZTnpv5BDi+xZS17OpdtW2iZ230G2fUZtjGjkSFZmbII/GzbdZ/by7erSjGmTyglY+r9BpMy9ARhabr/ajXqGbTVB4vTSb24FFp8k1dMq1H/A5BT499z2qAg99sYP1R9xsxizZ/SYr5iuWMJKkyjO8tl1LMtEBm3TMtQDIAvvdGEDPyozsnakvE1bR/v9XPGgvEHwvlo21zEmo4NPQPc3T1AQlFG6k2/31hTtat1KM5/AhFuau/DoJE4v9bFKps6jO6IxcGkg+eaYSACZQXwsVoYSowmm0Wv/p2JUAw27YZADA5dMKMEHct/jT5jvpgPpcVcxpIVhdahGH/ZfyG37lfD+bs7CSTkkff7Ygl2h4f4ZA5D4HzktIoSnjtwMefWOrl9GhvfbEgqCoNJmQdX1nNerZOz93QwXKZXVgBPWL3W/PJK0GkmD3mnMoJZjXDiKvjqatjtgcj7K6P6oeozpmM+lRbzrc8A9ejfE42Pi+OaoFVwGQ2ssVvQayQWmI3Um420DMVYrVevGZ/o87OutBD9FKOWuW7bVMylPmMsx5QWcvGE69C5bJfLqGddiR1JklhtsyAB3rRexWXQqWdDZ7qoemMxvNUDZr16/VlgUP81T7ONCgxQbIGeADSWzHgbzPjMeqqrmH+sauQfqxo5ssjGw/1+FEVhWzCSVZ/xrwMX89wBi/jLfnXUmYw5JSqoR9XH1jTy2JpGjiq2j8R6KxiZVjGhKAoP9/s5Mn3GGVZa3LmsJuO13XzGGmaVzULrUJyOoRjxVIrHev0cOeEsvK7UzqvpSRBvPEnLUIyaMarAf/T5+bxzah3DXLZtOmYaaypa05cAAM/5QtSZxg8f57JdRxXbeG1Q1Ty2DMVIKKN6lTKDXj07+tMzwZ0BdYKprkidzQX137oM+yEUg2T6WjWWBHdQnUV+H8yqIr+iKFzW6uZFf2hEnzE8lZ9JnzHsuhn+6WazN8DlrW68CRm7TsMyi4m7l6maA82EM4WiKFza0sMLvtCIimE41lgVw9uhoXGKiUvSU/NHvNlEPJWiaFjKNIXS4oOONfZ51ucGAvyyqYuUAl+uKOYHdU5uanaz0m7mqNJC1fi+t5ut3iBaSeL7teUcm07OzqE4X3qziZc/uh+aLGdvOTZ+cmu2bds8EOCylp7RfVZg4p796ma1HYf1GX3xJOUG3Yg+4/KWHl7yh9FLEnadhksaKlg8xU83s2lXPJXivL3d7ApH0Wskzq9zcljhqJ5kYaUEW1vUSSK7CT7VAIoCT+8d/enmqEXqtW1fCN7tU3/e6RiEV9rUYbKiwAoX7JdWc7zSDns9EE5AgR6WlkO7H6UvlHFn/tfqMyYm6/8S8/nw+cRkFbw/Fq6auVfoffHQzqzJKu5gEgjyBJGsAkGeIJJVIMgTRLIKBHmCSFaBIE8QySoQ5AkiWQWCPEEkq0CQJ4hkFQjyhBndwSRJUj/Q9sGtjkDw/z21iqJkfN5xRskqEAg+PMQwWCDIE0SyCgR5gkhWgSBPEMkqEOQJIlkFgjxBJKtAkCeIZBUI8gSRrAJBniCSVSDIE/4fya+r+pzQ9eMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_reward(irl_reward)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOsAAADrCAYAAACICmHVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAH+ElEQVR4nO3dPWzcdx3H8ff5qWntOA9tHJWkSpNKtFKrghqEOtEyIiSEYEAIxMzStVNHFhhYGUHqwEIl2JGgPIgIEQQSSIBoidu0Ck6b1I5dxTnH1+GCOP9th9iuHT7f3/s1Xe7Bn/tG+sSXn3337Q0GAyT9/xu7309A0r2xrFIIyyqFsKxSCMsqhbCsUoiJndz5cG98cILJ/XouUvOu0ufG4HZvq9t2VNYTTPIdznw8z0rSJq8wv+1tvgyWQlhWKYRllUJYVimEZZVCWFYphGWVQuzo56xdf2aFV1lgHXiRI3yJ4xtu77POD7jCJVaZYZyXeJT//FLFz7jG6ywyBnyLOZ5l2qx9zqo8W9WsUbv+zrrOgB+xwMuc4ns8zu9Y4jKrG+7zS5aYZpzvc5YvcJQfcxWAy6xygSW+yxle5jQ/ZIF1tn8TvFl7z6o8W9Wsrl2X9Q1ucpJJ5phigh7PM8tFVjbc5yLLfI5ZAD7LYf7KhwwYcJEVnmeWScaYY5KTTPIGN83ax6zKs1XN6tp1Wa+xxsMjr6KPM8F1+hvuc501jt+5zzg9HmKcZda5Tn/TY6+xZtY+ZlWerWpWlwdMUohdl/U4E7w/8q/CNdY41nlHzrGRfzluM+BDbjPDGMeY3PTY42x/1mXW3rMqz1Y1q2vXZT3HIa7QZ4E+awy4wBLnOydbzzHDr1gC4Pfc4GkeokeP80xzgSX6rLNAnyv0eYJDZu1jVuXZqmZ19XbyUaTneocGo2+R+xPLvMpV1oEXmOXLPMxPeI+zHOI8M9y6c4Q9zyrTjPESjzLHFAA/5X1eZ4lx4JvM8en/cYRt1t6zKs9WJesV5nlzcHPL97PuqaySPl53K6sHTFIIyyqFsKxSCMsqhbCsUgjLKoWwrFIIyyqFsKxSCMsqhbCsUgjLKoWwrFIIyyqFsKxSCMsqhbCsUgjLKoVwfUZDWZVnq5o1yvUZjWRVnq1qVpfrMxrJqjxb1awu12c0klV5tqpZXR4wSSFcn9FIVuXZqmZ1uT6jkazKs1XN6nJ9RkNZlWerkuX6DCmE6zOkAiyrFMKySiEsqxTCskohLKsUwrJKISyrFMKySiEsqxTCskohLKsUwrJKISyrFMKySiEsqxTCskohLKsUwvUZDWVVnq1q1ijXZzSSVXm2qlldrs9oJKvybFWzulyf0UhW5dmqZnV5wCSFcH1GI1mVZ6ua1eX6jEayKs9WNavL9RkNZVWerUqW6zOkEK7PkAqwrFIIyyqFsKxSCMsqhbCsUgjLKoWwrFIIyyqFsKxSCMsqhbCsUgjLKoWwrFIIyyqFsKxSCMsqhbCsUgjXZzSUVXm2qlmjXJ/RSFbl2apmdbk+o5GsyrNVzepyfUYjWZVnq5rV5QGTFML1GY1kVZ6talaX6zMayao8W9WsLtdnNJRVebYqWa7PkEK4PkMqwLJKISyrFMKySiEsqxTCskohLKsUwrJKISyrFMKySiEsqxTCskohLKsUwrJKISyrFMKySiEsqxTCskohLKsUwl03DWVVnq1q1ih33TSSVXm2qlld7rppJKvybFWzutx100hW5dmqZnV5wCSFcNdNI1mVZ6ua1eWum0ayKs9WNavLXTcNZVWerUqWu26kEO66kQqwrFIIyyqFsKxSCMsqhbCsUgjLKoWwrFIIyyqFsKxSCMsqhbCsUgjLKoWwrFIIyyqFsKxSCMsqhbCsUgjXZzSUVXm2qlmjXJ/RSFbl2apmdbk+o5GsyrNVzepyfUYjWZVnq5rV5QGTFML1GY1kVZ6talaX6zMayao8W9WsLtdnNJRVebYqWa7PkEK4PkMqwLJKISyrFMKySiEsqxTCskohLKsUwrJKISyrFMKySiEsqxTCskohLKsUwrJKISyrFMKySiEsqxTCskohXJ/RUFbl2apmjXJ9RiNZlWermtXl+oxGsirPVjWry/UZjWRVnq1qVpcHTFII12c0klV5tqpZXa7PaCSr8mxVs7pcn9FQVuXZqmS5PkMK4foMqQDLKoWwrFIIyyqFsKxSCMsqhbCsUgjLKoWwrFKIPX1ShNSKb3z72MEEvfbOtjf5nVUKYVmlEJZVCmFZpRCWVQphWaUQllUKYVmlEK7PaCir8mwH+vf4izdh/jo8OAlfe3bz7YMB/HYe3voAJsbg80/AiTtf8+9X4eKdX3w4fwqePHH3rBGuz2gkq/JsB77S4slH4ItPbX/7W4uweBO+/il44Sz8+l/D62+uwR/ega88A199Znh59QA+5LvqyoKqWZVnO/CVFp+YhQfu8qL00nX45CPQ68HJw7B6G1ZuwdsfwOkjcGhi+PjTR4bffe+R6zMayao82/1cabGllVsw88B//zwzNbxupT+8vOH6/ubHb8MDJimE6zMayao82/1cabGl6SlYHvk/8/Kt4XXTk8PLG66f3Pz4bbg+o5GsyrPdz5UWW3r8KPzjveGp8L9vwNT4sKyPHYXLi8NDpdW14eXHjt7zl3V9RkNZlWfb76wN72f9+T/h3aXh6e6DE/CZ07B+p0dPnxyW9DeX4O3F4Y9uXjwHczPD2/+2AH98d3j5uVPwVOdHN6/9hcHCsuszpN06uDefb19WD5ikEJZVCmFZpRCWVQphWaUQllUKYVmlEJZVCmFZpRA7+g2mXq93FZjfv6cjNe/MYDDY8uMjdlRWSfePL4OlEJZVCmFZpRCWVQphWaUQllUKYVmlEJZVCmFZpRAfATZ/1awQwN/0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_reward(true_reward)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Policy Comparison\n",
    "\n",
    "The cell at the top shows the policy derived from the learned reward function, while the cell below shows the expert policy. Differences in the policies are marked in green.  \n",
    "\n",
    "Overall, the learner-policy matches the expert policy well. States with different policies are barely visited and are not impactful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "↑ → → → → → → → \n",
      "↑ ↑ ↑ ↑ ↑ ↑ → → \n",
      "↑ ↑ ← ← → ↑ → → \n",
      "\u001b[6;30;42m↑\u001b[0m \u001b[6;30;42m↑\u001b[0m \u001b[6;30;42m↑\u001b[0m \u001b[6;30;42m↑\u001b[0m ← ← → → \n",
      "← ↑ ← ← → ↓ ↑ → \n",
      "← ← ← ↓ ↑ ← ← → \n",
      "← ← ↓ ← ← \u001b[6;30;42m→\u001b[0m ← → \n",
      "← ↓ ← ← ↓ → ↓ ← \n"
     ]
    }
   ],
   "source": [
    "visualize_policy(resulting_policy.reshape((8,8)), other_policy=expert_policy.reshape(8,8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "↑ → → → → → → → \n",
      "↑ ↑ ↑ ↑ ↑ ↑ → → \n",
      "↑ ↑ ← ← → ↑ → → \n",
      "← ← ← ↓ ← ← → → \n",
      "← ↑ ← ← → ↓ ↑ → \n",
      "← ← ← ↓ ↑ ← ← → \n",
      "← ← ↓ ← ← ← ← → \n",
      "← ↓ ← ← ↓ → ↓ ← \n"
     ]
    }
   ],
   "source": [
    "visualize_policy(expert_policy.reshape(8,8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is also apperent when comparing the ratio of successful episodes to total episodes over 1000 rollouts.\n",
    "Despite the policies not matching perfectly, expert and learner success rates are comparable. Both expert and learner achieve high percentages of successful episodes over 80%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expert success rate = 0.902\n",
      "IRL success rate = 0.888\n"
     ]
    }
   ],
   "source": [
    "number_of_rollouts = 1000\n",
    "rollouts = generate_rollouts(env, expert_policy, num_rollouts=number_of_rollouts)\n",
    "print(f\"Expert success rate = {sum([rollout[-1][1] for rollout in rollouts])/number_of_rollouts}\")\n",
    "rollouts = generate_rollouts(env, resulting_policy, num_rollouts=number_of_rollouts)\n",
    "print(f\"IRL success rate = {sum([rollout[-1][1] for rollout in rollouts])/number_of_rollouts}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Animation of the resulting policy\n",
    "\n",
    "The cell below can be executed to animate the execution of the learner policy on the environment.\n",
    "It shows that the environment is highly stochastic and throws the agent off the shortest path.\n",
    "The policy has to prevent the agent from stepping into a hole, while also respecting the time limit of 200 steps to reach the goal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (Right)\n",
      "SFFFFFFF\n",
      "FFFFFFFF\n",
      "FFFHFFFF\n",
      "FFFFFHFF\n",
      "FFFHFFFF\n",
      "FHHFFFHF\n",
      "FHFFHFHF\n",
      "FFFHFFF\u001b[41mG\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "# animate resulting policy\n",
    "_ = generate_rollouts(env, resulting_policy, num_rollouts=1, render=True, render_fps=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Changes made after the presentation:\n",
    "\n",
    "After the discussion of the project presentation, i made some changes to account for the criticism:\n",
    "\n",
    "* Calculate backward pass using value iteration instead of the version from the paper\n",
    "* Choose gamma 0.999 instead of 0.99\n",
    "* More optimization steps\n",
    "* Add L1-Loss graph and expert-learner-feature expectations to visualize convergence\n",
    "\n",
    "Resulting improvements (compare to slides):\n",
    "* Better convergence of feature expectations of learner to expert (small L1 norm of gradient, visualization similar)\n",
    "* Holes mostly have high negative rewards\n",
    "* Smaller gap between learner and expert success rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}