metadata.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6941d13d",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Filtering the characteristics' file and associating it with the segmentation masks\n",
    "\n",
    "This notebook can be viewed online :\n",
    "* Live, with MyBinder [at this URL](https://mybinder.org/v2/git/https%3A%2F%2Fgit.sophia.mines-paristech.fr%2Foie%2Fbdappv.git/HEAD?labpath=metadata.ipynb)\n",
    "* Static with NbViewer [at this URL](https://nbviewer.org/urls/git.sophia.mines-paristech.fr/oie/bdappv/-/raw/master/metadata.ipynb)\n",
    "\n",
    "In this notebook, we associate the raw characteristics file with the segmentation masks. We apply three filters : \n",
    "- Internal consistency : verify that the characteristics reported are coherent, according to a certain amount of criteria\n",
    "- Ambiguity : If the installation is associated with a mask that contains more than one polygon\n",
    "- External consistency : if the relationship between the surface estimated from the polygon and the surface reported in the characteristics file do not match\n",
    "\n",
    "The variable `controlled` indicates whether the installation passes the internal consistency criteria. The variables `IGNControlled` and `GoogleControlled` indicate whether the ambiguity and external consistency criteria are met, with respect to the masks of the IGN and Google campaign respectively. \n",
    "\n",
    "The application of these filters can be reproduced using this notebook. These filters have generated the `metadata.csv` file. You can also apply your own filters and conduct your own analyses using this notebook. \n",
    "\n",
    "<b> Prerequisites </b> \n",
    "\n",
    "- Specify the path to the BDAPPV database. It is then assumed that the structure is left unchanged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6627b94b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To download the data folder from the Zenodo repository \n",
    "# Skip this cell if you downloaded and placed it in the folder.\n",
    "!wget 'https://zenodo.org/record/7358126/files/data.zip?download=1' -O 'data.zip'\n",
    "# unzip the file\n",
    "!unzip 'data.zip' \n",
    "# delete the zip file\n",
    "!rm 'data.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0e728c1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-20T12:32:30.290741Z",
     "start_time": "2022-06-20T12:32:30.264729Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import datetime\n",
    "import plotly.express as px\n",
    "import plotly.graph_objects as go\n",
    "import os\n",
    "from PIL import Image\n",
    "from lib.utils import *\n",
    "#from rasterio import features, Affine\n",
    "#import shapely\n",
    "#from shapely import geometry\n",
    "import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce3087a5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-20T12:59:56.550465Z",
     "start_time": "2022-06-20T12:59:56.530082Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "JS_FILE_TEMPLATE=\"data/replication/campaign-{campaign}/polygon-analysis.json\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4222c8b2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-20T12:32:32.847877Z",
     "start_time": "2022-06-20T12:32:32.731516Z"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table = pd.read_csv(os.path.join(os.path.join(\"data\",\"raw\"), \"raw-metadata.csv\")) # it is assumed that the notebook is executed from the repository.\n",
    "table.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32062cd6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86745cfe",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Internal consistency\n",
    "\n",
    "We consider the database and filter the seemingly erroneous installations. To do so we plot a few descriptive statistics. Tag the variables that are coherent with an attribute `controlled`. Then, additional attributes will be made for the observations conssitent with IGN and consistent with Google (both handled independently)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bea762c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cb2184d",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Surface, arrays and installed capacity edits. \n",
    "\n",
    "Single variable edits :\n",
    "- Tag observations that contain 0 array\n",
    "- Tag observations for which the surface is lower than 0 or greater than 100 000\n",
    "- Tag observations for which the instaled capacity is lower than 0\n",
    "\n",
    "Bivariate edits :\n",
    "- Tag observations for which the capacity per array is lower than a minimum capacity (default : set to 0)\n",
    "- Tag observations for which the ratio between the surface and the installed capacity is strange and adjust the ratio if necessary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca6afc85",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def single_variable_filter(df, key, minimum = -1e6, maximum = 1e6):\n",
    "    \"\"\"\n",
    "    returns an additional column indicating whether the observation \n",
    "    lies within the defined boundaries\n",
    "    \"\"\"\n",
    "    # create a new key\n",
    "    coherent_key = 'coherent_{}'.format(key)\n",
    "    \n",
    "    # condition\n",
    "    df[coherent_key] = (df[key] < maximum) & (df[key] > minimum)\n",
    "    \n",
    "    print('Number of installations filtered for the {}:'.format(key), df[df[coherent_key] == False].shape[0])\n",
    "    \n",
    "    return None\n",
    "\n",
    "# tilt : below 90 degrees\n",
    "single_variable_filter(table, 'tilt', maximum = 60.)\n",
    "\n",
    "# installed capacity : positive\n",
    "single_variable_filter(table, 'kWp', minimum = 0)\n",
    "\n",
    "# surface\n",
    "single_variable_filter(table, 'surface', minimum = 0, maximum = 1e5)\n",
    "\n",
    "# number of arrays : ositive\n",
    "single_variable_filter(table, 'countArrays', minimum = 0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9e47afe",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# filter the items for which the capacity per array is lower than min capacity. \n",
    "# set to 0 by default\n",
    "min_capacity = 0 \n",
    "table['coherent_array'] = table['kWp'] / table['countArrays'] > min_capacity\n",
    "table[table['coherent_array'] == False].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1fa5dd61",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# consider the filtered table \n",
    "table['coherent'] = table['coherent_tilt'] * table['coherent_kWp'] * table['coherent_surface'] * table['coherent_countArrays'] * table['coherent_array']\n",
    "table[table['coherent'] == False].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dd05abf",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "px.scatter(table[table[\"coherent\"] == True], 'surface', 'kWp', hover_data = ['idInstallation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b3f9aa5",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "px.histogram((table[table['coherent'] == True]['kWp'] / 1000) / table[table['coherent'] == True]['surface'], log_y = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3a05101",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# rescale the observations for which the ratio is too high (above 500)\n",
    "\n",
    "threshold = 0.3425\n",
    "\n",
    "table['rescaled'] = (table['kWp'] / 1000) / table['surface'] >= threshold\n",
    "table['rescaled_kWp'] = table['rescaled'] * 1000 * table['kWp'] + (1 - table[\"rescaled\"]) * table['kWp'] \n",
    "table[table['rescaled'] == True].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92cd409b",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table[['kWp', 'rescaled_kWp', 'rescaled']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a09cf5c4",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a54acfd0",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "px.scatter(table[table['coherent'] == True], 'surface', 'rescaled_kWp', hover_data = ['idInstallation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c3499e6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "filter_back_installations, filter_back_indices = [36075, 10851, 4959, 31875], []\n",
    "\n",
    "for index in filter_back_installations:\n",
    "    filter_back_indices.append(table[table['idInstallation'] == index].index.item())\n",
    "    \n",
    "for index in filter_back_indices:\n",
    "    kWp = table.loc[index, 'kWp']\n",
    "    table.loc[index, 'rescaled_kWp'] = kWp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55d4f78e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "px.scatter(table[table['coherent'] == True], 'surface', 'rescaled_kWp', hover_data = ['idInstallation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "032871f1",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "surface_max = 50.\n",
    "kWp_min = 378000\n",
    "\n",
    "rescale_back = table[(table['coherent'] == True) & (table['surface'] <= surface_max) & (table['rescaled_kWp'] >= kWp_min)].index\n",
    "print(rescale_back)\n",
    "for index in rescale_back:\n",
    "    kWp = table.loc[index, 'kWp']\n",
    "    table.loc[index, 'rescaled_kWp'] = kWp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54f8bb14",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "px.scatter(table[table['coherent'] == True], 'surface', 'rescaled_kWp', hover_data = ['idInstallation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2be9214b",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "surface_min = 500.\n",
    "kWp_max = 10000\n",
    "\n",
    "rescale_back = table[(table['coherent'] == True) & (table['surface'] >= surface_min) & (table['rescaled_kWp'] <= kWp_max)].index\n",
    "print(rescale_back)\n",
    "\n",
    "for index in rescale_back:\n",
    "    kWp = table.loc[index, 'kWp']\n",
    "    table.loc[index, 'rescaled_kWp'] = kWp * 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdda39ed",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table['true_kWp'] = table['rescaled_kWp'] / 1000\n",
    "px.scatter(table[table['coherent'] == True], 'surface', 'true_kWp', hover_data = ['idInstallation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07ed7892",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# remaining ids are marked as strange\n",
    "\n",
    "surface_max = 6000.\n",
    "kWp_min = 1000\n",
    "\n",
    "table['coherent_surface_kWp'] = True \n",
    "rescale_back = table[(table['coherent'] == True) & (table['surface'] <= surface_max) & (table['true_kWp'] >= kWp_min)].index\n",
    "print(rescale_back)\n",
    "\n",
    "for index in rescale_back:\n",
    "    table.loc[index, 'coherent_surface_kWp'] = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35592bb7",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table['controlled'] = table['coherent'] * table['coherent_surface_kWp']\n",
    "\n",
    "px.scatter(table[table['controlled'] == True], 'surface', 'true_kWp', hover_data = ['idInstallation'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b0c875d",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# we finally clean the table of all the intermediary attributes.\n",
    "table = table.drop(['coherent_tilt', 'coherent_kWp', 'coherent_surface',\n",
    "       'coherent_countArrays', 'coherent_array', 'coherent', 'rescaled',\n",
    "       'rescaled_kWp', 'true_kWp', 'coherent_surface_kWp'], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1727bae6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table.shape[0] - table[table['controlled'] == True].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1619f26",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ba29987",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Association with the masks\n",
    "\n",
    "We create two new variables `hasUniqueGoogleMask` and `hasUniqueIGNMask` to associate the installations with the segmentation masks. We indicate whether the masks only contains one polygon or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "986bc1af",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def has_unique_poly(campaign, ids) :\n",
    "    # Load results for the given campaign\n",
    "    js = load_js(JS_FILE_TEMPLATE.format(campaign=campaign))\n",
    "\n",
    "    # Build a dict\n",
    "    res_polys={r.id: r for r in js}\n",
    "    \n",
    "    return list(id in res_polys and len(res_polys[id].polygons) == 1 for id in ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2052592",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table[\"hasUniqueGoogleMask\"] = has_unique_poly(\"google\", table.identifiant)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67b733f6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table[\"hasUniqueIGNMask\"] = has_unique_poly(\"ign\", table.identifiant)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbf3dac6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table[table['hasUniqueGoogleMask'] == True].shape[0], table[table['hasUniqueIGNMask'] == True].shape[0] "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73fdfa50",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## External consistency \n",
    "\n",
    "Finally, we check the adequacy between the masks and the characteristics by plotting the true projected surface against the estimated projected surface. For the values that do not match, we label the installations\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a95f0ae",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83d498be",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def filter_external_consistency(campaign, table, gsd):\n",
    "    \"\"\"\n",
    "    returns the table with an attribute filtering \n",
    "    \"\"\"\n",
    "    \n",
    "    # Load AREAs\n",
    "    js = load_js(JS_FILE_TEMPLATE.format(campaign=campaign.lower()))\n",
    "    \n",
    "    # Build a dict of areas\n",
    "    areas={r.id : sum(p.area for p in r.polygons) for r in js}\n",
    "    \n",
    "    # computes the projection from the mask and from the database\n",
    "    projection_items = []\n",
    "    targets = table[table['hasUnique{}Mask'.format(campaign)] == True].index\n",
    "\n",
    "    for index in tqdm.tqdm(targets):\n",
    "\n",
    "        id = table.loc[index, 'identifiant']\n",
    "        est_proj = areas[id] * (gsd ** 2)    \n",
    "        table_proj = table.loc[index, 'surface'] * np.cos(table.loc[index,'tilt'] * np.pi / 180)\n",
    "\n",
    "        projection_items.append([table.loc[index, 'identifiant'], est_proj, table_proj])\n",
    "\n",
    "\n",
    "    projection = pd.DataFrame(projection_items, columns = ['identifiant', 'estimated', 'target'])\n",
    "    projection['ratio'] = projection['estimated'] / projection['target']\n",
    "    \n",
    "    correct_identifiants = projection[(projection['ratio'] < 1.25) & (projection['ratio'] > 0.75)]['identifiant'].values\n",
    "\n",
    "    table['{}Controlled'.format(campaign)] = False\n",
    "    \n",
    "    for identifiant in correct_identifiants:\n",
    "        \n",
    "        # index \n",
    "        index = table[table['identifiant'] == identifiant].index        \n",
    "        # change the value depending on whether the\n",
    "        # observation is also controlled\n",
    "        \n",
    "        table.loc[index, '{}Controlled'.format(campaign)] = table.loc[index, 'controlled'] * True\n",
    "        \n",
    "    # return the projection table to generate the plots\n",
    "        \n",
    "    return projection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c52c0e7",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "projection_ign = filter_external_consistency('IGN', table, 0.2)\n",
    "projection_google = filter_external_consistency('Google', table, 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5aff6c62",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table[table['IGNControlled'] == True].shape[0], table[table['GoogleControlled'] == True].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7ddb4b6",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebdf9729",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "Throughout this quality control, we added indicator variables at three levels : \n",
    "- `controlled` : to indicate whether a characteristic is coherent,\n",
    "- `{provider}Controlled` to indicate whether the characteristics is unique and abides by external consistency\n",
    "\n",
    "Using these filters, we have 3127 installations attached and coherent with a IGN mask and 7753 installations coherent with a Google mask. Setting aside the masks, we have filtered 645 incoherent installations.\n",
    "\n",
    "The table below displays the remaining number of installations after each filter is applied. The final column indicates the total count of filtered characteristics that are unambiguously with a mask.\n",
    "\n",
    "|| Google | IGN | <i> Removed Google (%)  / Removed IGN (%) </i> |\n",
    "|---|---|---|---|\n",
    "Raw | 28408 | 28408 | <i> 0 (0) / 0 (0) </i> |\n",
    "Internal consistency | 27780 | 27780 | <i> 628 (2.21%) / 628 (2.21%) </i> |\n",
    "Mask uniqueness | 10523 | 5883 | <i> 17257 (62.12%) / 21897 (78.82%) </i> |\n",
    "External consistency | 8019 | 3658 | <i> 2504 (23.80%) / 2225 (37.82%) </i> |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "453ba6cd",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "table = table.drop([\"hasUniqueGoogleMask\", 'hasUniqueIGNMask'], axis = 1)\n",
    "table.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76bc982d",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Export the file\n",
    "# table.to_csv(os.path.join(os.path.join(\"data\",\"validation\"), \"metadata.csv\"), index = None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b50faa5",
   "metadata": {},
   "source": [
    "## Generation of the plots and statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "500b69bc",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# plot as an illustration the estimated installed capacity against the true installed capacity\n",
    "\n",
    "correct_ign = projection_ign[(projection_ign['ratio'] < 1.25) & (projection_ign['ratio'] > 0.75)]['identifiant'].values\n",
    "projection_ign['correct'] = projection_ign['identifiant'].isin(correct_ign)\n",
    "\n",
    "correct_identifiants = projection_google[(projection_google['ratio'] < 1.25) & (projection_google['ratio'] > 0.75)]['identifiant'].values\n",
    "projection_google['correct'] = projection_google['identifiant'].isin(correct_identifiants)\n",
    "\n",
    "# for index in range(projection_google.shape[0]):\n",
    "#    \n",
    "#    # get the estimation and the true value\n",
    "#    estimation, target = projection_google.loc[index, 'estimated'], projection_google.loc[index, 'target']\n",
    "#    distance = np.linalg.norm(estimation - target) #compute the element wise distance\n",
    "#    projection_google.loc[index, 'distance'] = distance\n",
    "\n",
    "for i in range(projection_google.shape[0]):\n",
    "    projection_google.loc[i, 'label'] = \"Filtered\" if projection_google.loc[i,'correct'] == True else \"Not filtered\"\n",
    "\n",
    "for i in range(projection_ign.shape[0]):\n",
    "    projection_ign.loc[i, 'label'] = \"Filtered\" if projection_ign.loc[i,'correct'] == True else \"Not filtered\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90a1628e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Correlation coefficients\n",
    "\n",
    "targets = projection_google[\"target\"].values\n",
    "preds = projection_google['estimated'].values\n",
    "\n",
    "\n",
    "filtered_targets = projection_google[projection_google[\"correct\"] == True][\"target\"].values\n",
    "filtered = projection_google[projection_google[\"correct\"] == True][\"estimated\"].values\n",
    "targets.shape, preds.shape\n",
    "\n",
    "np.corrcoef(targets, preds), np.corrcoef(filtered_targets, filtered)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "855a5aef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Correlation coefficients (IGN)\n",
    "\n",
    "targets = projection_ign[\"target\"].values\n",
    "preds = projection_ign['estimated'].values\n",
    "\n",
    "\n",
    "filtered_targets = projection_ign[projection_ign[\"correct\"] == True][\"target\"].values\n",
    "filtered = projection_ign[projection_ign[\"correct\"] == True][\"estimated\"].values\n",
    "targets.shape, preds.shape\n",
    "\n",
    "np.corrcoef(targets, preds), np.corrcoef(filtered_targets, filtered)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6fa01ab",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set_style(\"dark\")\n",
    "\n",
    "\n",
    "with open(\"data/validation/campaign-google/click-analysis-thres=1.0.json\", \"r\") as read_file: # considérer le jeu de données complet\n",
    "    dataPoint = json.load(read_file)\n",
    "    \n",
    "clicks = list(sum([x['clicks'] for x in dataPoint], [])) # some images have more than one click\n",
    "ListPerfClick = np.array([x['score'] for x in clicks])\n",
    "\n",
    "with open(\"data/validation/campaign-google/polygon-analysis-thres=1.0.json\", \"r\") as read_file:\n",
    "    dataPoly= json.load(read_file)\n",
    "    \n",
    "# extract the number of actors rom the raw data\n",
    "with open(\"data/raw/input-google.json\", \"r\") as read_file:\n",
    "    dataRaw= json.load(read_file)\n",
    "\n",
    "# create a dictionnary with the info on the polygons\n",
    "\n",
    "stats = {\n",
    "    item['id'] : dict( \n",
    "        id=item['id'],\n",
    "        nb_poly = len(item['polygons']),\n",
    "        nb_clicks = len(item['clicks']),\n",
    "        nb_poly_actors = len(set(poly['action']['actorId'] for poly in item['polygons'])))\n",
    "    for item in dataRaw\n",
    "}\n",
    "    \n",
    "# extract the scores and convert into relative scores by dividing by the number of actors.\n",
    "    \n",
    "items = [(x['polygons'], stats[x['id']]['nb_poly_actors']) for x in dataPoly] # extracts the polygons and the number of actors\n",
    "scores  = [[z['score'] / item[1] for z in item[0]] for item in items] # compute the relative PAC for each polygon\n",
    "# convert as a unidimensional np.array\n",
    "ListPerfPoly = np.array(list(sum(scores, [])))\n",
    "\n",
    "#%%\n",
    "fig,ax=plt.subplots(1,3,figsize=[23,6])\n",
    "\n",
    "plt.subplots_adjust(left=0.1,\n",
    "                    bottom=0.1, \n",
    "                    right=0.9, \n",
    "                    top=0.9, \n",
    "                    wspace=0.4, \n",
    "                    hspace=0.4)\n",
    "\n",
    "t=ax[0].hist(ListPerfClick,np.arange(0,15,1))\n",
    "values=t[0]\n",
    "base=t[1]\n",
    "plt.ylabel(\"Count\")\n",
    "ax_bisPoint = ax[0].twinx()\n",
    "values = np.append(values,0)\n",
    "ax_bisPoint.plot( base, 100*np.cumsum(values)/ np.cumsum(values)[-1], color='darkorange', marker='o', linestyle='-', markersize = 1, label = \"Cumulative Histogram\" )\n",
    "ax_bisPoint.set_ylabel(\"proportion [%]\")\n",
    "ax[0].set_ylabel(\"Count [-]\")\n",
    "ax[0].set_xlabel(\"Pixel annotation consensus (PAC) [-]\")\n",
    "ax_bisPoint.set_ylim([0,100])\n",
    "ax[0].set_title('Distribution of the PAC for image classification \\n (phase 1)')\n",
    "ax_bisPoint.legend()\n",
    "\n",
    "t=ax[1].hist(ListPerfPoly)\n",
    "values=t[0]\n",
    "base=t[1]\n",
    "ax_bisPoly = ax[1].twinx()\n",
    "values = np.append(values,0)\n",
    "ax_bisPoly.plot( base, 100*np.cumsum(values)/ np.cumsum(values)[-1], color='darkorange', marker='o', linestyle='-', markersize = 1, label = \"Cumulative Histogram\" )\n",
    "ax_bisPoly.set_ylabel(\"proportion [%]\")\n",
    "ax[1].set_ylabel(\"Count [-]\")\n",
    "ax[1].set_xlabel(\"Relative pixel annotation consensus (PAC) [-]\")\n",
    "ax_bisPoint.set_ylim([0,100])\n",
    "ax[1].set_title('Distribution of the relative PAC for polygon annotations \\n (phase 2)')\n",
    "ax_bisPoint.legend()\n",
    "\n",
    "#ax[2].set_title('Analysis of the comparison between \\n annotation and metadata')\n",
    "\n",
    "ax[2].scatter(projection_google[projection_google[\"correct\"] == False]['estimated'], projection_google[projection_google[\"correct\"] == False]['target'], color = \"blue\", alpha = 0.1, label = \"Discarded installations\")\n",
    "ax[2].scatter(projection_ign[projection_ign[\"correct\"] == False]['estimated'], projection_ign[projection_ign[\"correct\"] == False]['target'], color = \"blue\", alpha = 0.1)\n",
    "\n",
    "ax[2].plot(projection_google['target'], projection_google['target'], label = \"Perfect association\", color = \"black\", alpha = 0.5)\n",
    "\n",
    "ax[2].scatter(projection_google[projection_google[\"correct\"] == True]['estimated'], projection_google[projection_google[\"correct\"] == True]['target'], color = \"red\", label = \"Filtered installations\")\n",
    "ax[2].scatter(projection_ign[projection_ign[\"correct\"] == True]['estimated'], projection_ign[projection_ign[\"correct\"] == True]['target'], color = \"red\")\n",
    "\n",
    "ax[2].set_title(\"Correlation between the computed and recorded projected surface\")\n",
    "ax[2].set_xlabel(\"Projected surface reported in the registry [m²]\")\n",
    "ax[2].set_ylabel('Projected surface computed from the mask [m²]')\n",
    "ax[2].legend()\n",
    "\n",
    "plt.savefig('validation.pdf')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "163382b9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "bdappv",
   "language": "python",
   "name": "bdappv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}