hands-on.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands-on\n",
    "\n",
    "In this notebook, we extract the characteristics of a `.geojson` file obtained from DeepPVMapper. This file corresponds to the outputs of the example provided in the hands-on of DeepPVMapper's repository, accessible at this URL: [https://github.com/gabrielkasmi/dsfrance/blob/main/notebooks/hands-on.ipynb](https://github.com/gabrielkasmi/dsfrance/blob/main/notebooks/hands-on.ipynb).\n",
    "\n",
    "We work with the raw polygon file `arrays_69.geojson`. You can either generate your own file following the instruction on DeepPVMapper's repository, or use the file supplied in the folder `hands-on` on our Zenodo repository. \n",
    "\n",
    "## Imputation of the parameters\n",
    "\n",
    "`PyPVRoof` admits two ways to input the necessary parameters: either with a `config.yml` file or by passing a dictionnary of parameters as input. The parameter names are the same. In this notebook, we will pass a dictionnary with the parameters as input. The parameters can be categorized as follows:\n",
    "\n",
    "* General parameters:\n",
    "    * `has-data` (boolean). Specifies whether auxiliary data is accessible\n",
    "    * `has-dem` (boolean). Specifies whether surface models (as `.geotiff`) are accessible\n",
    "    * `tilt-method` (str). Specifies the tilt estimtation method. Can be `lut`, `theil-sen` or `constant`.\n",
    "    * `azimuth-method` (str). Specifies the azimuth estimation method. Can be `bounding-box` or `constant`.\n",
    "    * `regression-type` (str). Specifies the azimuth estimation method. Can be `bounding-box` or `constant`.\n",
    "    * `output-name` (str). Specifies the name of the output file. \n",
    "    * `data_dir` (str). Specifies the directory of the source data\n",
    "\n",
    "If you do not wish to specify some parameters, you can pass an empty string `''` for these fields.\n",
    "\n",
    "* Parameters related to a method:\n",
    "    *  If `tilt-method == 'constant'`:\n",
    "        *  `constant-tilt` (int or float). The value taken as a constant tilt.\n",
    "    * If `tilt-method == \"lut\"`. In this case, it is required that you have auxiliary data. So `has-data` should be `True`\n",
    "        * `data-directory` (str). The directory where the auxiliary file is located.\n",
    "        * `data-name` (str). The name of the auxiliary file (in the auxiliary directory).\n",
    "        * `latitude_var` (str). The name of the field corresponding to the latitude in the raw auxiliary file.\n",
    "        * `longitude_var`  (str). The name of the field corresponding to the latitude in the raw auxiliary file.\n",
    "        * `ic_var`  (str). The name of the field corresponding to the latitude in the raw auxiliary file.\n",
    "        * `surface_var`  (str). The name of the field corresponding to the latitude in the raw auxiliary file.\n",
    "        * `tilt_var` (str). The name of the field corresponding to the latitude in the raw auxiliary file.\n",
    "        * `lut-steps` (int). The number of steps used to generate the LUT. The LUT will be a `np.ndarray` of size `(lut-steps, lut-steps)`.\n",
    "        * `regression-clusters` (int). The number of clusters for the regression.\n",
    "    * If `tilt-method == \"theil-sen\"` or `azimuth-method == \"theil-sen\"`. In this case, it is required that you have auxiliary data. So `has-dem` should be `True`\n",
    "        * `conversion` (str or None): a string separated by a comma (,) indicating the coordinates system of the polygons and that of the raster. If both coordinates systems are the same, then leave it as `None`\n",
    "        * `M` and `N` (int or None). Number of subsamples and max subpopulation parameters for the Theil-Sen estimation. If left as empty, default parameters are used.\n",
    "        * `offset`: the offset between the polygon edges and the mask edges when computing the mask from the polygons. Also used to extract an array from the DEM raster.\n",
    "    * If `azimuth-method == \"bounding-box\"`:\n",
    "        * No additional parameter required.\n",
    "    *  If `regression-type == 'constant'`:\n",
    "        *  `default-coefficient` (int or float). Corresponds to the efficiency of the array, expressed in `kWp/m²`.\n",
    "    * If `regression-type == 'clustered'`:\n",
    "        * `regression-clusters`: (int). The number of clusters for the regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Library imports\n",
    "import geojson\n",
    "import pandas as pd\n",
    "import os\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "from src import main"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the file from Zenodo or from your local directory (uncomment what's relevant for you)\n",
    "\n",
    "# !wget 'https://zenodo.org/record/7586879/files/hands-on.zip?download=1' -O 'hands-on.zip'\n",
    "# !unzip 'hands-on.zip' \n",
    "# !rm 'hands-on.zip' # delete the zip file.\n",
    "\n",
    "\n",
    "# root directory (uncomment what irrelevant for you)\n",
    "# source_directory = \"path/to/your/data\"\n",
    "source_dir = 'hands-on'\n",
    "\n",
    "# file names\n",
    "source_data_name = \"bdappv-metadata.csv\"\n",
    "source_input_name = \"arrays_69.geojson\"\n",
    "lookup_name = \"lookup-table.json\"\n",
    "\n",
    "# load the files\n",
    "dataset = pd.read_csv(os.path.join(source_dir, source_data_name))\n",
    "arrays = geojson.load(open(os.path.join(source_dir, source_input_name)))\n",
    "lookup = json.load(open(os.path.join(source_dir, lookup_name)))\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up\n",
    "\n",
    "We will extract the characteristics of the `.geojson` file and load them locally in this notebook. We choose the following methods:\n",
    "\n",
    "* `tilt-method`: look-up table\n",
    "* `regression-type`: clustered regression\n",
    "* `azimuth-method`: bounding box\n",
    "\n",
    "We will use the method `extract_all_characteristics` of the `MetadataExtraction` extraction class. We need the following inputs: an auxiliary table and the file containing the polygons, both loaded in the previous cell.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the dictionnary of parameters\n",
    "\n",
    "params = {\n",
    "    # general parameters\n",
    "    \"has-data\" : True,\n",
    "    \"has-dem\"  : False,\n",
    "    'tilt-method' : \"lut\",\n",
    "    \"azimuth-method\" : 'bounding-box',\n",
    "    'regression-type' : \"linear\",\n",
    "    'output-name' : \"arrays_characteristics\",\n",
    "    'data-dir' : source_dir,\n",
    "\n",
    "    # parameters specific to our methods\n",
    "    \"data-directory\" : source_dir,\n",
    "    'data-name' : source_data_name,\n",
    "    \"latitude_var\" : \"latitude\",\n",
    "    \"longitude_var\" : \"longitude\",\n",
    "    \"ic_var\" : \"kWp\",\n",
    "    \"tilt_var\" : 'tilt',\n",
    "    \"surface_var\" : \"surface\",\n",
    "    \"lut-steps\" : 50,\n",
    "    'regression-clusters' : 4,\n",
    "    \n",
    "}\n",
    "\n",
    "# initialize the extractor object\n",
    "extraction = main.MetadataExtraction(p = params, lut = lookup)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our `extraction` object is initialized with our desired parameters. Alternatively, you can dump these parameters in a `config.yml` file and initialize as follows:\n",
    "```python\n",
    "    extraction = main.MetadataExtraction(cf = cf)\n",
    "```\n",
    "where `cf` is your configuration file. Now, we will extract the metadata for all our polygons and store the results in the directory `source_dir`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "extraction.extract_all_characteristics(arrays)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the results stored locally\n",
    "out = pd.read_csv(os.path.join(source_dir, \"{}.csv\".format(params['output-name'])))\n",
    "out.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And that's it! We can now plot the distribution of azimuths and installed capacities according to our estimation. We can see that most of them are pointing south, which is reassuring. Besides, most installations have a small installed capacity (beyond 100 kWp), as our algorithm mostly targets distributed PV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,3, figsize = (13,4))\n",
    "\n",
    "ax[0].set_title('Azimuth [°]')\n",
    "ax[0].hist(out[\"azimuth\"], density = True)\n",
    "\n",
    "ax[1].set_title('Installed capacity [kWp]')\n",
    "ax[1].hist(out['installed_capacity'])\n",
    "ax[1].set_yscale('log')\n",
    "\n",
    "ax[2].set_title('Tilt [°]')\n",
    "ax[2].hist(out['tilt'], density = True)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison of methods\n",
    "\n",
    "We now compare the tilt and azmimuth estimation using the LUT and the bounding-box method with the estimation using the Theil-Sen method. We do this for one installation as the raster is not available everywhere. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item = arrays['features'][1356] # specific array for which the DEM is available\n",
    "\n",
    "params['azimuth-method'] = 'bounding-box'\n",
    "params[\"tilt-method\"] = \"lut\"\n",
    "\n",
    "extraction = main.MetadataExtraction(p = params, lut = lookup)\n",
    "tilt_1, azim_1 = extraction.compute_tilt(item), extraction.compute_azimuth(item)\n",
    "\n",
    "\n",
    "params['azimuth-method'] = 'theil-sen'\n",
    "params[\"tilt-method\"] = \"theil-sen\"\n",
    "\n",
    "# Parameters for Theil-sen estimation\n",
    "params['offset'] = 25\n",
    "params['raster-folder'] = '../hands-on'\n",
    "params['conversion'] =  \"epsg:4326,epsg:2154\"\n",
    "\n",
    "extraction = main.MetadataExtraction(p = params, lut = lookup)\n",
    "tilt_2, azim_2 = extraction.compute_tilt(item), extraction.compute_azimuth(item)\n",
    " \n",
    "print(''' Characteristics extraction comparison:\n",
    "Tilt : {:0.2f} (LUT) / {:0.2f} (Theil-Sen)\n",
    "Azimuth : {:0.2f} (BB) / {:0.2f} (Theil-Sen)\n",
    "'''.format(tilt_1, tilt_2, azim_1, azim_2))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is an view of the installation (right) and the mask (left). We can see that Theil-Sen improves over the bouding box when the mask's boundaries are not parallel with the installations' edges. Mask and image are not at the same scale.\n",
    "\n",
    "\n",
    "<table align=\"center\"><tr>\n",
    "<td> <img src=\"assets/mask.png\" alt=\"Drawing\" style=\"width: 200px;\"/> </td>\n",
    "<td> <img src=\"assets/img.png\" alt=\"Drawing\" style=\"width: 200px;\"/> </td>\n",
    "</tr></table>\n",
    "\n",
    "<i> Image: Google Maps </i>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "stage",
   "language": "python",
   "name": "stage"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12 (main, Apr  5 2022, 01:53:17) \n[Clang 12.0.0 ]"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "0fe214ecfce0b87c57a362cc23dc898772443e26f8754ada0c142e4895ea3525"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}