GEE_ISMN_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The GEE_ISMN Package"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "1. Install & Import modules\n",
    "2. Setup\n",
    "3. Preprocessing\n",
    "4. Google Earth Engine (GEE)\n",
    "5. Postprocessing\n",
    "6. Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Install & Import modules\n",
    "\n",
    "The packages ismn and geehydro are necessary for some functions to work properly. <br>\n",
    "For more information about these packages please refer to these sources: <br>\n",
    "https://github.com/TUW-GEO/ismn <br>\n",
    "https://pypi.org/project/geehydro/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture \n",
    "import sys\n",
    "!{sys.executable} -m pip install ismn\n",
    "!{sys.executable} -m pip install geehydro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Google Earth Engine (GEE) Python API will be installed via conda. <br>\n",
    "For more information please refer to these sources: <br>\n",
    "https://anaconda.org/conda-forge/earthengine-api <br>\n",
    "https://developers.google.com/earth-engine/python_install-conda"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conda install -c conda-forge earthengine-api "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the different modules of the GEE_ISMN package can be imported from the `./GEE_ISMN` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from GEE_ISMN import setup_pkg as pkg\n",
    "from GEE_ISMN import preprocess as prep\n",
    "from GEE_ISMN import earthengine as earth\n",
    "from GEE_ISMN import postprocess as post\n",
    "from GEE_ISMN import visualization as vis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ISMN datasets that should be analysed need to be located in the directory `./data/ISMN/`. The function `setup_pkg()` checks if this directory exists and creats a new directory `./data/ISMN_Filt/`. <br>\n",
    "\n",
    "Also the Google Earth Engine Python API will be initialized and the function checks if credentials already exist. If not, an authentication process will be started. <br>\n",
    "\n",
    "Finally, the function asks for user input which is relevant for the extraction of Sentinel-1 backscatter values later on and will be stored in a dictionary called *user_input*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_input = pkg.setup_pkg()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `data_handling()` checks all files in the directory `./data/ISMN/` and copies all files that contain soil moisture measurements that were taken at a specific depth to the directory `.data/ISMN_Filt/`. If the parameter *measurement_depth* is not defined, the default value of 0.05 m is used. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prep.data_handling()\n",
    "\n",
    "## Example with parameter \"measurement_depth\" defined:\n",
    "# prep.data_handling(measurement_depth=0.05) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The soil moisture data from all files in the directory `.data/ISMN_Filt/` are then imported to a dictionary by using the function `data_import()`. Additionally, a CSV file with all dictionary keys and the corresponding coordinates of each station is saved in `./data/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dict = prep.data_import()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Google Earth Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ISMN datasets in the dictionary will than be filtered by landcover type. The landcover types can be defined through the parameter *landcover_ids*. <br>\n",
    "The IDs 40 (clutivated and managed vegetation / agriculture) and 60 (Bare / sparse vegetation) are used as default. <br>\n",
    "Other available landcover IDs can be found [here](https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_Landcover_100m_Proba-V_Global)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dict = earth.lc_filter(data_dict, user_input)\n",
    "\n",
    "## Example with parameter \"landcover_ids\" defined: \n",
    "# data_dict = earth.lc_filter(data_dict, user_input, landcover_ids=[30, 50, 100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now only the locations we are interested in remain. <br>\n",
    "The next step is to retrieve the Sentinel-1 backscatter time series for each location by using the function `get_s1_backscatter()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dict = earth.get_s1_backscatter(data_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Postprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the temporal resolution of the soil moisture data is much higher than the backscatter data, only soil moisture values taken immediatly before the backscatter recording will be kept by applying the function `ts_filter()`. <br> \n",
    "The resulting dataframes (one each for the descending and ascending orbits) contain the filtered soil moisture values & timestamps, as well as the backscatter values & timestamps and will be added to the dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dict = post.ts_filter(data_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For visualization, we need the station names that were created in preprocessing and saved to the CSV-File *stations.csv*. The name is a combination of the network name, the station name and the sensor type and is also used as the dictionary keys, so it's also possible to retrieve them by using `data_dict.keys()`. <br>\n",
    "For the following example a random station is retrieved from the current dictionary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "station_name = random.choice(list(data_dict.keys()))\n",
    "station_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `plot_data()` is used to create a graph in which the ISMN soil moisture data is plotted against the Sentinel-1 backscatter values. <br>\n",
    "If the parameters *orbit* and *pol* are not defined, the default settings are chosen: \"desc\" for descending orbit and \"VV\" as the polarisation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vis.plot_data(data_dict, station_name)\n",
    "\n",
    "## Example with parameters \"orbit\" and \"pol\" defined: \n",
    "# vis.plot_data(data_dict, station_name, orbit=\"asc\", pol=\"VH\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also it is possible to generate two maps. The function show_map() displays the chosen station on a map with an optical satellite imagery as a basemap. <br>\n",
    "The other map is generated by the function show_s1() which displays a Sentinel-1 scene that covers the location of the chosen station."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `show_map()` will show the location of the chosen station on a map that opens in the browser. A .html file is also saved in the working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vis.show_map(data_dict, station_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By also defining a date, the closest available Sentinel-1 scene can be visualized for a given station. Again, a .html file is saved in the working directory. <br>\n",
    "For this function the parameters *orbit* and *pol* can also be defined. Just like previously, \"desc\" and \"VV\" are used as default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "date = \"2018-08-20\"\n",
    "vis.show_s1(data_dict, station_name, date)\n",
    "\n",
    "## Example with parameters \"orbit\" and \"pol\" defined: \n",
    "# vis.show_s1(data_dict, station_name, date, orbit=\"asc\", pol=\"VH\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}