Data-Access.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">Copyright 2021, by the California Institute of Technology. ALL RIGHTS RESERVED. United States Government sponsorship acknowledged. Any commercial use must be negotiated with the Office of Technology Transfer at the California Institute of Technology.</font>\n",
    "    \n",
    "<font size=\"1\">This software may be subject to U.S. export control laws and regulations. By accepting this document, the user agrees to comply with all applicable U.S. export laws and regulations. User has the responsibility to obtain export licenses, or other export authority as may be required, before exporting such information to foreign countries or providing access to foreign persons.<font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Access\n",
    "This notebook provides simple examples for means of accessing datasets store on some of the more common stores.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# General Initialization\n",
    "#------------------------------------------------------------\n",
    "\n",
    "import os\n",
    "import requests, json, getpass\n",
    "from requests.auth import HTTPBasicAuth\n",
    "import urllib3\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "urllib3.disable_warnings()\n",
    "\n",
    "# this block makes sure the directory set-up/change is only done once and relative to the notebook's directory\n",
    "try:\n",
    "    start_dir\n",
    "except NameError:\n",
    "    start_dir = os.getcwd()\n",
    "    !mkdir -p ./notebook_output/02.1-Data-Access\n",
    "    os.chdir('notebook_output/02.1-Data-Access')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. PCM environment\n",
    "This is the storage of datasets produced as output of PCM processing. Stored in S3 buckets, the locations are obtained through the Pele API, which requires pre-registration as described in notebook 01-Introduction. This notebook also assumes that you have a ~/.netrc file containing an entry approprite to the <mozart_ip> specified below and have used the aws-login utility to refresh your AWS credentials. For the latter, run the following command in a terminal window:\n",
    "\n",
    "```aws-login -p default```\n",
    "\n",
    "and enter your aws login information if/when prompted.\n",
    "\n",
    "A more detailed explanation of the steps below is found in the 02-Datasets-geospatial notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pele.lib.client import PeleRequests\n",
    "\n",
    "# set the base url to interact with the goddess, Pele\n",
    "mozart_ip = \"100.64.122.98\"\n",
    "base_url = \"https://{}/pele/api/v0.1\".format(mozart_ip)\n",
    "print(\"Using Pele base url {}.\".format(mozart_ip))\n",
    "\n",
    "# Instantiate PeleRequests object\n",
    "#------------------------------------------------------------\n",
    "pr = PeleRequests(base_url, verify=False)                                               # <--- Initialize endpoint\n",
    "\n",
    "\n",
    "# Query for the dataset ID of the `L2_L_GSLC` dataset that satisifies search criteria.\n",
    "#------------------------------------------------------------\n",
    "search_poly=[[[-122,40],[-122,38.5],[-125,38.5],[-125,40],[-122,40]]]\n",
    "search_start_time = '2012-11-05'\n",
    "search_end_time = '2012-11-05T23:59:59Z'\n",
    "\n",
    "r = pr.post(base_url + '/pele/dataset/L2_L_GSLC/dataset_ids',                           # <--- Query for dataset(s)\n",
    "            json = { 'polygon' : search_poly, 'start_time' : search_start_time, 'end_time' : search_end_time })\n",
    "\n",
    "# Make sure the request succeeded\n",
    "assert r.status_code == 200\n",
    "\n",
    "# Obtain the dataset metadata\n",
    "#------------------------------------------------------------\n",
    "res = r.json()\n",
    "\n",
    "# Make sure there are qualifying datasets\n",
    "assert len(res['dataset_ids']) > 0\n",
    "\n",
    "dataset_id = res['dataset_ids'][0]\n",
    "print(\"Result dataset id {}\".format(dataset_id))\n",
    "r = pr.get(base_url + '/pele/dataset/{}'.format(dataset_id))                           # <--- Pull dataset metadata\n",
    "res = r.json()\n",
    "\n",
    "# print(json.dumps(res, indent=2))    # Uncomment to see the full metadata\n",
    "\n",
    "# Pull the dataset urls\n",
    "urls = res['result']['urls']\n",
    "# print(\"urls: {}\".format(urls))      # Uncomment to see the set of dataset URLs\n",
    "\n",
    "# Identify the S3 url\n",
    "s3_url = None\n",
    "for i in urls:\n",
    "    if i.startswith('s3://'): s3_url = i\n",
    "assert s3_url is not None\n",
    "\n",
    "url = 's3://{}'.format(urlparse(s3_url).path[1:])\n",
    "local_dir = os.path.basename(url)\n",
    "print (local_dir)\n",
    "\n",
    "# Copy the dataset files from S3 to the local filesystem\n",
    "!aws s3 sync {url} {local_dir}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. ASF (VERTEX) API\n",
    "\n",
    "This example submits a query for UAVSAR ground projected interferogram dataset covering the Point Reyes Lighthouse on January 16, 2020. Only some of the valid search terms are covered below, for more details about using the API go to: https://asf.alaska.edu/api/. The Vertex UI, an interactive tool useful for identifying example datasets is at: https://search.asf.alaska.edu/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_asf_search_url = \"https://api.daac.asf.alaska.edu/services/search/param\"     # <--- search URL for the ASF API\n",
    "\n",
    "# platform can be one or more (comma separated) of: \n",
    "#      ALOS, A3, AIRSAR, AS, ERS, ERS-1, E1, ERS-2, E2, JERS-1, J1, \n",
    "#      RADARSAT-1, R1, SEASAT, SS, S1, Sentinel, Sentinel-1, \n",
    "#      Sentinel-1A, SA, Sentinel-1B, SB, SMAP, SP, UAVSAR, UA\n",
    "platform = \"UAVSAR\"\n",
    "\n",
    "# processing level varies by platform - see API documentation noted above.\n",
    "processingLevel = \"INTERFEROMETRY_GRD\"\n",
    "\n",
    "# this is the search polygon, note the format, which differs from that used in the PCM example above.\n",
    "polygon = \"-123.0461,37.964,-122.9467,37.964,-122.9467,38.0524,-123.0461,38.0524,-123.0461,37.964\"\n",
    "\n",
    "# start/end dates, in standard YYYY-MM-DDTHH:MI:SSZ format\n",
    "start = \"2020-01-16T00:00:00Z\"\n",
    "end = \"2020-01-16T23:59:59Z\"\n",
    "\n",
    "# output format we want is JSON, other formats are: CSV, JSON, KML, METALINK, COUNT, DOWNLOAD, GEOJSON\n",
    "output_format = \"JSON\"\n",
    "\n",
    "# create the search\n",
    "search_url_template = \"{}?platform={}&processingLevel={}&polygon={}&start={}&end={}&output={}\"\n",
    "search_url = search_url_template.format(base_asf_search_url, platform, processingLevel, \n",
    "                                       polygon,start,end,output_format)                  # <-- create/submit search\n",
    "print(\"Submitting request {}\".format(search_url))\n",
    "\n",
    "r = requests.get(search_url)\n",
    "\n",
    "# Uncomment if you want to see the dataset metadata\n",
    "# print(\"dataset:\\n {}\".format(r.content.decode()))\n",
    "\n",
    "# The dataset file is provided by 'downloadUrl' in the response JSON\n",
    "downloadUrl = r.json()[0][0]['downloadUrl']                                              # <-- extract download URL\n",
    "print(\"\\ndownloadUrl : {}\".format(downloadUrl))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use the Osaka API to download the file\n",
    "We will use the PCM Osaka API to perform the download for us, which will perform necessary authentications and redirections. First, make sure your earthdata login is specified in the ~/.netrc file (**anywhere *before* the line 'macdef init'**):\n",
    "\n",
    "```machine urs.earthdata.nasa.gov login <earthdata_user> password <earthdata_password>```\n",
    "\n",
    "**Lock Files**:\n",
    "Note that Osaka employs lock files to ensure no download contention between processes and threads occurs. However, in the event the download fails (for example due to missing or incorrect credentials), the lock file is left behind which prevents subsequent attempts. If you encounter an error below to the effect of \"lock file already locked\", in a terminal window go to the nisar-on-demand-use-cases/notebook_output/02.1-Data-Access directory and remove the file with the \".lock\" extension and retry the download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import osaka.main\n",
    "\n",
    "print(\"Downloading {}\".format (downloadUrl))\n",
    "osaka.main.get(downloadUrl, \".\")                                                  # <-- download the file w/ Osaka\n",
    "print(\"Download complete.\\n\\n\")\n",
    "\n",
    "download_filename = os.path.basename(downloadUrl)\n",
    "!ls -l ./$download_filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": ""
    }
   },
   "source": [
    "## 3. ASF via MAAP\n",
    "#### (WIP : need to determine how to download a .vrt file and its source files.\n",
    "\n",
    "This example provides the means to access UAVSAR radar instrument data as extracted in the maap-plant-demo_20181206 notebook.\n",
    "\n",
    "For this example, you'll need to install the maap module through the folowing steps:\n",
    "\n",
    "    1. Open a terminal from the jupyter notebook home page.\n",
    "    2. In the jovyan home directory, run: git clone https://github.com/MAAP-Project/maap-py.git\n",
    "    3. Enter the newly created directory: cd maap-py\n",
    "    4. Perform the installation: python setup.py install\n",
    "    5. Create a copy of the maap.cfg file in the jovyan home directory: cp maap.cfg ~\n",
    "\n",
    "Note that since maap isn't a component of the underlying jupyter server image, you will need to repeate steps 3 and 4 each time you restart the jupyter server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from osgeo import gdal\n",
    "from maap.maap import MAAP, Granule\n",
    "from pprint import pprint\n",
    "import multiprocessing\n",
    "\n",
    "# Workaround for a bug in MAAP initializer\n",
    "if (os.environ.get(\"MAAP_CONF\") is None):\n",
    "    os.environ[\"MAAP_CONF\"] = \".\"\n",
    "\n",
    "# Initialize MAAP\n",
    "maap = MAAP()                                                                                        # <-- Initialize \n",
    "\n",
    "# Query for Lope National Park radar data for track 001\n",
    "#------------------------------------------------------------\n",
    "radarGranules = maap.searchGranule(site_name=\"Lope National Park Gabon\", \n",
    "                                   track_number=\"001\", \n",
    "                                   collectionConceptId='C1200000308-NASA_MAAP')                      # <-- Query\n",
    "# pprint(radarGranules)           # Uncomment to see the full metadata\n",
    "\n",
    "# extract the vrt filename from the first granule\n",
    "radarGranule = list(filter(lambda granule: granule._location.endswith('.vrt'), radarGranules))[0]\n",
    "# pprint(radarGranule)             # Uncomment to see the granule\n",
    "\n",
    "if not os.path.exists(os.path.basename(radarGranule._location)):                                     # <-- Download\n",
    "    print('Downloading {}'.format(os.path.basename(radarGranule._location)))\n",
    "    !aws s3 cp $radarGranule._location .\n",
    "\n",
    "# get the associated SLCs\n",
    "slcs_to_download = []\n",
    "for slc in radarGranules:\n",
    "    granule_id = slc[\"Granule\"][\"GranuleUR\"]\n",
    "    if (granule_id.startswith('uavsar_AfriSAR_v1_SLC-lopenp') and granule_id.endswith('slc')):\n",
    "       slcs_to_download.append(slc)\n",
    "\n",
    "for slc in slcs_to_download:\n",
    "    if not os.path.exists(slc[\"Granule\"][\"GranuleUR\"]):\n",
    "        print(f\"downloading {slc._location}\")\n",
    "        data = slc.getLocalPath()\n",
    "    else:\n",
    "        print(f\"{slc._location} already exists locally, skipping.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">This notebook is compatible with NISAR Jupyter Server Stack v1.4 and above</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "plant",
   "language": "python",
   "name": "plant"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}