Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sbApp_aria: move staged data dload to the top #29

Merged
merged 1 commit into from
Aug 27, 2021
+72 −55
Merged

sbApp_aria: move staged data dload to the top #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 71 additions & 54 deletions smallbaselineApp_aria.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,60 @@
"print('Go to work directory: {}'.format(work_dir))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Run the cell below to 1) download the staged stack of interferograms or 2) to download and prepare the stack of interferograms using ARIA-tools."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# download/prepare and load input dataset\n",
"# aws - download pre-processed data from AWS S3 bucket [recommended, requires awscli, which can be installed as `conda install -c conda-forge awscli`]\n",
"# zenodo - download pre-processed data from zenodo using wget\n",
"# False - download & pre-process from ARIA using ARIA-tools\n",
"use_staged_data = 'aws' #['aws' / 'zenodo' / False]\n",
"\n",
"# download / prepare the stack of interferograms from ARIA\n",
"stack_files = [os.path.join(work_dir, '../stack/{}Stack.vrt'.format(x)) for x in ['unwrap', 'coh', 'connComp']]\n",
"if all(os.path.isfile(x) for x in stack_files):\n",
" print('ARIA products already exists at: {}'.format(os.path.dirname(work_dir)))\n",
"\n",
"elif use_staged_data in ['aws', 'zenodo']:\n",
" # option 1: download the staged data from AWS or Zenodo\n",
" os.chdir(os.path.dirname(os.path.dirname(work_dir)))\n",
" tar_file = os.path.join(os.path.dirname(os.path.dirname(work_dir)), 'SanFranSenDT42.tar.xz')\n",
" if os.path.isfile(tar_file):\n",
" print('Staged ARIA product exists at: {}'.format(tar_file))\n",
" elif use_staged_data == 'aws':\n",
" !aws --region us-east-1 --no-sign-request s3 cp s3://asf-jupyter-data/SanFranSenDT42.tar.xz ./\n",
" elif use_staged_data == 'zenodo':\n",
" !wget https://zenodo.org/record/5152543/files/SanFranSenDT42.tar.xz ./\n",
"\n",
" # uncompress the tar file [it takes ~6 mins]\n",
" #!tar -xvJf SanFranSenDT42.tar.xz # if 'pv' is not installed\n",
" !pv SanFranSenDT42.tar.xz | tar -xJ\n",
" os.chdir(work_dir)\n",
"\n",
"elif use_staged_data is False:\n",
" # option 2: download / prepare ARIA products using ARIA-tools\n",
" print(\"Using ARIA-tools to download and prepare the input data for MintPy\")\n",
" os.chdir(os.path.dirname(work_dir))\n",
" !ariaDownload.py -b '37.25 38.1 -122.6 -121.75' --track 42\n",
" !ariaTSsetup.py -f 'products/*.nc' -b '37.25 38.1 -122.6 -121.75' --mask Download --num_threads 4 --verbose\n",
" os.chdir(work_dir)\n",
"\n",
"else:\n",
" msg = 'un-recognized \"use_staged_data\" setting: {}'.format(use_staged_data)\n",
" msg += '\\navailable settings: \"aws\", \"zenodo\", False.'\n",
" raise ValueError(msg)"
]
},
{
"cell_type": "markdown",
"metadata": {
Expand Down Expand Up @@ -364,7 +418,6 @@
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": "true",
"hidden": true,
"hideCode": false,
"hidePrompt": false
Expand Down Expand Up @@ -398,54 +451,12 @@
},
"outputs": [],
"source": [
"# download/prepare and load input dataset\n",
"# aws - download pre-processed data from AWS S3 bucket [recommended, requires awscli, which can be installed as `conda install -c conda-forge awscli`]\n",
"# zenodo - download pre-processed data from zenodo using wget\n",
"# False - download & pre-process from ARIA using ARIA-tools\n",
"use_staged_data = 'aws' #['aws' / 'zenodo' / False]\n",
"\n",
"if all(os.path.isfile(os.path.join(work_dir, 'inputs', x)) for x in ['ifgramStack.h5', 'geometryGeo.h5']):\n",
" print(\"ARIA products already loaded into MintPy. Skip re-loading.\")\n",
"\n",
"else:\n",
" # download / prepare the stack of interferograms from ARIA\n",
" stack_files = [os.path.join(work_dir, '../stack/{}Stack.vrt'.format(x)) for x in ['unwrap', 'coh', 'connComp']]\n",
" if all(os.path.isfile(x) for x in stack_files):\n",
" print('ARIA products already exists at: {}'.format(os.path.dirname(work_dir)))\n",
"\n",
" elif use_staged_data in ['aws', 'zenodo']:\n",
" # option 1: download the staged data from AWS or Zenodo\n",
" os.chdir(os.path.dirname(os.path.dirname(work_dir)))\n",
" tar_file = os.path.join(os.path.dirname(os.path.dirname(work_dir)), 'SanFranSenDT42.tar.xz')\n",
" if os.path.isfile(tar_file):\n",
" print('Staged ARIA product exists at: {}'.format(tar_file))\n",
" elif use_staged_data == 'aws':\n",
" !aws --region us-east-1 --no-sign-request s3 cp s3://asf-jupyter-data/SanFranSenDT42.tar.xz ./\n",
" elif use_staged_data == 'zenodo':\n",
" !wget https://zenodo.org/record/5152543/files/SanFranSenDT42.tar.xz ./\n",
"\n",
" # uncompress the tar file [it takes ~6 mins]\n",
" #!tar -xvJf SanFranSenDT42.tar.xz # if 'pv' is not installed\n",
" !pv SanFranSenDT42.tar.xz | tar -xJ\n",
" os.chdir(work_dir)\n",
"\n",
" elif use_staged_data is False:\n",
" # option 2: download / prepare ARIA products using ARIA-tools\n",
" print(\"Using ARIA-tools to download and prepare the input data for MintPy\")\n",
" os.chdir(os.path.dirname(work_dir))\n",
" !ariaDownload.py -b '37.25 38.1 -122.6 -121.75' --track 42\n",
" !ariaTSsetup.py -f 'products/*.nc' -b '37.25 38.1 -122.6 -121.75' --mask Download --num_threads 4 --verbose\n",
" os.chdir(work_dir)\n",
"\n",
" else:\n",
" msg = 'un-recognized \"use_staged_data\" setting: {}'.format(use_staged_data)\n",
" msg += '\\navailable settings: \"aws\", \"zenodo\", False.'\n",
" raise ValueError(msg)\n",
"\n",
" # load the stack of interferogrms to MintPy\n",
"if not all(os.path.isfile(os.path.join(work_dir, 'inputs', x)) for x in ['ifgramStack.h5', 'geometryGeo.h5']):\n",
" print(\"Load ARIA product into MintPy ...\")\n",
" os.chdir(work_dir)\n",
" !smallbaselineApp.py SanFranSenDT42.txt --dostep load_data"
" !smallbaselineApp.py SanFranSenDT42.txt --dostep load_data\n",
"else:\n",
" print(\"ARIA products already loaded into MintPy. Skip re-loading.\")"
]
},
{
Expand Down Expand Up @@ -623,7 +634,8 @@
"outputs": [],
"source": [
"!temporal_average.py ./inputs/ifgramStack.h5 -d coherence -o avgSpatialCoh.h5\n",
"view.main('avgSpatialCoh.h5 --noverbose'.split())"
"view.main('avgSpatialCoh.h5 --noverbose'.split())\n",
"# equivalent command in terminal: view.py avgSpatialCoh.h5 --noverbose"
]
},
{
Expand All @@ -648,7 +660,8 @@
"outputs": [],
"source": [
"!generate_mask.py inputs/geometryGeo.h5 waterMask --nonzero -o waterMask.h5\n",
"view.main('waterMask.h5 -c gray --noverbose'.split())"
"view.main('waterMask.h5 -c gray --noverbose'.split())\n",
"# equivalent command in terminal: view.py waterMask.h5 -c gray --noverbose"
]
},
{
Expand Down Expand Up @@ -740,7 +753,6 @@
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": "true",
"hidden": true,
"hideCode": false,
"hidePrompt": false
Expand Down Expand Up @@ -818,7 +830,8 @@
},
"outputs": [],
"source": [
"view.main('timeseries.h5 -v -5 5 --noaxis'.split())"
"view.main('timeseries.h5 -v -5 5 --noaxis'.split())\n",
"# equivalent command in terminal: view.py timeseries.h5 -v -5 5 --noaxis"
]
},
{
Expand All @@ -838,7 +851,6 @@
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": "true",
"hidden": true,
"hideCode": false,
"hidePrompt": false
Expand Down Expand Up @@ -881,7 +893,7 @@
},
"outputs": [],
"source": [
"view.main('velocity.h5 velocity -v -1 1'.split())"
"view.main('velocity.h5 velocity -v -1 1 --figsize 10 8'.split())"
]
},
{
Expand Down Expand Up @@ -1005,6 +1017,12 @@
"hidePrompt": false
},
"source": [
"Anatomy of interferometric phases:\n",
"\n",
"$$ \\Large \\Delta \\phi = \\color{green}{ \\Delta\\phi_{defo} + \\Delta\\phi_{atm} + \\Delta\\phi_{geom} + \\Delta\\phi_{tidal}} + \\color{royalblue}{ \\Delta\\phi_{non-closure}} $$\n",
"\n",
"where $\\color{green}{\\Delta\\phi_{atm}}$ includes ionospheric and tropospheric delays, $\\color{green}{\\Delta\\phi_{geom}}$ includes the topographic residual (DEM error) and orbital error, $\\color{green}{\\Delta\\phi_{tidal}}$ includes solid Earth tides, ocean tidal loading, etc. $\\color{royalblue}{\\Delta\\phi_{non-closure}}$ includes all phase contributions that does not fulfill the zero phase closure of interferogram triplets, including decorrelation noise, potential phase-unwrapping errors, processing inconsistency due to filtering and multilooking, and non-closure phases from short temporal baseline interferograms.\n",
"\n",
"Uncertainty of the ground displacement products derived from InSAR time-series, depends on the quality of the inversion of the stack of interferograms and the accuracy in separating the ground displacement from other components of the InSAR data. Therefore the definition of signal vs noise is different at the two main steps in mintpy: \n",
"\n",
"1) During the inversion: \n",
Expand Down Expand Up @@ -1136,7 +1154,6 @@
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": "true",
"hideCode": false,
"hidePrompt": false
},
Expand Down
2 changes: 1 addition & 1 deletion visualization/tsview.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"tsview.py /home/jovyan/data/SanFranSenDT42/mintpy/timeseries_ERA5_demErr.h5 --ms 4 --noverbose --yx 840 796\n",
"tsview.py /home/jovyan/data/SanFranSenDT42/mintpy/timeseries.h5 --ms 4 --noverbose --yx 840 796\n",
"No lookup table info range/lat found in files.\n",
"estimate deformation model with the following assumed time functions:\n",
" polynomial : 1\n",
Expand Down