diff --git a/README.org b/README.org index e256f6b..499c2db 100644 --- a/README.org +++ b/README.org @@ -3,80 +3,84 @@ * Table of Contents :toc_2:noexport: - [[#sentinel-3a-s3a-albedo-euro-data-cube-edc-processing-pipeline][Sentinel-3A (S3A) Albedo Euro Data Cube (EDC) Processing Pipeline]] - - [[#fetch-s3-olci--slstr-products][Fetch S3 OLCI & SLSTR products]] - - [[#process-with-snap][Process with SNAP]] + - [[#divide-area-into-tiles][Divide Area Into Tiles]] + - [[#fetch-s3a-olci--slstr-products-using-sentinelhub-sh][Fetch S3A OLCI & SLSTR products using SentinelHub (SH)]] + - [[#start-multiprocessing][Start Multiprocessing]] + - [[#scda--ice-mask][SCDA & Ice Mask]] + - [[#cams-aerosol][cams Aerosol]] - [[#pysice][pySICE]] - [[#mosaic][Mosaic]] - [[#outputs][Outputs]] -- [[#debugging--testing][Debugging & Testing]] -- [[#development-environment][Development Environment]] - - [[#bash][Bash]] +- [[#development-environment][Development Environment] - [[#python][Python]] - - [[#parallel][Parallel]] - - [[#grass][GRASS]] - - [[#gpt][GPT]] - - [[#snap][SNAP]] -- [[#misc-notes--code-snippets][Misc Notes & Code Snippets]] - - [[#generate-ice-mask][Generate ice mask]] - - [[#footprint][Footprint]] - - [[#regional-masks][Regional masks]] -- [[#post-processing-analysis-and-visualisation][Post-processing, analysis and visualisation]] * Sentinel-3A (S3A) Albedo Euro Data Cube (EDC) Processing Pipeline -1. Fetch S3 OLCI & SLSTR products -2. Process OLCI & SLSTR using SNAP GPT -3. Process using pySICE -4. Build daily mosaic +1. Divide Area Into Tiles +2. Fetch S3A OLCI & SLSTR products using SentinelHub (SH) +3. Start Multiprocessing +4. Simple Cloud Detection Algorithm (SCDA) & Masking Non-Ice/Snow areas +5. Get Forecasted Aerosol data from Copernicus Atmospgeric Monotoring Service (cams) +6. Compute outputs with pySICE +7. Merge all tiles into daily mosaics +8. Load all daily mosiacis to one multidimensional NetCDF file +9. Upload to Dataverse and Thredds Server -The steps above are encapsulated in [[./S3_wrapper.sh]]. +The steps above are encapsulated in [[./src/pysicehub.py]]. In more detail: -** Fetch S3 OLCI & SLSTR products +** Divide Area Into Tiles -+ Download OLCI EFR and SLSTR RBT products for a specified day with [[./dhusget_wrapper.sh]]. ++ Every area is divided into 50km x 50km tiles with a 10 km buffer -** Process with SNAP +** Fetch S3A OLCI & SLSTR products -+ This step is a combination of running GPT on [[./S3.xml]] with [[./S3_proc.sh]]. -+ Inputs: OLCI and SLSTR scenes -+ Outputs: A folder OLCI timestamp ++ Download OLCI and SLSTR products for each tile concurrently through SH ++ Donwloads the S3A scene with the lowest Sun Zenith Angle + +** Start Multiprocessing + ++ Use a specified number of cores (default is 12) to process each tile in parrallel ' + +** Simple Cloud Detection Algorithm (SCDA) & Masking Non-Ice/Snow areas + ++ Using SLSTR, clouds in every tile are detected and masked ++ Using a predifened mask, all areas in the tile that are not snow or ice are masked + +** Get Forecasted Aerosol data from Copernicus Atmospgeric Monotoring Service (cams) ++ Download raw forecasted aerosol optical depth at 550 nm and 670 nm for each tile ++ interpolate and regrid to SICE 500m grid ++ compute ångstrøm paramter ** pySICE -+ Run [[./sice.py]] passing in one of the folders generated in the previous step. -+ *WARNING*: This step is slow, and may take > 24 hours. ++ Run [[./pysice.py]] on all tiles ** Mosaic - -+ As implemented in [[./dm.sh]] and [[./dm.grass.sh]], for each day - + Combine all the files from to form a mosaic - + Mask out clouds, then... - + When scenes overlap, use minimum SZA ++ Merge all tiles into moasic ** Outputs | File Name | Description | Units | |-------------------------------+------------------------------------------------------------------------+-------------| | grain_diameter | | mm | | snow_specific_surface_area | | m^{2}kg^{-1}| -| ndsi | normalized difference snow index | | -| ndbi | normalized difference blue ice index | | -| auto_cloud | in development | | | sza | sun zenith angle | | | vza | viewing zenith angle | | | saa | sun azimuth angle | | | vaa | viewing azimuth angle | | +| AOD_550 | Aerosol Optical Depth (550nm) | m | +| ANG | ångstrøm parameter | | | albedo_bb_planar_nir | near infrared broadband planar albedo | 700-2400 nm | | albedo_bb_planar_sw | shortwave broadband planar albedo | 300-2400 nm | | albedo_bb_planar_vis | visible broadband planar albedo | 300-700 nm | | albedo_bb_spherical_nir | near infrared broadband spherical albedo | 700-2400 nm | | albedo_bb_spherical_sw | shortwave broadband spherical albedo | 300-2400 nm | | albedo_bb_spherical_vis | visible broadband spherical albedo | 300-700 nm | -| Oa01..21_reflectance | top of the atmosphere reflectance (OLCI bands 1 through 21) | | +| r_TOA_01..21_reflectance | top of the atmosphere reflectance (OLCI bands 1 through 21) | | | rBRR_01..21 | bottom of the atmosphere surface reflectance (OLCI bands 1 through 21) | | | albedo_spectral_planar_01..21 | spectral planar albedo (OLCI bands 1 through 21) | | -| diagnostic_retrieval | See next table. | | +| isnow | See next table. | | | Diagnostic Code | Description | @@ -90,193 +94,18 @@ In more detail: | 104 | grain_diameter < 0.1, no retrieval, potential cloud flag | | -n | impossible to solve polluted snow albedo equation at band n | -* Debugging & Testing - -Change the year and day for loops in [[./S3_wrapper.sh]] to one year and one day. E.g. - -#+BEGIN_SRC bash :results verbatim -for year in 2017; do # one year - for doy in 227 180; do # two example days - # loop contents - done -done -#+END_SRC - -#+RESULTS: * Development Environment -:PROPERTIES: -:header-args:bash+: :eval no-export -:END: - -This work is developed with the following software versions - -** Bash -#+BEGIN_SRC bash :results verbatim :exports both -bash --version -#+END_SRC - -#+RESULTS: -: GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu) -: Copyright (C) 2016 Free Software Foundation, Inc. -: License GPLv3+: GNU GPL version 3 or later -: -: This is free software; you are free to change and redistribute it. -: There is NO WARRANTY, to the extent permitted by law. ** Python -Run the following using [[./SICE.yml]] in order to set up an identical Anaconda Python environment. +Run the following using [[./environement.yml]] in order to set up an identical Anaconda Python environment. #+BEGIN_SRC bash :results verbatim -conda env create -f SICE.yml +conda env create -f environement.yml #+END_SRC And then run =conda activate SICE= to activate the SICE environment. =ResolvePackageNotFound= error can be raised. In that case, run =conda env export --no-builds > environment.yml= instead. - -** Parallel - -#+BEGIN_SRC bash :results verbatim :exports both -parallel --version -#+END_SRC - -#+RESULTS: -#+begin_example -GNU parallel 20161222 -Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016 -Ole Tange and Free Software Foundation, Inc. -License GPLv3+: GNU GPL version 3 or later -This is free software: you are free to change and redistribute it. -GNU parallel comes with no warranty. - -Web site: http://www.gnu.org/software/parallel - -When using programs that use GNU Parallel to process data for publication -please cite as described in 'parallel --citation'. -#+end_example - -** GRASS -#+BEGIN_SRC bash :results verbatim :exports both -grass --version 2>&1 -#+END_SRC - -#+RESULTS: -#+begin_example -GRASS GIS 7.4.0 - -Geographic Resources Analysis Support System (GRASS) is Copyright, -1999-2018 by the GRASS Development Team, and licensed under terms of the -GNU General Public License (GPL) version >=2. - -This GRASS GIS 7.4.0 release is coordinated and produced by -the GRASS Development Team with contributions from all over the world. - -This program is distributed in the hope that it will be useful, but -WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -General Public License for more details. - -#+end_example - -** GPT - -#+BEGIN_SRC sh :results verbatim :exports both -~/local/snap/bin/gpt --diag -#+END_SRC - -#+RESULTS: -#+begin_example -SNAP Release version 7.0 -SNAP home: /home/kdm/local/snap/bin//.. -SNAP debug: null -SNAP log level: null -Java home: /home/kdm/local/snap/jre -Java version: 1.8.0_202 -Processors: 8 -Max memory: 18.7 GB -Cache size: 1024.0 MB -Tile parallelism: 8 -Tile size: 512 x 512 pixels - -To configure your gpt memory usage: -Edit snap/bin/gpt.vmoptions - -To configure your gpt cache size and parallelism: -Edit .snap/etc/snap.properties or gpt -c ${cachesize-in-GB}G -q ${parallelism} -#+end_example - -** SNAP - -SNAP extensions used in this toolchain are listed in [[SNAP_extensions.txt]]. - -* Misc Notes & Code Snippets -:PROPERTIES: -:header-args:bash+: :eval no -:END: - -** Generate ice mask - -+ Use the icemask from BedMachine v3 - -#+BEGIN_SRC bash :results verbatim :eval no -grass72 -c EPSG:3413 ./Gtmp -r.in.gdal input=NetCDF:~/data/Greenland/Morlighem_2017/BedMachineGreenland-2017-09-20.nc:mask output=icemask - -g.region raster=icemask -g.region res=500 -ap -g.region zoom=icemask - -d.mon start=wx0 -d.erase -d.rast icemask - -r.mapcalc "mask = if(icemask == 4, null(), icemask)" --o -d.rast mask -g.region zoom=mask - -r.out.gdal -c -m input=mask output=mask.tif type=Byte createopt=COMPRESS=DEFLATE --o -exit -trash Gtmp -#+END_SRC -*** Buffered ice mask -#+BEGIN_SRC bash :results verbatim -grass -c mask.tif ./Gtmp -r.in.gdal input=mask.tif output=mask -# 50 cells = 25 km @ 500 m -r.mapcalc "ice = if(mask == 2, 1, null())" -r.grow input=ice output=ice_grow radius=50 new=1 - -r.null ice_grow null=100 -r.clump input=ice_grow output=clumps -r.stats -c clumps sort=asc -for ID in $(r.stats -c clumps sort=asc | head -n7 | cut -d" " -f1); do - r.mapcalc "ice_grow = if(clumps == ${ID}, 1, ice_grow)" --o -done -r.null ice_grow setnull=100 - -r.out.gdal input=ice_grow output=ice_mask_buffer.tif format=GTiff type=Byte createopt="COMPRESS=DEFLATE" -exit -trash Gtmp -#+END_SRC - - -** Footprint - -Get GL outline by drawing in Google Earth, export KML, then: - -#+BEGIN_SRC bash :results verbatim :eval no-export -ogrinfo -al GL_outline.kml | grep LINESTRING | sed s/\ 0//g -#+END_SRC - -** Regional masks - -Regional masks stored in the [[https://github.com/GEUS-SICE/SICE/tree/master/masks][masks]] folder are clips from the -[[https://www.esa-landcover-cci.org/?q=node/197][ESA global Land Cover products]] (download [[https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-land-cover?tab=form][here]]). Masks were downloaded for 2018 and are available -at 300m and 1km resolutions ({region}_300m.tif and {region}_1km.tif, respectively). A description of the 22 [[https://www.esa-landcover-cci.org/?q=node/197][ESA global Land Cover products]] can be found [[https://www.esa-landcover-cci.org/?q=webfm_send/84][here]]. - -* Post-processing, analysis and visualisation - - Post-processing, analysis and visualisation tools have been implemented in the [[https://github.com/GEUS-SICE/SICE-toolbox][SICE toolbox]].