Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
  • Loading branch information
@RasmusBahbah
RasmusBahbah committed Oct 31, 2023
1 parent 0177548 commit fec6892
Show file tree
Hide file tree
Showing 3 changed files with 706 additions and 2 deletions.
2 changes: 0 additions & 2 deletions .cdsapirc
View file

This file was deleted.

282 changes: 282 additions & 0 deletions README.org
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,282 @@
[[https://www.gnu.org/licenses/gpl-3.0][https://img.shields.io/badge/License-GPLv3-blue.svg]]
[[https://github.com/GEUS-SICE/SICE/actions][file:https://github.com/GEUS-SICE/SICE/workflows/CI/badge.svg]]

* Table of Contents :toc_2:noexport:
- [[#sentinel-3a-s3a-albedo-euro-data-cube-edc-processing-pipeline][Sentinel-3A (S3A) Albedo Euro Data Cube (EDC) Processing Pipeline]]
- [[#fetch-s3-olci--slstr-products][Fetch S3 OLCI & SLSTR products]]
- [[#process-with-snap][Process with SNAP]]
- [[#pysice][pySICE]]
- [[#mosaic][Mosaic]]
- [[#outputs][Outputs]]
- [[#debugging--testing][Debugging & Testing]]
- [[#development-environment][Development Environment]]
- [[#bash][Bash]]
- [[#python][Python]]
- [[#parallel][Parallel]]
- [[#grass][GRASS]]
- [[#gpt][GPT]]
- [[#snap][SNAP]]
- [[#misc-notes--code-snippets][Misc Notes & Code Snippets]]
- [[#generate-ice-mask][Generate ice mask]]
- [[#footprint][Footprint]]
- [[#regional-masks][Regional masks]]
- [[#post-processing-analysis-and-visualisation][Post-processing, analysis and visualisation]]

* Sentinel-3A (S3A) Albedo Euro Data Cube (EDC) Processing Pipeline

1. Fetch S3 OLCI & SLSTR products
2. Process OLCI & SLSTR using SNAP GPT
3. Process using pySICE
4. Build daily mosaic

The steps above are encapsulated in [[./S3_wrapper.sh]].

In more detail:

** Fetch S3 OLCI & SLSTR products

+ Download OLCI EFR and SLSTR RBT products for a specified day with [[./dhusget_wrapper.sh]].

** Process with SNAP

+ This step is a combination of running GPT on [[./S3.xml]] with [[./S3_proc.sh]].
+ Inputs: OLCI and SLSTR scenes
+ Outputs: A folder OLCI timestamp

** pySICE

+ Run [[./sice.py]] passing in one of the folders generated in the previous step.
+ *WARNING*: This step is slow, and may take > 24 hours.

** Mosaic

+ As implemented in [[./dm.sh]] and [[./dm.grass.sh]], for each day
+ Combine all the files from to form a mosaic
+ Mask out clouds, then...
+ When scenes overlap, use minimum SZA

** Outputs
| File Name | Description | Units |
|-------------------------------+------------------------------------------------------------------------+-------------|
| grain_diameter | | mm |
| snow_specific_surface_area | | m^{2}kg^{-1}|
| ndsi | normalized difference snow index | |
| ndbi | normalized difference blue ice index | |
| auto_cloud | in development | |
| sza | sun zenith angle | |
| vza | viewing zenith angle | |
| saa | sun azimuth angle | |
| vaa | viewing azimuth angle | |
| albedo_bb_planar_nir | near infrared broadband planar albedo | 700-2400 nm |
| albedo_bb_planar_sw | shortwave broadband planar albedo | 300-2400 nm |
| albedo_bb_planar_vis | visible broadband planar albedo | 300-700 nm |
| albedo_bb_spherical_nir | near infrared broadband spherical albedo | 700-2400 nm |
| albedo_bb_spherical_sw | shortwave broadband spherical albedo | 300-2400 nm |
| albedo_bb_spherical_vis | visible broadband spherical albedo | 300-700 nm |
| Oa01..21_reflectance | top of the atmosphere reflectance (OLCI bands 1 through 21) | |
| rBRR_01..21 | bottom of the atmosphere surface reflectance (OLCI bands 1 through 21) | |
| albedo_spectral_planar_01..21 | spectral planar albedo (OLCI bands 1 through 21) | |
| diagnostic_retrieval | See next table. | |


| Diagnostic Code | Description |
|-----------------+-----------------------------------------------------------------------------------------------|
| 0 | clean snow |
| 1 | polluted snow |
| 6 | polluted snow for which r0 was calculated and not derived from observations |
| 7 | polluted snow of calculated spherical albedo in bands 1 and 2 >0.98 reprocessed as clean snow |
| 100 | sza>75, no retrival |
| 102 | TOA reflectance at band 21 < 0.1, no retrieval |
| 104 | grain_diameter < 0.1, no retrieval, potential cloud flag |
| -n | impossible to solve polluted snow albedo equation at band n |

* Debugging & Testing

Change the year and day for loops in [[./S3_wrapper.sh]] to one year and one day. E.g.

#+BEGIN_SRC bash :results verbatim
for year in 2017; do # one year
for doy in 227 180; do # two example days
# loop contents
done
done
#+END_SRC

#+RESULTS:

* Development Environment
:PROPERTIES:
:header-args:bash+: :eval no-export
:END:

This work is developed with the following software versions

** Bash
#+BEGIN_SRC bash :results verbatim :exports both
bash --version
#+END_SRC

#+RESULTS:
: GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
: Copyright (C) 2016 Free Software Foundation, Inc.
: License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
:
: This is free software; you are free to change and redistribute it.
: There is NO WARRANTY, to the extent permitted by law.

** Python

Run the following using [[./SICE.yml]] in order to set up an identical Anaconda Python environment.

#+BEGIN_SRC bash :results verbatim
conda env create -f SICE.yml
#+END_SRC

And then run =conda activate SICE= to activate the SICE environment.

=ResolvePackageNotFound= error can be raised. In that case, run =conda env export --no-builds > environment.yml= instead.


** Parallel

#+BEGIN_SRC bash :results verbatim :exports both
parallel --version
#+END_SRC

#+RESULTS:
#+begin_example
GNU parallel 20161222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: http://www.gnu.org/software/parallel

When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.
#+end_example

** GRASS
#+BEGIN_SRC bash :results verbatim :exports both
grass --version 2>&1
#+END_SRC

#+RESULTS:
#+begin_example
GRASS GIS 7.4.0

Geographic Resources Analysis Support System (GRASS) is Copyright,
1999-2018 by the GRASS Development Team, and licensed under terms of the
GNU General Public License (GPL) version >=2.

This GRASS GIS 7.4.0 release is coordinated and produced by
the GRASS Development Team with contributions from all over the world.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.

#+end_example

** GPT

#+BEGIN_SRC sh :results verbatim :exports both
~/local/snap/bin/gpt --diag
#+END_SRC

#+RESULTS:
#+begin_example
SNAP Release version 7.0
SNAP home: /home/kdm/local/snap/bin//..
SNAP debug: null
SNAP log level: null
Java home: /home/kdm/local/snap/jre
Java version: 1.8.0_202
Processors: 8
Max memory: 18.7 GB
Cache size: 1024.0 MB
Tile parallelism: 8
Tile size: 512 x 512 pixels

To configure your gpt memory usage:
Edit snap/bin/gpt.vmoptions

To configure your gpt cache size and parallelism:
Edit .snap/etc/snap.properties or gpt -c ${cachesize-in-GB}G -q ${parallelism}
#+end_example

** SNAP

SNAP extensions used in this toolchain are listed in [[SNAP_extensions.txt]].

* Misc Notes & Code Snippets
:PROPERTIES:
:header-args:bash+: :eval no
:END:

** Generate ice mask

+ Use the icemask from BedMachine v3

#+BEGIN_SRC bash :results verbatim :eval no
grass72 -c EPSG:3413 ./Gtmp
r.in.gdal input=NetCDF:~/data/Greenland/Morlighem_2017/BedMachineGreenland-2017-09-20.nc:mask output=icemask

g.region raster=icemask
g.region res=500 -ap
g.region zoom=icemask

d.mon start=wx0
d.erase
d.rast icemask

r.mapcalc "mask = if(icemask == 4, null(), icemask)" --o
d.rast mask
g.region zoom=mask

r.out.gdal -c -m input=mask output=mask.tif type=Byte createopt=COMPRESS=DEFLATE --o
exit
trash Gtmp
#+END_SRC
*** Buffered ice mask
#+BEGIN_SRC bash :results verbatim
grass -c mask.tif ./Gtmp
r.in.gdal input=mask.tif output=mask
# 50 cells = 25 km @ 500 m
r.mapcalc "ice = if(mask == 2, 1, null())"
r.grow input=ice output=ice_grow radius=50 new=1

r.null ice_grow null=100
r.clump input=ice_grow output=clumps
r.stats -c clumps sort=asc
for ID in $(r.stats -c clumps sort=asc | head -n7 | cut -d" " -f1); do
r.mapcalc "ice_grow = if(clumps == ${ID}, 1, ice_grow)" --o
done
r.null ice_grow setnull=100

r.out.gdal input=ice_grow output=ice_mask_buffer.tif format=GTiff type=Byte createopt="COMPRESS=DEFLATE"
exit
trash Gtmp
#+END_SRC


** Footprint

Get GL outline by drawing in Google Earth, export KML, then:

#+BEGIN_SRC bash :results verbatim :eval no-export
ogrinfo -al GL_outline.kml | grep LINESTRING | sed s/\ 0//g
#+END_SRC

** Regional masks

Regional masks stored in the [[https://github.com/GEUS-SICE/SICE/tree/master/masks][masks]] folder are clips from the
[[https://www.esa-landcover-cci.org/?q=node/197][ESA global Land Cover products]] (download [[https://cds.climate.copernicus.eu/cdsapp#!/dataset/satellite-land-cover?tab=form][here]]). Masks were downloaded for 2018 and are available
at 300m and 1km resolutions ({region}_300m.tif and {region}_1km.tif, respectively). A description of the 22 [[https://www.esa-landcover-cci.org/?q=node/197][ESA global Land Cover products]] can be found [[https://www.esa-landcover-cci.org/?q=webfm_send/84][here]].

* Post-processing, analysis and visualisation

Post-processing, analysis and visualisation tools have been implemented in the [[https://github.com/GEUS-SICE/SICE-toolbox][SICE toolbox]].
Loading

0 comments on commit fec6892

Please sign in to comment.