Ready for batch

[yellowcap]

What is changing

• Better retry strategy for Sentinel-1 tiles matching with Sentinel-2
• Storing data into S3 as compressed netCDF files
• Uint16 for Sentinel-2 data to save memory and disk space

How it was done

• Loop through S2 scenes by cloud cover (least first) until a S1 scenes is found within 3 days
• Use tempfile to write a compressed netcdf and then upload.

Will add batch configs later if this works. Its currently in active testing on AWS Batch

Refs

#10

[weiji14]

Managed to save one MGRS tile, and tried dragging and dropping the NetCDF into QGIS, but it ends up being georeferenced to (0, 0). We can try to add the georeferencing later (using rioxarray) to the xr.Dataset to make it easier to plot in QGIS.

[yellowcap]

That is a good idea, I'll see if I can get this to work for the tiles.

[yellowcap]

I did quite some work on the xarrays so that they are proper rioxarrays. The algorithm now stores tiles that can be opened in Qgis and friends.

[lillythomas]

The script stack_tile.py works for calling this version of the datacube module and the tile module. We just need to add the file writing to tile.py now.

[yellowcap]

Ok its working locally. Outputting compressed TIF tiles with 13 bands. I think that is even easier to use than netCDF. I'll try to run this in the cloud later today to see if it works there too.

[yellowcap]

I will add a section for this in the README and add the configurations for the jobs etc about how to run it on Batch over the next few days in a separate PR