- ENH: Add a new type (
BandsType
) for list of BandType - ENH: Add a new environment variable
EOREADER_NOF_BANDS_IN_CHUNKS
to control the number of the bands in chunks when usingdask
. (#178) - ENH: Allow
'auto'
inEOREADER_TILE_SIZE
, to setchunks="auto"
when reading data. (#178) - FIX: Fix stack
save_as_int
to use updated int values - by @TabeaW - FIX: Fixed PAZ Product Regex to properly indentify PAZ ST products as
PAZProduct
- by @guillemc23 - FIX: Fixed PNEO Product Regex to properly indentify PNEO products as
PneoProduct
- by @guillemc23 - FIX: Fixed preprocessing graph paths in order to support relative paths in more complex environments or contexts - by @guillemc23
- FIX: Remove useless
_norm_diff
functionindices.py
- FIX: Add a fallback in case
map-overlay.kml
is not readable forSentinel-1
data (#180,#182) - FIX: Remove warning about Dask's lock and client
- FIX: Don't throw an error in case of missing cloud coverage, only a warning and set the cloud coverage to 0 #159
- FIX: Use the sun elevation angle rather than the sun zenith angle for STAC #158
- FIX: Create comparison operators for
BandNames
, removing thexarray RuntimeWarning
aboutsort order is undefined for incomparable objects
. - FIX: Add some missing
@cache
around time-consuming functions - FIX: Set correctly the SAR product type, with adding two types (
ORTHO
andGEOCODED
) - FIX: Fix the computation of parametric spectral indices #193
- OPTIM: Cache the access to any archived file list, as this operation is expensive when done with large archives stored on the cloud (and thus better done only once).
- CI: Remove useless verbosity in CI
- CI: GDAL performance tuning by tweaking
rasterio
's env - INTERNAL: Switch from
setup.py
topyproject.toml
#109 - INTERNAL: Use
ruff
instead ofblack
+flake8
+isort
- DOC: Update
conf.py
(remove useless hunks and set Sphinx 7 as base) - DOC: Added the PAZ product guide to the PAZ Product documentation instead of the TerraSAR-X one - by @guillemc23
- DEPS: Pin
sertit>=1.44.1
- FIX: Handle
ICEYE
products with missing quicklook - FIX: Fix
Sentinel-1
name with weird PDFs names (i.e. ending with.SAFE-report...
) - FIX: Remove multi-swath workaround for
Cosmo
products if SNAP > 11.0 - FIX: By default, try to assign a constellation (in a pure dummy way) to any
Product
created - FIX: Add ways of knowing if a constellation is a real one or not (i.e.
CUSTOM
or template such asMaxar
) - FIX: Create
TDX
andPAZ
(completely inherited) classes to disambiguate their constellations
- FIX: Fix (really) window's name coming from a vector with an underscore after it
- FIX: Fix clean band path for Sentinel-3 SLSTR products
- FIX: Remove an ignored exception when deleting a Product (
ValueError: Unknown '__class__' name in 'covariance.compute' hyperparameters
)
- FIX: Fix window's name coming from a vector with an underscore after it
- FIX: Allow to load numpy pickles stored in S3 buckets
- FIX: Add MS and PAN resolution in Landsat Products
- DEPS: Don't force using geopandas 1.0.0, 0.14.4 should be enough.
- ENH: Allow the process of Sentinel-1 COGs (provided by the Copernicus DataSpace) for SNAP >= 10 (#172)
- ENH: Add a
BandType
alias for any types that could be a band: a string, aBandNames
or any of its children: Spectral, SAR, DEM or Cloud band names - FIX: Anticipate Sentinel C and D platforms in Reader's regexes
- FIX: Resolve the inversion of resolution and pixel size between
stripmap
andsliding_spotlight
types forCapella
products - FIX: Get better window name (if available) when writing bands on disk (in tmp folder)
- FIX: Reject buggy Maxar products (with version 28.4) as the workaround would be too heavy to implement. (#106)
- FIX: Fix Despeckle graph with SNAP10 (#177)
- OPTIM: Save rasterized masks of DIMAP V2 products on disk to avoid recomputing them (
features.rasterize
could be a heavy computation that shouldn't be done twice) - COMPAT: EOReader works correctly with SNAP 10 (#165)
- PUBLISH: Use PyPI's Trusted Publisher Management mechanism
- ENH:
to_str
andto_band
: add aas_list
argument defaulting toTrue
. When set as False, return a str fromto_str
and a band fromto_band
(#138). Thanks @jsetty! - FIX:
Sentinel-2
product withStopIteration
error (#142) - FIX: Fix error in looking for bands in
Sentinel-2 L1C
archived products (#168) - FIX: Fix issue with geocoding with unzipped
Sentinel-3 OLCI
product (#137) - FIX: In
SPOT
products, METADATA.DIM and IMAGERY.TIF must be at the root of the product (#145) - FIX: Fix
Maxar
product withQB02
satellite ID (#140) - FIX: Fix
ICEYE
product when extent file (*.kml) not found (#135) - FIX: Handle
RCM
andRS2
products that doesn't bundle their extent in a KML file (#155) - FIX: Handle wrongly recognized
Planet
products because of the recursive nested mtd in the Reader (#169) - FIX: Fix an unknown
Planet
bug that just appeared ('...Path' has no len()
) - FIX: Force the loading of
DimapV1
bands infloat32
- FIX: Handle the case where
fiona
isn't installed anymore (withgeopandas 1.0
) - FIX: Don't make
pystac
a mandatory requirement - OPTIM: Search correctly nested metadata in the Reader (without accidentally using a recursive glob)
- CI: Fix S3 endpoint management with
sertit>=1.37
- CI: Remove for now end-to-end tests with Python 3.11 and 3.10.
- INSTALL: Remove
pystac[validation]
(as it is an optional dependency) from setup.py, and create astac
extra feature.
- ENH: Add a
is_stacked
parameters for EOReader'sProduct
to document either its bands are delivered stacked or file by file. - FIX: Correct
SWIR_CIRRUS
spectral band's enum value (toSWIR_CIRRUS
instead ofCIRRUS
), avoiding shadowing cloud bandCIRRUS
(#131) - FIX: Raise proper exception (
UnhandledArchiveError
) for archived data that needs to be extracted before use. A warning wasn't enough. - FIX: Remove unused
pixel_spacing
for SAR Products - FIX: Fix workaround for corrupted
Sentinel-2
mask.
- FIX: Don't force install
planetary-computer
orstac-asset
to use EOReader - DOC: Remove Twitter from README
- BREAKING CHANGES: Rename
utils.stack_dict
toutils.stack
since we are stacking datasets and not dict anymore. - BREAKING CHANGES: Band ID for Sentinel-3 OLCI are now int instead of band names (i.e.
7
instead ofOa07
. The names don't change). - ENH: Allow to use bands IDs, names and common name added to mapped names when trying to load a spectral band. (#111)
- ENH: Manage Sentinel-2 as formatted on the cloud (Element84 or Sinergise's way). (#104)
- ENH: Handle Python 3.12. (#113)
- ENH: Guard against S1 COG format, not yet handled by SNAP.
- ENH: Calibration step for
Capella
products now exists in ESA SNAP. Add it in pre-processing. - ENH: Handling of Sentinel-1 ASF and MPC RTC products. (#112, #118)
- ENH: Handling of Sentinel-1 SM products.
- ENH: Better handling of calibration step in SNAP for SAR data.
- FIX: Fix jpg, png... quicklooks management when plotting
- FIX: Fix an
xarray
issue when trying to compute percentiles when stacking bands - DEPS: Remove as many mention as possible to
cloudpathlib
- DEPS: Update minimum versions of some libraries
- DOC: Add example about the output management (in
base
notebook, #117) - DOC: Update copyright to 2024
- GITHUB: Update bug template
- CI: Enabling pre-commit.ci and dependabot bots
- CI: Update pre-commit hooks
- CI: Revamping
test_satellites
- CI: Some refactoring and speed-ups
- FIX: Don't collocate a raster on itself
- FIX: Better management of default pixel size for multi-resolution products (such as PAN band in Landsat)
- FIX: Fixing the PAN GSD for Landsat-OLI products
- FIX: Update some code to match
sertit>=1.29.0
- FIX: Use
auto
as default dask chunk (instead of2048
) - FIX: Minor fix in RasterioError handling when reading bands
- FIX: Fix Landsat L2 surface reflectance and temperature computation (#99)
- FIX: Fixing TIR temperature conversion for Landsat-7
- FIX: Test thermal bands in CI
- FIX: Upgraded to EO STAC extension v1.1.0 (#83)
- FIX: Use already computed bands stored in
tmp
for Planet products
- FIX: Allow band aliases (such as
"CA"
instead of"COASTAL_AEROSOL"
) in string in band mapping when creating Custom Stacks
- COMPAT: Add the alias
GREEN_1
for GREEN I band of PlanetScope data, in order to stay in the same pattern asVRE_x
,SWIR_x
...GREEN1
will be deprecated in another release.
- BREAKING CHANGES: Switching from
resolution
topixel_size
to avoid confusion about the definitions (especially for SAR data) (#82) - BREAKING CHANGES:
load
function now returns axarray.Dataset
(#88)
- FIX: Collocate bands before trying to create spectral indices: resolve the case where their size mismatches (i.e. in case of window or change of native pixel size)
- FIX: Landsat band masking when specifying a custom resolution and a custom window
- FIX: Round the default pixel_size of custom stacks
- FIX: Convert some fields of STAC items from non JSON serializable dtypes to correct ones
- FIX: Fix erroneous property set to
_get_raw_crs
for Maxar products
- ENH: Don't load data into memory when computing indices, ensuring tasks are delayed a bit longer (#58)
- DOCS: Add documentation about default CRS (#87)
- DEPS: Dropping support of Python 3.8 (#81)
- DEPS: Pin sertit to 1.27.0
- INTERNAL: Better management of logs for deprecation warnings
- INTERNAL: Refactoring
simplify_footprint
insertit
library - CI: Test that STAC items are serializable when added to a catalog
- FIX: Removing calibration step from SNAP pre-processing graph for multi-swath
Cosmo-SkyMed 1st GEN
products (to avoid ending up with empty images after pre-process) - FIX: Fixing the paths to Sentinel-2 quicklooks: using PVI instead of TCI file if no .jpg preview file is found (#84 , #85, thanks a lot @floriandeboissieu)
- STAC: Updates in STAC management
- INTERNAL: Use
geopandas.estimate_utm_crs()
when possible
- OPTIM: Don't recompute stacks if already existing on disk
- FIX: Fixing
Custom Stacks
when specifyingdatetime=None
on creation - FIX: Fix regression for multi-swath DGM CSK data (huge region) (#78)
- FIX: Fix calibration issues with CSK HR data (using fallback GPT graph by default)
- OPTIM: Always use chunks when reading rasters (#58)
- OPTIM: Speed up VRT virtual warping
- OPTIM: Better management of dask's usage
- CI: Fix projection STAC extension's new version number (1.1.0)
- FIX: Fixing stack when saved as integer for some special cases
- FIX: Clipping negative reflectances to 0 (#79)
- FIX: Fixing nodata management for Theia product
- FIX: Fixing handling of SCS multi-swath
Cosmo-SkyMed
products (#78) - FIX: Writing spectral indices on disk to align with other bands (#80)
- FIX: By default, calibration is not applied to slant range
CSG
data, avoiding producing an empty raster (#48)
- OPTIM: Using warped VRT instead of reprojecting DEM/VHR stacks to UTM (#58)
- TYP: Fixing typos in typing
- INTERNAL: Moving
EOREADER_NAME
andDATETIME_FMT
into__init__.py
- INTERNAL: Moving stacking function into
utils
- INTERNAL: Removing unused
cache_property
decorator - INTERNAL: Factorizing
_load
function - CI: Using
assert_raster_almost_equal_magnitude
in CI to better check according to bands' content (sertit 1.24.0) - DEPS: Officially handling Python 3.11 (adding weekly tests on Python 3.11) (#71)
- DOC: Updating
Custom
notebook - DOC: Updating jupyter cache to match new way of handling outputs in readthedocs
- FIX: Fixing a bug for DIMAP V2 products with GML masks opening without CRS: assigning first the raw CRS before converting to the product's CRS
- FIX: Fixing index creation when exotic bands not handled by ASI have been loaded in the same time (i.e. stacking
NDWI
withOa21
band)
- CLEAN: Removing useless GCP functions regarding Sentinel-3 data
- DOC: Adding a
Remove Clouds
notebook - LIB: Pinning
sertit
to 1.22.0 - CI: Don't run tests when only
__init__
or__meta__
is updated - CI: Some factorizing in
gitlab-ci
- ENH: Adding the support of Capella constellation (#74)
- ENH: Allow the user to load bands with a window (pixels and geo) (#25) , notebook)
- FIX: Fix extent computation for
CSG
products with Shapely 2.0 - FIX: Shapely 2.0 deprecation warnings
- DEPR: Add deprecation warning for EOReader spectral indices (used for legacy in 0.18.0) that are aliases of ASI names (#72):
AFRI_1_6
:AFRI1600
,AFRI_2_1
:AFRI2100
,BSI
:BI
,NDGRI
:NGRDI
,NDRE1
:NDREI
,RGI
:RGRI
,WV_BI
:NHFD
,WI
:WI2015
,RDI
:DSI
,DSWI
:DSWI5
,GRI
:DSWI4
,WV_SI
:NDSIWV
,PANI
:BITM
- DOC: Changing copyright from 2022 to 2023
- FIX: Fix regression for missing EOReader aliases for
spyndex
spectral indices - FIX: Fix bug in footprint computation of DIMAP V1 data
- DOC: Add latest DOI link
- LIB: Pass to
sertit==1.21.0
to handle windowed data in read natively
- BREAKING CHANGES: Refactoring spectral indices management (#47)
- Using spyndex library, allowing to use all spectral indices listed here
- SAR products may now compute indices if possible (see this list)
- Old EOReader indices are still available for legacy purposes, with some changes:
- For Sentinel-2 data, the band
NIR
andNARROW_NIR
may be interchanged for some index ( see this discussion) - OSAVI formula has changed to stick with the original paper definition (see issue)
NDRE2/3
formula are fixed, now usingVRE_2/3
andNDRE1
corresponds toNDREI
and usesVRE_1
CI1
is renamedCI32
andCI2
is renamedCI21
for readability purposesNDWI21
can be writtenNDWI2100
for homogeneity purposesRDI
(orDSI
) uses nowSWIR_1
instead ofSWIR_2
(see this issue)PANI
equivalent is nowBITM
and is normalised ! (divided by 3)SBI
is normalized (divided by 2) to fit withBIXS
definition- ⚠ You may need to install the last
spyndex
directly from GitHub latest version to have all available indices
- For Sentinel-2 data, the band
- BREAKING CHANGES: Using
pyresample
to geocode Sentinel-3 data (#55)- Cleaner: better conversion from swath to grid
- Faster: Up to 4 times faster
- Allows code refactoring between OLCI and SLSTR
- BREAKING CHANGES: For SAR product types that are not available in the Data Access Portfolio, default resolution is now the pixel spacing instead of the rg x az resolution
- Changes mainly Sentinel-1 default resolutions (except from IW mode)
- ENH: Adding the support of Harmonized Landsat-Sentinel constellation (#49)
- ENH: Adding the support of GEOSAT-2 constellation (#59)
- FIX: Fixing
CustomProduct
initialization when fields are set to None (instead of not declaring them) - FIX: SNAP cannot handle float predictors other than 1! Set it to 1 when saving ortho SAR images to disk, in order for SNAP to be able to despeckle them. See SNAP issue. (#62)
- FIX: Fixing mix in
Sentinel-2
mapping forB8
(NIR
, 10m resolution, large spectral bandwidth) andB8A
(NARROW_NIR
, 20m resolution, narrow spectral bandwidth)
- DOC: Add FAQ entry concerning SAR constellations extent KML files failing to be read (TLDR: needs
ogr2ogr
in your PATH) - DOC: Add Technical Note published in Remote Sensing MDPI in Readme
- DOC: Update optical band mapping graphs (fix regression to 0.15.0 supported constellation)
- DOC: Add information about DEM management in SAR notebook (#61)
- DOC: Updating indices paragraphs
- CI: Using actions/checkout@v3
- CI: Updating versions of pre-commit hooks
- LIBS: Updating
requirements.txt
andsetup.py
to addpyresample
andzarr
- ENH: Adding the support of RapidEye constellation
- ENH: Handling Planet data with multiple subdatasets (#45)
- ENH: Adding the support of Landsat Level-2 products (#49)
- ENH: Adding the support of Pleides Neo SEN and PRJ products (needs GDAL 3.5+ or rasterio 1.3.0+)
- ENH: Adding the function
bands.is_thermal_band
- ENH: Adding the ability for optical custom stacks to load indices
- ENH: Adding BAIM (MODIS Burned Area Index) spectral index
- ENH: Better management of raw units of the bands of optical products
- ENH: Copying files from
tmp_process
when changing product's output
- FIX: Stacks saved as integers on disk keep their original dtype (float32) in Python
- FIX: Stacks with bands loaded "as is" are correctly saved as integers on disk (#52)
- FIX: Using stack CRS (if projected) for
DIMAP
products instead of recomputing from lat/lon, solving potential discrepancies between stack and product CRS - FIX: Workaround for JP2 bug when updating an existing raster (maybe related to this bug)
- FIX: Better management of SkySat datetime conversion from JSON to XML (deterministic way)
- FIX: Fixing computation of invalid pixels for
Sentinel-2
andDIMAP
products (do not remove straylight mask) - FIX: Fixing reprojection resolution of VHR data
- FIX: Computing Brightness Temperature of
Landsat
TIR bands instead of leaving them as is - FIX: Better management of Landsat Instrument values
- FIX: Better radiometry attribute (adding
brightness temperature
andreflectance and brightness temperature
values) - FIX: Changing
Brilliance Temperature
to the correctBrightness Temperature
- FIX: Fixing pandas FutureWarning
The frame.append method is deprecated and will be removed from pandas in a future version.
- FIX: Fixing DeprecationWarning
invalid escape sequence \.
- FIX: Manage correctly Planet dubious pixels (especially for 8 bands products)
- OPTIM: Reduce memory usage when updating all the bands attributes
- OPTIM: Reduce memory usage when stacking as integers
- DOC: Add the need of using SNAP 8.0 up-to-date or SNAP 9.0 (#42)
- DOC: Add the STAC session in API documentation
- DOC: Add warnings for shifts when orthorectifying DIMAP SEN products (using RPCs) (#53)
- DOC: Add limitations to custom stacks
- DEPS: Dropping support of Python 3.7 (#18)
- DEPS: Update minimum version of libs (geopandas 0.11.0+, rasterio 1.3.0+...)
- FIX: Add the missing conversion to reflectance for
Sentinel-3 OLCI
- FIX: Better condition for the conversion to reflectance for
Sentinel-2 THEIA
- FIX: Add logs for
SkySat
data that cannot been converted to reflectance and fix theradiometry
field of its band xarrays - FIX: Add the correct nodata (when overridden by the user) to stacks saved as uint16
- OPTIM: Reduce memory usage during stacking
- CI: Test reflectance values
- ENH: Adding the support of SuperView-1 constellation (#21)
- ENH: Adding the support of SPOT-4/5 constellations (#39)
- ENH: Allow the possibility to pass a constellation (or a constellation list) to
Reader().open()
to speed up the opening of a product - ENH: Add a quicklook search for
Sentinel-3
products
- FIX: Fix quicklook media type with
JP2
files - FIX: Fix
Sentinel-3 SLSTR
F1
bands based on F grid - FIX: Correct the UTM projection for
Sentinel-3
data - FIX: Fix handling of zipped
Sentinel-2 L2Ap
- FIX: Fix zipped
Sentinel-2
with other XML files in GRANULE subdirectories
- Renaming
master
branch tomain
- OPTIM: Try to create
Vision-1
footprint from the preview file instead of from the stack. - OPTIM: Create footprints for stacked products (i.e.
Maxar
,SkySat
,Custom
...) without mask by opening only the first band of the stack - OPTIM: Create footprints for
Maxar
Products with a resolution 10 times lower. - OPTIM: Footprints have now maximum 50 vertices in order to avoid pixelized footprints
- FIX: Fixing condensed name to avoid duplicates:
- adding the
job_id
forVHR
products - adding the polarization channels for
SAR
products
- adding the
- FIX: Remove import of pystac in
stac_utils
- FIX: Fix bug for
Vision-1
data looking for non-existing RPC files in case ofORTP
product type - FIX: Fix quicklook regex for
Vision-1
data - FIX: Fix regex for raw bands for extracted
Sentinel-3 OLCI
products - FIX: Fix
PlanetScope
identifying regex to handle products with a satellite_id containing a letter - FIX: Force metadata regex for
Maxar
products to look for a file with pattern{name}.XML
to avoid other misplaced XML to be found in place of the true XML. - FIX: Fix regression for
Landsat-7
footprint - FIX: Manage the case with
cloud_cover = -999.0
forMaxar
products (returnsNone
)
- CI: Add new optical products to be tested for end-to-end tests
- BREAKING CHANGES:
Optical
becomesSpectral
when more appropriate - BREAKING CHANGES:
Platform
andSensor
becomeConstellation
when more appropriate, to fit STAC vocabulary (#29):Platform
enum becomesConstellation
prod.platform
becomesprod.constellation
prod.sat_id
becomesprod.constellation_id
- BREAKING CHANGES: File
alias
is removed, replaced by*_bands
files and proper imports inbands.__init__
- BREAKING CHANGES: Product attribute
band_names
becomesbands
in order to be STAC compliant (#29) - BREAKING CHANGES: Better use of
NIR
andNARROW_NIR
in theindices
file (according to the gsd ofSentinel-2
bands composing the indices) - BREAKING CHANGES: Correcting Landsat product types to better manage processing levels and instrument. Landsat-8/9 condensed name may change!
- ENH: Adding the support of
SkySat
(Collect) products (#20) - ENH: Bands in mapping are now objects, instead of just IDs (#29). This allows us to:
- Add band metadata (such as center wavelength, bandwidth...)
- Map spectral bands between STAC spec and EOReader format (#29)
- Add a better
__repr__
functions
- ENH: Handling 8 bands
PlanetScope
data (#20) - ENH: Adding the
GREEN1
mapped band, corresponding to PlanetScopeGREEN I
andSentinel-3 OLCI
Oa05
band - ENH: Handle some slightly broken
Sentinel-2
products:- when the metadata files are corrupted or when the detfoo vectors are empty (#34)
- with missing MSK prefix for QI_DATA files (i.e.
DETFOO
instead ofMSK_DETFOO
)
- ENH: Handle exception for corrupted bands (in
Sentinel-2
andutils.read
) (#34) - ENH: Add a STAC object that can be used to retrieve STAC Items from every Product (
prod.stac.create_item()
) (#29) - ENH: Add a
get_mean_viewing_angles
for Optical Products to fill STAC View Extension (#29) - ENH: Extending
get_raw_band_paths
to every product (#31) - ENH: Adding a
is_ortho
attribute corresponding to when the product is already orthorectified/geocoded, in order to avoid computing heavy processes without wanting it (i.e. footprint...) - ENH: Adding the instrument name of every constellation, under
prod.instrument
- ENH: Handling
COSMO
product with only theh5
file in it (if missing XML metadata file) (#36)
- OPTIM: Retrieve name from filename if possible
- OPTIM: Retrieve extent from metadata when possible (for VHR data)
- FIX: Fixing the band mapping of
WorldView-2/3 Multi
(8 bands) - FIX: Retrieval (if possible) of Sentinel-1 unique ID (was missing from the product name, as it is not in the product preview)
- FIX: Fixing PAZ/TDX MTD regex
- FIX: Optical products: Only set cloud cover and radiometry attributes if spectral bands are asked
- INTERNAL: File
spot_6
andspot_7
are removed, replaced by a uniquespot
file. This shouldn't affect the user. - INTERNAL: Refactoring Landsat-XX products into
LandsatProduct
, this should be invisible for user. - INTERNAL: Some refactoring in
VHR
files - WARNINGS: Filter warnings from
__init__
- CI: Do not process two times the zipped Sentinel-1 in end-to-end tests and manage when the runner kills SNAP
- DOC: Adding a STAC notebook
- DOC: Various updates
- BREAKING CHANGES:
footprint
,extent
,wgs84_extent
andcrs
properties are converted back to methods in order to prevent side effects of expensive computation when displaying the object when debugging (rollback before version 0.8.0) - BREAKING CHANGES:
get_all_index
becomesget_all_indices
- BREAKING CHANGES:
acquisition_datetime
becomesdatetime
anddefault_resolution
becomesresolution
forCustomProduct
in accepted keywords, and the metadata fields have been renamed according to theCustomFields
enum
- ENH: Adding spectral indices:
- Shadow Index (
SI
) - Global Vegetation Moisture Index (
GVMI
) - Soil Brightness Index (
SBI
), Soil Cuirass Index (SCI
) - Panchromatic mocking Index (
PANI
) - Green-to-Red ratio Index (
GRI
) - Soil Adjusted Vegetation Index (
SAVI
) - Optimized Soil Adjusted Vegetation Index (
OSAVI
) - Visible Atmospherically Resistant Index (Green) (
VARI
) - Enhanced Vegetation Index (
EVI
) - Chlorophyll Index RedEdge VRE_3/VRE_2 (
CI1
) - Chlorophyll Index RedEdge VRE_2/VRE_1 (
CI2
) - Normalized Difference Moisture Index (with SWIR_21) (
NDMI21
)
- Shadow Index (
- ENH: Making SAR attribute
snap_filename
public - ENH: Handling
ICEYE
pure SLC products - ENH: Allowing the user to choose if they want the GRD or SLC image for
ICEYE
products - ENH: Add the possibility to directly load the cloud cover for optical data (and add it in the band attributes) (#28)
- ENH: Add the possibility to retrieve the quicklook path (if existing) and add the
plot
function allowing the user to plot the quicklook (if existing) (#28) - ENH: Add the possibility to retrieve the orbit direction (and add it in the band attributes) (#28)
- FIX: Fixing the inversion between
8
and8A
bands forSentinel-2
andSentinel-2 Theia
products - FIX: Loading every optical band in reflectance (fixed for
Sentinel-2 THEIA
,Maxar
,Planet
andVision-1
data) (#30) - FIX: Fixing
ReferenceError: weakly-referenced object no longer exists
when deleting an object - FIX: Do not set sea values to nodata when orthorectifying SAR data with SNAP
- FIX: Handle
Sentinel-2
data with PB < 02.07 asL2Ap
products - FIX: Fixing nodata and offset for
Sentinel-2
data with PB > 04.00 - FIX: Handle new
ICEYE
metadata name's nomenclature - FIX: Fixing harmless regex error when searching for B1 path for
Landsat
products - FIX: Fixing platform for
Sentinel-2 Theia
- DOC: Creating a real
base
notebook and renaming the old one tooptical
- DOC: Better type hints (replacing
XDS_TYPE
byxr.DataArray
) - CI: Using
sertit.ci.reduce_verbosity
instead of recreating the function
- FIX: Handling
Sentinel-2 L2Ap
data - FIX: Do not use
--no-binary fiona,rasterio
directly inrequirements.txt
(breaks on Windows) - FIX: Fixing stacking with string bands
- FIX: Better
__repr__
function - FIX: Read README as UTF-8 in setup.py
- CI: Adding a tag for choosing the runners
- DOC: Fixing cartopy/GEOS conflicts making the documentation build to fail
- ENH: Adding the support of
Landsat-9
sensor - ENH: Support Sentinel-2 with missing datatake metadata file(sometimes happens with data downloaded from AWS buckets and converted to .SAFE)
- FIX: Using default SAR resolution from official Copernicus Data Access Portfolio (2014-2022) ( Sentinel-2 default resolution goes to 10.0 m !)
- FIX: Use
--no-binary fiona,rasterio
directly inrequirements.txt
- FIX: Removing useless
outputComplex
line in GPT graphs that is breaking SNAP on Linux - FIX: Removing the workarounds caused by some bugs of
cloudpathlib
and enabling retrieval of nested SAR products (TSX, TDX, PAZ, RCM) from S3 compatible storage. - FIX: Do not process nodata for a band already existing
- FIX: Fixing an error when reading
TIR
bands with Landsat-7 - FIX: Fixing an error when additive/multiplicative coefficients are set to
NULL
for Landsat data - FIX: Returning sun angles always as float (some
Sentinel-3
angles were returned asnp.array
)
- CI: Do not try to process SAR end to end if GPT cannot be found
- CI: Publishing wheel from GitHub instead of Gitlab
- REPO: Setting GitHub as the main repository and using new Gitlab runners
- ENH: Adding the support of
Pleiades-Neo
,Vision-1
andSAOCOM
sensors - ENH: Adding a keyword to allow passing a specific DEM path in
load
/stack
(for VHR orthorectification andDEM
bands) - ENH: Adding the name of the DEM in DEM band (i.e. allow to compute the
HILLSHADE
with a DEM and theSLOPE
with a DTM)
- FIX:
Sentinel-2
Processing Baseline 04.00:NARROW_NIR
bands are now loaded correctly - FIX:
Maxar
products (withMulti
band ID) are now correctly handled - FIX: Using
COPDEM-30
(GLO-30
) by default for SNAP as it appears that the retrieval has been fixed. - FIX: Fixing the default name for cleaned bands for
Sentinel-3 SLSTR
data (was set onCLEAN
instead ofNODATA
) - FIX: Fixing default band for Custom stacks
- FIX: Fixing
get_existing_band_paths
behavior for Custom stacks - FIX: Remove other never covered lines of code (archived
RCM
products, complexICEYE
products, others...) - FIX: Re-enabling loading str bands (regression)
- FIX: Proper check for empty fields when parsing metadata
- FIX: VHR
_get_dem_path
raisesValueError
instead ofTypeError
- FIX: Pre-process SAR bands before despeckling if not existing (was OK in most of the cases, but broke in some cases, especially with CI folder activated and S3 compatible storage)
- FIX: Remove warning
invalid escape sequence \.
,\w
,\D
and\s
- FIX: Do not set
long_name
forRAW_CLOUDS
arrays - FIX: Providing a URL DEM on Windows throws a
OSError
instead of a bareException
- OPTIM: Do not pre-process existing Sentinel-3 geocoded bands
- OPTIM: Do not look for valid metadata further than a given nested level in product's directory (for extracted products)
- CI: Using another (faster) runner
- CI: Add on disk and end-to-end tests
- CI: Do not write tmp files when running on disk tests
- CI: Coverage:
- Get coverage as HTML
- Remove useless lines from coverage
- Combine coverage of S3 and on disk tests
- DOC: Adding a DEM notebook
- FIX: Fixing archived SAR processing
- FIX: Needs extraction for
RS2-SLC
data as SNAP does not handle the product - FIX: Fixing the default name for cleaned bands for optical data (was set on
CLEAN
instead ofNODATA
)
- FIX: Fixing complex and orthorectified products for
SAR
data - FIX: Fixing
RADARSAT-2
SLC
product type
- OPTIM: Only preprocessing wanted SAR bands (instead of all existing)
- OPTIM: Do not interpolate nan values by default when writing SAR bands to disk (using a keyword instead)
- DOC: Updating the SAR notebook and documentation
- BREAKING CHANGES: Renamed
is_band
tois_sat_band
to better reflect that this function only checks optical and SAR bands - BREAKING CHANGES: Invalid pixels are not processed by default anymore! Only the nodata is set (to go a bit faster)
- ENH: Allowing the user to choose the pixel processing for optical bands: raw band, only nodata or total cleaning of defective pixels (#16)
- ENH: Adding a CustomProduct, allowing the user to load any stack as an EOReader Product !
- ENH: Check if a band exists before trying to load it
- FIX: Better handling of
__all__
in__init__.py
files - FIX: Ensure that extents and footprints are in UTM
- FIX: Removing docs from wheel
- FIX: Fixing
TIR
bands reading for Landsat data
- OPTIM: Optimizing
manage_invalid_pixels
forSentinel-2
data (processing baseline >= 04.00)
- DOC: Update README, documentation and notebooks
- DOC: Water Extraction notebook has been refined to show how to manage multiple products
- DOC: Update the installation paragraph in README
- DOC: Adding a
For Contributors
section in the documentation (contributing, release history and GitHub repository) - DOC: Remove doc testing in GitHub (as the docs are built with readthedocs)
- INTERNAL: Better management of project metadata (version...) in a dedicated file
- FIX: Resolve a bug when
methodtools
is not present (for conda package)
- ENH: Adding
has_bands
to products, ingesting lists as a shortcut for testing the availability of multiple bands - ENH: Simplifying imports. Now you can replace:
from eoreader.bands.alias import RED, NDVI
byfrom eoreader.bands import RED, NDVI
,from eoreader.products.optical.optical_product import OpticalProduct
byfrom eoreader.products import OpticalProduct
,from eoreader.products.optical.s3_slstr_product import SlstrRadAdjustTuple
byfrom eoreader.products import SlstrRadAdjustTuple
, ...
- OPTIM: Writing cloud bands on disk to speed up multiple calls to
load
orstack
functions (#17)
- FIX: Correctly naming cloud xarrays
- FIX: Add missing
SLEA
(Spot Extended Area) product type toICEYE
data - FIX: Sentinel-2 clouds (with processing baseline >= 4.0) are now given with a rasterio shape (
count
,height
,width
)
- CI: Remove
pages
stage and run only the tests when a Python file has changed - DOC: Updating notebooks
- DOC: Updating copyright to 2022
- FIX: Do not force import
methodtools
(not existing lib in conda) - FIX: Using
GRD
resolution given by the constructors as default values forSLC
products. Do not look it up in metadata as SLC resolution is NOT the GRD resolution !
- FIX: Caching properties and functions only for object instances
- FIX: Fixing metadata reading for
COSMO-SkyMed 1st Generation
withWide Region
and complex product type (handling of multiple swaths) - FIX: Updates of SNAP GPT graphs for complex SAR data
- FIX: Interpolate nodata inside SAR images (badly handled by SNAP -> fill the gaps that shouldn't exist)
- INTERNAL: Creation of a class
CosmoProduct
handling generic methods for bothCOSMO-SkyMed
generations
- FIX: Fixing the search for
.TIL
files forMaxar
products (with on disk files) - FIX: Fixing the search for metadata files for
Landsat
products (with on disk files) - FIX: Fixing the search for metadata files for
TerraSAR-X
,TanDEM-X
andPAZ SAR
products (with on disk files) - FIX: Fixing SNAP files for
TerraSAR-X
,TanDEM-X
andPAZ SAR
products - FIX: Fixing when reading CRS code for
DIMAP
products
- FIX: Fixing flag type for
Sentinel-3
data - FIX: Do not multiply the flags values by the radiance adjustment factor for
Sentinel-3 SLSTR
! - FIX: Fixing flag exception threshold for
Sentinel-3 SLSTR
- FIX: Fixing preprocessed band filenames for
Sentinel-3 SLSTR
- FIX:
Reader().valid_mtd
now correctly accepts strings instead of onlyPlatform
objects - FIX: Better handling of
Sentinel-2
product type - FIX: Save bands' new attributes in
str
(to pickle them) - FIX: Add a
clear()
function to clear products cache
- ENH: Adding the support of the ICEYE sensor
- ENH: Adding the support of the COSMO-SkyMed 2nd Generation sensor
- ENH: Adding some attributes to bands and stack:
sensor
,sensor_id
,product_type
,acquisition_date
,condensed_name
#7 - ENH: Replace name by filename and read directly the true name of the product in the metadata (#15)
- FIX:
Sentinel-1
metadata file with archived products (discarding RFI folder in its search). - FIX: Add
Quickbird
,GeoEye
andWorldView
sensors inreader
regexes. - FIX: Add scipy in
requirements.txt
andsetup.py
- DOC: Fix references to
pcigeomatics
that doesn't exist anymore (RADARSAT-2 and Constellation) - REQ: Update
dask
to fix a security issue (only in requirements asdask
is not mandatory)
- FIX: Do not force
chunk
inutils.read
if dask is not installed
- BREAKING CHANGE:
crs
,footprint
,extent
,wgs84_extent
are now properties ! - BREAKING CHANGE: Removing raw
gdaldem
CLI from EOReader (theHILLSHADE
andSLOPE
bands are now slightly different !) (#10) - BREAKING CHANGE:
HILLSHADE
is given infloat32
instead ofuint8
- BREAKING CHANGE:
SLOPE
is given in degrees instead of percents
- ENH: Adding the support of the PAZ SAR sensor
- ENH: Adding the support of the Sentinel-2 processed with the processing baseline 4.0 (#11)
- ENH: Removing SNAP from Sentinel-3 pre-process -> Freeing optical data from SNAP dependency ! (#12)
- ENH: Enabling the use of other S3-SLSTR suffixes than
an
(stripe A at nadir position) - ENH: Thermal bands of Sentinel-3 SLSTR can now be used
- ENH: All bands of Sentinel-3 SLSTR/OLCI can now be used (
S7
,F1
,F2
for SLSTR,Oaxx
for OLCI) (#14) - ENH:
YELLOW
band is mapped toOa07
band of Sentinel-3 OLCI - ENH: Zipped Sentinel-3 products can now be processed
- ENH: Allow the use of
kwargs
inload
, mainly forrasters.read
(and allowing i.e. radiance adjustment in S3-SLSTR)
- OPTIM:
crs
,footprint
,extent
,default_transform
,wgs84_extent
are cached ( using@cached_property
) (#13) - OPTIM:
get_mean_sun_angles
anddefault_transform
are now cached ( using@cache
) (#13) - OPTIM:
get_datetime
: Look for the date only ifdatetime
attribute is None (#13) - OPTIM: Better management of
fspath
for cloud-stored products (download the files only once) - OPTIM: Stop downloading/extracting files if not necessary
- FIX: Bands are correctly ordered in stacks
- FIX: Only load a band once, even if asked several time in the bands
- FIX: Use band size for cleaning optical pixel (instead of user resolution/size)
- FIX: Always take the absolute value of the resolution when converting it to strings (for filenames)
- FIX: Take the default resolution if nothing is given when converting it to strings (for filenames)
- FIX: Always use
utils.read/write
instead ofrasters.read/write
(for Dask management) - FIX: Fixing a bug in
utils.write
- FIX: Add .xml files from
eoreader/data
in the MANIFEST.in - FIX: Add forgotten
@abstractmethod
where needed - FIX: Better management of
_tmp_process
- FIX: Fixing minor bug when trying to read metadata with a POSIX path
- FIX: Fixing the
**kwargs
omission inutils.read
- FIX: Better management of
_temp_process
directory - FIX: Landsats and TSX: Can use other filenames now
- DEPR:
FAR_NIR
band is removed - REQ: Using
h5netcdf
instead ofnetCDF4
- DOC: Add a Context paragraph in the README
- DOC: Add a Conda x SNAP question in the FAQ
- DOC: Creation of a Sentinel-3 notebook
- DOC: Updates of notebooks
- DOC: Numerous updates
- FIX: Fixing a bug when opening archived Sentinel-1 (wrong metadata file found)
- DOC: Updating CSS and readme
- ENH: Implementing RADARSAT-Constellation products (as
RCM
) - ENH: Implementing Maxar products (such as
GE01, WV02, WV03, WV04
, but others should be supported too) - ENH: Implementing TanDEM-X products (as
TDX
) - ENH: Adding
RH
,RV
,RH_DSPK
andRV_DSPK
SAR bands - ENH: Adding the
YELLOW
optical band (forWorldView-2
,WorldView-3
andSentinel-3 OLCI
) - ENH: Adding WorldView index (without the ones using SWIR)
- ENH: Loading by size -> round resolution to the closest meter (or decimeter for resolution < 1.0m)
- ENH: Super class for VHR data
- FIX: Fixing reading PlanetScope archived products (error in read band)
- FIX: Fix band name with complex resolutions
- FIX: Fixing minor bug in RADARSAT-2 data when looking for product type
- FIX: Fixing SAR band search in BEAM-DIMAP files
- FIX: Fixing python version in environment.yml
- FIX: Discard unused MIR and FNIR bands
- FIX: Check for existence of given path when reading any product
- FIX: Workaround for a bug involving some downloaded but badly formatted archives for Sentinel-2
- FIX: Allow NARROW_NIR for and DIMAP data (== NIR)
- FIX: Better management of writeable band folder
- DOC: Fix documentation of the NDWI index
- DOC: Update graph for optical band mapping
- CI: Adding a test loading invalid band name
- CI: Setting CI log level to DEBUG
- CI: Accelerating the CI processes
- FIX: Sentinel-3 band mapping (
Coastal Aerosol
<->03
,BLUE
<->04
)
- DOC: Adding an interactive graph for optical band mapping
- FIX: Load works with string bands (
prod.load('BLUE')
) - FIX: Fixing missing
_remove_tmp_process
for products needing extraction - FIX: Remove multi converting for Sentinel-3
- FIX: Better handling of archives for products that needs extraction
- FIX: TerraSAR-X products need to be extracted to be processed by SNAP !
- FIX: Fixing critical bug for Sentinel-3 (mapping between clean bands and SNAP bands)
- ENH: Ensuring EOReader supports Dask
- FIX: Fixing and adding BAIS2 index in alias
- FIX: Fixing GLI index
- FIX: Fixing a bug when writing reprojected DIMAP band
- FIX: Fixing a bug with SCS Cosmo-SkyMed data
- DOC: Adding a DASK notebook
- DOC: Updating notebooks
- BREAKING CHANGE: Read metadata/namespaces only once and store it as a private member. Keep accessing it through the
read_mtd
function (#9) WARNING: Breaking change for Landsat:read_mtd()
loses the argumentforce_pd=True
as it always returns an Etree
- ENH: Adding the BAIS2 index
- ENH: Reads Sentinel-3 global attributes as metadata:
absolute_orbit_number
comment
contact
creation_time
history
institution
netCDF_version
product_name
references
resolution
source
start_offset
start_time
stop_time
title
ac_subsampling_factor
(OLCI
only)al_subsampling_factor
(OLCI
only)track_offset
(SLSTR
only)
- ENH: Refining Despeckle Graph (#6) to use a more usual filter (
Refined Lee
) - ENH: Allowing the user to open the datatake metadata for Sentinel-2 products
- FIX: Decoupling classic metadata reading from the name as EOReader accepts now modified product names (#9)
- FIX: Better handling of cloud-stored DEM (raising an exception for non-ortho DIMAP data as GDAL and rasterio does not handle that case)
- FIX:
environment.yml
to respect the stricter use offile:
syntax. See here for more information. - FIX: Fixing bug when opening an archive product with
name
mode and nested dictionary (when looking for a filename instead of the directory name)
- CI: Fixing
test_dems_https
and resetting DEM afterward - CI: Fixing DEM and ds2 database management
- DOC: Adding a FAQ page and enhancing the Main Features page (#3)
- DOC: Read Metadata has its own paragraph in Main Features
- ENH: Allowing
stack
to take single band in input instead of a list
- FIX: Fixing a regression loading optical bands which have been previously cleaned (Landsat, Theia, possibly PlanetScope)
- FIX:
load
andstack
always returnsfloat32
arrays
- CI: Testing if loading 2 times a band gives the same result
- FIX: Fixing a regression loading Landsat bands which have been previously cleaned
- ENH: Adding a
default_transform
function returning data from default band (without warping it) -> mappingcalculate_default_transform
fromrasterio
- ENH: Adding a
clean_tmp
function allowing the user to clean the product's temporary output by hand - ENH: Simplifying DEM warping code
- FIX:
DIMAP
products return always projected (in UTM) default bands (get_default_band_path
uses_get_default_utm_band
) - FIX: Theia Footprint returns a
GeoDataFrame
instead of aGeoSeries
- FIX: Better management of the
size
keyword withload
andstack
functions - FIX: Landsat retrieval of multipart cleaned bands (like
SWIR_1
) - FIX: Some typehints fixes
- FIX: Sentinel-3 open sun angles with NetCDF files stored in the cloud
- FIX: Silently pass the writing of clean bands if we cannot (like if permission error)
- CI: Do not reinstall everything if not needed (only PyYAML)
- CI: renaming
build
stage intolint
- CI: simplifying geometry comparison
- CI: Testing the
size
keyword for every sensor
- FIX: Fixing no data for Sentinel-3 cloud bands
- FIX: In alias:
DeprecationWarning: using non-Enums in containment checks will raise TypeError in Python 3.8
- CI: Set default S3 client to point to unistra's bucket
- CI: Tox SNAP relay S3_DB_URL_ROOT
- FIX: Adding condensed name in the search when loading S3-SLSTR clouds
- FIX: Fixing no data for Sentinel-3 bands processed by SNAP
- FIX: Fix bug when stack path's directory doesn't exist
- FIX: Do not verbose empty lists when loading optical bands
- FIX: Sentinel-3: Bands processed by SNAP are written with the condensed name as suffix
- FIX: Fixing another bug for DEM_PATH using S3 Paths
- DOC: Add a DOI
- FIX: Fixing DEM_PATH using S3 Paths
- ENH: Optimizing loading cloud bands for DIMAP Products
stack
accepts**kwargs
in order to pass options torioxarray.to_raster()
- FIX: Fixing not found masks with S3+zip Sentinel-2 products
- FIX: Fixing some type hints
- CI: Fixing network directories with pathlib
- ENH: Enabling the use of products stored in the cloud
(S3, S3 compatible storage, Google, Azure...) through
cloudpathlib
- ENH: Using correct band names in long_name
- CI: Use pre-computed cleaned band if existing
- DOC: Adding examples for using S3 data, especially for S3 compatible storage
- FIX: cloud mask values were inverted in Sentinel-2 cloud masks
- FIX: Landsat collection 2 cloud masks are now OK
- FIX: Improving stacks saved as uint16:
- Only satellite bands and index are scaled (*10.000)
- DEM bands are just rounded
- Cloud bands (booleans) are saved as is
- FIX: Fixing a rasterization bug affecting S2 and DIMAP masks, happening when the vectors have another size than the image
- FIX: Adding a warning on bad georeferencing when using GS and GT Landsat products
- CI: Minor updates in documentation and code
- ENH: Adding THR data support:
- PlanetScope
- Pleiades
- SPOT-6/7
- ENH: Better handling of SNAP DEMs (using External DEM and other available SNAP DEMs)
- FIX: More robust way of looking for
data
directory - FIX: Bug fix in
stack
that causes some bands to be inexplicably empty sometimes - FIX: Bug fix in
alias.isindex
- FIX: Forcing extent to UTM
- OPTIM: Write clean bands on disk to avoid redoing invalid pixel computation and allow the user to remove them on deletion
- OPTIM:
prod.has_bands
/prod.get_existing_bands
do not orthorectify/despeckle SAR bands anymore
- CI: Adding a bimonthly test for SNAP processes
- DOC: Adding two new notebooks (SAR and VHR data)
- DOC: Completing the documentation
- BREAKING CHANGES:
read_mtd()
returns a dict for the namespace map in order to manage multi namespace XMLs - ENH: Introduced support for DEM files from web urls (starting with http(s)://)
- FIX: Landsat Zenith angle computation (mixing elevation and zenith angle)
- CI: Adding weekly tests (
tox
on Python 3.7, 3.8, 3.9 on Linux and Windows) - DOC: Updating documentation, setting DEM path as an environment variable
- Migrating documentation to sphinx and readthedocs
- Refactoring documentation and ReadMe
- Bug correction when loading invalid masks for S2 sensor (fiona.errors.UnsupportedGeometryTypeError)
- Bug when reading cloud mask on Linux
- Log update in
product._check_dem_path()
- README & documentation updates
- Do not use psutil in requirements and setup.py
- Typo in setup.py
- In case of DEM bands, checks if the DEM is set and exists and raise an exception if not
- Setting minimum versions in setup.py and in requirements.txt
- ReadMe improvement
- Updating docs
- Bug in NDVI formula, typo when passing to xarray
- Correcting a broken link in setup.py
- Setting a minimum version of sertit
- Footprint:
- Fixing a bug in the default function
- Optimization for S2 and S2-Theia sensors
- Fixing L7 footprint which was too complex due to nodata stripes
- CI:
- Testing footprints and extent
- Resolving non update of gitlab documentation
- Changing the resolution of CI processes (to a multiple of the real resolution to speed up the computation)
- Multiple optimizations when reading and processing the bands
- Bug resolutions:
- S2-Theia masks
- SAR DSPK bands always recomputed
- SAR nodata set to 0 as SNAP expects it
- Bad mask nodata setting when computing invalid pixels
- Inverted default resolution between L4/5 MSS and TM sensors
- Removing useless logs when missing DEM but no computed DEM bands
- Adding copyright headers to every python files
- Fixing and adding examples
- Going Open Source