PER_BAND mask and MaskedArray #580

vincentsarago · 2023-03-06T23:12:30Z

Since early version of rio-tiler we always used either the alpha band or dataset_mask as mask, but in reality mask can be PER_BAND (e.g nodata masked value can be different from a band to another).

This PR Sketch out the switch to numpy.MaskedArray within the ImageData class and as output from rio_tiler.reader.read function.

ToDo

add more tests
update PointData class
update mosaic methods
documentation

rio_tiler/io/rasterio.py

vincentsarago · 2023-03-06T23:13:31Z

rio_tiler/models.py



 @attr.s
 class ImageData:
    """Image Data class.

    Attributes:
-        data (numpy.ndarray): pixel values.
-        mask (numpy.ndarray): rasterio mask values.
+        data (numpy.ma.MaskedArray): image values.


data -> array

vincentsarago · 2023-03-06T23:14:16Z

rio_tiler/models.py

+    def mask(self) -> numpy.ndarray:
+        """Return Mask in form of rasterio dataset mask."""
+        return numpy.logical_or.reduce(~self.array.mask) * numpy.uint8(255)
+


we can still use ImageData().data and ImageData.mask for compatibility

vincentsarago · 2023-03-06T23:14:39Z

rio_tiler/models.py

-            mask = ~numpy.logical_or.reduce(numpy.ma.getmaskarray(arr))
-            return cls(data, mask * numpy.uint8(255))
-
+        # TODO: Deprecate


from array is not needed anymore

vincentsarago · 2023-03-06T23:16:11Z

rio_tiler/reader.py

@@ -31,7 +31,7 @@ class Options(TypedDict, total=False):
    vrt_options: Optional[Dict]
    resampling_method: Optional[Resampling]
    unscale: Optional[bool]
-    post_process: Optional[Callable[[numpy.ndarray, numpy.ndarray], DataMaskType]]
+    post_process: Optional[Callable[[numpy.ma.MaskedArray], numpy.ma.MaskedArray]]


we should assume that post_process method will only work with MaskedArray

vincentsarago · 2023-03-06T23:16:30Z

rio_tiler/reader.py

-                out_shape=(height, width) if height and width else None,
-                resampling=resampling,
-                boundless=boundless,
+                masked=True,


👋 dataset_mask

vincentsarago · 2023-03-06T23:17:23Z

rio_tiler/reader.py

@@ -441,7 +437,7 @@ def point(
    resampling_method: Resampling = "nearest",
    unscale: bool = False,
    post_process: Optional[
-        Callable[[numpy.ndarray, numpy.ndarray], DataMaskType]
+        Callable[[numpy.ma.MaskedArray], numpy.ma.MaskedArray]


PointData will also be updated to use MaskedArray

vincentsarago · 2023-03-06T23:17:45Z

rio_tiler/utils.py

+                else numpy.nan,
+                "minority": float(keys[counts.tolist().index(counts.min())].tolist())
+                if valid_pixels
+                else numpy.nan,
                "unique": float(counts.size),
                **dict(zip(percentiles_names, percentiles_values)),
                "histogram": histogram,


Fix some issues when no valid data is present in the file

vincentsarago · 2023-03-08T16:05:01Z

tests/test_utils.py

+        arr = src.read(
+            masked=True,
+            out_shape=(src.count, int(src.height / 10), int(src.width / 10)),
+        )


speed up tests by reducing the array size

vincentsarago · 2023-03-08T16:05:25Z

tests/test_io_rasterio.py

        data = data * 2
-        return data, mask
+        data.mask = False  # set mask to False
+        return data


post_process now require a MaskedArray

vincentsarago · 2023-03-08T22:24:05Z

rio_tiler/reader.py

        if force_binary_mask:
-            mask = numpy.where(mask != 0, numpy.uint8(255), numpy.uint8(0))
+            pass
+            # mask = numpy.where(mask != 0, numpy.uint8(255), numpy.uint8(0))


This is the biggest breaking change! now that we use rasterio masked and numpy.ma.MaskedArray for the PER_BAND mask representation we won't support non-binary mask. It will always be True (0) or False (255).

This is a major downside for people using non-binary alpha bands, but sometimes we have to take hard decisions. (ref: rasterio/rasterio#1721 (comment))

vincentsarago · 2023-03-08T23:41:25Z

🤔 we may hit #105 again

…TilerV5

* update readme * PER_BAND mask and MaskedArray (#580) * sketch use of MaskedArray with PER_BAND mask * update PointData and fix tests * update mosaics * update nodata mask * remove unused * update changelog * migration guide * split resampling option to RasterIO and Warp options (#590) * refactor Mask tests and Benchmark * refactor Mask tests and Benchmark (#591) * change arguments in dynamic_tiler.md (#592) * change arguments The argument ("tile", {"z": "{z}", "x": "{x}", "y": "{y}"}) causes errors below. File "/home/ubuntu/app/app.py", line 48, in tilejson tile_url = request.url_for("tile", {"z": "{z}", "x": "{x}", "y": "{y}"}) TypeError: HTTPConnection.url_for() takes 2 positional arguments but 3 were given ("tile", z='{z}', x='{x}', y='{y}') is correct. * Update docs/src/advanced/dynamic_tiler.md --------- Co-authored-by: Vincent Sarago <vincent.sarago@gmail.com> * refactor MosaicMethods (#594) * refactor MosaicMethods * fix tests * Optional boto3 (#597) * make boto3 an optional dependency * update migration * update test dependencies * add flake8 rules in ruff (#600) * update dev version * update mosaic example notebook custom pixel_selection class (#602) * improve cutline handling (#598) * Improve cutline handling Closes #588 * handle projection --------- Co-authored-by: vincentsarago <vincent.sarago@gmail.com> * remove useless cache * save benchmarks * add benchmarking in docs * fix nodata/mask/alpha forwarding for GCPS dataset (#604) * remove boto3 * Allow clip-box to auto expand (#608) * Allow clip-box to auto expand * Add test * Update tests/test_io_xarray.py * Fix import order * Changes from black linter * Change default to true * use auto_expand in part and update changelog --------- Co-authored-by: Vincent Sarago <vincent.sarago@gmail.com> * allow morecantile 4.0 (#606) * allow morecantile 4.0 * set morecantile min version to 4.0 * update changelog * forward statistics from STAC raster:bands (#611) * forward statistics from STAC raster:bands * forward tags as metadata and forward metadata when creating ImageData from list * update changelog * handle nodata in XarrayReader (#612) * handle nodata in XarrayReader * update changelog * add AWS credential overrides for S3 stac (#613) * add AWS credential overrides for S3 stac * catch warnings * deprecated model methods (#615) * update readme --------- Co-authored-by: TTY6335 <36815385+TTY6335@users.noreply.github.com> Co-authored-by: Daniel Wiesmann <yellowcap@users.noreply.github.com> Co-authored-by: Aimee Barciauskas <aimee@developmentseed.org>

sketch use of MaskedArray with PER_BAND mask

e4ff9bd

vincentsarago commented Mar 6, 2023

View reviewed changes

rio_tiler/io/rasterio.py Show resolved Hide resolved

vincentsarago commented Mar 6, 2023

View reviewed changes

vincentsarago self-assigned this Mar 6, 2023

vincentsarago added the 5.0.0 label Mar 6, 2023

update PointData and fix tests

b4b6ea7

vincentsarago commented Mar 8, 2023

View reviewed changes

update mosaics

583dfd4

vincentsarago commented Mar 8, 2023

View reviewed changes

vincentsarago added the blocked label Mar 13, 2023

vincentsarago added 4 commits March 20, 2023 10:01

update nodata mask

e121eae

remove unused

afb5749

update changelog

ecb1440

migration guide

606102f

vincentsarago marked this pull request as ready for review March 23, 2023 15:50

vincentsarago added the breaking breaking change label Mar 23, 2023

vincentsarago changed the title ~~sketch use of MaskedArray with PER_BAND mask~~ PER_BAND mask and MaskedArray Mar 23, 2023

vincentsarago removed the blocked label Mar 23, 2023

vincentsarago added 2 commits March 24, 2023 18:21

update from main

2e1fb5c

update from main

ac45d52

vincentsarago changed the base branch from main to dev March 24, 2023 20:23

Merge branch 'dev' of https://github.com/cogeotiff/rio-tiler into rio…

2eb924a

…TilerV5

vincentsarago merged commit a5502ce into dev Mar 27, 2023

vincentsarago deleted the rioTilerV5 branch March 27, 2023 13:45

vincentsarago mentioned this pull request Jun 16, 2023

get_array_statistics fails when data is fully masked #617

Closed

vincentsarago mentioned this pull request Aug 22, 2023

enable partial transparency for output image when using Colormap developmentseed/titiler#689

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PER_BAND mask and MaskedArray #580

PER_BAND mask and MaskedArray #580

vincentsarago commented Mar 6, 2023 •

edited

Loading

vincentsarago Mar 6, 2023

vincentsarago Mar 6, 2023

vincentsarago Mar 6, 2023

vincentsarago Mar 6, 2023

vincentsarago Mar 6, 2023

vincentsarago Mar 6, 2023

vincentsarago Mar 6, 2023

vincentsarago Mar 8, 2023 •

edited

Loading

vincentsarago Mar 8, 2023 •

edited

Loading

vincentsarago Mar 8, 2023

vincentsarago commented Mar 8, 2023

PER_BAND mask and MaskedArray #580

PER_BAND mask and MaskedArray #580

Conversation

vincentsarago commented Mar 6, 2023 • edited Loading

ToDo

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincentsarago Mar 8, 2023 • edited Loading

Choose a reason for hiding this comment

vincentsarago Mar 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vincentsarago commented Mar 8, 2023

vincentsarago commented Mar 6, 2023 •

edited

Loading

vincentsarago Mar 8, 2023 •

edited

Loading

vincentsarago Mar 8, 2023 •

edited

Loading