t.rast.univar: Add region_relation option for spatial filtering STDS by computational region #2793

ninsbl · 2023-01-28T23:11:02Z

This PR adds a spatial_relation filter to t.rast.univar that makes the module compute univariate statistis only for maps with the given spatial relation to the computational region.

It also fixes an issue in the temporal library.

This is intended to be a first of a series of PRs that add the spatial filter module to other relevant temporal modules. So, the PR is in a way a template and a careful review would be very much appreciated.

Tests are added for the new option.

Two questions in this regards:

If no maps are returned because of a given where clause or because the computational region is else where, should that really lead to a fatal error? Or is not a warning and no output more appropriate?
Is more documentation of the added option in the manual needed?

veroandreo · 2023-02-05T20:04:29Z

I went through the changes in the test file, looks good to me 👍

As per your questions:

If the where returns nothing or the region is elsewhere do not seem like errors to me, so perhaps just a warning with a proper indication is enough - i.e., the selection yielded nothing or the region is elsewhere, please check or so.
Yes, definitely. However, as it will be something common to more temporal modules, perhaps we can include an explanation here: https://github.com/OSGeo/grass/blob/main/temporal/temporalintro.html

ninsbl · 2023-02-05T21:13:21Z

I went through the changes in the test file, looks good to me +1

Thanks, @veroandreo for looking into this and your feedback.

Will add something to the documentation and turn the error into a warning then.

ninsbl · 2023-02-12T21:50:29Z

@veroandreo Now I added some documentation. Please have a look, if that is an OK basis for adding functionality to other relevant modules!?!

temporal/t.rast.univar/t.rast.univar.html

temporal/temporalintro.html

wenzeslaus · 2023-02-13T17:18:03Z

Is this meant for 8.3.0? Given that it is a first of a series of PRs, there might be some benefit in leaving this for 8.4.0.

wenzeslaus · 2023-02-13T17:20:20Z

python/grass/temporal/abstract_space_time_dataset.py

-        elif spatial_relation == "is_contained":
-            spatial_where_template += " AND top <= {t}" " AND bottom >= {b}"
-        elif spatial_relation == "contains":
-            spatial_where_template += " AND top >= {t}" " AND bottom <= {b}"
+            elif spatial_relation == "is_contained":
+                spatial_where_template += " AND top <= {t}" " AND bottom >= {b}"
+            elif spatial_relation == "contains":
+                spatial_where_template += " AND top >= {t}" " AND bottom <= {b}"


This piece is a bug fix, so having it in a separate PR to go to 8.3.0 immediately would be nice, but it depends on how serious that is. Does not actually cause the code to fail?

Yes, the code would fail, but the library function is not yet used... Could indeed be separated out, maybe together with corrections / improvements of the library function doc....

wenzeslaus · 2023-02-13T17:22:21Z

python/grass/temporal/univar_statistics.py

@@ -126,6 +127,12 @@ def print_gridded_dataset_univar_statistics(
    :param rast_region: If set True ignore the current region settings
           and use the raster map regions for univar statistical calculation.
           Only available for strds.
+    :param dict spatial_relation: Spatial relation to the provided


It is str not dict. Same issue is the in the documentation of underlying functions.

wenzeslaus · 2023-02-13T17:28:17Z

python/grass/temporal/univar_statistics.py

-        err = "Space time %(sp)s dataset <%(i)s> is empty"
+        warn = "Space time %(sp)s dataset <%(i)s> is empty"
+        if spatial_relation:
+            warn += " or no maps with the requested spatial_relation to the computational region exist"
        if where:
-            err += " or where condition is wrong"
-        gs.fatal(
-            _(err) % {"sp": sp.get_new_map_instance(None).get_type(), "i": sp.get_id()}
+            warn += " or where condition does not return any maps"
+        gs.warning(
+            _(warn) % {"sp": sp.get_new_map_instance(None).get_type(), "i": sp.get_id()}


The original code is wrong. The message cannot be constructed using +=, but .format() needs to be used (or maybe f-string in some cases). The _() call needs to be around an actual string (which can be later combined with others).

Additionally, in major rewrites like this, try to replace things like %(sp)s by .format() (or f-strings if the base string is not translatable).

Addressed in cf49c4e

The .format() call needs to be on the result of _(). _() needs to be around the actual string "...".

Additionally, a sentence cannot be constructed from multiple strings (regardless of what is used to construct it, += or multiple function calls). While this particular case may work for most languages, generally, you need to work around it, here, e.g., by stating something general and then creating a list of possible detailed explanations.

This should result in one verbose(...) call which probably cleanest and fastest way of producing the message.

python/grass/temporal/univar_statistics.py

wenzeslaus · 2023-02-13T17:47:14Z

If no maps are returned...

I agree with @veroandreo that the it is simply an empty output, not an error. I would add that this usually the desired behavior when batch processing and automating. There is couple of these cases in the code. For db.univar, the new JSON output gives "nulls" while the original output formats fails (PR code). Notably, I decided not to include any warning at all. Text won't help in automation, but you can always test for it in the parsed JSON if needed. I think it can be argued that warning is not appropriate, it is an expected output given the inputs, but a verbose message might be useful when you did not expect the output and need to investigate.

wenzeslaus · 2023-02-13T18:00:44Z

As for the concept, I think the functionality is useful, but the option name is unclear. Spatial relation to what? The underlying function uses spatial relation but in combination with spatial extent. However, the new interface is always spatial relation to computational region extent. If your plan is to add spatial relation to other things besides the computational region extent, then the name makes sense.

I'm not sure why -r cannot be combined with spatial_extent. You can select the maps, but compute for the whole map, no? Perhaps the "ignore" in the description would have to be more nuanced (e.g., "Use the raster map regions...instead current region"). This would be more pronounced if the spatial relation is to some other things not only the region.

Co-authored-by: Veronica Andreo <veroandreo@gmail.com>

ninsbl · 2023-02-13T22:46:12Z

If no maps are returned...

I agree with @veroandreo that the it is simply an empty output, not an error. (...). Notably, I decided not to include any warning at all. Text won't help in automation, but you can always test for it in the parsed JSON if needed. I think it can be argued that warning is not appropriate, it is an expected output given the inputs, but a verbose message might be useful when you did not expect the output and need to investigate.

Changed to verbose message...

ninsbl · 2023-02-13T22:48:14Z

As for the concept, I think the functionality is useful, but the option name is unclear. Spatial relation to what? The underlying function uses spatial relation but in combination with spatial extent. However, the new interface is always spatial relation to computational region extent. If your plan is to add spatial relation to other things besides the computational region extent, then the name makes sense.

Renamed the module option to region_relation. Hope that is a bit more clear/precise. Any alternative suggestions are welcome...

I'm not sure why -r cannot be combined with spatial_extent. You can select the maps, but compute for the whole map, no? Perhaps the "ignore" in the description would have to be more nuanced (e.g., "Use the raster map regions...instead current region"). This would be more pronounced if the spatial relation is to some other things not only the region.

Indeed, removed the rule....

ninsbl · 2023-02-13T23:33:06Z

I am happy to separate out bug-fixes for 8.3 and keep the rest for 8.4, esp if a first 8.3 RC is around the corner. However, if 8.3 is on a more long term schedule, the alternative would be to speed up adding functionality to other relevant modules. As of now, there is no 8.3 branch yet, so the PR would practically be on hold until the 8.3 branch is created...

wenzeslaus · 2023-02-14T02:59:12Z

@neteler can comment on the schedule for 8.3, but in last few days, number of open PRs against main was reduced from >80 to <40 (by merging or moving the milestone), so we are getting reading for branching out.

neteler · 2023-02-14T08:58:30Z

@neteler can comment on the schedule for 8.3, but in last few days, number of open PRs against main was reduced from >80 to <40 (by merging or moving the milestone), so we are getting reading for branching out.

Seems all blockers are gone. Will write to grass-dev asap as the discussion doesn't fit here in a PR.

python/grass/temporal/abstract_space_time_dataset.py

python/grass/temporal/univar_statistics.py

wenzeslaus · 2023-02-21T15:30:15Z

python/grass/temporal/univar_statistics.py

-        err = "Space time %(sp)s dataset <%(i)s> is empty"
+        warn = "Space time %(sp)s dataset <%(i)s> is empty"
+        if spatial_relation:
+            warn += " or no maps with the requested spatial_relation to the computational region exist"
        if where:
-            err += " or where condition is wrong"
-        gs.fatal(
-            _(err) % {"sp": sp.get_new_map_instance(None).get_type(), "i": sp.get_id()}
+            warn += " or where condition does not return any maps"
+        gs.warning(
+            _(warn) % {"sp": sp.get_new_map_instance(None).get_type(), "i": sp.get_id()}


The .format() call needs to be on the result of _(). _() needs to be around the actual string "...".

Additionally, a sentence cannot be constructed from multiple strings (regardless of what is used to construct it, += or multiple function calls). While this particular case may work for most languages, generally, you need to work around it, here, e.g., by stating something general and then creating a list of possible detailed explanations.

This should result in one verbose(...) call which probably cleanest and fastest way of producing the message.

temporal/temporalintro.html

Co-authored-by: Vaclav Petras <wenzeslaus@gmail.com>

ninsbl · 2023-02-21T20:52:10Z

Thanks for your code review effort, @wenzeslaus . Good catches. Should all be addressed now.

ninsbl · 2023-02-27T20:22:58Z

@wenzeslaus Anything left to address here, or is the PR ready to merge?

wenzeslaus

Thank you for updating it!

ninsbl · 2023-09-29T10:12:22Z

OK to merge? I so, I will merge and take it as a template for adding spatial filtering to other temporal modules...

…by computational region (OSGeo#2793) * add spatial extent filter * pass only spatial relation * add spatial extent filter * fix 3D where * add tests for spatial filter * add more tests for spatial filter * fix failing test * add spatial relation doc * lower message severity * spatial_relation intro * Update temporal/t.rast.univar/t.rast.univar.html Co-authored-by: Veronica Andreo <veroandreo@gmail.com> * Update temporal/temporalintro.html Co-authored-by: Veronica Andreo <veroandreo@gmail.com> * rewrite user message * fix tests for empty strds * rename spatial filter option * rename spatial filter option * update r-flag description * rename spatial filter option * rename spatial filter option * improve spatial filter docs * fix failing test * Update python/grass/temporal/abstract_space_time_dataset.py Co-authored-by: Vaclav Petras <wenzeslaus@gmail.com> * Update python/grass/temporal/univar_statistics.py Co-authored-by: Vaclav Petras <wenzeslaus@gmail.com> * address code review * address code review * address code review, some linting * add back percentiles * black --------- Co-authored-by: Veronica Andreo <veroandreo@gmail.com> Co-authored-by: Vaclav Petras <wenzeslaus@gmail.com>

ninsbl added 6 commits January 28, 2023 22:00

add spatial extent filter

1cba081

pass only spatial relation

99a4760

add spatial extent filter

2e30546

fix 3D where

fafa82c

add tests for spatial filter

c081811

add more tests for spatial filter

9159eac

ninsbl added enhancement New feature or request temporal Related to temporal data processing Python Related code is in Python labels Jan 28, 2023

ninsbl added this to the 8.3.0 milestone Jan 28, 2023

ninsbl requested review from wenzeslaus and veroandreo January 28, 2023 23:11

fix failing test

59f2fbe

ninsbl added 3 commits February 12, 2023 22:10

add spatial relation doc

cd4ce6c

lower message severity

7945b9a

spatial_relation intro

be9f222

veroandreo approved these changes Feb 13, 2023

View reviewed changes

temporal/t.rast.univar/t.rast.univar.html Outdated Show resolved Hide resolved

temporal/temporalintro.html Outdated Show resolved Hide resolved

wenzeslaus requested changes Feb 13, 2023

View reviewed changes

ninsbl and others added 4 commits February 13, 2023 21:16

Update temporal/t.rast.univar/t.rast.univar.html

643bc4d

Co-authored-by: Veronica Andreo <veroandreo@gmail.com>

Update temporal/temporalintro.html

550947c

Co-authored-by: Veronica Andreo <veroandreo@gmail.com>

rewrite user message

cf49c4e

fix tests for empty strds

37fa7ff

ninsbl modified the milestones: 8.3.0, 8.4.0 Feb 13, 2023

ninsbl added 2 commits February 13, 2023 23:36

rename spatial filter option

2e12174

rename spatial filter option

1fc915f

improve spatial filter docs

0896d00

ninsbl requested a review from wenzeslaus February 13, 2023 23:22

fix failing test

0d7afa6

wenzeslaus reviewed Feb 21, 2023

View reviewed changes

wenzeslaus requested changes Feb 21, 2023

View reviewed changes

temporal/temporalintro.html Outdated Show resolved Hide resolved

ninsbl and others added 5 commits February 21, 2023 18:43

Update python/grass/temporal/abstract_space_time_dataset.py

9f7735a

Co-authored-by: Vaclav Petras <wenzeslaus@gmail.com>

Update python/grass/temporal/univar_statistics.py

f8496ec

Co-authored-by: Vaclav Petras <wenzeslaus@gmail.com>

address code review

593da28

address code review

403072a

address code review, some linting

52eab35

ninsbl requested a review from wenzeslaus February 21, 2023 20:52

wenzeslaus approved these changes Feb 27, 2023

View reviewed changes

ninsbl added 3 commits September 28, 2023 13:37

Merge branch 'main' into t_rast_univar_sr

37a1418

add back percentiles

493697d

black

3f52fca

ninsbl changed the title ~~t.rast.univar: Add spatial_relation filter~~ t.rast.univar: Add region_relation option for spatial filtering STDS by computational region Sep 29, 2023

ninsbl merged commit b20af2c into OSGeo:main Sep 29, 2023
18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

t.rast.univar: Add region_relation option for spatial filtering STDS by computational region #2793

t.rast.univar: Add region_relation option for spatial filtering STDS by computational region #2793

ninsbl commented Jan 28, 2023

veroandreo commented Feb 5, 2023

ninsbl commented Feb 5, 2023

ninsbl commented Feb 12, 2023

wenzeslaus commented Feb 13, 2023

wenzeslaus Feb 13, 2023

ninsbl Feb 13, 2023

wenzeslaus Feb 13, 2023

wenzeslaus Feb 13, 2023

ninsbl Feb 13, 2023

wenzeslaus Feb 21, 2023

wenzeslaus commented Feb 13, 2023

wenzeslaus commented Feb 13, 2023

ninsbl commented Feb 13, 2023

ninsbl commented Feb 13, 2023

ninsbl commented Feb 13, 2023

wenzeslaus commented Feb 14, 2023

neteler commented Feb 14, 2023 •

edited

Loading

wenzeslaus Feb 21, 2023

ninsbl commented Feb 21, 2023

ninsbl commented Feb 27, 2023

wenzeslaus left a comment

ninsbl commented Sep 29, 2023

t.rast.univar: Add region_relation option for spatial filtering STDS by computational region #2793

t.rast.univar: Add region_relation option for spatial filtering STDS by computational region #2793

Conversation

ninsbl commented Jan 28, 2023

veroandreo commented Feb 5, 2023

ninsbl commented Feb 5, 2023

ninsbl commented Feb 12, 2023

wenzeslaus commented Feb 13, 2023

wenzeslaus Feb 13, 2023

Choose a reason for hiding this comment

ninsbl Feb 13, 2023

Choose a reason for hiding this comment

wenzeslaus Feb 13, 2023

Choose a reason for hiding this comment

wenzeslaus Feb 13, 2023

Choose a reason for hiding this comment

ninsbl Feb 13, 2023

Choose a reason for hiding this comment

wenzeslaus Feb 21, 2023

Choose a reason for hiding this comment

wenzeslaus commented Feb 13, 2023

wenzeslaus commented Feb 13, 2023

ninsbl commented Feb 13, 2023

ninsbl commented Feb 13, 2023

ninsbl commented Feb 13, 2023

wenzeslaus commented Feb 14, 2023

neteler commented Feb 14, 2023 • edited Loading

wenzeslaus Feb 21, 2023

Choose a reason for hiding this comment

ninsbl commented Feb 21, 2023

ninsbl commented Feb 27, 2023

wenzeslaus left a comment

Choose a reason for hiding this comment

ninsbl commented Sep 29, 2023

neteler commented Feb 14, 2023 •

edited

Loading