t.register: support mapset name in input file #2863

ninsbl · 2023-02-27T21:48:21Z

Before this PR, input files for registering maps with fully classified map names like here:

prec_1@$MAPSET|2001-01-01|2001-04-01|semantic_label
prec_2@$MAPSET|2001-04-01|2001-07-01|semantic_label
prec_3@$MAPSET|2001-07-01|2001-10-01|semantic_label
prec_4@$MAPSET|2001-10-01|2002-01-01|semantic_label
prec_5@$MAPSET|2002-01-01|2002-04-01|semantic_label
prec_6@$MAPSET|2002-04-01|2002-07-01|semantic_label

threw an error. Now, maps in the register file can be specified with a fully qualified map name (including the mapset name), which should be even faster than using unqualified map names, as existence check is skipped.

wenzeslaus

Thanks for updating the test. Can you please enable the test by moving the file to testsuite directory? (I did that recently in b59a047 for r3.out.vtk.) There it would run in CI and fail it if the test fails. A different question is what the test actually tests automatically, but just testing that the modules don't fail is not bad for starters.

python/grass/temporal/register.py

ninsbl · 2023-02-28T09:57:45Z

Thanks for updating the test. Can you please enable the test by moving the file to testsuite directory? (I did that recently in b59a047 for r3.out.vtk.) There it would run in CI and fail it if the test fails. A different question is what the test actually tests automatically, but just testing that the modules don't fail is not bad for starters.

Sure. There are more test files though. And I am not sure if all tests are currently successful. I will have a look at them and evtl. port them to Python...

wenzeslaus · 2023-03-01T03:49:38Z

Seems totally unrelated, but v.to.lines test failed for macOS, have we seen this before?

 Running ./scripts/v.to.lines/testsuite/test_v_to_lines.py...
========================================================================
.========================================================================
FAILED ./scripts/v.to.lines/testsuite/test_v_to_lines.py - Timeout >300.0s

nilason · 2023-03-01T08:48:26Z

Seems totally unrelated, but v.to.lines test failed for macOS, have we seen this before?

 Running ./scripts/v.to.lines/testsuite/test_v_to_lines.py...
========================================================================
.========================================================================
FAILED ./scripts/v.to.lines/testsuite/test_v_to_lines.py - Timeout >300.0s

Mac timeout caused fails do occur, but rarely enough for me to not see any pattern.

wenzeslaus

I like that the similar code is replaced by a function, but I have doubts about when the function actually does and what is considered an error.

wenzeslaus · 2023-03-03T00:16:57Z

python/grass/temporal/register.py

+    element = {
+        "rast": "cell",
+        "raster": "cell",
+        "rast3d": "grid3",
+        "raster3d": "grid3",
+        "raster_3d": "grid3",
+        "vector": "vector",
+    }[type]


How comes it worked before without this?

It worked because

a) the find_file function replaces "raster" with "cell" (and then gives a message if in verbose mode that the user may not be able to understand, see:

grass/python/grass/script/core.py

Lines 1345 to 1347 in ca3b0a8

if element == "raster" or element == "rast":

verbose(_('Element type should be "cell" and not "%s"') % element)

element = "cell"

) and
b) if a 3d map was not found (which I guess never was the case) the function simply assumed it is in the current mapset, which may be true for the tests.

Makes sense, but perhaps best hidden in a library function which actually understands the user-facing types and does not complain about them, but that's for a different PR.

Addressed in find_file

python/grass/temporal/register.py

ninsbl · 2023-03-05T15:19:23Z

In order to disentangle the issues a bit here, I created two different feature branches in my fork from which I can create PRs if the direction to solve this is right.

Regarding find_file, I replaced the current translation of raster and rast to cell with a more comprehensive replacement for such element types that are used in the code. Here, the logic does not change but message level is decreased from verbose to debug, as it is more relevant for development any way. See: main...ninsbl:grass:find_file_elements for proposed changes.
Regarding the build_id function in abstract_map_dataset: Here, I would suggest to add a keyword argument check_mapset_element, which searches for a given element with the given name on the current search path, but only if an element type and no mapset is given. So the behavior remains the same. The costs for that function increase by one if-check, though. See here for how I would propose to implement it: main...ninsbl:grass:bstract_map_dataset_build_id

With those two changes in place, the function added in this PR would be obsolet and we could do:

# Vector maps
# With checks executed
AbstractMapDataset.build_id("streams:1", None, None, check_mapset_element="vector")
# Without existence checks
AbstractMapDataset.build_id("streams:1", "PERMANENT", None, check_mapset_element="vector")
AbstractMapDataset.build_id("streams", "PERMANENT", 1, check_mapset_element="vector")
AbstractMapDataset.build_id("streams:1@PERMANENT", None, None, check_mapset_element="vector")
AbstractMapDataset.build_id("streams:1@PERMANENT")
# Note that the existence of a layer is not checked for vector maps

# Raster maps
# With checks executed
AbstractMapDataset.build_id("elevation", None, None, check_mapset_element="raster")
AbstractMapDataset.build_id("elevation", None, None, check_mapset_element="cell")
# Without existence checks
AbstractMapDataset.build_id("elevation", "PERMANENT", None, check_mapset_element="raster")
AbstractMapDataset.build_id("elevation", "PERMANENT")
AbstractMapDataset.build_id("elevation@PERMANENT")
AbstractMapDataset.build_id("elevation@PERMANENT", "mapset_that_does_not_exist", None)

All would basically return the same id.

That would appear to me as the clearest implementation with least duplication / repetition of functionality and without changing current behavior.

Any thoughts, should I open PRs for the suggested changes to the two functions in question?

ninsbl · 2023-03-17T22:56:35Z

a7c0030 illustrates how this would be solved if #2882 was merged... (because of missing kwarg that comes with #2882, the tests should currently fail. Would need a rebase in case)...

ninsbl · 2023-03-23T10:11:10Z

Should be ready to merge as requested changes are addressed...

wenzeslaus

Thanks a lot. Addressing the other issues in separate PRs helped quite a bit here. Impressive amount of enabled tests and a test for the new and old functionality.

ninsbl · 2023-03-23T22:23:46Z

Thanks @wenzeslaus
There remain some shell script tests that have not been ported to Python, but that would be another PR.

wenzeslaus · 2023-03-24T00:38:43Z

...There remain some shell script tests that have not been ported to Python...

Simple git mv will take care of enabling them. Shell scripts are supported (at least on unix-like systems) so that these legacy scripts can at least run with minimal or no changes (another reason was to have a shell-user friendly way of creating tests, but that really did not help with contributions).

* support mapset in file * move some tests to Python * use new build_id_from_search_path * add documentation in the manual

ninsbl added 2 commits February 27, 2023 22:05

support mapset in file

0471798

test mapset in file

a96ce33

ninsbl added bug Something isn't working temporal Related to temporal data processing Python Related code is in Python labels Feb 27, 2023

ninsbl added this to the 8.3.0 milestone Feb 27, 2023

ninsbl requested a review from metzm February 27, 2023 21:58

wenzeslaus requested changes Feb 27, 2023

View reviewed changes

python/grass/temporal/register.py Outdated Show resolved Hide resolved

python/grass/temporal/register.py Outdated Show resolved Hide resolved

python/grass/temporal/register.py Outdated Show resolved Hide resolved

ninsbl added 6 commits February 28, 2023 23:17

handle find_file elements and results

6e81fb9

first Python code

c8bf7ca

port to Python

f9aa6e4

cleanup

552ac14

port to Python

70e83db

Delete accidentally created file

46eb80e

ninsbl added 5 commits March 2, 2023 00:12

fail early if map is not found

c521277

check register result

8d736bb

use function for mapid

2dc70bb

move test to Python

d7a06f8

port test to Python

b5c478d

wenzeslaus requested changes Mar 3, 2023

View reviewed changes

ninsbl added 3 commits March 5, 2023 14:01

check semantic labels

3f0f185

split tests

5c89a60

update

4195f98

black 23

9f4f9c9

This was referenced Mar 6, 2023

grass.script: find_file: translate more legacy elements #2881

Merged

grass.temporal: Add option for mapset check to AbstractMapDataset #2882

Merged

ninsbl added 2 commits March 12, 2023 00:11

rename classes

f5bbab7

assume OSGeo#2882 is merged

a7c0030

ninsbl force-pushed the t_register_mapset branch 2 times, most recently from de6ba10 to a7c0030 Compare March 20, 2023 21:21

ninsbl added 2 commits March 20, 2023 22:27

use build_id_from_search_path

557263b

add documentation

ff6ee68

ninsbl requested a review from wenzeslaus March 23, 2023 10:03

wenzeslaus approved these changes Mar 23, 2023

View reviewed changes

wenzeslaus removed the request for review from metzm March 23, 2023 13:56

ninsbl merged commit 6686aa7 into OSGeo:main Mar 23, 2023

neteler pushed a commit to nilason/grass that referenced this pull request Nov 7, 2023

t.register: support mapset name in input file (OSGeo#2863)

01c1347

* support mapset in file * move some tests to Python * use new build_id_from_search_path * add documentation in the manual

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

t.register: support mapset name in input file #2863

t.register: support mapset name in input file #2863

ninsbl commented Feb 27, 2023

wenzeslaus left a comment

ninsbl commented Feb 28, 2023

wenzeslaus commented Mar 1, 2023

nilason commented Mar 1, 2023

wenzeslaus left a comment

wenzeslaus Mar 3, 2023

ninsbl Mar 3, 2023

wenzeslaus Mar 3, 2023

ninsbl Mar 23, 2023

ninsbl commented Mar 5, 2023

ninsbl commented Mar 17, 2023

ninsbl commented Mar 23, 2023

wenzeslaus left a comment

ninsbl commented Mar 23, 2023

wenzeslaus commented Mar 24, 2023

	if element == "raster" or element == "rast":
	verbose(_('Element type should be "cell" and not "%s"') % element)
	element = "cell"

t.register: support mapset name in input file #2863

t.register: support mapset name in input file #2863

Conversation

ninsbl commented Feb 27, 2023

wenzeslaus left a comment

Choose a reason for hiding this comment

ninsbl commented Feb 28, 2023

wenzeslaus commented Mar 1, 2023

nilason commented Mar 1, 2023

wenzeslaus left a comment

Choose a reason for hiding this comment

wenzeslaus Mar 3, 2023

Choose a reason for hiding this comment

ninsbl Mar 3, 2023

Choose a reason for hiding this comment

wenzeslaus Mar 3, 2023

Choose a reason for hiding this comment

ninsbl Mar 23, 2023

Choose a reason for hiding this comment

ninsbl commented Mar 5, 2023

ninsbl commented Mar 17, 2023

ninsbl commented Mar 23, 2023

wenzeslaus left a comment

Choose a reason for hiding this comment

ninsbl commented Mar 23, 2023

wenzeslaus commented Mar 24, 2023