Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

t.rast.list: Add CSV, JSON, YAML outputs #2258

Merged
merged 4 commits into from
Mar 31, 2022

Conversation

wenzeslaus
Copy link
Member

@wenzeslaus wenzeslaus commented Mar 10, 2022

  • This adds CSV, JSON, and YAML output and generalizes the other outputs.
  • Method is now meant only for different data gathering methods, not for the output formatting. method=cols and method=comma are now obsolete.
  • All formatting is available with all methods.
  • Most code is shared between the two main branches of code (simple list versus delta/gran methods code).
  • Columns can be now specified for any option (previously only for cols/list and partially for comma/line output).
  • The change is backwards compatible both on module interface level and on Python API level.

There is no performance penalty for the more general, step-by-step code (at least with smaller data; several tests with perf give all times around 1.7s with the 6 map test).

Binder link

* This adds CSV, JSON, and YAML output and generalizes the other outputs.
* Method is now meant only for different data gathering methods, not for the output formatting. method=cols and method=comma are now obsolete.
* All formatting is avaiable with all methods.
* Most code is shared between the two main branches of code (simple list versus delta/gran methods code).
* Columns can be now specified for any option (previously only for cols/list and partically for comma/line output).
* The change is backwards compatible both on module interface level and on Python API level.
@wenzeslaus wenzeslaus force-pushed the exchange-formats-for-t_rast_list branch from 0bf4f38 to fc49350 Compare March 11, 2022 02:15
@wenzeslaus
Copy link
Member Author

New usage: Load JSON

import json

import grass.script as gs

result = json.loads(
    gs.read_command("t.rast.list", method="gran", input="precip_abs0", format="json")
)
names = [item["name"] for item in result["data"]]
dates = [item["start_time"] for item in result["data"]]
print(names)
print(dates)

Classic usage: Custom parsing

import grass.script as gs

rows = gs.read_command("t.rast.list", method="gran", input="precip_abs0").splitlines()

new_rows = [row.split("|") for row in rows]
new_array = [list(row) for row in zip(*new_rows)]

for column in new_array:
    if column[0] == "name":
        names = column[1:]
    if column[0] == "start_time":
        dates = column[1:]
print(names)
print(dates)

Both take approximately same time, approx 3.33s, with variations bigger than the difference.

@wenzeslaus wenzeslaus added enhancement New feature or request temporal Related to temporal data processing Python Related code is in Python labels Mar 15, 2022
@wenzeslaus
Copy link
Member Author

This has tests, it doesn't change default behavior, and doesn't seem to have any performance issues, I plan to merge this soon still to be included to 8.2. Comments welcome.

@veroandreo
Copy link
Contributor

I'd like to test it, but I can during the weekend only. Is that still good timing @wenzeslaus ?

@wenzeslaus
Copy link
Member Author

That would be great. Thank you!

@veroandreo
Copy link
Contributor

I've been testing different combinations of methods, columns and format options. I really like that method refers now to data gathering and format to printing.

I found however, that not all column possibilities can go with all methods, eg., columns min and max are not available for method gran or delta given the type of gathering I assume. Maybe document that (if not yet there) saying something like: columns can be specified for all methods and formats, but note that depending on the method some columns might not be available.

@wenzeslaus
Copy link
Member Author

So, it actually needed fix for the checking, some valid columns were rejected and some other caused a traceback (relevant exception from the underlying function, just not appropriate in the module context). I added the fix, doc, and tests.

@wenzeslaus wenzeslaus merged commit d2317a2 into OSGeo:main Mar 31, 2022
@wenzeslaus wenzeslaus deleted the exchange-formats-for-t_rast_list branch March 31, 2022 19:09
ninsbl pushed a commit to ninsbl/grass that referenced this pull request Oct 26, 2022
* This adds CSV, JSON, and YAML output and generalizes the other outputs.
* Method is now meant only for different data gathering methods, not for the output formatting. method=cols and method=comma are now obsolete.
* All formatting is available with all methods.
* Most code is shared between the two main branches of code (simple list versus delta/gran methods code).
* Columns can be now specified for any option (previously only for cols/list and partially for comma/line output).
* The change is backwards compatible both on module interface level and on Python API level.
* Activate and fix old bash test (-i by t.register is not accepted without start_time).
ninsbl pushed a commit to ninsbl/grass that referenced this pull request Feb 17, 2023
* This adds CSV, JSON, and YAML output and generalizes the other outputs.
* Method is now meant only for different data gathering methods, not for the output formatting. method=cols and method=comma are now obsolete.
* All formatting is available with all methods.
* Most code is shared between the two main branches of code (simple list versus delta/gran methods code).
* Columns can be now specified for any option (previously only for cols/list and partially for comma/line output).
* The change is backwards compatible both on module interface level and on Python API level.
* Activate and fix old bash test (-i by t.register is not accepted without start_time).
neteler pushed a commit to nilason/grass that referenced this pull request Nov 7, 2023
* This adds CSV, JSON, and YAML output and generalizes the other outputs.
* Method is now meant only for different data gathering methods, not for the output formatting. method=cols and method=comma are now obsolete.
* All formatting is available with all methods.
* Most code is shared between the two main branches of code (simple list versus delta/gran methods code).
* Columns can be now specified for any option (previously only for cols/list and partially for comma/line output).
* The change is backwards compatible both on module interface level and on Python API level.
* Activate and fix old bash test (-i by t.register is not accepted without start_time).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Python Related code is in Python temporal Related to temporal data processing
Projects
Development

Successfully merging this pull request may close these issues.

2 participants