Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

r.univar: parallelization for extended statistics #2683

Merged
merged 14 commits into from
Nov 17, 2023

Conversation

aaronsms
Copy link
Contributor

@aaronsms aaronsms commented Dec 5, 2022

This PR addressed #2564, completing parallelization for extended statistics.

The key idea is to maintain "local buckets" for each thread to sort their cells according to the different zones before transfering to the global buffer for computing the order statistics.

In the case of nprocs=1, care is taken to ensure no major overhead, as the first transfer would just be a redirection to the local bucket.

Performance benchmark still needs to be done.

@aaronsms aaronsms marked this pull request as draft December 5, 2022 10:53
@aaronsms aaronsms changed the title [Draft] r.univar parallelization for extended statistics r.univar parallelization for extended statistics Dec 5, 2022
@nilason nilason added enhancement New feature or request C Related code is in C labels Feb 15, 2023
@nilason nilason added this to the 8.4.0 milestone Feb 15, 2023
@nilason nilason added the raster Related to raster data processing label Feb 15, 2023
@aaronsms aaronsms marked this pull request as ready for review March 4, 2023 08:38
@petrasovaa petrasovaa self-requested a review March 6, 2023 15:43
@petrasovaa
Copy link
Contributor

I didn't test yet, but you need to remove the restriction that prevented r.univar -e to run in parallel:
https://github.com/OSGeo/grass/blob/main/raster/r.univar/r.univar_main.c#L155

@petrasovaa
Copy link
Contributor

petrasovaa commented Mar 16, 2023

Weird, the tests in CI work, I am getting locally a lot of errors, e.g.:

GRASS nc_spm_08_grass7/user1:testsuite > r.univar elevation zones=basin_50K -e nprocs=3
WARNING: ZLIB decompression error -3: data error
WARNING: ZLIB decompression error -3: data error
ERROR: Error uncompressing fp raster data for row 0 of <elevation>: error
       code -1
Segmentation fault (core dumped)


@aaronsms aaronsms marked this pull request as draft May 10, 2023 16:24
@aaronsms
Copy link
Contributor Author

aaronsms commented Jun 2, 2023

It did not

Weird, the tests in CI work, I am getting locally a lot of errors, e.g.:

GRASS nc_spm_08_grass7/user1:testsuite > r.univar elevation zones=basin_50K -e nprocs=3
WARNING: ZLIB decompression error -3: data error
WARNING: ZLIB decompression error -3: data error
ERROR: Error uncompressing fp raster data for row 0 of <elevation>: error
       code -1
Segmentation fault (core dumped)

The last commit should fix this. We should test it more.

@aaronsms aaronsms marked this pull request as ready for review June 2, 2023 16:36
@petrasovaa petrasovaa merged commit c307930 into OSGeo:main Nov 17, 2023
HuidaeCho pushed a commit to HuidaeCho/grass that referenced this pull request Jan 9, 2024
@neteler neteler changed the title r.univar parallelization for extended statistics r.univar: parallelization for extended statistics Jun 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C Related code is in C enhancement New feature or request raster Related to raster data processing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants