Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PostGIS Raster integration #112

Merged
merged 22 commits into from
Jan 19, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
fa6a6c4
PostGIS Raster integration
Dec 27, 2022
48a6530
PostGIS Raster integration: creating postgis_raster extension in tests
Dec 28, 2022
a869134
PostGIS Raster integration: explicit conversion of records to return …
Dec 28, 2022
b9acb57
PostGIS Raster integration: "CREATE OR REPLACE AGGREGATE" removed for…
Dec 28, 2022
6e0f7dc
PostGIS Raster integration: documentation generation: "RETURNS TABLE"…
Dec 29, 2022
ee3373c
PostGIS Raster integration: type creation fix
Dec 29, 2022
3d00489
PostGIS Raster integration: raster geometry transform fix
Dec 30, 2022
423470f
PostGIS Raster integration: rewritten to re-use raster polygon
Jan 2, 2023
cc81a6a
PostGIS Raster integration: minor documentation update
Jan 2, 2023
a95b8ce
PostGIS Raster integration: redundant subqueries removed
Jan 4, 2023
b7e2fb4
PostGIS Raster integration: documentation update
Jan 4, 2023
ff8bd85
PostGIS Raster integration: method selection based on pixels per cell…
Jan 9, 2023
de88f47
PostGIS Raster integration: STRICT removed for helper functions that …
Jan 12, 2023
51142e7
PostGIS Raster integration: ST_Buffer "join=mitre" parameter added fo…
Jan 12, 2023
7c753f7
PostGIS Raster integration: cell geometry filtering fix
Jan 13, 2023
4f4bfb3
PostGIS Raster integration: method selection condition adjusted
Jan 13, 2023
42625ac
PostGIS Raster integration: always using hexagons for cell area estim…
Jan 13, 2023
18ea23b
PostGIS Raster integration: documentation update
Jan 17, 2023
711d80c
PostGIS Raster integration: pixel area estimation fix
Jan 17, 2023
1c29e8b
PostGIS Raster integration: documentation update
Jan 18, 2023
9a08ab7
PostGIS Raster integration: documentation update
Jan 18, 2023
94c4dfc
PostGIS Raster integration: geometry to geography conversion fix
Jan 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/pgxn.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
sudo systemctl start postgresql.service
sudo -u postgres -i createuser --superuser runner
sudo -u postgres -i createdb runner
psql -c "CREATE EXTENSION postgis;"
psql -c "CREATE EXTENSION postgis; CREATE EXTENSION postgis_raster;"

- name: Bundle
run: scripts/bundle
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-linux.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
sudo apt-get -y install postgresql-${{ matrix.pg }}-postgis-3
sudo -u postgres -i createuser --superuser runner
sudo -u postgres -i createdb runner
psql -c "CREATE EXTENSION postgis;"
psql -c "CREATE EXTENSION postgis; CREATE EXTENSION postgis_raster;"

- name: Generate
run: cmake -B build -DCMAKE_BUILD_TYPE=${{ matrix.config }}
Expand Down
163 changes: 163 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -618,3 +618,166 @@ Create a LinkedGeoPolygon describing the outline(s) of a set of hexagons, conver
Splits polygons when crossing 180th meridian.


# Raster processing functions

## Continuous raster data
For rasters with pixel values representing continuous data (temperature, humidity,
elevation), the data inside H3 cells can be summarized by calculating number of
pixels, sum, mean, standard deviation, min and max for each cell inside a raster
and grouping these stats across multiple rasters by H3 index.
```
SELECT
(summary).h3 AS h3,
(h3_raster_summary_stats_agg((summary).stats)).*
FROM (
SELECT h3_raster_summary(rast, 8) AS summary
FROM rasters
) t
GROUP BY 1;
h3 | count | sum | mean | stddev | min | max
-----------------+-------+--------------------+---------------------+--------------------+-------+------------------
882d638189fffff | 10 | 4.607657432556152 | 0.46076574325561526 | 1.3822972297668457 | 0 | 4.607657432556152
882d64c4d1fffff | 10 | 3.6940908953547478 | 0.3694090895354748 | 1.099336879464068 | 0 | 3.667332887649536
882d607431fffff | 11 | 6.219290263950825 | 0.5653900239955295 | 1.7624673707119065 | 0 | 6.13831996917724
<...>
```


*Since vunreleased*


### h3_raster_summary_stats_agg(setof `h3_raster_summary_stats`)
*Since vunreleased*


### h3_raster_summary_clip(rast `raster`, resolution `integer`, [nband `integer` = 1]) ⇒ TABLE (h3 `h3index`, stats `h3_raster_summary_stats`)
*Since vunreleased*


Returns `h3_raster_summary_stats` for each H3 cell in raster for a given band. Clips the raster by H3 cell geometries and processes each part separately.


### h3_raster_summary_centroids(rast `raster`, resolution `integer`, [nband `integer` = 1]) ⇒ TABLE (h3 `h3index`, stats `h3_raster_summary_stats`)
*Since vunreleased*


Returns `h3_raster_summary_stats` for each H3 cell in raster for a given band. Finds corresponding H3 cell for each pixel, then groups values by H3 index.


### h3_raster_summary_subpixel(rast `raster`, resolution `integer`, [nband `integer` = 1]) ⇒ TABLE (h3 `h3index`, stats `h3_raster_summary_stats`)
*Since vunreleased*


Returns `h3_raster_summary_stats` for each H3 cell in raster for a given band. Assumes H3 cell is smaller than a pixel. Finds corresponding pixel for each H3 cell in raster.


### h3_raster_summary(rast `raster`, resolution `integer`, [nband `integer` = 1]) ⇒ TABLE (h3 `h3index`, stats `h3_raster_summary_stats`)
*Since vunreleased*


Returns `h3_raster_summary_stats` for each H3 cell in raster for a given band. Attempts to select an appropriate method based on number of pixels per H3 cell.


## Discrete raster data
For rasters where pixels have discrete values corresponding to different classes
of land cover or land use, H3 cell data summary can be represented by a JSON object
with separate fields for each class. First, value, number of pixels and approximate
area are calculated for each H3 cell and value in a raster, then the stats are
grouped across multiple rasters by H3 index and value, and after that stats for
different values in a cell are combined into a single JSON object.
The following example query additionally calculates a fraction of H3 cell pixels
for each value, using a window function to get a total number of pixels:
```
WITH
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This query frightens me as documentation, likely because it lacks sample output to grasp what it does.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added sample output and a description.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interested and innocent bystander: It would be good to have an example of this but where the return value is simpler, e.g. just the value at the H3 cell centre, or the return value is the mode value, rather than these complex objects.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alpha-beta-soup
Thank you, that's a good idea. I've added an example of getting the most frequent value in each cell.
The other example shows an intended use case, and I also tried to show how to calculate a total # of pixels in each cell, since it's not immediately obvious.

summary AS (
-- get aggregated summary for each H3 index/value pair
SELECT h3, val, h3_raster_class_summary_item_agg(summary) AS item
FROM
rasters,
h3_raster_class_summary(rast, 8)
GROUP BY 1, 2),
summary_total AS (
-- add total number of pixels per H3 cell
SELECT h3, val, item, sum((item).count) OVER (PARTITION BY h3) AS total
FROM summary)
SELECT
h3,
jsonb_object_agg(
concat('class_', val::text),
h3_raster_class_summary_item_to_jsonb(item) -- val, count, area
|| jsonb_build_object('fraction', (item).count / total) -- add fraction value
ORDER BY val
) AS summary
FROM summary_total
GROUP BY 1;
h3 | summary
----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------
88194e6f3bfffff | {"class_1": {"area": 75855.5748, "count": 46, "value": 1, "fraction": 0.4509}, "class_2": {"area": 92345.9171, "count": 56, "value": 2, "fraction": 0.5490}}
88194e6f37fffff | {"class_1": {"area": 255600.3064, "count": 155, "value": 1, "fraction": 0.5}, "class_2": {"area": 255600.3064, "count": 155, "value": 2, "fraction": 0.5}}
88194e6f33fffff | {"class_1": {"area": 336402.9840, "count": 204, "value": 1, "fraction": 0.5125}, "class_2": {"area": 319912.6416, "count": 194, "value": 2, "fraction": 0.4874}}
<...>
```
Area covered by pixels with the most frequent value in each cell:
```
SELECT DISTINCT ON (h3)
h3, val, (item).area
FROM (
SELECT
h3, val, h3_raster_class_summary_item_agg(summary) AS item
FROM
rasters,
h3_raster_class_summary(rast, 8)
GROUP BY 1, 2
) t
ORDER BY h3, (item).count DESC;
h3 | val | area
-----------------+-----+--------------------
88194e6f3bfffff | 5 | 23238.699360251427
88194e6f37fffff | 9 | 60863.26022922993
88194e6f33fffff | 8 | 76355.72646939754
<...>
```


*Since vunreleased*


### h3_raster_class_summary_item_to_jsonb(item `h3_raster_class_summary_item`) ⇒ `jsonb`
*Since vunreleased*


Convert raster summary to JSONB, example: `{"count": 10, "value": 2, "area": 16490.3423}`


### h3_raster_class_summary_item_agg(setof `h3_raster_class_summary_item`)
*Since vunreleased*


### h3_raster_class_summary_clip(rast `raster`, resolution `integer`, [nband `integer` = 1]) ⇒ TABLE (h3 `h3index`, val `integer`, summary `h3_raster_class_summary_item`)
*Since vunreleased*


Returns `h3_raster_class_summary_item` for each H3 cell and value for a given band. Clips the raster by H3 cell geometries and processes each part separately.


### h3_raster_class_summary_centroids(rast `raster`, resolution `integer`, [nband `integer` = 1]) ⇒ TABLE (h3 `h3index`, val `integer`, summary `h3_raster_class_summary_item`)
*Since vunreleased*


Returns `h3_raster_class_summary_item` for each H3 cell and value for a given band. Finds corresponding H3 cell for each pixel, then groups by H3 and value.


### h3_raster_class_summary_subpixel(rast `raster`, resolution `integer`, [nband `integer` = 1]) ⇒ TABLE (h3 `h3index`, val `integer`, summary `h3_raster_class_summary_item`)
*Since vunreleased*


Returns `h3_raster_class_summary_item` for each H3 cell and value for a given band. Assumes H3 cell is smaller than a pixel. Finds corresponding pixel for each H3 cell in raster.


### h3_raster_class_summary(rast `raster`, resolution `integer`, [nband `integer` = 1]) ⇒ TABLE (h3 `h3index`, val `integer`, summary `h3_raster_class_summary_item`)
*Since vunreleased*


Returns `h3_raster_class_summary_item` for each H3 cell and value for a given band. Attempts to select an appropriate method based on number of pixels per H3 cell.


2 changes: 2 additions & 0 deletions h3_postgis/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ PostgreSQL_add_extension(postgresql_h3_postgis
REQUIRES
h3
postgis
postgis_raster
SOURCES
src/init.c
src/wkb_bbox3.c
Expand All @@ -23,6 +24,7 @@ PostgreSQL_add_extension(postgresql_h3_postgis
sql/install/05-regions.sql
sql/install/20-casts.sql
sql/install/30-wkb.sql
sql/install/40-rasters.sql
sql/install/99-deprecated.sql
UPDATES
sql/updates/h3_postgis--4.0.0.sql
Expand Down
Loading