DGGRID Roadmap

[sahrk]

What do you want to see for future DGGRID features?

[danlooo]

Add function to transform points using numbers instead of strings

Thank you very much for developing such a low distorted DGGS! We're working on DGGS native data cubes that utilize the Q2DI index to store high resolution satellite imagery datasets at global scale. This is done in various programming languages, emphasising the need to have efficient bindings of TRANSFORM_POINTS and others.

We'd like to have a function in the library that directly operates on numbers instead of strings containing the coordinates, avoiding the creation, loading, and parsing of text files for coordinates in packages like dggrid4py and DGGS.jl. This would also enable the use of DGGRID in xarray-contrib/xdggs.

We discussed the issue in danlooo/DGGS.jl#121. @d-consoli started to implement this at d-consoli@20fecf3.

[allixender]

Hi Daniel,

We'd like to have a function in the library that directly operates on numbers instead of strings containing the coordinates, avoiding the creation, loading, and parsing of text files for coordinates in packages like dggrid4py and DGGS.jl. This would also enable the use of DGGRID in xarray-contrib/xdggs.

We already building an Xarray/XDDGS prototype for use with DGGRID through dggrid4py: https://github.com/landscapeGeoinformatics/xdggs

We will have to refactor though to not make it a dependency but rather a plugin (due to heavy dependency and license)

[sahrk]

@danlooo ,

Add function to transform points using numbers instead of strings

The string addresses are just for input/output, internally everything is done with numeric addresses for almost all address forms. Early versions of DGGRID only supported the text file format ARCINFO GENERATE, so text was assumed for all input/output and it permeated the operations code. In Version 8 we internally decoupled (mostly) the operations from the input/output systems. The next step is to create an API that exposes the operations directly. Many comments below seem to be in line with that approach.

While the TRANSFORM operation is still buried in DGGRID, I would point out that the functionality to perform address conversions is in dglib, and thus technically exposed , though it is not well documented. The example program appex.c (directory src/apps/appex) demonstrates a little bit of this, and I could add more conversion examples. The program does transform a location from lat/lon to Q2DI and back to lat/lon using dglib and c++, so might be helpful if that is your use case.

[MichaelJendryke]

The example program appex.c (directory src/apps/appex) demonstrates a little bit of this, and I could add more conversion examples.

This is great! This is the type of abstraction/API I was thinking about. It would be great if we can expand that and create a dglib API that expose a few frequently used functions. I think that generating cells globally, generating cells for a given parent cell, and point to cell conversion could be a start. I am happy to support this and will look into the current appex.cpp. I see a lot of potential for this and it would make a lot of ISEA-based grids available very quickly.

[sahrk]

@MichaelJendryke

This is great! This is the type of abstraction/API I was thinking about. It would be great if we can expand that and create a dglib API that expose a few frequently used functions. I think that generating cells globally, generating cells for a given parent cell, and point to cell conversion could be a start. I am happy to support this and will look into the current appex.cpp. I see a lot of potential for this and it would make a lot of ISEA-based grids available very quickly.

All of this is in the immediate roadmap. Basically we want to make all functionality accessible in dglib (or maybe a slightly higher-level glue library), without the need for DGGRID and its dependency on file input/output. Support is always welcome, thanks :).

[sahrk]

@danlooo I just took a quick look at the discussion in your DGGS.jl issue. I definitely think appex.cpp will be helpful.

btw my long-time colleague and mentor Ross Kiester keeps raving about Julia to me, so I am going to have to spend some time trying it out. Ross was a major force in the design of DGGRID.

[allixender]

Hi, from my interests I'd like to propose a couple of directions which I might have mentioned earlier:

• Arrow/GeoArrow support as a main data sharing/access mechanism to read/write and pass tabular data with and without geometries around
• and I feel (Geo)Arrow should be usable independent of GDAL, building DGGRID with GeoArrow but without GDAL should be possible

This could also be an agreeable entry point for potential external programming language bindings, as Arrow is robustly supported across C/C++, Python, R, Rust, etc. and becoming a preferred efficient tabular in-memory format (plus Parquet/Feather as file format)

I would also good to highlight which primary DGGRID functions we use in order to propose test cases for the dglib interoperability.

• OUTPUT_STATS (good lookup to have)
• GENERATE_GRID (prime use case, often with clip_subsets)
• TRANSFORM_POINTS (not intensively yet)

Please everyone commenting here consider proposing funding options if you use DGGRID in projects funded by like European Commission, ESA, NASA, NSF, or national research funding 💰

We have supported DGGRID through EU funding (ERC, WaterSmartLand, 101125476, Interreg-BSR, HyTruck, #C031) and will likely provide another round next year.

[changliao1025]

We have used DGGRID in the US DOE Biological and Environmental Research program ICoM and InteRFACE projects. A manuscript is currently under review at: https://essd.copernicus.org/preprints/essd-2023-398/

[changliao1025]

"and I feel (Geo)Arrow should be usable independent of GDAL, building DGGRID with GeoArrow but without GDAL should be possible"
I think for the core capability of DGGRID, it should be a minimal dependency. But for most wrapper features, including the format support, dependency can be optional such as GDAL and GeoArrow, which allows for maximal flexibility. For example, if a user is only interested in the Shapefile or Geojson, then only corresponding dependency packages are required but the core dependency (C++) is always required.

[danlooo]

Apache Arrow / GeoArrow would also solve our issues. We just need an in-memory data structure that doesn't require additional serialization or writing results to disk. It makes writing bindings to other languages much easier.

[sahrk]

I think we're all in agreement, Arrow/GeoArrow should be supported, and it should be an optional dependancy independent of GDAL.

[MichaelJendryke]

Hello,

I have a few ideas for DGGRID (and dglib). Most of these ideas are focused on increasing adoption and expanding the community around these tools.

Tooling
DGGRID would benefit from easier options for building tooling around it. Currently, I have developed a few wrappers in different languages that generate a metafile and call DGGRID in a subprocess. While this approach works, I believe leveraging dglib would be a more efficient option. Given that dglib serves as the engine, having bindings available in different languages would be valuable. These bindings don’t necessarily have to come from the DGGRID project itself; they could also be generated externally. For instance, the bindgen tool in Rust could be used to create bindings, as long as dglib's header files are compatible. This approach would allow for a direct class binding.

An even better option would be to develop a set of abstractions or an API for dglib that would support common tasks such as:

• Generating global ISEA grids
• Creating grids for a given polygon
• Generating grids by intersection with points
• Creating grids for a parent cell

This API could be integrated into dglib to expose core functionalities, similar to what H3 offers. The goal would be to establish dglib as the core of a broader ecosystem and potentially turn it into a reference system for DGGS, much like PROJ.

Documentation
Although I am now familiar with DGGRID, new users often find the number of options overwhelming. Creating searchable online documentation would greatly improve the user experience. I could help by converting the existing LaTeX (?) files using pandoc. I recommend using Asciidoc and Antora for static site generation and GitHub Pages for hosting.

Precompiled Executables
I’ve noticed that some potential users are hesitant to approach DGGRID because they need to compile it themselves, which is unfortunate. Implementing GitHub's CI/CD pipeline to compile and publish precompiled executables as releases would be a great solution.

Indexing
This point is more of a question: Would it be useful to modify the Z3 and Z7 indices to include refinement level information in the cell ID? Currently, the resolution is not encoded in the cell ID (at least for the non-string version), requiring it to be stored in a separate column. One idea would be to merge the ID and resolution as a000000000000000rr, where rr represents the resolution. However, parsing such strings may not be very efficient.

Lastly, we funded DGGRID through a national research transfer grant (Ruhr-University Bochum/GeoInsight Project) and plan to provide additional funding through GeoInsight in 2025, if possible.

[allixender]

Hi Michael,

regarding documentation:

Documentation
Although I am now familiar with DGGRID, new users often find the number of options overwhelming. Creating searchable online documentation would greatly improve the user experience. I could help by converting the existing LaTeX (?) files using pandoc. I recommend using Asciidoc and Antora for static site generation and GitHub Pages for hosting.

I have already a prototype conversion of the original user manual of Kevin into a markdown-based static web doc system (Sphinx/ReadTheDocs format). We had agreed now to postpone that and wait for the latest update. But maybe we should just push forward to show what's possible and to improve upon.

[MichaelJendryke]

I have already a prototype conversion of the original user manual of Kevin into a markdown-based static web doc system (Sphinx/ReadTheDocs format).

Will the markdown files live in this repo here? I am open to preparing a showcase if that is desired. I am more in favour of asciidoc.

[sahrk]

@MichaelJendryke

All of your requests closely track the discussions that @allixender and I have been having. As briefly discussed in a comment above, we’ve been steadily working towards making all functionality available independent of the DGGRID interface. I’m hoping we can complete that transition in the first quarter of 2025.

Indexing This point is more of a question: Would it be useful to modify the Z3 and Z7 indices to include refinement level information in the cell ID? Currently, the resolution is not encoded in the cell ID (at least for the non-string version), requiring it to be stored in a separate column. One idea would be to merge the ID and resolution as a000000000000000rr, where rr represents the resolution. However, parsing such strings may not be very efficient.

We’ve been discussing this as well, apropos of some of the OGC API work. I see two options for incorporating the resolution in an index:

1. the resolution is recoverable from existing Z7 indexes because digits beyond the encoded resolution are set to the Z7 invalid digit 7. Z7 uses 3 bits/digit, with valid digits being 0-6, leaving 7 to use for an invalid digit. So one option would be to supply a function that returns the resolution given a Z7 index. This would have to search the digits for the first 7, so would be relatively expensive to execute.

Z3 uses 2 bits/digit, with valid digits being 0-2, so we could use digit 3 for the invalid resolutions. Unfortunately when implementing Z3 I chose to use 0 for the invalid resolutions, but I think you’re the only one using Z3 so far so we could change that, and also use a function approach.

However this approach will not work for aperture 4 indexes like ZORDER, which uses 2 bits/pixel with all possible digits 0-3 being valid digits. ZORDER uses 0 for invalid resolutions (which is why I chose 0 for Z3). So if we went with this approach we could only recover resolutions from Z3 and Z7 indexes, not ZORDER.

2. We could store the resolution in the index, as you suggest. I would probably put it at the front of the index before the quadrilateral number. This would be consistent with the structure of H3 indexes, and also allow for the future creation of high resolution multi-integer indexes. We would still need to provide a function to parse out the resolution.

Option 2 would reduce the number of index bits available for the actual index. So it would trim 2 or 3 resolutions off of the maximum resolution representable in a 64 bit integer (I haven’t thought through the math yet). But the function to determine the resolution would be trivial compared to Option 1 function that would need to parse the digits to see when they become invalid. And as mentioned above only Option 2 will work for aperture 4 indexes.

If anyone has any preference or input on the options let us know. I guess my current preference is to use Option 2, because it will work for any potential integer index. We could even provide a SEQNUM address with encoded resolution, though that may be a bridge too far.

[sahrk]

I have already a prototype conversion of the original user manual of Kevin into a markdown-based static web doc system (Sphinx/ReadTheDocs format).

Will the markdown files live in this repo here? I am open to preparing a showcase if that is desired. I am more in favour of asciidoc.

I'm leaving the decisions on the new documentation to @allixender, since he's done all the work.

@allixender I guess it's silly at this point not to put up the new documentation, especially if it's on a different site than the main GitHub repo. I was just trying to keep everything in sync but I've become an impediment to advancement....

[MichaelJendryke]

2. and also allow for the future creation of high resolution multi-integer indexes.

+1 for high resolution indices. would it be possible to go for 128 bit? Adding the resolution in the front makes sense.

[changliao1025]

I want to point out two visualization efforts that might be useful for the dggrid output visualization:
Uxarray: https://github.com/UXARRAY/uxarray, based on matplotlib, cartopy.
And GeoVista: https://github.com/bjlittle/geovista, based largely on VTK.
If we can also provide some features that export output compatible with one of the visualization tools, it would be easier for users who do not use QGIS or need customized visualizations.

[changliao1025]

Other than the existing GIS-focused grid formats, there are other options often used in the Earth system model community, including the MPAS mesh specs (https://mpas-dev.github.io/files/documents/MPAS-MeshSpec.pdf)
and UGrid standard (https://github.com/ugrid-conventions/ugrid-conventions).
NetCDF is widely used in ESMs, so a conversion to a grid standard may be important.

[bjlittle]

@changliao1025 et al, do you guys have some example data that you want to visualize? GeoVista can render rectilinear, curvilinear and unstructured meshes. Happy to help 👍

[sahrk]

@changliao1025 @bjlittle I understand that these visualization platforms (including QGIS) can render unstructured meshes. The issue is that DGGS are not unstructured! They have a high degree of structure, they just aren't planar square or triangle meshes. What we really need is a visualization platform that takes advantage of the DGGS structure to do efficient rendering.

In the meantime I wonder if GDAL can produce output formats that these tools can visualize? I just checked and NetCDF is supported, but I haven't looked further (I don't know offhand what formats to search for).

[allixender]

@changliao1025 @bjlittle
In the meantime I wonder if GDAL can produce output formats that these tools can visualize? I just checked and NetCDF is supported, but I haven't looked further (I don't know offhand what formats to search for).

The only issue for the global visualisation with QGIS or Cartopy is that we regularly run into is the dateline wrap. GDAL/OGR can approach that a little bit with something like:

ogr2ogr -wrapdateline ...

But efficient web visualisation or at least within jupyter notebooks is still a bit challenging when working with many millions of cells and treating them as vector geometries. lonboard is quite promising, or another approach would be via datashader like mentioned above, or just using the centroid points when the number of cells approaches the number of pixel you can physically show on a display/plot.

[sahrk]

@allixender

The only issue for the global visualisation with QGIS or Cartopy is that we regularly run into is the dateline wrap.

Since version 7.4 DGGRID can fix the dateline wrap as well; see parameters longitude_wrap_mode and unwrap_points, and the example planetRiskGridNoWrap. Of course that won't fix issues with cells that contain a pole.

[bjlittle]

@changliao1025 @bjlittle
In the meantime I wonder if GDAL can produce output formats that these tools can visualize? I just checked and NetCDF is supported, but I haven't looked further (I don't know offhand what formats to search for).

The only issue for the global visualisation with QGIS or Cartopy is that we regularly run into is the dateline wrap. GDAL/OGR can approach that a little bit with something like:

ogr2ogr -wrapdateline ...

But efficient web visualisation or at least within jupyter notebooks is still a bit challenging when working with many millions of cells and treating them as vector geometries. lonboard is quite promising, or another approach would be via datashader like mentioned above, or just using the centroid points when the number of cells approaches the number of pixel you can physically show on a display/plot.

Do you have any example datasets that you could point me to? Are they in NetCDF format or a bespoke serialisation tied to your package?