Thoughts on PROJJSON for CRS encoding?

[kylebarron]

I'm not an expert by any means on CRS, but I'd be interested in hearing others' thoughts about requiring PROJJSON instead of (or as an optional counterpart to) a WKT2 string.

Benefits:

• According to the PROJJSON spec, "PROJJSON is a JSON encoding of WKT2:2019". So in effect it's the same metadata we already require, but stored as JSON instead of as a WKT string.

• A stated issue with requiring CRS information is that the WKT string is hard to parse. From Should we recommend EPSG:4326 or something else? #52 (comment):

  Keywords are case-insensitive. Space outside of quoted strings is insignificant. Delimiters can be [ or ( and ] or ). Certain keywords (like USAGE and ID) are optional. Anyone reading the spec will likely reach for proj or something else to parse and compare equality. If CRS is mandatory, this essentially means working with geoparquet adds a dependency on proj, even if you just want to know if data is in the recommended CRS (and even proj trips up if the delimiters following the ensemble keyword are parens instead of brackets - proper wkt parsing is hard).

  Using PROJJSON instead of WKT could at least remove the difficulty of parsing the CRS information, though the difficulty of interpreting the parsed information would remain.

• The rest of our metadata is already stored as JSON. So we already expect readers and writers to be able to handle JSON.

• PROJ defines a PROJJSON schema: https://proj.org/schemas/v0.4/projjson.schema.json. We would be able to reference that schema within our own JSON Schema definition, and so be able to validate our own metadata and CRS in one go. (Though I assume it's possible to create a PROJJSON string that validates against the schema but is non-sensical?)

• Might be easier to support than WKT2 for proj4.js: Support for PROJJSON proj4js/proj4js#400 (comment)

Drawbacks:

• Not usable pre PROJ version 6.2.
• Probably others. Please enlighten me 😄

References:

• PROJJSON definition: https://proj.org/specifications/projjson.html

[hobu]

Ok, since I kind of started this discussion. Here's my take. I think a lot of the posturing for or against CRS metadata is due to WKT2 being poorly suited for what geospatial file formats and data transports need from CRS interoperability. I think it was the same thing with GeoJSON, but there was no full-featured WKT2 library at that time, which really tilted us away from imposing its full complexity on all implementors as a requirement.

I think @kylebarron's description of the terrible parts of WKT2 being both parsing and interpretation is accurate, and PROJJSON conveniently addresses the parsing and syntax validation in combination with JSON Schema usage. This means that applications can at least be protected against propagating junk syntax definitions. Kyle's point about being able to write nonsense definitions that still syntax validate is always true, and the data model what WKT or PROJJSON describes is going to complex no matter its container.

Another +1 for PROJJSON is applications can peek into it and consume parts that they do know. It's tough to do that with WKT because of the parsing and validation challenge. The definitions are conveniently available, so answers to questions of the CRS like "Is this a geographic one?" or "What are the X and Y units?" are possible with some JSON walking/scanning. Doing that with WKT means writing some awful regexs, writing a parser, or pulling in a giant dependency like PROJ.

It also would be nice if someone could take PROJJSON through the OGC Community Specification process so it could be referenced. That's a money/time/desire request, but some kind of alternative encoding CRS definition is needed to sidestep WKT's problem of requiring custom parsers, and PROJJSON uptake by other specs likely requires/desires that kind of referencing. @rouault made PROJJSON partly to see if it could work and partly to start a discussion about WKT's failures, but maybe now is the time to get more serious about its uptake.

[rouault]

I'd be obviously supportive of PROJJSON, and concur with @hobu and @kylebarron above remarks.
Parsing WKT is a non trivial exercice, in part because the latest spec also encompasses all the previous ones. So implicitly a WKT CRS reader should be able to understand the 4 different OGC WKT CRS specs. Depending on which spec you refer to, keywords change. e.g that can be GEOGCS, GEOGCRS, GEODCRS for a GeographicCRS. Probably the main issue with WKT is that it tries to be at the same time both human and computer friendly, but fails a bit at both. PROJJSON is clearly more oriented to be computer friendly, at least on the production side as it is quite verbose
PROJJSON (could be named in a more neutral way as CRSJSON if it was to be more widely adopted) can be losslessly converted to WKT2:2019 and vice-versa. My take is that with the PROJJSON JSON schema (for the general structure) and the CRS WKT spec (for domain specific rules that can't be encoded as JSON schema constraints), you've implicitly the PROJJSON spec. And probably add to that the EPSG dataset which brings the real interoperability with respect to datum names, projection method and parameters, etc etc and codes for all those objects (but that obviously holds true for CRS WKT, for the reason mentioned before that PROJJSON and WKT encode the same information)
Re "Not usable pre PROJ version 6.2.", the soon to be old-stable Ubuntu 20.04 release ships with PROJ 6.3.1, so in the PROJ ecosystem that isn't much an issue. Adoption in other software stacks would probably be more problematic.

[jorisvandenbossche]

I am generally positive on the idea to move to PROJJSON. I think it would be a nice improvement in usability (easier to check eg the id of a "crs", without the need to have a full WKT parser).

From GeoPandas' point of view, this is also fine because we rely on PROJ (through pyproj) anyway for handling CRS related functionality, and so we have the functionalities to work with either WKT or PROJJSON available.
For GDAL, it is generally the same (PROJ is typically available).

It would be good to hear what the impact of moving to PROJJSON would be for some other implemenations?
cc @ashar236 @Sarwat (Sedona), @pomadchin, @alasarr (CARTO Analytics toolbox?)

[kylebarron]

Does R have support out of the box? I assume it also links to PROJ? cc @paleolimbot

[hobu]

Does R have support out of the box?

Quite likely due to R geospatial stuff being based on PROJ. @edzer may also have thoughts and experience to relay in regards to CRS definitions and their interactions with users.

[rouault]

I'm also curious about projjson support in the overall Java ecosystem, like GeoTools/GeoServer.

https://github.com/geotools/geotools/search?q=projjson ==> 0 match
But GeoTools doesn't support WKT2 as well as far as I can see looking for WKT2 specific keywords in the code base, and at least was confirmed by Andrea in 2019: https://sourceforge.net/p/geotools/mailman/message/36704702/ )

[edzer]

Does R have support out of the box?

Only R packages linking to PROJ or GDAL can do something useful with WKT2. Such packages are available but are considered a "heavy" dependency by many developers. Having a JSON representation of CRS will make it much easier to figure out simple things such as whether coordinates are degrees or meters, lifting the need for the heavy deps.

[cholmes]

Cool. I'm definitely up to run the 'experiment' of making projjson a requirement for geoparquet and seeing how much if any pushback we get. Like if there's some major packages where it'd just be too much work to support it then we could consider other options. Or better yet just fund them to actually support projjson and accelerate its adoption.

[brendan-ward]

I'm 👍 on trying PROJJSON as a replacement for WKT2. This seems like it addresses the shortcomings of making WKT required for common use cases around geographic coordinates but still provides the ability to clearly document the CRS, and since folks will have to parse JSON anyway, this is probably even better.

[paleolimbot]

I like the idea of PROJJSON in GeoParquet metadata...while somebody doing coordinate transforms will almost certainly be using PROJ in some way (thus WKT is fine), there are some questions like "are the coordinates longitudes and latitudes?" and "can we consider this CRS equal to this other CRS?" that PROJJSON facilitates in a way that WKT2 doesn't. Probably the most useful thing that can be done with PROJ JSON that is difficult to do with WKT is to print out a compact definition of the CRS for a user to read.

From a 'geoarrow' encoding perspective, we're (currently, depending on geoarrow/geoarrow#17) stuck with a single level of key/value metadata where the value has to be a string. One thread has probed at the idea of allowing multiple CRS encodings (geoarrow/geoarrow#8), so there's no reason a PROJJSON string couldn't be one of the types. The set of C++ headers that power the 'geoarrow' R package could probably include enough parsing capability to at least extract WKT and/or JSON into C++ structures that can be inspected (if not interpreted to the extent that the PROJ WKT parser is able to do).

[alasarr]

I'm 👍 too with the idea of PROJJSON

[cayetanobv]

+1 with PROJJSON. I've already shared my thoughts about this issue in #52 (comment)

[jorisvandenbossche]

@rouault to what extent would a json object like {'type': 'GeographicCRS', 'id': {'authority': 'EPSG', 'code': 4326}} be valid / make sense?

It's not valid, because it is missing a bunch of required fields (following the JSON schema). And I suppose it also doesn't make much sense because PROJJSON (or WKT) is meant to describe the actual information you need to know instead of having to rely on some (database) source to look up that information for an authority/code?

The reason that I was wondering the above is to understand to what extent using PROJJSON could also help the desire of some people to only use an authority:code combo.

[rouault]

And I suppose it also doesn't make much sense because PROJJSON (or WKT) is meant to describe the actual information you need to know instead of having to rely on some (database) source to look up that information for an authority/code?

yes, exactly. That would be a different concept of what PROJJSON/WKT CRS intends to be. That discussion reminds me a lot GeoJSON 2008 ( https://geojson.org/geojson-spec#coordinate-reference-system-objects ) where there were different proposed ways of setting the CRS (none of them had provision for online definition like PROJJSON though)

[hobu]

Additionally, online definitions of CRSs wasn't quite a thing in 2008 when we proposed that. We weren't quite sure what would work (as evidenced by the ambiguity), but we knew that requiring users to plop in WKT1 was not the solution. In the end we backed away when we did the IETF standardization because years of use in the wild had shown GeoJSON was a wire/transport format, and most of that transport was in EPSG:4326.

Is GeoParquet for archive, transport, or exploitation? If it is all three, what's the rank ordering?

[jorisvandenbossche]

With "exploitation", you mean a data source as a direct starting point for analysis?

I think the challenge is that it is indeed all three, and that different people have different rankings / goals with geoparquet.

When we originally started using Parquet in GeoPandas, I think it was clearly in the "exploitation" bucket. To have a "high-performance, low transformation internal use" format (to quote @brendan-ward from #89), that can be used as a starting point for analysis workflows (or as intermediate storage between analysis steps). For that case, we want to be able to have the data as close as possible in the format / projection we need for analysis (and so for this reason we argued before that a required winding order, or a hypothetical requirement on only WGS84 data, .. gives unnecessary overhead for this use case)
I think that's also how Parquet is often used in the general ecosystem (eg there are nowadays a lot of data lakes that can use Parquet as the actual storage format on which queries are run)

[cayetanobv]

Transport is an obvious and important role in his motivation. But we need to take into account (as @jorisvandenbossche said) that Parquet is used intensively for archive/exploitation purposes. Aside from tools and platforms in the more classical Hadoop ecosystem, newer big data platforms like DeltaLake from DataBricks are using Parquet under the hood to build the database (with a powerful query engine).

[rouault]

I've just submitted enhancements to the PROJJSON specification in OSGeo/PROJ#3190 . Feedback welcome. It is hard to find the right balance between being exhaustive (which require a ~ 100 page document like the WKT CRS spec) and not saying enough. I've tried to concentrate on the gaps / particularities of the JSON encoding compared to the WKT one