Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(datatypes): proper handling of srid in geospatial datatypes #9519

Merged
merged 20 commits into from
Jul 5, 2024

Conversation

ncclementi
Copy link
Contributor

This is still broken, some how the srid value is not being passed properly to _from_sqlglot_GEOMETRY, I think it's because I need to change something in from_ibis too, but not sure how to handle passing the param.

I'm constructing well, the ibis_type in the test, I see

pp ibis_type
GeoSpatial(geotype='geometry', srid=1234, nullable=True)

But when this is passed to SqlglotType.from_ibis(ibis_type) I'm missing somthing that I don't quite pin point yet.

Any comments are welcome

@cpcloud
Copy link
Member

cpcloud commented Jul 4, 2024

This isn't hitting the codepath because the round trip assertion first produces a SQLGlot type, but without the SRID, which is incorrect, and then converts that back to an Ibis type that doesn't have the SRID (because it wasn't produced).

@cpcloud
Copy link
Member

cpcloud commented Jul 4, 2024

You have to also adjust the _from_ibis_GeoSpatial method to include the SRID in the SQLGlot output.

@ncclementi
Copy link
Contributor Author

I added the srid in the _from_ibis_GeoSpatial and that seems to be generating the right thing, but for the roundtrip, I'm not sure how to modify the to_ibis pass the srid properly to the method. As is, with the test example I'm working, I get

-> restored_dtype = SqlglotType.to_ibis(sqlglot_result)
(Pdb) pp sqlglot_result
DataType(
  this=Type.GEOMETRY,
  expressions=[
    DataTypeParam(
      this=Literal(this=1234, is_string=False))])
(Pdb) pp SqlglotType.to_ibis(sqlglot_result)
*** KeyError: '1234'

Clearly I'm missing something here, but not sure what the syntax is to be ale to pass the srid,

 if method := getattr(cls, f"_from_sqlglot_{typecode.name}", None):
            dtype = method(*typ.expressions)

@cpcloud
Copy link
Member

cpcloud commented Jul 4, 2024

Pushed up a few changes to get this working for both geometry and geography, was a bit gnarlier than I thought. The issue was that all the Point/MultiPoint etc types need to be handled separately so that they can be passed as a DataTypeParam argument when coming from Ibis, and then also that they are transformed to the right bits when going to Ibis.

ibis/tests/strategies.py Outdated Show resolved Hide resolved
return st.builds(
dt.GeoSpatial,
geotype=st.just("geometry"),
nullable=nullable,
srid=st.just(srid),
srid=st.one_of(st.just(None), st.just(4321)),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this to build the two separate strategies? If so, pretty neat : )

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep!

@cpcloud
Copy link
Member

cpcloud commented Jul 4, 2024

Looks like I broke a few things. Have an idea how to fix it, but checking out for the day, happy 4th!

@cpcloud cpcloud added this to the 9.2 milestone Jul 5, 2024
@cpcloud cpcloud added geospatial Geospatial related functionality datatypes Issues relating to ibis's datatypes (under `ibis.expr.datatypes`) labels Jul 5, 2024
@cpcloud cpcloud added the bug Incorrect behavior inside of ibis label Jul 5, 2024
@cpcloud cpcloud marked this pull request as ready for review July 5, 2024 11:13
@cpcloud cpcloud changed the title fix: proper handling of srid in geo dtype [WIP] fix(datatypes): proper handling of srid in geospatial datatypes Jul 5, 2024
@cpcloud
Copy link
Member

cpcloud commented Jul 5, 2024

Gonna add the test case from the original issue here as well!

Copy link
Member

@cpcloud cpcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ncclementi This LGTM if you want to give it another look!

@@ -27,8 +27,6 @@
typecode.ENUM16: dt.String,
typecode.FLOAT: dt.Float32,
typecode.FIXEDSTRING: dt.String,
typecode.GEOMETRY: partial(dt.GeoSpatial, geotype="geometry"),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small question, we don't need these ones any more because we will always git this path
method := getattr(cls, f"_from_sqlglot_{typecode.name}", None) in to_ibis() now?

Copy link
Member

@cpcloud cpcloud Jul 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's the reason, but it was that way before this PR too.

Copy link
Contributor Author

@ncclementi ncclementi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for taking this across the line, looks like it got a bit gnarly.

I left a comment with one question, but it's mostly for my understanding.

@cpcloud
Copy link
Member

cpcloud commented Jul 5, 2024

Thanks @ncclementi! Merging.

@cpcloud cpcloud merged commit a3ceb59 into ibis-project:main Jul 5, 2024
76 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Incorrect behavior inside of ibis datatypes Issues relating to ibis's datatypes (under `ibis.expr.datatypes`) geospatial Geospatial related functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug: Loading geometry from postgres with SRID gives error
2 participants