Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for SQL Server working copy #362

Merged
merged 2 commits into from
Mar 6, 2021
Merged

Add support for SQL Server working copy #362

merged 2 commits into from
Mar 6, 2021

Conversation

olsen232
Copy link
Collaborator

@olsen232 olsen232 commented Feb 25, 2021

Description

A working implementation of a SQL server working copy.
Tests are passing - checkouts, diffs, and commits are working.

TODO:
Documentation is not yet updated.
Dependencies are not yet updated - so, this works for me locally, and all new sqlserver tests are passing, but this won't work for anyone else because they don't have the appropriate SQL server driver etc. I installed these manually to my local Sno environment. However, this won't be immediately obvious in github actions, since all of the relevant tests will be skipped anyway since no SQL server database is configured.

@olsen232
Copy link
Collaborator Author

olsen232 commented Mar 1, 2021

@craigds @rcoup This working copy is mostly done now

Copy link
Member

@craigds craigds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😅 phew

nice work! I had some questions and suggestions but on the whole it looks great 😄

sno/working_copy/sqlserver.py Outdated Show resolved Hide resolved
sno/working_copy/sqlserver.py Outdated Show resolved Hide resolved
sno/working_copy/sqlserver.py Outdated Show resolved Hide resolved
sno/working_copy/sqlserver.py Outdated Show resolved Hide resolved
sno/working_copy/sqlserver.py Outdated Show resolved Hide resolved
sno/working_copy/sqlserver_adapter.py Outdated Show resolved Hide resolved
sno/working_copy/sqlserver_adapter.py Outdated Show resolved Hide resolved
sno/working_copy/table_defs.py Outdated Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
Copy link
Member

@rcoup rcoup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, is impressive how little code is getting changed here 🌟

sno/geometry.py Show resolved Hide resolved
sno/geometry.py Show resolved Hide resolved
sno/geometry.py Show resolved Hide resolved
sno/sqlalchemy.py Outdated Show resolved Hide resolved
sno/working_copy/base.py Show resolved Hide resolved
sno/working_copy/sqlserver.py Show resolved Hide resolved
sno/working_copy/sqlserver_adapter.py Outdated Show resolved Hide resolved
sno/working_copy/sqlserver_adapter.py Show resolved Hide resolved
sno/working_copy/sqlserver.py Outdated Show resolved Hide resolved


def quote(ident):
return _PREPARER.quote(ident)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to drop these for SQLAlchemy's rather than writing more :)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is sqlalchemy's quote - I just wrapped it since writing quote() is easier than writing IdentifierPreparer(MSDialect()).quote()

But you are right - I should be trying to avoid manually quoting things. This quote here is only called when generating a table spec eg ("FID" bigint, "GEOM" geography, "CRT_DATE" datetime) - it could go away if, instead of generating a table specification in SQL, I created one using sqlalchemy types and created that. It's about the same amount of work, but I just haven't switched to it yet. (It also might not end up being as symmetrical as the current code with regards to writing table specs and reading table specs - I don't know exactly unless I try to write it).

Copy link
Member

@rcoup rcoup Mar 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One option is to use use Table()/etc, but my understanding is you can also use the SA SQL expression builder which has escaping functions/templating/etc for SQL strings

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes - will revisit this soon to try and get rid of lots of f-strings in all 3 working copies.

@olsen232 olsen232 requested review from craigds and rcoup March 3, 2021 02:17
@olsen232 olsen232 force-pushed the sqlserver branch 2 times, most recently from 22d98b1 to 00c1429 Compare March 3, 2021 23:44
@olsen232 olsen232 force-pushed the sqlserver branch 3 times, most recently from e4abd0b to 7955906 Compare March 4, 2021 08:39
@olsen232
Copy link
Collaborator Author

olsen232 commented Mar 4, 2021

Missed all the hidden comments last time - didn't realise github likes to hide comments. Think I've addressed everything now.
@rcoup @craigds

sno/sqlalchemy.py Outdated Show resolved Hide resolved
Copy link
Member

@rcoup rcoup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly there 😄

sno/working_copy/base.py Outdated Show resolved Hide resolved
sno/working_copy/base.py Show resolved Hide resolved
@@ -603,6 +634,19 @@ def _apply_meta_metadata_dataset_json(self, sess, dataset, src_value, dest_value
def _update_last_write_time(self, sess, dataset, commit=None):
self._update_gpkg_contents(sess, dataset, commit)

def _get_geom_extent(self, sess, dataset):
"""Returns the envelope around the entire dataset as (min_x, min_y, max_x, max_y)."""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using GetSpatialIndexExtent() should be a lot quicker

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not done.
Ugh just spent a good half hour on this - for no particular reason, calling GetSpatialIndex from within Sno / sqlalchemy reliably segfaults for me, although I can call it from my sqlite3 CLI. My spatialite version should be the same for both. Left this as-is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough.

It's documented as returning a Spatialite geometry, so would need to cast it to GPKG/WKB or extract min/max values from it. Not sure if that's the issue though. 🤷

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably not the issue - I was calling ST_MinX etc on it, it was working in sqlite3, just not in Sno

sno/working_copy/postgis.py Outdated Show resolved Hide resolved
sno/working_copy/postgis.py Outdated Show resolved Hide resolved
sno/working_copy/sqlserver_adapter.py Outdated Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
tests/test_working_copy_sqlserver.py Outdated Show resolved Hide resolved
tests/test_working_copy_sqlserver.py Show resolved Hide resolved
sno/working_copy/sqlserver.py Outdated Show resolved Hide resolved
sno/working_copy/postgis.py Outdated Show resolved Hide resolved
Copy link
Member

@rcoup rcoup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! 🏅

@olsen232 olsen232 merged commit 4f12a22 into master Mar 6, 2021
@olsen232 olsen232 deleted the sqlserver branch March 6, 2021 23:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants