Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix problematic Date fiels in shapefile #176

Merged
merged 1 commit into from
Nov 17, 2020

Conversation

cipri-tom
Copy link
Contributor

Hi,

I have a shapefile which contains a Date field. This is returned as bytes, which is not writable to JSON.
Here is a PR which check which fields are dates and converts them to strings, using the same convention as pyShp.

It is related to this issue in pyShp.

Of course, the better fix would be that pyShp provides a way to represent dates as strings. Then, the whole shp_to_json() function would be only a call to reader.__geo_interface__.

On the same note, we can make the shp_to_json() function not write an auxiliary file. Or, ar least, it should be optional, since we can json.dumps and return that string directly. Would you be OK to refactor this method?

Thanks,
Ciprian

@cipri-tom
Copy link
Contributor Author

I tried to run tests on my side, but they are failing as I cannot connect to earth-engine with the default way, only with initialization from service account.

Also, I tried to run flake8, but thre are a lot of warnings in places not related to these code changes.

@giswqs giswqs added the enhancement improvements to geemap label Nov 17, 2020
@giswqs giswqs merged commit 0ee092e into gee-community:master Nov 17, 2020
@giswqs
Copy link
Member

giswqs commented Nov 17, 2020

This is great! Thanks for the fix. And yes, we can make creating the geojson file optional. Pull request is welcome. Thanks.

@karimbahgat
Copy link

Just pushed new commits to pyshp to address this. Now, date fields are loaded into python date objects for convenience to the user as usual, but calls to geo_interface will automatically convert them back to the original date string, so that they are directly able to be json serialized. This means you can simply call .__geo_interface__ on the entire shapefile, or on each individual feature/shaperecord.

A minor correction regarding the previous behavior: Date fields were returned as datetime.date objects, as None if null-value, or as bytes if the datestring was faulty and could not be parsed. However, it's possible some null-values were not correctly identified and failed to bytestring, but I've now made this more robust.

Until I push this to a new release version, you can test it out from the latest GitHub repo.

@giswqs
Copy link
Member

giswqs commented Dec 21, 2020

@karimbahgat Thanks for the fix.

@cipri-tom cipri-tom deleted the bugfix/shp-to-geojson-bytes branch January 7, 2021 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement improvements to geemap
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants