Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download #1050

Merged
merged 38 commits into from
Oct 9, 2019
Merged

Download #1050

merged 38 commits into from
Oct 9, 2019

Conversation

simon-contreras-deel
Copy link
Contributor

@simon-contreras-deel simon-contreras-deel commented Oct 2, 2019

@simon-contreras-deel simon-contreras-deel marked this pull request as ready for review October 7, 2019 14:44
Copy link
Contributor

@alrocar alrocar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a comment about geography tables. Let's be sure the downloaded geometry comes in a format we can then save in the CSV and work with.


def _download_query(project, dataset, table, limit=None, offset=None):
full_table_name = '`{}.{}.{}`'.format(project, dataset, table)
query = 'SELECT * FROM {}'.format(full_table_name)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will use this for both downloading a plain dataset or a geographies dataset. I'd like to be sure of the geometry format downloaded and written in the CSV.

Could we add a test to make sure that we can read/upload to CARTO the geometry in the downloaded CSV file for a geography table? (or any other type of test that you think it might be interesting for this case)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The geometry is saved as WKT as string in the file

"POLYGON((-4.22608307660106 43.3126868301895, ...))"

We could use a carto-do-public-data for that, but I would not really like to make a tests end to end against big query and carto.

(still thinking about it)

@simon-contreras-deel simon-contreras-deel removed their assignment Oct 8, 2019
Copy link
Contributor

@alrocar alrocar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice implementation.

Added some suggestions, the one about the notebook I'd like to be addressed 🙏

cartoframes/data/observatory/dataset.py Outdated Show resolved Hide resolved
cartoframes/data/observatory/entity.py Outdated Show resolved Hide resolved
cartoframes/data/observatory/entity.py Outdated Show resolved Hide resolved
cartoframes/data/observatory/geography.py Show resolved Hide resolved
examples/08_data_observatory/download.ipynb Outdated Show resolved Hide resolved
simon-contreras-deel and others added 6 commits October 8, 2019 19:29
Co-Authored-By: Alberto Romeu <alrocar@users.noreply.github.com>
Co-Authored-By: Alberto Romeu <alrocar@users.noreply.github.com>
@simon-contreras-deel simon-contreras-deel removed their assignment Oct 8, 2019
Copy link
Contributor

@alrocar alrocar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀 🦉

@alrocar
Copy link
Contributor

alrocar commented Oct 8, 2019

Acceptance 🍏

Let's address my last comment (about ™️ ) and this is ready to merge.

@simon-contreras-deel simon-contreras-deel merged commit 243e3a0 into develop Oct 9, 2019
@simon-contreras-deel simon-contreras-deel deleted the feature/do-bq-download branch October 9, 2019 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants