Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BigQuery: Allow subset of schema to be passed into load_table_from_dataframe. #9064

Merged
merged 9 commits into from
Aug 22, 2019
Merged

BigQuery: Allow subset of schema to be passed into load_table_from_dataframe. #9064

merged 9 commits into from
Aug 22, 2019

Conversation

tswast
Copy link
Contributor

@tswast tswast commented Aug 20, 2019 •
edited
Loading

Closes #8140

  • Allow subset of schema to be passed into load_table_from_dataframe.
  • The types of any remaining columns will be auto-detected.
  • When the type of any remaining column can't be determined, warn that the schema was ignored because the type of column X could not be determined.
  • Update snippets to show overriding schema for columns whose types can't be autodetected. Blocked by BigQuery: Load to table from dataframe without index #5572 (comment).

This PR is branched from #9049.

@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Aug 20, 2019
@tswast tswast marked this pull request as ready for review August 21, 2019 18:59
@tswast tswast requested review from a team and plamut August 21, 2019 18:59
@tswast tswast mentioned this pull request Aug 21, 2019
Closed
@tswast tswast mentioned this pull request Aug 21, 2019
Closed
@tswast
Copy link
Contributor Author

tswast commented Aug 22, 2019 •
edited
Loading

Blocked by #9074

Test failures are because the schema in the snippets samples cannot be autodetected and we send a schema of {"fields": []}, instead of None.

@tswast tswast added api: bigquery Issues related to the BigQuery API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. labels Aug 22, 2019
plamut
plamut reviewed Aug 22, 2019
View reviewed changes
Copy link
Contributor

@plamut plamut left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, I can see how this can be useful to users. 👍

Had a few sanity check remarks/questions, please just check if any of them apply.

Edit: What about the last point in the PR checklist, i.e. updating snippets?

bigquery/tests/unit/test_client.py Show resolved Hide resolved
bigquery/tests/system.py Outdated Show resolved Hide resolved
bigquery/google/cloud/bigquery/_pandas_helpers.py Show resolved Hide resolved
bigquery/tests/system.py Show resolved Hide resolved
@tswast tswast mentioned this pull request Aug 22, 2019
Closed
@tswast
Copy link
Contributor Author

tswast commented Aug 22, 2019

On second thought, since the snippets show writing the indexes of a DataFrame, I can't actually update them until #5572 (comment) is fixed. I'll work on that issue next.

@tswast tswast requested a review from plamut August 22, 2019 18:21
@tswast tswast removed the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Aug 22, 2019
plamut
plamut approved these changes Aug 22, 2019
View reviewed changes
Copy link
Contributor

@plamut plamut left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, and I agree with leaving out the snippets until their blocker is resolved.

Edit: One of the unit tests is failing, needs a fix.

@tswast tswast merged commit adefce3 into googleapis:master Aug 22, 2019
@tswast tswast deleted the issue8140-partial-schema branch August 22, 2019 19:36
HemangChothani pushed a commit to HemangChothani/google-cloud-python that referenced this pull request Aug 29, 2019
@tswast
Allow subset of schema to be passed into load_table_from_dataframe. (…
a4739bd 
…googleapis#9064)

* Allow subset of schema to be passed into `load_table_from_dataframe`.

The types of any remaining columns will be auto-detected.

* Warn when it's not possible to determine a column type.
emar-kar pushed a commit to MaxxleLLC/google-cloud-python that referenced this pull request Sep 18, 2019
@tswast @emar-kar
Allow subset of schema to be passed into load_table_from_dataframe. (…
e3dcf4a 
…googleapis#9064)

* Allow subset of schema to be passed into `load_table_from_dataframe`.

The types of any remaining columns will be auto-detected.

* Warn when it's not possible to determine a column type.
@tswast tswast mentioned this pull request Nov 12, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@plamut plamut plamut approved these changes

Assignees

@plamut plamut

Labels
api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

BigQuery: Allow partial schema in load_table_from_dataframe
3 participants
@tswast @plamut @googlebot