Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Download BigQuery table to pyarrow table for python-based ingestion flow #1366

Merged
merged 11 commits into from
Mar 12, 2021
Merged

Download BigQuery table to pyarrow table for python-based ingestion flow #1366

merged 11 commits into from
Mar 12, 2021

Commits on Mar 12, 2021

  1. WIP Pull table_ref from BQ 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    b09f8b7 View commit details
    Browse the repository at this point in the history

  2. More additions 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    f2997f3 View commit details
    Browse the repository at this point in the history

  3. Refactor a bit + address comments 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    8710676 View commit details
    Browse the repository at this point in the history

  4. lint 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    c7350a5 View commit details
    Browse the repository at this point in the history

  5. Rebase and add created_timestamp_column 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    13c288c View commit details
    Browse the repository at this point in the history

  6. Filter by timestamp in BQ first 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    0f251d7 View commit details
    Browse the repository at this point in the history

  7. Lint 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    6052d7a View commit details
    Browse the repository at this point in the history

  8. Move field mapping to standalone functions 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    9198c09 View commit details
    Browse the repository at this point in the history

  9. Update docstrings 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    b864271 View commit details
    Browse the repository at this point in the history

  10. Address comments 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    36beacd View commit details
    Browse the repository at this point in the history

  11. Rebase 

    Signed-off-by: Jacob Klegar <jacob@tecton.ai>
    @jklegar
    jklegar committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    85465a1 View commit details
    Browse the repository at this point in the history