Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Destination bigquery: rerelease 1s1t behind gate #27936

Merged
merged 30 commits into from
Jul 14, 2023

Commits on Jul 3, 2023

  1. Configuration menu
    Copy the full SHA
    0ed9c0b View commit details
    Browse the repository at this point in the history
  2. version bumps+changelog

    edgao committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    308422e View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. Configuration menu
    Copy the full SHA
    151e675 View commit details
    Browse the repository at this point in the history
  2. 🐛 Destination Bigquery: fix bug in standard inserts for syncs >10K re…

    …cords (#27856)
    
    * only run t+d code if it's enabled
    
    * dockerfile+changelog
    
    * remove changelog entry
    edgao authored Jul 5, 2023
    Configuration menu
    Copy the full SHA
    6196133 View commit details
    Browse the repository at this point in the history
  3. Destinations V2: handle optional fields for object and array types (

    #27898)
    
    * catch null schema
    
    * fix null properties
    
    * clean up
    
    * consolidate + add more tests
    
    * try catch
    
    * empty json test
    cynthiaxyin authored Jul 5, 2023
    Configuration menu
    Copy the full SHA
    452df77 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    c915af0 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. remove todo

    edgao committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    8969f73 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. destination bigquery: misc updates to 1s1t code (#28057)

    * switch to checkedconsumer
    
    * add unit test for buildColumnId
    
    * use flag
    
    * restructure prefix check
    
    * fix build
    edgao authored Jul 7, 2023
    Configuration menu
    Copy the full SHA
    3ca51b6 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2023

  1. more type-parsing fixes (#28100)

    * more type-parsing fixes
    
    * handle duplicates
    
    * Automated Commit - Format and Process Resources Changes
    
    * add tests for asColumns
    
    * Automated Commit - Format and Process Resources Changes
    
    * log warnings instead of throwing exception
    
    * better log message
    
    * error level
    
    ---------
    
    Co-authored-by: edgao <edgao@users.noreply.github.com>
    edgao and edgao authored Jul 10, 2023
    Configuration menu
    Copy the full SHA
    57422c1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2a63e8b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    353db7b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a65eff3 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    9660632 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2023

  1. Change from T&D every 10k records to an increasing time based interval (

    #28130)
    
    * fifteen minute t&d
    
    * add typing and deduping operation valve for increased intervals of typing and deduping
    
    * Automated Commit - Format and Process Resources Changes
    
    * resolve bizarre merge conflict
    
    * Automated Commit - Format and Process Resources Changes
    
    ---------
    
    Co-authored-by: jbfbell <jbfbell@users.noreply.github.com>
    jbfbell and jbfbell authored Jul 11, 2023
    Configuration menu
    Copy the full SHA
    2dc7d77 View commit details
    Browse the repository at this point in the history
  2. Simplify and speed up CDC delete support [DestinationsV2] (#28029)

    * Simplify and speed up CDC delete support [DestinationsV2]
    
    * better QUOTE
    
    * spotbugs?
    
    * recompile dbt image for local arch and use that when building images
    
    * things compile, but tests fail
    
    * tests working-ish
    
    * comment
    
    * fix logic to re-insert deleted records for cursor comparison.
    
    tests pass!
    
    * remove comment
    
    * Skip CDC re-include logic if there are no CDC columns
    
    * stop hardcoding pk (#28092)
    
    * wip
    
    * remove TODOs
    
    ---------
    
    Co-authored-by: Edward Gao <edward.gao@airbyte.io>
    evantahler and edgao authored Jul 11, 2023
    Configuration menu
    Copy the full SHA
    7efc294 View commit details
    Browse the repository at this point in the history
  3. update method name

    jbfbell committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    57b54ec View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    aa9b318 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    a16881e View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2023

  1. Configuration menu
    Copy the full SHA
    7d23572 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    610ea35 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    8792461 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    344908f View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2023

  1. Configuration menu
    Copy the full SHA
    f2da68c View commit details
    Browse the repository at this point in the history
  2. implement 1s1t DATs for destination-bigquery (#27852)

    * intiial implementation
    
    * Automated Commit - Formatting Changes
    
    * add second sync to test
    
    * do concurrent things
    
    * Automated Commit - Formatting Changes
    
    * clarify comment
    
    * minor tweaks
    
    * more stuff
    
    * Automated Commit - Formatting Changes
    
    * minor cleanup
    
    * lots of fixes
    
    * handle sql vs json null better
    * verify extra columns
    * only check deleted_at if in DEDUP mode and the column exists
    * add full refresh append test case
    
    * Automated Commit - Formatting Changes
    
    * add tests for the remaining sync modes
    
    * Automated Commit - Formatting Changes
    
    * readability stuff
    
    * Automated Commit - Formatting Changes
    
    * add test for gcs mode
    
    * remove static fields
    
    * Automated Commit - Formatting Changes
    
    * add more test cases, tweak test scaffold
    
    * cleanup
    
    * Automated Commit - Formatting Changes
    
    * extract recorddiffer
    
    * and use it in the sql generator test
    
    * fix
    
    * comment
    
    * naming+comment
    
    * one more comment
    
    * better assert
    
    * remove unnecessary thing
    
    * one last thing
    
    * Automated Commit - Formatting Changes
    
    * enable concurrent execution on all java integration tests
    
    * add test for default namespace
    
    * Automated Commit - Formatting Changes
    
    * implement a 2-stream test
    
    * Automated Commit - Formatting Changes
    
    * extract methods
    
    * invert jsonNodesNotEquivalent
    
    * Automated Commit - Formatting Changes
    
    * fix conditional
    
    * pull out diffSingleRecord
    
    * Automated Commit - Formatting Changes
    
    * handle nulls correctly
    
    * remove raw-specific handling; break up methods
    
    * Automated Commit - Formatting Changes
    
    ---------
    
    Co-authored-by: edgao <edgao@users.noreply.github.com>
    Co-authored-by: octavia-approvington <octavia-approvington@users.noreply.github.com>
    3 people authored Jul 13, 2023
    Configuration menu
    Copy the full SHA
    bf65992 View commit details
    Browse the repository at this point in the history
  3. Destinations V2: move create raw tables earlier (#28255)

    * move create raw tables
    
    * better log message
    cynthiaxyin authored Jul 13, 2023
    Configuration menu
    Copy the full SHA
    1b10cdb View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f8cd863 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    5a93265 View commit details
    Browse the repository at this point in the history
  6. fix ability to run tests

    edgao committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    aa7b029 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. disable incremental t+d for now

    edgao committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    ae9a5b5 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    105517f View commit details
    Browse the repository at this point in the history