🐛 Source Google Search Console: incremental stream not working incrementally

[danieldiamond]

Environment

• Airbyte version: 0.33.11-alpha
• OS Version / Instance: AWS EC2
• Deployment: Docker
• Source Connector and version: GSC 0.1.7
• Destination Connector and version: Snowflake 0.3.20
• Severity: High
• Step where error happened: Sync Job

Current Behavior

Incremental stream performs full refresh

Expected Behavior

Incremental stream performs incremental sync

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG

second_sync.txt
first_sync.txt

Steps to Reproduce

Are you willing to submit a PR?

No

Possibly information related: #7479

[grubberr]

Hay @danieldiamond yes you probably right !
current implementation of connector did NOT PASS test_incremental.
as for me current implementation of "incremental streaming" looks buggy

it's not fully comply CDK concepts:

1. it slices over conceptually wrong pair: [url, type]
   but CDK assume you slice over something comparable like date or datetime

2. For one stream it keeps pretty complex state object

{
  "url1": {
    "web": {"date": "date"},
    "news": {"date": "date"}
  },
  "url2": {
    "web": {"date": "date"},
    "news": {"date": "date"}
  },
  "date": "date"  <- IT PUT HERE JUST TO MAKE TESTS SILENT
}

and it ignores concept of cursor_field.

I bet all this can produce to incorrect results
slicing and keeped state - has to be re-implemented to use right approach.