Encountered an exception while reading stream SourceHubspot

[coadan]

Environment

• Airbyte version: example is 0.35.3-alpha
• OS Version / Instance: GCP Kubernetes Autopilot instances
• Deployment: Kubernetes
• Source Connector and version: HubSpot - Latest
• Destination Connector and version: Postgres - Latest
• Severity: Low
• Step where error happened: Sync job

Current Behavior

The sync job is set to failed because it seems to be an issue with a preceding PY-script in the Hubspot source connector

Logs

logs-63-0.txt

LOG

2022-01-06 08:16:13 �[1;31mERROR�[m i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - Encountered an exception while reading stream SourceHubspot
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/deprecated/base_source.py", line 67, in read
    yield from self._read_stream(logger=logger, client=client, configured_stream=configured_stream, state=total_state)
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/deprecated/base_source.py", line 79, in _read_stream
    use_incremental = configured_stream.sync_mode == SyncMode.incremental and client.stream_has_state(stream_name)
  File "/airbyte/integration_code/source_hubspot/client.py", line 76, in stream_has_state
    return hasattr(self._apis[name], "state")
KeyError: 'quotes'
2022-01-06 08:16:13 �[1;31mERROR�[m i.a.w.p.a.DefaultAirbyteStreamFactory(internalLog):95 - 'quotes'
Traceback (most recent call last):
  File "/airbyte/integration_code/main.py", line 13, in <module>
    launch(source, sys.argv[1:])
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 116, in launch
    for message in source_entrypoint.run(parsed_args):
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/entrypoint.py", line 107, in run
    for message in generator:
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/deprecated/base_source.py", line 67, in read
    yield from self._read_stream(logger=logger, client=client, configured_stream=configured_stream, state=total_state)
  File "/usr/local/lib/python3.7/site-packages/airbyte_cdk/sources/deprecated/base_source.py", line 79, in _read_stream
    use_incremental = configured_stream.sync_mode == SyncMode.incremental and client.stream_has_state(stream_name)
  File "/airbyte/integration_code/source_hubspot/client.py", line 76, in stream_has_state
    return hasattr(self._apis[name], "state")
KeyError: 'quotes'

Steps to Reproduce

1. Run a hubspot sync

Are you willing to submit a PR?

Seems to be an issue related to this file in the error log: "/airbyte/integration_code/source_hubspot/client.py", line 76?

[harshithmullapudi]

Hey @coadan looks like there is some issue with scope which I found in the logs
Stream product cannot be procced. This app hasn't been granted all required scopes to make this call. Read more about required scopes here: https://developers.hubspot.com/scopes.

[coadan]

Yes, but this should not set the sync job in a failed state, the others just provide a warning? I tried skipping sync of the quotes object and it started working again so it might be an issue there when it comes to handling the missing oauth scope.

[keu]

Possible reproducing steps:

• create connector and select API Key Credentials as credentials option
• select quotes stream to sync
• update connector configuration to use OAuth Credentials as credentials option
• run sync (see error in the logs)

possible fix:

• check that stream exist in client before reading and emit warning, skip stream afterwards.