Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redshift to Snowflake Connector fails but sync shows succeeded #4628

Closed
danieldiamond opened this issue Jul 8, 2021 · 2 comments
Closed

Redshift to Snowflake Connector fails but sync shows succeeded #4628

danieldiamond opened this issue Jul 8, 2021 · 2 comments
Labels
connectors/destination/snowflake connectors/destinations-warehouse connectors/source/redshift connectors/sources-database type/bug Something isn't working

Comments

@danieldiamond
Copy link
Contributor

Enviroment

  • Airbyte version: 0.27.0-alpha
  • OS Version / Instance: AWS EC2
  • Deployment: Docker
  • Source Connector and version: Redshift 0.3.1
  • Destination Connector and version: Snowflake 0.3.9
  • Severity: Critical
  • Step where error happened: Sync job

Current Behavior

Job fails due to incorrect user permissions but job shows successful sync

Expected Behavior

Airbyte should error when a job sync actually fails

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG 

2021-07-08 12:06:53 INFO () WorkerRun(call):62 - Executing worker wrapper. Airbyte version: 0.27.0-alpha
2021-07-08 12:06:53 INFO () TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.27.0-alpha
2021-07-08 12:06:53 INFO () DefaultReplicationWorker(run):102 - start sync worker. job id: 1767 attempt id: 0
2021-07-08 12:06:53 INFO () DefaultReplicationWorker(run):111 - configured sync modes: {<redacted>.<redacted>=full_refresh - overwrite}
2021-07-08 12:06:53 INFO () DefaultAirbyteDestination(start):78 - Running destination...
2021-07-08 12:06:53 INFO () LineGobbler(voidCall):85 - Checking if airbyte/destination-snowflake:0.3.9 exists...
2021-07-08 12:06:53 INFO () LineGobbler(voidCall):85 - airbyte/destination-snowflake:0.3.9 was found locally.
2021-07-08 12:06:53 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/1767/0 --network host --log-driver none airbyte/destination-snowflake:0.3.9 write --config destination_config.json --catalog destination_catalog.json
2021-07-08 12:06:53 INFO () LineGobbler(voidCall):85 - Checking if airbyte/source-redshift:0.3.1 exists...
2021-07-08 12:06:53 INFO () LineGobbler(voidCall):85 - airbyte/source-redshift:0.3.1 was found locally.
2021-07-08 12:06:53 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/1767/0 --network host --log-driver none airbyte/source-redshift:0.3.1 read --config source_config.json --catalog source_catalog.json
2021-07-08 12:06:53 INFO () DefaultReplicationWorker(lambda$getDestinationOutputRunnable$3):243 - Destination output thread started.
2021-07-08 12:06:53 INFO () DefaultReplicationWorker(run):139 - Waiting for source thread to join.
2021-07-08 12:06:53 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):207 - Replication thread started.
2021-07-08 12:06:58 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:58 �[32mINFO�[m i.a.i.s.r.RedshiftSource(main):67 - {} - starting source: class io.airbyte.integrations.source.redshift.RedshiftSource
2021-07-08 12:06:58 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:58 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):78 - {} - Running integration: io.airbyte.integrations.source.redshift.RedshiftSource
2021-07-08 12:06:58 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:58 �[32mINFO�[m i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {read=null, catalog=source_catalog.json, config=source_config.json}
2021-07-08 12:06:58 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:58 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):82 - {} - Command: READ
2021-07-08 12:06:59 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:59 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):83 - {} - Integration config: IntegrationConfig{command=READ, configPath='source_config.json', catalogPath='source_catalog.json', statePath='null'}
2021-07-08 12:06:59 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:59 �[32mINFO�[m i.a.i.d.s.SnowflakeDestination(main):81 - {} - starting destination: class io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2021-07-08 12:06:59 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:59 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):78 - {} - Running integration: io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2021-07-08 12:06:59 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:59 �[32mINFO�[m i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=destination_catalog.json, write=null, config=destination_config.json}
2021-07-08 12:06:59 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:59 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):82 - {} - Command: WRITE
2021-07-08 12:06:59 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:59 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):83 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='destination_config.json', catalogPath='destination_catalog.json', statePath='null'}
2021-07-08 12:06:59 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:59 �[32mINFO�[m i.a.i.s.j.JdbcCdcStateManager(<init>):46 - {} - Initialized CDC state with: null
2021-07-08 12:06:59 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:59 �[32mINFO�[m i.a.i.s.j.JdbcStateManager(createCursorInfoForStream):138 - {} - No cursor field set in catalog but not present in state. Stream: AirbyteStreamNameNamespacePair{name='<redacted>', namespace='<redacted>'}, New Cursor Field: null. Resetting cursor value
2021-07-08 12:07:00 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:06:59 �[32mINFO�[m i.a.i.d.j.c.SwitchingDestination(getConsumer):83 - {} - Using destination type: INSERT
2021-07-08 12:07:00 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:00 �[32mINFO�[m i.a.i.d.b.BufferedStreamConsumer(startTracked):120 - {} - class io.airbyte.integrations.destination.buffered_stream_consumer.BufferedStreamConsumer started.
2021-07-08 12:07:00 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:00 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):145 - {} - Preparing tmp tables in destination started for 1 streams
2021-07-08 12:07:00 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:00 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):149 - {} - Preparing tmp table in destination started for stream <redacted>. schema: <redacted>, tmp table name: _airbyte_1625746020058__airbyte_raw_<redacted>
2021-07-08 12:07:04 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:04 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onStartFunction$1):155 - {} - Preparing tables in destination completed.
2021-07-08 12:07:21 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:21 �[32mINFO�[m i.a.i.s.j.AbstractJdbcSource(queryTableFullRefresh):520 - {} - Queueing query for table: <redacted>
2021-07-08 12:07:22 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:22 �[32mINFO�[m i.a.i.s.j.AbstractJdbcSource(lambda$queryTableFullRefresh$29):525 - {} - Preparing query for table: <redacted>
2021-07-08 12:07:22 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:22 �[32mINFO�[m i.a.i.s.j.AbstractJdbcSource(lambda$read$4):193 - {} - Closing database connection pool.
2021-07-08 12:07:22 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:22 �[32mINFO�[m i.a.i.s.j.AbstractJdbcSource(lambda$read$4):195 - {} - Closed database connection pool.
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - Exception in thread "main" java.lang.RuntimeException: java.sql.SQLException: [Amazon](500310) Invalid operation: permission denied for schema <redacted>;
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.db.jdbc.StreamingJdbcDatabase.query(StreamingJdbcDatabase.java:108)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableFullRefresh$30(AbstractJdbcSource.java:523)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:57)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:58)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:58)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:83)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:58)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:105)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.source.redshift.RedshiftSource.main(RedshiftSource.java:68)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - Caused by: java.sql.SQLException: [Amazon](500310) Invalid operation: permission denied for schema <redacted>;
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.client.messages.inbound.ErrorResponse.toErrorException(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.client.PGMessagingContext.handleErrorResponse(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.client.PGMessagingContext.handleMessage(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.jdbc.communications.InboundMessagesPipeline.getNextMessageOfClass(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.client.PGMessagingContext.doMoveToNextClass(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.client.PGMessagingContext.getParameterDescription(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.client.PGClient.prepareStatement(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.dataengine.PGQueryExecutor.<init>(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.dataengine.PGDataEngine.prepare(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.jdbc.common.SPreparedStatement.<init>(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.jdbc.jdbc41.S41PreparedStatement.<init>(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.jdbc.jdbc42.S42PreparedStatement.<init>(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.core.jdbc42.PGJDBC42PreparedStatement.<init>(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.redshift.core.jdbc42.PGJDBC42ObjectFactory.createPreparedStatement(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.jdbc.common.SConnection.prepareStatement(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.amazon.jdbc.common.SConnection.prepareStatement(Unknown Source)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:303)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at org.apache.commons.dbcp2.DelegatingConnection.prepareStatement(DelegatingConnection.java:303)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableFullRefresh$29(AbstractJdbcSource.java:529)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.db.jdbc.StreamingJdbcDatabase.query(StreamingJdbcDatabase.java:95)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.source.jdbc.AbstractJdbcSource.lambda$queryTableFullRefresh$30(AbstractJdbcSource.java:523)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.LazyAutoCloseableIterator.computeNext(LazyAutoCloseableIterator.java:57)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:58)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:58)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.CompositeIterator.computeNext(CompositeIterator.java:83)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.commons.util.DefaultAutoCloseableIterator.computeNext(DefaultAutoCloseableIterator.java:58)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:105)
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - Caused by: com.amazon.support.exceptions.ErrorException: [Amazon](500310) Invalid operation: permission denied for schema <redacted>;
2021-07-08 12:07:22 ERROR () LineGobbler(voidCall):85 - 	... 40 more
2021-07-08 12:07:24 INFO () DefaultReplicationWorker(run):141 - Source thread complete.
2021-07-08 12:07:24 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:24 �[32mINFO�[m i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):80 - {} - Airbyte message consumer: succeeded.
2021-07-08 12:07:24 INFO () DefaultReplicationWorker(run):142 - Waiting for destination thread to join.
2021-07-08 12:07:24 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:24 �[32mINFO�[m i.a.i.d.b.BufferedStreamConsumer(close):190 - {} - executing on success close procedure.
2021-07-08 12:07:24 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:24 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):182 - {} - Finalizing tables in destination started for 1 streams
2021-07-08 12:07:24 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:24 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):187 - {} - Finalizing stream <redacted>. schema <redacted>, tmp table _airbyte_1625746020058__airbyte_raw_<redacted>, final table _airbyte_raw_<redacted>
2021-07-08 12:07:25 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:25 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):200 - {} - Executing finalization of tables.
2021-07-08 12:07:40 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:40 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):202 - {} - Finalizing tables in destination completed.
2021-07-08 12:07:40 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:40 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):205 - {} - Cleaning tmp tables in destination started for 1 streams
2021-07-08 12:07:40 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:40 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):209 - {} - Cleaning tmp table in destination started for stream <redacted>. schema <redacted>, tmp table name: _airbyte_1625746020058__airbyte_raw_<redacted>
2021-07-08 12:07:41 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:41 �[32mINFO�[m i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):214 - {} - Cleaning tmp tables in destination completed.
2021-07-08 12:07:41 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:41 �[32mINFO�[m i.a.i.b.IntegrationRunner(run):118 - {} - Completed integration: io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2021-07-08 12:07:41 INFO () DefaultAirbyteStreamFactory(lambda$create$0):73 - 2021-07-08 12:07:41 �[32mINFO�[m i.a.i.d.s.SnowflakeDestination(main):83 - {} - completed destination: class io.airbyte.integrations.destination.snowflake.SnowflakeDestination
2021-07-08 12:07:43 INFO () DefaultReplicationWorker(run):144 - Destination thread complete.
2021-07-08 12:07:43 WARN () DefaultAirbyteSource(close):132 - Source process might not have shut down correctly. source process alive: false, source process exit value: 1. This warning is normal if the job was cancelled.
2021-07-08 12:07:43 INFO () DefaultReplicationWorker(run):169 - sync summary: io.airbyte.config.ReplicationAttemptSummary@74eee169[status=completed,recordsSynced=0,bytesSynced=0,startTime=1625746013254,endTime=1625746063725]
2021-07-08 12:07:43 INFO () DefaultReplicationWorker(run):178 - Source did not output any state messages
2021-07-08 12:07:43 WARN () DefaultReplicationWorker(run):189 - State capture: No state retained.
2021-07-08 12:07:43 INFO () TemporalAttemptExecution(get):133 - Stopping cancellation check scheduling...
2021-07-08 12:07:43 INFO () RetryingTemporalAttemptExecution(get):118 - Last output present: true. Should attempt again: false
2021-07-08 12:07:43 INFO () SyncWorkflow$ReplicationActivityImpl(replicate):202 - attempt summaries: 
2021-07-08 12:07:43 INFO () SyncWorkflow$ReplicationActivityImpl(replicate):203 - sync summary: 
2021-07-08 12:07:43 INFO () TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.27.0-alpha
2021-07-08 12:07:43 INFO () DefaultNormalizationWorker(run):61 - Running normalization.
2021-07-08 12:07:43 INFO () LineGobbler(voidCall):85 - Checking if airbyte/normalization:0.1.34 exists...
2021-07-08 12:07:43 INFO () LineGobbler(voidCall):85 - airbyte/normalization:0.1.34 was found locally.
2021-07-08 12:07:43 INFO () DockerProcessFactory(create):146 - Preparing command: docker run --rm --init -i -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -w /data/1767/0/normalize --network host --log-driver none airbyte/normalization:0.1.34 run --integration-type snowflake --config destination_config.json --catalog destination_catalog.json
2021-07-08 12:07:46 INFO () LineGobbler(voidCall):85 - Running: transform-config --config destination_config.json --integration-type snowflake --out /data/1767/0/normalize
2021-07-08 12:07:46 INFO () LineGobbler(voidCall):85 - Namespace(config='destination_config.json', integration_type=<DestinationType.snowflake: 'snowflake'>, out='/data/1767/0/normalize')
2021-07-08 12:07:46 INFO () LineGobbler(voidCall):85 - transform_snowflake
2021-07-08 12:07:46 INFO () LineGobbler(voidCall):85 - Running: transform-catalog --integration-type snowflake --profile-config-dir /data/1767/0/normalize --catalog destination_catalog.json --out /data/1767/0/normalize/models/generated/ --json-column _airbyte_data
2021-07-08 12:07:46 INFO () LineGobbler(voidCall):85 - Processing destination_catalog.json...
2021-07-08 12:07:46 INFO () LineGobbler(voidCall):85 -   Generating airbyte_ctes/<redacted>/<redacted>_AB1.sql from <redacted>
2021-07-08 12:07:46 INFO () LineGobbler(voidCall):85 -   Generating airbyte_ctes/<redacted>/<redacted>_AB2.sql from <redacted>
2021-07-08 12:07:46 INFO () LineGobbler(voidCall):85 -   Generating airbyte_ctes/<redacted>/<redacted>_AB3.sql from <redacted>
2021-07-08 12:07:46 INFO () LineGobbler(voidCall):85 -   Generating airbyte_tables/<redacted>/<redacted>.sql from <redacted>
2021-07-08 12:07:48 INFO () LineGobbler(voidCall):85 - Running with dbt=0.19.1
2021-07-08 12:07:50 INFO () LineGobbler(voidCall):85 - [�[33mWARNING�[0m]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
2021-07-08 12:07:50 INFO () LineGobbler(voidCall):85 - There are 1 unused configuration paths:
2021-07-08 12:07:50 INFO () LineGobbler(voidCall):85 - - models.airbyte_utils.generated.airbyte_views
2021-07-08 12:07:50 INFO () LineGobbler(voidCall):85 - 
2021-07-08 12:07:51 INFO () LineGobbler(voidCall):85 - Found 4 models, 0 tests, 0 snapshots, 0 analyses, 369 macros, 0 operations, 0 seed files, 1 source, 0 exposures
2021-07-08 12:07:51 INFO () LineGobbler(voidCall):85 - 
2021-07-08 12:07:55 INFO () LineGobbler(voidCall):85 - 12:07:55 | Concurrency: 32 threads (target='prod')
2021-07-08 12:07:55 INFO () LineGobbler(voidCall):85 - 12:07:55 | 
2021-07-08 12:07:55 INFO () LineGobbler(voidCall):85 - 12:07:55 | 1 of 1 START table model <redacted>.<redacted>................................................... [RUN]
2021-07-08 12:07:57 INFO () LineGobbler(voidCall):85 - 12:07:57 | 1 of 1 OK created table model <redacted>.<redacted>.............................................. [�[32mSUCCESS 1�[0m in 2.39s]
2021-07-08 12:07:59 INFO () LineGobbler(voidCall):85 - 12:07:59 | 
2021-07-08 12:07:59 INFO () LineGobbler(voidCall):85 - 12:07:59 | Finished running 1 table model in 8.23s.
2021-07-08 12:07:59 INFO () LineGobbler(voidCall):85 - 
2021-07-08 12:07:59 INFO () LineGobbler(voidCall):85 - �[32mCompleted successfully�[0m
2021-07-08 12:07:59 INFO () LineGobbler(voidCall):85 - 
2021-07-08 12:07:59 INFO () LineGobbler(voidCall):85 - Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
2021-07-08 12:08:01 INFO () DefaultNormalizationWorker(run):77 - Normalization executed in 0.
2021-07-08 12:08:01 INFO () TemporalAttemptExecution(get):133 - Stopping cancellation check scheduling...

Steps to Reproduce

  1. Set up redshift connector with write only permissions
  2. Run job sync to read from redshift
  3. Watch it fail but still show a successful sync
@danieldiamond danieldiamond added the type/bug Something isn't working label Jul 8, 2021
@ChristopheDuong
Copy link
Contributor

ChristopheDuong commented Jul 8, 2021 •
edited
Loading

This seem similar to the same topic here :
#4298
#4236
#4538

@sherifnada
Copy link
Contributor

closed in #4617

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
connectors/destination/snowflake connectors/destinations-warehouse connectors/source/redshift connectors/sources-database type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@sherifnada @ChristopheDuong @karinakuz @danieldiamond @igrankova