Bootloader crashes when using image tags that don't follow semantic versioning #21006

SebastienN15 · 2023-01-04T08:56:35Z

Environment

Airbyte version: example is 0.40.26
OS Version / Instance: AWS EC2
Deployment: Kubernetes
Step where error happened: Deploy - the bootloader throws an error and stops

Current Behavior

I’m trying to update Airbyte from 0.40.14 to 0.40.26, and the booloader fails because it tries to upgrade my custom connectors, which don’t follow semantic versioning for tag names. I know this issue already happened before, and was solved with this PR.

Expected Behavior

The bootloader shouldn't check for most recent versions of custom connector images, as explained in this PR
I believe (but I could be wrong) that the PR that introduced this issue is #18749

Logs

Bootloader logs

2022-12-30 11:44:25,488 main INFO Loading mask data from '/seed/specs_secrets_mask.yaml
2022-12-30 11:44:25 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable CONFIG_DATABASE_USER: 'main'
2022-12-30 11:44:25 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable CONFIG_DATABASE_PASSWORD: '*****'
2022-12-30 11:44:25 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable CONFIG_DATABASE_URL: 'jdbc:postgresql://airbyte-test-postgres-db.cwjsqgvfpeim.us-east-1.rds.amazonaws.com:5433/airbyte_test'
2022-12-30 11:44:25 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting...
2022-12-30 11:44:26 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed.
2022-12-30 11:44:26 INFO c.z.h.HikariDataSource(<init>):80 - HikariPool-2 - Starting...
2022-12-30 11:44:26 INFO c.z.h.HikariDataSource(<init>):82 - HikariPool-2 - Start completed.
2022-12-30 11:44:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-12-30 11:44:26 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable SECRET_PERSISTENCE: 'TESTING_CONFIG_DB_TABLE'
2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(load):194 - Initializing databases...
2022-12-30 11:44:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable CONFIGS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-12-30 11:44:27 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2022-12-30 11:44:27 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte configs database connection...
2022-12-30 11:44:27 INFO i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
2022-12-30 11:44:27 INFO i.a.d.i.DatabaseInitializer(initializeSchema):90 - The airbyte configs database is initialized
2022-12-30 11:44:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable JOBS_DATABASE_INITIALIZATION_TIMEOUT_MS: '60000'
2022-12-30 11:44:27 WARN i.a.d.c.DatabaseAvailabilityCheck(check):38 - Waiting for database to become available...
2022-12-30 11:44:27 INFO i.a.d.c.DatabaseAvailabilityCheck(lambda$isDatabaseConnected$1):75 - Testing airbyte jobs database connection...
2022-12-30 11:44:27 INFO i.a.d.c.DatabaseAvailabilityCheck(check):57 - Database available.
2022-12-30 11:44:27 INFO i.a.d.i.DatabaseInitializer(initializeSchema):90 - The airbyte jobs database is initialized
2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(load):200 - Databases initialized.
2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(load):202 - Setting up config database and default workspace...
2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(assertNonBreakingMigration):340 - Checking illegal upgrade..
2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(isLegalUpgrade):384 - Current Airbyte version: AirbyteVersion{version='0.40.26', major='0', minor='40', patch='26'}
2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(isLegalUpgrade):385 - Future Airbyte version: AirbyteVersion{version='0.40.26', major='0', minor='40', patch='26'}
2022-12-30 11:44:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable AIRBYTE_PROTOCOL_VERSION_MIN: '0.0.0'
2022-12-30 11:44:27 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable AIRBYTE_PROTOCOL_VERSION_MAX: '0.3.0'
2022-12-30 11:44:27 INFO i.a.b.ProtocolVersionChecker(validate):77 - Using AirbyteProtocolVersion range [0.0.0:0.3.0]
2022-12-30 11:44:27 INFO i.a.b.BootloaderApp(trackProtocolVersion):374 - AirbyteProtocol version support range [0.0.0:0.3.0]
2022-12-30 11:44:27 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-30 11:44:27 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://airbyte-test-postgres-db.cwjsqgvfpeim.us-east-1.rds.amazonaws.com:5433/airbyte_test (PostgreSQL 12.11)
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema history table "public"."airbyte_configs_migrations" already initialized with (0.29.0.001,Baseline from file-based migration v1). Skipping.
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Database: jdbc:postgresql://airbyte-test-postgres-db.cwjsqgvfpeim.us-east-1.rds.amazonaws.com:5433/airbyte_test (PostgreSQL 12.11)
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema history table "public"."airbyte_jobs_migrations" already initialized with (0.29.0.001,Baseline from file-based migration v1). Skipping.
2022-12-30 11:44:28 INFO i.a.c.EnvConfigs(getEnvOrDefault):1161 - Using default value for environment variable RUN_DATABASE_MIGRATION_ON_STARTUP: 'true'
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(runFlywayMigration):396 - Migrating configs database
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Successfully validated 33 migrations (execution time 00:00.019s)
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Current version of schema "public": 0.40.23.002
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema "public" is up to date. No migration necessary.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(runFlywayMigration):398 - Migrating jobs database
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Flyway Community Edition 7.14.0 by Redgate
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Successfully validated 11 migrations (execution time 00:00.007s)
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Current version of schema "public": 0.40.18.002
2022-12-30 11:44:28 INFO o.f.c.i.l.s.Slf4jLog(info):49 - Schema "public" is up to date. No migration necessary.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):216 - Ran Flyway migrations.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(createWorkspaceIfNoneExists):317 - workspace already exists for the deployment.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):219 - Default workspace created.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(createDeploymentIfNoneExists):307 - running deployment: 5c1e816f-dada-4e28-bcf2-fd430ae877c1
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):222 - Default deployment created.
2022-12-30 11:44:28 INFO i.a.b.BootloaderApp(load):225 - Set version to AirbyteVersion{version='0.40.26', major='0', minor='40', patch='26'}
2022-12-30 11:44:28 INFO i.a.c.p.ActorDefinitionMigrator(updateConfigsFromSeed):72 - Updating connector definitions from the seed if necessary...
2022-12-30 11:44:28 INFO i.a.c.p.ActorDefinitionMigrator(updateConfigsFromSeed):75 - Connectors in use: [airbyte/destination-snowflake, 464622532012.dkr.ecr.us-east-1.amazonaws.com/dd-airbyte]
Exception in thread "main" java.lang.IllegalArgumentException: Invalid version string: source-s3-0.1.18
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
	at io.airbyte.commons.version.Version.<init>(Version.java:37)
	at io.airbyte.commons.version.AirbyteVersion.<init>(AirbyteVersion.java:15)
	at io.airbyte.config.persistence.ActorDefinitionMigrator.lambda$getConnectorRepositoryToInfoMap$3(ActorDefinitionMigrator.java:144)
	at java.base/java.util.HashMap.merge(HashMap.java:1391)
	at java.base/java.util.stream.Collectors.lambda$toMap$68(Collectors.java:1673)
	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
	at java.base/java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:1003)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1921)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at io.airbyte.config.persistence.ActorDefinitionMigrator.getConnectorRepositoryToInfoMap(ActorDefinitionMigrator.java:110)
	at io.airbyte.config.persistence.ActorDefinitionMigrator.updateConfigsFromSeed(ActorDefinitionMigrator.java:77)
	at io.airbyte.config.persistence.ActorDefinitionMigrator.lambda$migrate$0(ActorDefinitionMigrator.java:59)
	at io.airbyte.db.Database.lambda$transaction$0(Database.java:27)
	at org.jooq.impl.DefaultDSLContext.lambda$transactionResult0$0(DefaultDSLContext.java:564)
	at org.jooq.impl.Tools$12$1.block(Tools.java:4926)
	at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3744)
	at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3689)
	at org.jooq.impl.Tools$12.get(Tools.java:4923)
	at org.jooq.impl.DefaultDSLContext.transactionResult0(DefaultDSLContext.java:616)
	at org.jooq.impl.DefaultDSLContext.transactionResult(DefaultDSLContext.java:533)
	at io.airbyte.db.Database.transaction(Database.java:27)
	at io.airbyte.db.ExceptionWrappingDatabase.transaction(ExceptionWrappingDatabase.java:31)
	at io.airbyte.config.persistence.ActorDefinitionMigrator.migrate(ActorDefinitionMigrator.java:57)
	at io.airbyte.config.persistence.ConfigRepository.seedActorDefinitions(ConfigRepository.java:1190)
	at io.airbyte.config.init.ApplyDefinitionsHelper.apply(ApplyDefinitionsHelper.java:71)
	at io.airbyte.config.init.ApplyDefinitionsHelper.apply(ApplyDefinitionsHelper.java:47)
	at io.airbyte.bootloader.BootloaderApp.lambda$new$0(BootloaderApp.java:178)
	at io.airbyte.bootloader.BootloaderApp.load(BootloaderApp.java:227)
	at io.airbyte.bootloader.BootloaderApp.main(BootloaderApp.java:300)
2022-12-30 11:44:28 INFO c.z.h.HikariDataSource(close):350 - HikariPool-1 - Shutdown initiated...

Steps to Reproduce

Add a source definition with an image tag that don't follow semantic versioning (ie foo-1.2.3)
Re-deploy / re-start your airbyte instance, this should trigger the bootloader
The bootloader then fails with the error mentioned above
Note : the behavior is the same in Docker and in k8s deployment

Are you willing to submit a PR?

Yes, but I would definitely need some guidance on what should be changed.

The text was updated successfully, but these errors were encountered:

rach-r · 2023-04-04T17:34:44Z

I can confirm this is still happening in Airbyte latest version as of today (0.42.1). It indeed looks like a regression

marcosmarxm · 2023-04-06T13:02:59Z

@malikdiarra can you give your thoughts here?

SebastienN15 · 2023-04-07T07:08:26Z

Hey @rach-r @marcosmarxm, is there a planned ETA for this?

rach-r · 2023-04-07T07:55:48Z

@SebastienN15 I'm not from the Airbyte team so I can't give you an ETA, but I've encountered the same issue and here's a workaround until this gets fixed

SebastienN15 · 2023-04-07T08:01:16Z

Oh that's awesome, thank you! 🙇

marcosmarxm · 2023-04-07T13:38:07Z

@rach-r @SebastienN15 did you try to disable the connectors.versionOverridesEnabled flags.yaml
https://github.com/airbytehq/airbyte-platform/blob/4f5ac8c328bc02e3238ed18c82599c029611c095/flags.yml#L8-L9

rach-r · 2023-04-07T15:19:55Z

@marcosmarxm tried it on my side with this line changed to false

- name: connectors.versionOverridesEnabled
  serve: false

but still not working unfortunately (exact same error)

SebastienN15 added needs-triage type/bug Something isn't working labels Jan 4, 2023

octavia-squidington-iv added autoteam community team/tse Technical Support Engineers team/triage and removed team/triage labels Jan 4, 2023

sajarin added area/platform issues related to the platform team/prod-eng and removed needs-triage team/tse Technical Support Engineers autoteam labels Jan 5, 2023

c-p-b assigned octavia-squidington-iii Apr 13, 2023

malikdiarra self-assigned this Apr 14, 2023

c-p-b added team/compose and removed team/prod-eng labels Apr 25, 2023

octavia-squidington-iii added autoteam team/compose team/platform-move labels Feb 22, 2024

malikdiarra removed team/compose team/platform-move labels Feb 26, 2024

timroes added team/compose and removed team/compose labels Feb 29, 2024

octavia-squidington-iii added team/compose team/platform-move labels Feb 29, 2024

timroes removed the team/compose label Feb 29, 2024

bleonard added the frozen Not being actively worked on label Mar 22, 2024

marcosmarxm removed team/platform-move autoteam labels May 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bootloader crashes when using image tags that don't follow semantic versioning #21006

Bootloader crashes when using image tags that don't follow semantic versioning #21006

SebastienN15 commented Jan 4, 2023 •

edited

Loading

rach-r commented Apr 4, 2023

marcosmarxm commented Apr 6, 2023

SebastienN15 commented Apr 7, 2023 •

edited

Loading

rach-r commented Apr 7, 2023

SebastienN15 commented Apr 7, 2023

marcosmarxm commented Apr 7, 2023

rach-r commented Apr 7, 2023 •

edited

Loading

Bootloader crashes when using image tags that don't follow semantic versioning #21006

Bootloader crashes when using image tags that don't follow semantic versioning #21006

Comments

SebastienN15 commented Jan 4, 2023 • edited Loading

Environment

Current Behavior

Expected Behavior

Logs

Steps to Reproduce

Are you willing to submit a PR?

rach-r commented Apr 4, 2023

marcosmarxm commented Apr 6, 2023

SebastienN15 commented Apr 7, 2023 • edited Loading

rach-r commented Apr 7, 2023

SebastienN15 commented Apr 7, 2023

marcosmarxm commented Apr 7, 2023

rach-r commented Apr 7, 2023 • edited Loading

SebastienN15 commented Jan 4, 2023 •

edited

Loading

SebastienN15 commented Apr 7, 2023 •

edited

Loading

rach-r commented Apr 7, 2023 •

edited

Loading