Kube acceptance test is flaky due to missing config volume

[tuliren]

Enviroment

• Airbyte version: commit #e577b4987e8b9d1306eed833ac04a17f7a1a890c
• OS Version / Instance: CI
• Deployment: CI
• Severity: Very Low / Low / Medium / High / Critical
• Step where error happened: Kube acceptance test

Current Behavior

This acceptance test is flaky. Here is the log:

2021-07-19 16:46:39 INFO i.a.s.ServerApp(runServer):212 - ***workspace_app_root=/workspace/server/logs*** - Creating config repository...
2021-07-19 16:46:39 INFO i.a.c.p.ConfigPersistenceBuilder(getDbPersistenceWithFileSeed):103 - ***workspace_app_root=/workspace/server/logs*** - Creating db-based config persistence, and loading seed and existing data from files
2021-07-19 16:46:39 INFO i.a.c.p.ConfigPersistenceBuilder(getDbPersistence):114 - ***workspace_app_root=/workspace/server/logs*** - Use database config persistence.
2021-07-19 16:46:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):288 - ***workspace_app_root=/workspace/server/logs*** - CONFIG_DATABASE_USER not found or empty, defaulting to docker
2021-07-19 16:46:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):288 - ***workspace_app_root=/workspace/server/logs*** - CONFIG_DATABASE_PASSWORD not found or empty, defaulting to docker
2021-07-19 16:46:39 INFO i.a.c.EnvConfigs(getEnvOrDefault):288 - ***workspace_app_root=/workspace/server/logs*** - CONFIG_DATABASE_URL not found or empty, defaulting to jdbc:postgresql://airbyte-db-svc:5432/airbyte
2021-07-19 16:46:39 WARN i.a.d.Databases(createPostgresDatabaseWithRetry):84 - ***workspace_app_root=/workspace/server/logs*** - Waiting for database to become available...
2021-07-19 16:46:39 INFO i.a.d.Databases(lambda$static$2):57 - ***workspace_app_root=/workspace/server/logs*** - Testing config database connection...
2021-07-19 16:46:40 INFO i.a.d.Databases(createPostgresDatabaseWithRetry):101 - ***workspace_app_root=/workspace/server/logs*** - Database available!
2021-07-19 16:46:40 INFO i.a.c.p.DatabaseConfigPersistence(lambda$initialize$0):81 - ***workspace_app_root=/workspace/server/logs*** - Config database has not been initialized
2021-07-19 16:46:40 INFO i.a.c.p.DatabaseConfigPersistence(lambda$initialize$0):82 - ***workspace_app_root=/workspace/server/logs*** - Creating tables with schema: -- tables
 CREATE
    TABLE
        IF NOT EXISTS AIRBYTE_CONFIGS(
            id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
            config_id VARCHAR(36) NOT NULL,
            config_type VARCHAR(60) NOT NULL,
            config_blob JSONB NOT NULL,
            created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
            updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
        );

-- indices
 CREATE
    UNIQUE INDEX IF NOT EXISTS airbyte_configs_type_id_idx ON
    AIRBYTE_CONFIGS(
        config_type,
        config_id
    );

CREATE
    INDEX IF NOT EXISTS airbyte_configs_id_idx ON
    AIRBYTE_CONFIGS(config_id);

2021-07-19 16:46:40 INFO i.a.c.p.DatabaseConfigPersistence(lambda$loadData$3):101 - ***workspace_app_root=/workspace/server/logs*** - Loading data to config database...
2021-07-19 16:46:40 INFO i.a.c.p.DatabaseConfigPersistence(lambda$loadData$3):118 - ***workspace_app_root=/workspace/server/logs*** - Config database data loading completed with 0 records
2021-07-19 16:46:40 ERROR i.a.s.ServerApp(setCustomerIdIfNotSet):197 - ***workspace_app_root=/workspace/server/logs*** - Could not find workspace with id: 5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6
io.airbyte.config.persistence.ConfigNotFoundException: config type: STANDARD_WORKSPACE id: 5ae6b09b-fdec-41af-aaf7-7d94cfc33ef6
	at io.airbyte.config.persistence.DatabaseConfigPersistence.getConfig(DatabaseConfigPersistence.java:133) ~[io.airbyte.airbyte-config-persistence-0.27.3-alpha.jar:?]
	at io.airbyte.config.persistence.ValidatingConfigPersistence.getConfig(ValidatingConfigPersistence.java:55) ~[io.airbyte.airbyte-config-persistence-0.27.3-alpha.jar:?]
	at io.airbyte.config.persistence.ConfigRepository.getStandardWorkspace(ConfigRepository.java:55) ~[io.airbyte.airbyte-config-persistence-0.27.3-alpha.jar:?]
	at io.airbyte.server.ServerApp.setCustomerIdIfNotSet(ServerApp.java:185) [io.airbyte-airbyte-server-0.27.3-alpha.jar:?]
	at io.airbyte.server.ServerApp.runServer(ServerApp.java:218) [io.airbyte-airbyte-server-0.27.3-alpha.jar:?]
	at io.airbyte.server.ServerApp.main(ServerApp.java:264) [io.airbyte-airbyte-server-0.27.3-alpha.jar:?]

The reason that the standard workspace could not be found is that it was not written to the db config persistence, highlighted by these two lines:

2021-07-19 16:46:40 INFO i.a.c.p.DatabaseConfigPersistence(lambda$loadData$3):101 - ***workspace_app_root=/workspace/server/logs*** - Loading data to config database...
2021-07-19 16:46:40 INFO i.a.c.p.DatabaseConfigPersistence(lambda$loadData$3):118 - ***workspace_app_root=/workspace/server/logs*** - Config database data loading completed with 0 records

The db config persistence read in initial data from the config volume. This means somehow in the acceptance test, the config volume is empty or not ready.

Are you willing to submit a PR?

Yes.

[tuliren]

Resolved by #4835.