-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[5889] Bmoric/log application name for all workers [2/2] #7268
Merged
Merged
Changes from 38 commits
Commits
Show all changes
41 commits
Select commit
Hold shift + click to select a range
99f4546
Add the log source to the MDP
benmoriceau ed4bcc6
Add helper to safely update the MDC
benmoriceau 85b81a2
Format
benmoriceau 3b1a65a
Add an application name
benmoriceau 0b27e02
Tmp
benmoriceau 2e794b1
Log the application name
benmoriceau 1103348
PR Comments
benmoriceau dd8a123
Merge branch 'bmoric/mdc-manipulation-tools' of github.com:airbytehq/…
benmoriceau 20964f0
PR Comments
benmoriceau f5ff0dc
Merge branch 'bmoric/add-application-name' of github.com:airbytehq/ai…
benmoriceau 80d5aab
Update related to merge
benmoriceau d536342
Format
benmoriceau bd4a7ff
Merge branch 'bmoric/mdc-manipulation-tools' of github.com:airbytehq/…
benmoriceau ee6b7e9
Format
benmoriceau c3c82bb
Merge branch 'bmoric/add-application-name' of github.com:airbytehq/ai…
benmoriceau a041a7d
Format
benmoriceau 988c9d8
Fix one test
benmoriceau bbdb3e9
Remove the need of an application interface
benmoriceau 7e64527
restore dbt runner
benmoriceau 2c1d9fc
Rollback changes non related to the issue.
benmoriceau 0c08536
Add missing file
benmoriceau 2f5e0dc
limit the labelling to logs coming from a container and not a worker
benmoriceau b5796b2
Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/log…
benmoriceau 734bc1a
Add the right colot to the prefix
benmoriceau c1c46d9
Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/log…
benmoriceau 3fa1260
Rm useless print of a stack trace
benmoriceau 0001cc2
Format
benmoriceau 1b313a1
Add a ScopeMdc to the line gobbler
benmoriceau b0cb21e
Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/log…
benmoriceau be40626
Fix the check connection activity
benmoriceau f1c7bbb
Merge branch 'bmoric/fix-temporal' into bmoric/log-application-name
benmoriceau 2861087
Merge branch 'master' of github.com:airbytehq/airbyte into bmoric/log…
benmoriceau 62fa485
Add tests
benmoriceau e1b1fd8
Test compilation
benmoriceau cfa5192
Change to logJobRoot name
benmoriceau 89e732d
Fix test setup
benmoriceau 430f3dd
Add tag to normalization and dbt
benmoriceau d5abecd
Update test
benmoriceau af40532
PR comments
benmoriceau 1f6f2c4
Update test
benmoriceau cc019c9
Format
benmoriceau File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -10,6 +10,9 @@ | |||||
import com.google.common.collect.ImmutableMap; | ||||||
import io.airbyte.commons.io.LineGobbler; | ||||||
import io.airbyte.commons.json.Jsons; | ||||||
import io.airbyte.commons.logging.LoggingHelper.Color; | ||||||
import io.airbyte.commons.logging.MdcScope; | ||||||
import io.airbyte.commons.logging.MdcScope.Builder; | ||||||
import io.airbyte.config.OperatorDbt; | ||||||
import io.airbyte.config.ResourceRequirements; | ||||||
import io.airbyte.protocol.models.ConfiguredAirbyteCatalog; | ||||||
|
@@ -27,6 +30,10 @@ | |||||
public class DefaultNormalizationRunner implements NormalizationRunner { | ||||||
|
||||||
private static final Logger LOGGER = LoggerFactory.getLogger(DefaultNormalizationRunner.class); | ||||||
private static final MdcScope CONTAINER_LOG_MDC = new Builder() | ||||||
.setLogPrefix("normalization-container-log") | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
.setPrefixColor(Color.GREEN) | ||||||
.build(); | ||||||
|
||||||
private final DestinationType destinationType; | ||||||
private final ProcessFactory processFactory; | ||||||
|
@@ -58,24 +65,26 @@ public boolean configureDbt(final String jobId, | |||||
final ResourceRequirements resourceRequirements, | ||||||
final OperatorDbt dbtConfig) | ||||||
throws Exception { | ||||||
final Map<String, String> files = ImmutableMap.of( | ||||||
WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, Jsons.serialize(config)); | ||||||
final String gitRepoUrl = dbtConfig.getGitRepoUrl(); | ||||||
if (Strings.isNullOrEmpty(gitRepoUrl)) { | ||||||
throw new WorkerException("Git Repo Url is required"); | ||||||
} | ||||||
final String gitRepoBranch = dbtConfig.getGitRepoBranch(); | ||||||
if (Strings.isNullOrEmpty(gitRepoBranch)) { | ||||||
return runProcess(jobId, attempt, jobRoot, files, resourceRequirements, "configure-dbt", | ||||||
"--integration-type", destinationType.toString().toLowerCase(), | ||||||
"--config", WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, | ||||||
"--git-repo", gitRepoUrl); | ||||||
} else { | ||||||
return runProcess(jobId, attempt, jobRoot, files, resourceRequirements, "configure-dbt", | ||||||
"--integration-type", destinationType.toString().toLowerCase(), | ||||||
"--config", WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, | ||||||
"--git-repo", gitRepoUrl, | ||||||
"--git-branch", gitRepoBranch); | ||||||
try (CONTAINER_LOG_MDC) { | ||||||
final Map<String, String> files = ImmutableMap.of( | ||||||
WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, Jsons.serialize(config)); | ||||||
final String gitRepoUrl = dbtConfig.getGitRepoUrl(); | ||||||
if (Strings.isNullOrEmpty(gitRepoUrl)) { | ||||||
throw new WorkerException("Git Repo Url is required"); | ||||||
} | ||||||
final String gitRepoBranch = dbtConfig.getGitRepoBranch(); | ||||||
if (Strings.isNullOrEmpty(gitRepoBranch)) { | ||||||
return runProcess(jobId, attempt, jobRoot, files, resourceRequirements, "configure-dbt", | ||||||
"--integration-type", destinationType.toString().toLowerCase(), | ||||||
"--config", WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, | ||||||
"--git-repo", gitRepoUrl); | ||||||
} else { | ||||||
return runProcess(jobId, attempt, jobRoot, files, resourceRequirements, "configure-dbt", | ||||||
"--integration-type", destinationType.toString().toLowerCase(), | ||||||
"--config", WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, | ||||||
"--git-repo", gitRepoUrl, | ||||||
"--git-branch", gitRepoBranch); | ||||||
} | ||||||
} | ||||||
} | ||||||
|
||||||
|
@@ -87,14 +96,16 @@ public boolean normalize(final String jobId, | |||||
final ConfiguredAirbyteCatalog catalog, | ||||||
final ResourceRequirements resourceRequirements) | ||||||
throws Exception { | ||||||
final Map<String, String> files = ImmutableMap.of( | ||||||
WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, Jsons.serialize(config), | ||||||
WorkerConstants.DESTINATION_CATALOG_JSON_FILENAME, Jsons.serialize(catalog)); | ||||||
|
||||||
return runProcess(jobId, attempt, jobRoot, files, resourceRequirements, "run", | ||||||
"--integration-type", destinationType.toString().toLowerCase(), | ||||||
"--config", WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, | ||||||
"--catalog", WorkerConstants.DESTINATION_CATALOG_JSON_FILENAME); | ||||||
try (CONTAINER_LOG_MDC) { | ||||||
final Map<String, String> files = ImmutableMap.of( | ||||||
WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, Jsons.serialize(config), | ||||||
WorkerConstants.DESTINATION_CATALOG_JSON_FILENAME, Jsons.serialize(catalog)); | ||||||
|
||||||
return runProcess(jobId, attempt, jobRoot, files, resourceRequirements, "run", | ||||||
"--integration-type", destinationType.toString().toLowerCase(), | ||||||
"--config", WorkerConstants.DESTINATION_CONFIG_JSON_FILENAME, | ||||||
"--catalog", WorkerConstants.DESTINATION_CATALOG_JSON_FILENAME); | ||||||
} | ||||||
} | ||||||
|
||||||
private boolean runProcess(final String jobId, | ||||||
|
@@ -104,7 +115,7 @@ private boolean runProcess(final String jobId, | |||||
final ResourceRequirements resourceRequirements, | ||||||
final String... args) | ||||||
throws Exception { | ||||||
try { | ||||||
try (CONTAINER_LOG_MDC) { | ||||||
LOGGER.info("Running with normalization version: {}", normalizationImageName); | ||||||
process = processFactory.create(jobId, attempt, jobRoot, normalizationImageName, false, files, null, resourceRequirements, | ||||||
Map.of(KubeProcessFactory.JOB_TYPE, KubeProcessFactory.SYNC_JOB, KubeProcessFactory.SYNC_STEP, KubeProcessFactory.NORMALISE_STEP), args); | ||||||
|
@@ -126,14 +137,16 @@ private boolean runProcess(final String jobId, | |||||
|
||||||
@Override | ||||||
public void close() throws Exception { | ||||||
if (process == null) { | ||||||
return; | ||||||
} | ||||||
try (CONTAINER_LOG_MDC) { | ||||||
if (process == null) { | ||||||
return; | ||||||
} | ||||||
|
||||||
LOGGER.debug("Closing normalization process"); | ||||||
WorkerUtils.gentleClose(process, 1, TimeUnit.MINUTES); | ||||||
if (process.isAlive() || process.exitValue() != 0) { | ||||||
throw new WorkerException("Normalization process wasn't successful"); | ||||||
LOGGER.debug("Closing normalization process"); | ||||||
WorkerUtils.gentleClose(process, 1, TimeUnit.MINUTES); | ||||||
if (process.isAlive() || process.exitValue() != 0) { | ||||||
throw new WorkerException("Normalization process wasn't successful"); | ||||||
} | ||||||
} | ||||||
} | ||||||
|
||||||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.