Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update ConfigRepository to read protocol version #16670

Merged
merged 10 commits into from
Sep 14, 2022
Merged

Update ConfigRepository to read protocol version #16670

merged 10 commits into from
Sep 14, 2022

Conversation

gosusnp
Copy link
Contributor

@gosusnp gosusnp commented Sep 13, 2022

What

When reading actor definitions, we need to copy the protocol version from the connector spec into the top level protocol version value of the standard definitions for it to be stored in the database.

@gosusnp gosusnp temporarily deployed to more-secrets September 13, 2022 21:51 Inactive
@gosusnp gosusnp temporarily deployed to more-secrets September 13, 2022 21:58 Inactive
@gosusnp gosusnp temporarily deployed to more-secrets September 13, 2022 22:30 Inactive
@gosusnp gosusnp temporarily deployed to more-secrets September 13, 2022 23:08 Inactive
benmoriceau
benmoriceau reviewed Sep 13, 2022
View reviewed changes
...-config/config-persistence/src/main/java/io/airbyte/config/persistence/ConfigRepository.java
@@ -206,6 +207,8 @@ public List<StandardSourceDefinition> listStandardSourceDefinitions(final boolea
final List<StandardSourceDefinition> sourceDefinitions = new ArrayList<>();
for (final StandardSourceDefinition sourceDefinition : persistence.listConfigs(ConfigSchema.STANDARD_SOURCE_DEFINITION,
StandardSourceDefinition.class)) {
sourceDefinition.withProtocolVersion(AirbyteProtocolVersion
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be change to a common method or something like:

return persistence.listConfigs(ConfigSchema.STANDARD_SOURCE_DEFINITION,
StandardSourceDefinition.class)
.map(extract the protocol version)
.filter(filter the tombstone)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, we should use set since we don't use the returned value of withProtocolVersion

benmoriceau
benmoriceau approved these changes Sep 14, 2022
View reviewed changes
Copy link
Contributor

@benmoriceau benmoriceau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline

jdpgrailsdev
jdpgrailsdev approved these changes Sep 14, 2022
View reviewed changes
Copy link
Contributor

@jdpgrailsdev jdpgrailsdev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@gosusnp gosusnp temporarily deployed to more-secrets September 14, 2022 15:59 Inactive
@gosusnp gosusnp temporarily deployed to more-secrets September 14, 2022 17:14 Inactive
@gosusnp gosusnp temporarily deployed to more-secrets September 14, 2022 17:22 Inactive
@gosusnp gosusnp temporarily deployed to more-secrets September 14, 2022 18:05 Inactive
@gosusnp gosusnp temporarily deployed to more-secrets September 14, 2022 20:06 Inactive
@benmoriceau benmoriceau merged commit 2630031 into master Sep 14, 2022
@benmoriceau benmoriceau deleted the gosusnp/15980-store_versions_from_applyseedtodefinitions branch September 14, 2022 21:25
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Sep 14, 2022
Closed
letiescanciano added a commit that referenced this pull request Sep 15, 2022
@letiescanciano
Merge branch 'master' into leti/add-segment-cross-domain-tracking
f701c59 
* master: (200 commits)
  🪟 🧹 Display returned error messages on replication view (#16280)
  🎉 Source mixpanel: Use "Retry-After" header for backoff (#16770)
  🐛 Source google ads: mark custom query fields required (#15858)
  🪟 🔧Remove useRouter hook (#16598)
  CDK: improve TypeTransformer to convert simple types to array of simple types (#16636)
  CDK: TypeTransformer - warning message more informative (#16695)
  Source MySQL: Add Python SAT to detect backwards breaking changes (#16445)
  remove eager (#16756)
  bump com.networknt:json-schema-validator to latest version (#16619)
  Remove Cloud from Kafka docs (#16753)
  Normalization Summaries table and read/write methods (#16655)
  comment out flaky test suite while it is being investigated (#16752)
  Update ConfigRepository to read protocol version (#16670)
  Use LOG4J2 to wrap connectors logs to JSON format (#15668)
  Update connector catalog (#16749)
  🪟 🎨 Remove feedback modal from UI (#16548)
  Add missing env var for Kube overlays (#16747)
  Prepare for React v18 upgrade (#16694)
  🪟 🐛 Fix direct job linking to work with pagination (#16517)
  Fix formatting (#16743)
  ...
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Sep 15, 2022
Merged
@gosusnp gosusnp mentioned this pull request Sep 16, 2022
Merged
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
@gosusnp @robbinhan
Update ConfigRepository to read protocol version (airbytehq#16670)
6e7e9ed 
When reading actor definitions, we need to copy the protocol version from the connector spec into the top level protocol version value of the standard definitions for it to be stored in the database.