Glue Schema Registry schema replication converter #352
Conversation
…ation-mm2-converter New cross region replication mm2 converter
…into gsr-schema-replication-converter
…rsions of the existing methods to ensure source schema compatibility with target schema
…2 versions of the existing methods to ensure source schema compatibility with target schema
… versions of the existing methods to ensure source schema compatibility with target schema
…nSchemaReplicationConverter
…ixed issue with how default configs are set with TARGET specific configs in AWSGlueCrossRegionSchemaReplicationConverter and GlueSchemaRegistryConfiguration
| import software.amazon.awssdk.services.glue.model.RegisterSchemaVersionResponse; | ||
| import software.amazon.awssdk.services.glue.model.RegistryId; | ||
| import software.amazon.awssdk.services.glue.model.SchemaId; | ||
| import software.amazon.awssdk.services.glue.model.*; |
There was a problem hiding this comment.
We avoid using * imports in this code base. Can you please change this back?
| glueSchemaRegistryConfiguration.getSourceRegion() != null && | ||
| glueSchemaRegistryConfiguration.getSourceEndPoint() != null) { | ||
|
|
||
| glueSourceClientBuilder = GlueClient |
There was a problem hiding this comment.
Instead of creating multiple clients here, can we pass in a region and create multiple instances of this client in different regions?
| * @return Map of SchemaV2 with VersionIds | ||
| * @throws AWSSchemaRegistryException on any error during the schema creation | ||
| */ | ||
| public Map<SchemaV2, UUID> createSchemaV2(String schemaName, |
|
|
||
| import java.util.Map; | ||
| import java.util.UUID; | ||
| import java.util.*; |
There was a problem hiding this comment.
Same as above on imports.
| */ | ||
| @AllArgsConstructor | ||
| @Value | ||
| public class SchemaV2 { |
There was a problem hiding this comment.
Can we re-use the existing Schema class?
| /** | ||
| * Target Registry Name. | ||
| */ | ||
| public static final String TARGET_REGISTRY_NAME = "target.registry.name"; |
There was a problem hiding this comment.
Same as above. Converter specific code must be in converter.
|
|
||
| private void validateRequiredConfigsIfPresent(Map<String, ?> configs) { |
There was a problem hiding this comment.
Is it possible to re-use any existing configs here?
| import java.util.stream.Collectors; | ||
|
|
||
| @Slf4j | ||
| public class KafkaHelper { |
There was a problem hiding this comment.
Isn't this already present?
out.log
Outdated
| @@ -0,0 +1 @@ | |||
| nohup: ./run-local-tests.sh: No such file or directory | |||
There was a problem hiding this comment.
Can we remove this file?
| @@ -60,6 +63,9 @@ public class GlueSchemaRegistryDeserializationFacade implements Closeable { | |||
| @VisibleForTesting | |||
| protected LoadingCache<UUID, Schema> cache; | |||
|
|
|||
| @VisibleForTesting | |||
| protected LoadingCache<UUID, SchemaV2> cacheV2; | |||
There was a problem hiding this comment.
Can this be removed if we can figure out how to re-use Schema?
…ions to schema replication converter.
…tial MM2 schema sync
…schema replciation converter module
…schema replciation converter module
Issue #, if available:
Replicate GSR schema and schema versions from one region to another
Description of changes:
Written a new Kafka Converter to replicate schema and schema versions cross-region
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.