Metadata config and pipeline options #2081

akashthawaitcc · 2024-12-23T11:22:03Z

This PR introduces a comprehensive refactor and enhancement of the migration workflow to streamline schema handling and extend support for Cassandra sources. Key changes include:

Schema Population for Cassandra Sources:
Refactored logic to efficiently populate schemas for Cassandra sources, ensuring better maintainability and readability.
Schema Population for Spanner Using DDL:
Added robust handling for schema population from DDL statements to support Spanner targets.
Pipeline Option Enhancements for Cassandra:
Updated pipeline options to seamlessly incorporate Cassandra-related configurations.
Enhanced SourceWriterFn for Cassandra Functionality:
Extended the SourceWriterFn implementation to support Cassandra-specific operations, enabling improved functionality.
Implementation of Cassandra Connection Helper:
Introduced a CassandraConnectionHelper to abstract and simplify connection handling for Cassandra, ensuring consistency and reducing redundancy.

Sync master

Sync main branch

tests: Adding Forward Migration Tests (GoogleCloudPlatform#2001)

Sync main branch

Use [self-hosted, it] for prepare java cache workflow (GoogleCloudPlatform#2080)

bharadwaj-aditya

I've added a few comments. The major concerns here are

There seems to be overlap with - Cassandra Driver Config Loader for Cassandra Driver Config in GCS #2077. Please rebase from there once it is merged (ETA - Today EOD)
There are no tests added here.

...nner-to-sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/SpannerToSourceDb.java

...n/java/com/google/cloud/teleport/v2/spanner/migrations/metadata/CassandraSourceMetadata.java

...main/java/com/google/cloud/teleport/v2/spanner/migrations/schema/cassandra/SourceColumn.java

.../main/java/com/google/cloud/teleport/v2/spanner/migrations/schema/cassandra/SourceTable.java

...in/java/com/google/cloud/teleport/v2/spanner/migrations/utils/CassandraConfigFileReader.java

...nner-to-sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/SpannerToSourceDb.java

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java

Sync main branch

...nner-to-sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/SpannerToSourceDb.java

.../java/com/google/cloud/teleport/v2/spanner/migrations/utils/CassandraDriverConfigLoader.java

v2/common/src/main/resources/udf-samples/README.md

...n/java/com/google/cloud/teleport/v2/spanner/migrations/metadata/CassandraSourceMetadata.java

...mmon/src/main/java/com/google/cloud/teleport/v2/spanner/migrations/shard/CassandraShard.java

Sync main branch

Co-authored-by: Narendra Rajput <narendra.rajput@ollion.com>

* Removed * * Create README.md for UDF samples (GoogleCloudPlatform#2083) This commit adds a README.md file to the directory. The README file provides descriptions for each of the sample Javascript UDF files in the directory, including their purpose and usage examples. Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com> * CassandraDriverConfigLoader from GCS (GoogleCloudPlatform#2077) * Added Config File Path * Added Fix for Loading Driver Options * Added Dependecy Fixes * Fix UT --------- Co-authored-by: liferoad <huxiangqian@gmail.com> Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com> Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>

* Convert it to builder Pattern * Convert Waring to Error * remove the unwanted comments

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java

...mmon/src/main/java/com/google/cloud/teleport/v2/spanner/migrations/shard/CassandraShard.java

...n/java/com/google/cloud/teleport/v2/spanner/migrations/metadata/CassandraSourceMetadata.java

...mmon/src/main/java/com/google/cloud/teleport/v2/spanner/migrations/shard/CassandraShard.java

...in/java/com/google/cloud/teleport/v2/spanner/migrations/utils/CassandraConfigFileReader.java

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java

...sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/transforms/SourceWriterFn.java

shreyakhajanchi

Please add unit tests for all the java classes added/modified

...mmon/src/main/java/com/google/cloud/teleport/v2/spanner/migrations/shard/CassandraShard.java

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java

codecov · 2024-12-31T06:42:33Z

Codecov Report

Attention: Patch coverage is 11.35371% with 203 lines in your changes missing coverage. Please review.

Project coverage is 46.19%. Comparing base (587fc05) to head (245cd6e).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...r/migrations/metadata/CassandraSourceMetadata.java	0.00%	51 Missing ⚠️
.../dbutils/connection/CassandraConnectionHelper.java	7.31%	38 Missing ⚠️
...rt/v2/spanner/migrations/shard/CassandraShard.java	0.00%	35 Missing ⚠️
...t/v2/spanner/migrations/spanner/SpannerSchema.java	0.00%	30 Missing ⚠️
...cloud/teleport/v2/templates/SpannerToSourceDb.java	0.00%	19 Missing ⚠️
...er/migrations/utils/CassandraConfigFileReader.java	0.00%	10 Missing ⚠️
.../v2/templates/dbutils/dao/source/CassandraDao.java	0.00%	9 Missing ⚠️
...leport/v2/templates/transforms/SourceWriterFn.java	0.00%	7 Missing and 1 partial ⚠️
.../teleport/v2/spanner/migrations/schema/Schema.java	0.00%	2 Missing ⚠️
...ates/dbutils/processor/SourceProcessorFactory.java	95.83%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2081      +/-   ##
============================================
- Coverage     46.39%   46.19%   -0.20%     
- Complexity     3930     4260     +330     
============================================
  Files           859      866       +7     
  Lines         51300    51553     +253     
  Branches       5394     5410      +16     
============================================
+ Hits          23800    23817      +17     
- Misses        25786    26021     +235     
- Partials       1714     1715       +1

Components	Coverage Δ
spanner-templates	`67.48% <11.35%> (-0.77%)`	⬇️
spanner-import-export	`65.57% <ø> (ø)`
spanner-live-forward-migration	`74.20% <0.00%> (-2.47%)`	⬇️
spanner-live-reverse-replication	`74.75% <11.35%> (-3.19%)`	⬇️
spanner-bulk-migration	`85.55% <0.00%> (-1.76%)`	⬇️

Files with missing lines	Coverage Δ
...ort/v2/spanner/migrations/constants/Constants.java	`0.00% <ø> (ø)`
...plates/dbutils/processor/InputRecordProcessor.java	`92.45% <ø> (ø)`
...ates/dbutils/processor/SourceProcessorFactory.java	`90.47% <95.83%> (+0.89%)`	⬆️
.../teleport/v2/spanner/migrations/schema/Schema.java	`88.59% <0.00%> (-1.59%)`	⬇️
...leport/v2/templates/transforms/SourceWriterFn.java	`79.67% <0.00%> (-5.55%)`	⬇️
.../v2/templates/dbutils/dao/source/CassandraDao.java	`67.85% <0.00%> (-32.15%)`	⬇️
...er/migrations/utils/CassandraConfigFileReader.java	`0.00% <0.00%> (ø)`
...cloud/teleport/v2/templates/SpannerToSourceDb.java	`0.00% <0.00%> (ø)`
...t/v2/spanner/migrations/spanner/SpannerSchema.java	`0.00% <0.00%> (ø)`
...rt/v2/spanner/migrations/shard/CassandraShard.java	`0.00% <0.00%> (ø)`
... and 2 more

... and 3 files with indirect coverage changes

bharadwaj-aditya

Please add unit tests for all the classes added.

...n/java/com/google/cloud/teleport/v2/spanner/migrations/metadata/CassandraSourceMetadata.java

v2/spanner-to-sourcedb/pom.xml

...nner-to-sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/SpannerToSourceDb.java

...n/java/com/google/cloud/teleport/v2/spanner/migrations/metadata/CassandraSourceMetadata.java

bharadwaj-aditya · 2024-12-31T06:47:38Z

The test coverage on this PR has dropped 3% across the project. Can you please add unit tests here ?

* Added Dummy Test case * Removed * from import * removed unwanted * Added Fixes * Added test case and fixes * Added Some more PR comments * Added Test case for Cassandra Reader * Added New testcase * Added Dependecny

taherkl and others added 7 commits November 27, 2024 20:19

Merge remote-tracking branch 'upstream/main'

15b621d

Merge pull request #9 from GoogleCloudPlatform/main

d314c70

Sync master

Merge pull request #20 from GoogleCloudPlatform/main

71a6477

Sync main branch

Merge pull request #24 from GoogleCloudPlatform/main

7512825

Sync main branch

Merge pull request #26 from GoogleCloudPlatform/main

5aa21dc

tests: Adding Forward Migration Tests (GoogleCloudPlatform#2001)

Merge pull request #28 from GoogleCloudPlatform/main

04bac39

Sync main branch

Merge pull request #29 from GoogleCloudPlatform/main

f618128

Use [self-hosted, it] for prepare java cache workflow (GoogleCloudPlatform#2080)

pull-request-size bot added the size/XL label Dec 23, 2024

akashthawaitcc marked this pull request as ready for review December 23, 2024 11:23

akashthawaitcc requested a review from a team as a code owner December 23, 2024 11:23

akashthawaitcc requested review from manitgupta and shreyakhajanchi December 23, 2024 11:23

pull-request-size bot added size/XXL and removed size/XL labels Dec 23, 2024

bharadwaj-aditya reviewed Dec 24, 2024

View reviewed changes

VardhanThigle reviewed Dec 24, 2024

View reviewed changes

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java Show resolved Hide resolved

VardhanThigle reviewed Dec 24, 2024

View reviewed changes

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java Show resolved Hide resolved

VardhanThigle reviewed Dec 24, 2024

View reviewed changes

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java Outdated Show resolved Hide resolved

VardhanThigle reviewed Dec 24, 2024

View reviewed changes

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java Outdated Show resolved Hide resolved

VardhanThigle reviewed Dec 24, 2024

View reviewed changes

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java Outdated Show resolved Hide resolved

Merge pull request #32 from GoogleCloudPlatform/main

e53e8e4

Sync main branch

shreyakhajanchi reviewed Dec 29, 2024

View reviewed changes

taherkl and others added 8 commits December 30, 2024 10:54

Merge pull request #36 from GoogleCloudPlatform/main

407a593

Sync main branch

Added Connection Helper

b83eacd

Added Formatter

aa78dae

Added File Reader

17558a8

Refectored

6da159f

Added Changes for Schema and Spanner Schema

87b1146

Added Schema Changes to read Spanner Table in Schema object

10fc0c4

Added Schema Changes

55b89d7

pawankashyapollion and others added 5 commits December 30, 2024 11:59

Fixed UT for Metadata config (#30)

fe3f17c

Co-authored-by: Narendra Rajput <narendra.rajput@ollion.com>

PR Review Comments (#35)

f7c9a1f

* Convert it to builder Pattern * Convert Waring to Error * remove the unwanted comments

Removed Unwanted options

11db7ff

Address the PR

928fba9

pawankashyapollion force-pushed the metadata_config_and_pipeline_options branch from 7a2d80a to 928fba9 Compare December 30, 2024 06:30

pawankashyapollion added 2 commits December 30, 2024 13:59

Address the PR Comments

71dc086

Added Missing Getter for Configuration

b1eb73e

VardhanThigle reviewed Dec 30, 2024

View reviewed changes

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java Show resolved Hide resolved

VardhanThigle reviewed Dec 30, 2024

View reviewed changes

...mmon/src/main/java/com/google/cloud/teleport/v2/spanner/migrations/shard/CassandraShard.java Outdated Show resolved Hide resolved

shreyakhajanchi reviewed Dec 30, 2024

View reviewed changes

...mmon/src/main/java/com/google/cloud/teleport/v2/spanner/migrations/shard/CassandraShard.java Outdated Show resolved Hide resolved

VardhanThigle reviewed Dec 30, 2024

View reviewed changes

...mmon/src/main/java/com/google/cloud/teleport/v2/spanner/migrations/shard/CassandraShard.java Outdated Show resolved Hide resolved

pawankashyapollion added 3 commits December 30, 2024 18:21

Address to removed configuration changes

709f2cf

removed unwanted getters

124d571

Removal of getter from cassandra dao and test case fixes

b91a825

shreyakhajanchi reviewed Dec 30, 2024

View reviewed changes

pawankashyapollion added 2 commits December 30, 2024 20:16

Handle Optionmap in cassandra DAO for serialization

4934edc

removed dependecy of getKeyOrder

b79f21e

shreyakhajanchi reviewed Dec 31, 2024

View reviewed changes

VardhanThigle reviewed Dec 31, 2024

View reviewed changes

...mmon/src/main/java/com/google/cloud/teleport/v2/spanner/migrations/shard/CassandraShard.java Outdated Show resolved Hide resolved

VardhanThigle reviewed Dec 31, 2024

View reviewed changes

...ava/com/google/cloud/teleport/v2/templates/dbutils/connection/CassandraConnectionHelper.java Outdated Show resolved Hide resolved

pawankashyapollion added 2 commits December 31, 2024 11:25

Removed And Update Exception

cb7b016

Fix Checkstyle Voilation

fe6f6fc

bharadwaj-aditya requested changes Dec 31, 2024

View reviewed changes

Missing UT Added (#37)

245cd6e

* Added Dummy Test case * Removed * from import * removed unwanted * Added Fixes * Added test case and fixes * Added Some more PR comments * Added Test case for Cassandra Reader * Added New testcase * Added Dependecny

akashthawaitcc requested review from bharadwaj-aditya, VardhanThigle and shreyakhajanchi December 31, 2024 09:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metadata config and pipeline options #2081

Metadata config and pipeline options #2081

akashthawaitcc commented Dec 23, 2024

bharadwaj-aditya left a comment

shreyakhajanchi left a comment

codecov bot commented Dec 31, 2024 •

edited

Loading

bharadwaj-aditya left a comment

bharadwaj-aditya commented Dec 31, 2024

Metadata config and pipeline options #2081

Are you sure you want to change the base?

Metadata config and pipeline options #2081

Conversation

akashthawaitcc commented Dec 23, 2024

bharadwaj-aditya left a comment

Choose a reason for hiding this comment

shreyakhajanchi left a comment

Choose a reason for hiding this comment

codecov bot commented Dec 31, 2024 • edited Loading

Codecov Report

bharadwaj-aditya left a comment

Choose a reason for hiding this comment

bharadwaj-aditya commented Dec 31, 2024

codecov bot commented Dec 31, 2024 •

edited

Loading