CassandraToBigtable classic template feature: Cassandra Writetime replication

[georgecma]

Add Cassandra writetime replication feature to CassandraToBigtable classic template.

Cassandra writetime can now be replicated to Bigtable as Bigtable cell timestamp. For this feature to work, users have to upload a json-like schema file to a GCS bucket location first for the table they wish to replicate writetimes for. The command to generate this schema file is:

cqlsh -e "select json * from system_schema.columns
                where keyspace_name='$CASSANDRA_KEYSPACE'
                and table_name='$CASSANDRA_TABLE'"
    > column_schema.json

Then, to upload the file:

gcloud storage cp column_schema $WRITETIME_CASSANDRA_COLUMN_SCHEMA

The schema GCS file path should then be set as $WRITETIME_CASSANDRA_COLUMN_SCHEMA to be parsed by the template at run time.

The template replication behavior now sets Bigtable cell time as replication time (i.e. now) instead of epoch start time previously. SetZeroTimestamp is the backwards compatibility option to set 0 as the timestamp.

@GoogleCloudPlatform/dataflow-templates-wg

[georgecma]

@GoogleCloudPlatform/dataflow-templates-wg

[codecov]

Codecov Report

Attention: Patch coverage is 56.77419% with 67 lines in your changes missing coverage. Please review.

Project coverage is 44.00%. Comparing base (c4df310) to head (8a38b50).
Report is 108 commits behind head on main.

Files with missing lines	Patch %	Lines
...port/bigtable/CassandraWritetimeQueryProvider.java	0.00%	22 Missing ⚠️
...e/cloud/teleport/bigtable/BeamRowToBigtableFn.java	81.69%	7 Missing and 6 partials ⚠️
...gle/cloud/teleport/util/GCSAwareValueProvider.java	0.00%	10 Missing ⚠️
...cloud/teleport/bigtable/CassandraColumnSchema.java	76.31%	4 Missing and 5 partials ⚠️
...e/cloud/teleport/bigtable/CassandraToBigtable.java	0.00%	8 Missing ⚠️
.../java/com/google/cloud/teleport/util/GCSUtils.java	0.00%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1653      +/-   ##
============================================
+ Coverage     42.98%   44.00%   +1.01%     
- Complexity     3405     3479      +74     
============================================
  Files           818      829      +11     
  Lines         47779    49186    +1407     
  Branches       5127     5168      +41     
============================================
+ Hits          20537    21642    +1105     
- Misses        25581    25883     +302     
  Partials       1661     1661              

Components	Coverage Δ
spanner-templates	63.56% <ø> (-0.32%)	⬇️
spanner-import-export	63.76% <ø> (-0.55%)	⬇️
spanner-live-forward-migration	75.24% <ø> (+0.07%)	⬆️
spanner-live-reverse-replication	68.40% <ø> (-3.23%)	⬇️
spanner-bulk-migration	84.29% <ø> (+0.44%)	⬆️

Files with missing lines	Coverage Δ
...gle/cloud/teleport/bigtable/CassandraKeyUtils.java	95.65% <ø> (ø)
.../cloud/teleport/bigtable/CassandraRowMapperFn.java	89.88% <100.00%> (+0.11%)	⬆️
.../java/com/google/cloud/teleport/util/GCSUtils.java	40.00% <0.00%> (-13.34%)	⬇️
...e/cloud/teleport/bigtable/CassandraToBigtable.java	0.00% <0.00%> (ø)
...cloud/teleport/bigtable/CassandraColumnSchema.java	76.31% <76.31%> (ø)
...gle/cloud/teleport/util/GCSAwareValueProvider.java	0.00% <0.00%> (ø)
...e/cloud/teleport/bigtable/BeamRowToBigtableFn.java	87.11% <81.69%> (-4.32%)	⬇️
...port/bigtable/CassandraWritetimeQueryProvider.java	0.00% <0.00%> (ø)

... and 163 files with indirect coverage changes

[Polber]

Thanks for the contribution! I left a couple small comments for core templates related files. Please tag a member from the BigTable team to review BigTable-related changes

[vermas2012]

Please go through all the warnings of lines not covered by tests and add any relevant Tests.

[Abacn]

ack. will merge after fixing the format (mv. spotless:apply)

[Abacn]

cc: @Polber looks like merge is still blocked pending your approval

[vermas2012]

@Polber Is there anything pending from the CBT side? can we merge this change?

[Polber]

Thanks!