CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

worryg0d · 2024-09-16T08:01:11Z

Overview

C* 4.0 introduced Native Protocol 5, so we should make gocql to support it.
This PR consists of other my PRs, which also provides support for the v5 version of the protocol.

This PR provides support of Native Protocol 5 features as:

New frames format CASSANDRA-15299
result_metadata_id field for EXECUTE and PREPARED messages CASSANDRA-10786
now_in_seconds field for BATCH and QUERY messages CASSANDRA-14664
keyspace field for BATCH and QUERY messages

worryg0d · 2024-10-01T15:01:29Z

This PR is merged from #1778, #1773, #1782, and #1781, so we could close them in favor of this one

joao-r-reis · 2024-10-08T15:17:34Z

What about #1783

lukasz-antoniak

Copied unsolved comments fromCopied unresolved comments from previous PRs.

frame_test.go

lukasz-antoniak · 2024-10-08T15:43:12Z

What about #1783

@worryg0d, IMHO changes in #1783 are still needed and improve code maintainability.

worryg0d · 2024-10-09T04:17:51Z

Thanks a lot for reaching out!

This PR is lacking changes from #1783. I will ask @testisnullus if we could merge it into this PR.

testisnullus · 2024-10-09T09:29:35Z

@worryg0d @lukasz-antoniak I think we can keep the current implementation and merge the #1783 PR into this one.

worryg0d · 2024-10-10T12:36:12Z

I merged changes from #1783 into this one.

joao-r-reis

First pass review. I'll need some extra time to properly review the envelope/message format changes.

batch_test.go

cassandra_test.go

compressor.go

conn.go

session.go

frame.go

conn.go

worryg0d · 2024-10-15T18:01:13Z

Hey, @joao-r-reis. Thanks a lot for your feedback! I'm pleased this PR is moving. I will take a look at your comments and suggestions tomorrow. I hope everything will be fine here.

…to ver < 5. Using ptr based logic for now_in_seconds field instead of additional flag

worryg0d · 2024-10-16T12:47:09Z

Hello, @joao-r-reis. I added some changes following your comments and suggestions. Could you please take a look at this one again? Also, I have a question regarding to cache of prepared statements. Could you please look at this one too?

Thanks a lot.

…orthIt integration tests

conn.go

joao-r-reis · 2024-10-29T15:44:43Z

Added semver-major label due to the Compressor interface changes

2. The driver should relly on No_metadata flag instead of Skip_metadata when handling RESULT/ROWS responses from C*

…_changed handling logic by forcing the driver to execute queries with old metadata 2. Added checking of integrity of the result to the test

…rent code base uses initial keyspace provided by the user to construct the keys. Since proto v5 we also should account for keyspace bounding for a specific query, so the driver should use the bounded keyspace instead of the initial to construct the key. 2. Changed the way how routing key cache keys are constructed to account the keyspace overriding as well.

joao-r-reis

Good stuff 👍 , I left 2 conversations open so that we get input from the other committer that looks at this:

bumping go.mod version to 1.19 so we can use atomic.Bool (current minimum version is 1.17 but go.mod is not up to date
Compressor interface changes

I'm +1 on both of these though.

joao-r-reis · 2024-10-31T15:48:19Z

oh I just noticed the integration tests on GH are running with snappy compression, we should change it to lz4 so we get coverage on compression+v5

worryg0d · 2024-10-31T17:32:11Z

Good stuff 👍 , I left 2 conversations open so that we get input from the other committer that looks at this:

bumping go.mod version to 1.19 so we can use atomic.Bool (current minimum version is 1.17 but go.mod is not up to date

Compressor interface changes

I'm +1 on both of these though.

Thanks a lot! I'm really happy to hear that. I'm waiting for a review from another committer and hoping everything will be fine here.

oh I just noticed the integration tests on GH are running with snappy compression, we should change it to lz4 so we get coverage on compression+v5

Oh, yeah. I'll try to change CI to run tests here.

worryg0d · 2024-10-31T18:03:01Z

I changed CI here to run tests over proto 5. One thing I had to replace lz4 dependency in go.mod to use the updated version of the pkg...

…CacheUsesOverriddenKeyspace integration tests when running on proto < 5 because they requires overridding keyspace feature that is available for proto 5 and higher

joao-r-reis · 2024-11-06T15:11:34Z

Hmm it does look weird to have gocql's go.mod list gocql/lz4 as a dependency when gocql/lz4's go.mod lists gocql as a dependency... A possible solution is to make lz4 a package instead of its own module, I brought this up in slack and here.

If lz4 stays a separate module then I'm not sure what is the best way to use lz4 in the integration tests... Maybe your approach is the best one, if I'm not mistaken the lz4 dependency will not actually be downloaded by users since it's only used in tests.

go.mod

ribaraka

lgtm

worryg0d · 2024-11-12T16:03:39Z

@joao-r-reis I adjusted the CI in this PR to run integration tests over proto 5 and the lz4 compressor. Could you please approve running the workflow to see if nothing breaks?

jameshartig · 2025-01-29T16:25:46Z

conn.go

+}
+
+// connReader implements ConnReader.
+// It retries to read data up to 5 times or returns error.


I'm curious why it retries up to 5 times. Is that recommended somewhere or consistent with other drivers? I'm not used to seeing a Read call not respect the timeout and instead take 5x the timeout.

Here I focused on extracting this code to the specific connReader type to avoid unintentional reading from Conn.conn or Conn.r. I didn't aim to change this behavior, because this is a bit out of the scope of this PR. However, I already added a connReader... Probably changing this behavior should be handled here as well

jameshartig · 2025-01-29T16:39:43Z

conn.go

@@ -1394,14 +1578,18 @@ func (c *Conn) executeQuery(ctx context.Context, qry *Query) *Iter {
 		params.skipMeta = !(c.session.cfg.DisableSkipMetadata || qry.disableSkipMetadata)


Can you update this like I did in #1852 or we can just merge that MR since I don't think it'll conflict? We shouldn't be sending skipMeta if we didn't get it from the prepare. @joao-r-reis

Yeah now I see the difference, I think it's better if we just merge #1852 right away, I don't think it will conflict

I think we can merge #1852. It should not cause conflicts, but if it does I'll resolve them

worryg0d · 2025-02-13T15:26:59Z

Hello. This PR has already been floating for quite a while. Is there anything I can do to push it forward?

worryg0d mentioned this pull request Sep 16, 2024

Support for result_metadata_id in PREPARED and EXECUTE messages #1782

Closed

worryg0d force-pushed the native-proto-5-support-all-merged branch 4 times, most recently from d67fcb1 to 93933c6 Compare September 19, 2024 12:49

worryg0d mentioned this pull request Sep 19, 2024

Proto v5 support was added #1773

Closed

worryg0d force-pushed the native-proto-5-support-all-merged branch 3 times, most recently from 47a3ae9 to b617638 Compare October 1, 2024 07:03

joao-r-reis added the protocol_v5 Tickets related to supporting protocol version 5 label Oct 1, 2024

This was referenced Oct 1, 2024

Support for now_in_seconds option field for QUERY, BATCH and EXECUTE messages #1781

Closed

Support of keyspace field for BATCH message #1778

Closed

lukasz-antoniak reviewed Oct 8, 2024

View reviewed changes

frame_test.go Show resolved Hide resolved

worryg0d force-pushed the native-proto-5-support-all-merged branch from b617638 to 3f50a90 Compare October 10, 2024 12:06

Support for Native Protocol 5 release version

fbdaa2b

worryg0d force-pushed the native-proto-5-support-all-merged branch from 3f50a90 to fbdaa2b Compare October 10, 2024 12:14

tengu-alt mentioned this pull request Oct 15, 2024

CASSGO-11 Support vector type #1828

Open

joao-r-reis reviewed Oct 15, 2024

View reviewed changes

worryg0d added 3 commits October 16, 2024 13:45

Added 'IF NOT EXISTS' to table creation queries in integration tests

93fb4ad

Clarified comment on startup completed

d926108

Returned panics on setting keyspace and now_in_seconds field when pro…

a1baea2

…to ver < 5. Using ptr based logic for now_in_seconds field instead of additional flag

Added description for TestLargeSizeQuery and TestQueryCompressionNotW…

b77aa18

…orthIt integration tests

joao-r-reis reviewed Oct 29, 2024

View reviewed changes

conn.go Show resolved Hide resolved

joao-r-reis added the semver-major label Oct 29, 2024

worryg0d added 2 commits October 30, 2024 09:57

1. Updating info to ensure the code looking at the updated prepared stmt

bf0d7fa

2. The driver should relly on No_metadata flag instead of Skip_metadata when handling RESULT/ROWS responses from C*

1. Updated TestPrepareExecuteMetadataChangedFlag to validate Metadata…

0592a90

…_changed handling logic by forcing the driver to execute queries with old metadata 2. Added checking of integrity of the result to the test

worryg0d force-pushed the native-proto-5-support-all-merged branch 2 times, most recently from 6370ae0 to b5d76bd Compare October 30, 2024 16:58

worryg0d force-pushed the native-proto-5-support-all-merged branch from b5d76bd to 0298a00 Compare October 31, 2024 14:56

joao-r-reis approved these changes Oct 31, 2024

View reviewed changes

Added skip for TestStmtCacheUsesOverriddenKeyspace and TestRoutingKey…

a1a613c

…CacheUsesOverriddenKeyspace integration tests when running on proto < 5 because they requires overridding keyspace feature that is available for proto 5 and higher

joao-r-reis changed the title ~~CASSGO-1 Native Protocol 5 Support~~ CASSGO-1 CASSGO-30 Native Protocol 5 Support Nov 6, 2024

joao-r-reis mentioned this pull request Nov 6, 2024

CASSGO-30 DSE Search Results May Not Be Queryable #1689

Open

joao-r-reis reviewed Nov 6, 2024

View reviewed changes

go.mod Show resolved Hide resolved

Go bump to 1.19

e779e73

ribaraka approved these changes Nov 7, 2024

View reviewed changes

ribaraka mentioned this pull request Nov 11, 2024

CASSGO-21 Remove HostPoolHostPolicy from gocql package #1770

Merged

Changed CI to run integration tests over proto 5 + lz4 compressor

5ca5382

This was referenced Nov 20, 2024

CASSGO-32 move lz4 compressor to lz4 package within the gocql module #1842

Open

WIP-Cassgo 33 snappy has been moved to a separate package #1844

Open

testisnullus approved these changes Jan 24, 2025

View reviewed changes

joao-r-reis mentioned this pull request Jan 29, 2025

CASSGO-40: skip metadata only if prepared result included metadata #1852

Merged

jameshartig reviewed Jan 29, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

worryg0d commented Sep 16, 2024 •

edited

Loading

worryg0d commented Oct 1, 2024 •

edited

Loading

joao-r-reis commented Oct 8, 2024

lukasz-antoniak left a comment

lukasz-antoniak commented Oct 8, 2024

worryg0d commented Oct 9, 2024

testisnullus commented Oct 9, 2024

worryg0d commented Oct 10, 2024

joao-r-reis left a comment

worryg0d commented Oct 15, 2024

worryg0d commented Oct 16, 2024

joao-r-reis commented Oct 29, 2024

joao-r-reis left a comment

joao-r-reis commented Oct 31, 2024

worryg0d commented Oct 31, 2024

worryg0d commented Oct 31, 2024 •

edited

Loading

joao-r-reis commented Nov 6, 2024

ribaraka left a comment

worryg0d commented Nov 12, 2024

jameshartig Jan 29, 2025

worryg0d Jan 29, 2025

jameshartig Jan 29, 2025

joao-r-reis Jan 29, 2025

worryg0d Jan 29, 2025

worryg0d commented Feb 13, 2025

		@@ -1394,14 +1578,18 @@ func (c Conn) executeQuery(ctx context.Context, qry Query) *Iter {
		params.skipMeta = !(c.session.cfg.DisableSkipMetadata \|\| qry.disableSkipMetadata)

CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

Are you sure you want to change the base?

CASSGO-1 CASSGO-30 Native Protocol 5 Support #1822

Conversation

worryg0d commented Sep 16, 2024 • edited Loading

Overview

worryg0d commented Oct 1, 2024 • edited Loading

joao-r-reis commented Oct 8, 2024

lukasz-antoniak left a comment

Choose a reason for hiding this comment

lukasz-antoniak commented Oct 8, 2024

worryg0d commented Oct 9, 2024

testisnullus commented Oct 9, 2024

worryg0d commented Oct 10, 2024

joao-r-reis left a comment

Choose a reason for hiding this comment

worryg0d commented Oct 15, 2024

worryg0d commented Oct 16, 2024

joao-r-reis commented Oct 29, 2024

joao-r-reis left a comment

Choose a reason for hiding this comment

joao-r-reis commented Oct 31, 2024

worryg0d commented Oct 31, 2024

worryg0d commented Oct 31, 2024 • edited Loading

joao-r-reis commented Nov 6, 2024

ribaraka left a comment

Choose a reason for hiding this comment

worryg0d commented Nov 12, 2024

jameshartig Jan 29, 2025

Choose a reason for hiding this comment

worryg0d Jan 29, 2025

Choose a reason for hiding this comment

jameshartig Jan 29, 2025

Choose a reason for hiding this comment

joao-r-reis Jan 29, 2025

Choose a reason for hiding this comment

worryg0d Jan 29, 2025

Choose a reason for hiding this comment

worryg0d commented Feb 13, 2025

worryg0d commented Sep 16, 2024 •

edited

Loading

worryg0d commented Oct 1, 2024 •

edited

Loading

worryg0d commented Oct 31, 2024 •

edited

Loading