Support TLS for replication #1630

PragmaTwice · 2023-08-02T23:30:03Z

It closes #1501.

Currently we do not support splitting cert between server and client.

This PR does not plan to support TLS for cluster mode, like slot migration.

@gofort It will be nice if you guys still want this feature and would like to try to do some tests on the patch.

gofort · 2023-08-03T09:37:40Z

I have asked @sheyt0 to test, but please don't block this PR with our reply. We will anyway test this, but not sure if it will be during this week or next.

PragmaTwice · 2023-08-03T11:55:00Z

I have asked @sheyt0 to test, but please don't block this PR with our reply. We will anyway test this, but not sure if it will be during this week or next.

Thanks! It is OK to test it in next week since we do not require this PR to be merged ASAP. There is still work to be done before we can merge this PR.

src/common/io_util.cc

…-replica

sheyt0 · 2023-08-14T08:27:44Z

@PragmaTwice сould you leave a little doc on how to use TLS for replication?

With this config
on master

kvrocks --tls-key-file ... --tls-cert-file ... --tls-ca-cert-file ...

and on slave

kvrocks --tls-replication yes --tls-key-file ... --tls-cert-file ... --tls-ca-cert-file ...

I get such errors

E20230814 08:22:48.497459 957356 redis_connection.cc:103] [connection] Going to remove the client: ...:59388, while encounter error: Success, SSL Error: error:0A0000C7:SSL routines::peer did not return a certificate
E20230814 08:22:49.507014 957356 redis_connection.cc:103] [connection] Going to remove the client: ...:59392, while encounter error: Success, SSL Error: error:0A0000C7:SSL routines::peer did not return a certificate

PragmaTwice · 2023-08-15T05:24:50Z

@sheyt0 Sure. Here are my test steps:

Generate or obtain some certs (e.g. using minica):

ca.crt
server.crt
server.key

(all of them are in PEM format)

Start a kvrocks server instance

./kvrocks --dir datadir1 --port 6666 --tls-port 6676 --log-dir stdout --tls-cert-file cert/server.crt --tls-key-file cert/server.key --tls-ca-cert-file cert/ca.crt

Start a replica (in same host, for convenience)

./kvrocks --dir datadir2 --port 6667 --tls-port 6677 --log-dir stdout --tls-cert-file cert/server.crt --tls-key-file cert/server.key --tls-ca-cert-file cert/ca.crt --tls-replication yes --slaveof "127.0.0.1 6676"

Test

➜  build git:(tls-replica) ✗ redis-cli -p 6666
127.0.0.1:6666> get a
(nil)
127.0.0.1:6666> set a 1
OK
127.0.0.1:6666> set b 2
OK
127.0.0.1:6666> set c 3
OK
127.0.0.1:6666> exit
➜  build git:(tls-replica) ✗ redis-cli -p 6667
127.0.0.1:6667> get a
"1"
127.0.0.1:6667> get b
"2"
127.0.0.1:6667> get c
"3"
127.0.0.1:6667>

Server logs:

WARNING: No config file specified, using the default configuration. In order to specify a config file use 'kvrocks -c /path/to/kvrocks.conf'
I20230815 13:19:36.561048 20476 main.cc:328] kvrocks unstable (commit 83ad999c)
I20230815 13:19:36.591779 20476 storage.cc:332] [storage] Success to load the data from disk: 24 ms
I20230815 13:19:36.595307 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20230815 13:19:36.595387 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6676
I20230815 13:19:36.595523 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20230815 13:19:36.595558 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6676
I20230815 13:19:36.595630 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20230815 13:19:36.595645 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6676
I20230815 13:19:36.595757 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20230815 13:19:36.595808 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6676
I20230815 13:19:36.595898 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20230815 13:19:36.595913 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6676
I20230815 13:19:36.595993 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20230815 13:19:36.596009 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6676
I20230815 13:19:36.596077 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20230815 13:19:36.596091 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6676
I20230815 13:19:36.596154 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6666
I20230815 13:19:36.596168 20476 worker.cc:72] [worker] Listening on: 0.0.0.0:6676
W20230815 13:19:36.596231 20476 server.cc:1629] [server] Increased maximum number of open files to 18464 (it's originally set to 1024)
I20230815 13:19:36.596359 20476 worker.cc:516] [worker] Thread #139993055979072 started
I20230815 13:19:36.596398 20476 worker.cc:516] [worker] Thread #139993064371776 started
I20230815 13:19:36.596438 20476 worker.cc:516] [worker] Thread #139993072764480 started
I20230815 13:19:36.596958 20476 worker.cc:516] [worker] Thread #139993127319104 started
I20230815 13:19:36.597033 20476 worker.cc:516] [worker] Thread #139993118926400 started
I20230815 13:19:36.597082 20476 worker.cc:516] [worker] Thread #139993110533696 started
I20230815 13:19:36.597136 20476 worker.cc:516] [worker] Thread #139993102140992 started
I20230815 13:19:36.597185 20476 worker.cc:516] [worker] Thread #139993093748288 started
I20230815 13:19:36.597337 20476 server.cc:208] [server] Ready to accept connections
I20230815 13:22:06.778137 20577 cmd_replication.cc:59] Slave 127.0.0.1:43632, listening port: 6667, announce ip: 127.0.0.1 asks for synchronization with next sequence: 1 replication id: not supported, and local sequence: 2
I20230815 13:22:06.778566 20577 cmd_replication.cc:112] New replica: 127.0.0.1:43632 was added, start incremental syncing

Replica logs:

WARNING: No config file specified, using the default configuration. In order to specify a config file use 'kvrocks -c /path/to/kvrocks.conf'
I20230815 13:22:06.714430 20842 main.cc:328] kvrocks unstable (commit 83ad999c)
I20230815 13:22:06.767542 20842 storage.cc:332] [storage] Success to load the data from disk: 12 ms
I20230815 13:22:06.769820 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6667
I20230815 13:22:06.769878 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6677
I20230815 13:22:06.770000 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6667
I20230815 13:22:06.770045 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6677
I20230815 13:22:06.770133 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6667
I20230815 13:22:06.770148 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6677
I20230815 13:22:06.770259 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6667
I20230815 13:22:06.770275 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6677
I20230815 13:22:06.770339 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6667
I20230815 13:22:06.770357 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6677
I20230815 13:22:06.770440 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6667
I20230815 13:22:06.770459 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6677
I20230815 13:22:06.770542 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6667
I20230815 13:22:06.770557 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6677
I20230815 13:22:06.770632 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6667
I20230815 13:22:06.770645 20842 worker.cc:72] [worker] Listening on: 0.0.0.0:6677
W20230815 13:22:06.770705 20842 server.cc:1629] [server] Increased maximum number of open files to 18464 (it's originally set to 1024)
W20230815 13:22:06.770941 20842 replication.cc:348] Clean old synced checkpoint successfully
I20230815 13:22:06.770998 20842 worker.cc:516] [worker] Thread #140695450900032 started
I20230815 13:22:06.771035 20842 worker.cc:516] [worker] Thread #140695459292736 started
I20230815 13:22:06.771433 20842 worker.cc:516] [worker] Thread #140695532729920 started
I20230815 13:22:06.771507 20842 worker.cc:516] [worker] Thread #140695522240064 started
I20230815 13:22:06.771544 20842 worker.cc:516] [worker] Thread #140695513847360 started
I20230815 13:22:06.771581 20842 worker.cc:516] [worker] Thread #140695505454656 started
I20230815 13:22:06.771620 20842 worker.cc:516] [worker] Thread #140695497061952 started
I20230815 13:22:06.771665 20842 worker.cc:516] [worker] Thread #140695488669248 started
I20230815 13:22:06.771783 20842 server.cc:208] [server] Ready to accept connections
I20230815 13:22:06.775859 20954 replication.cc:421] [replication] Check db name request was sent, waiting for response
I20230815 13:22:06.777531 20954 replication.cc:441] [replication] DB name is valid, continue...
I20230815 13:22:06.777562 20954 replication.cc:459] [replication] replconf request was sent, waiting for response
I20230815 13:22:06.777760 20954 replication.cc:485] [replication] replconf is ok, start psync
I20230815 13:22:06.777988 20954 replication.cc:515] [replication] Try to use psync, next seq: 1
I20230815 13:22:06.779289 20954 replication.cc:552] [replication] PSync is ok, start increment batch loop

sheyt0 · 2023-08-15T05:46:46Z

@PragmaTwice

Currently we do not support splitting cert between server and client.

Do I understand correctly that one set of certs should be used (same certs for master and slave)?
Or is it just about config options?

PragmaTwice · 2023-08-15T05:53:15Z

@PragmaTwice

Currently we do not support splitting cert between server and client.

Do I understand correctly that one set of certs should be used (same certs for master and slave)? Or is it just about config options?

Yeah, certs for master and slave must be the same.

sheyt0 · 2023-08-15T07:33:23Z

Yeah, certs for master and slave must be the same.

In the current implementation, this is not suitable for us. We need mTLS, not just TLS.

PragmaTwice · 2023-08-15T08:00:07Z

In the current implementation, this is not suitable for us. We need mTLS, not just TLS.

Although I previously do not plan to support it in this PR, the feature is not hard to implement.

I will try to support it later.

PragmaTwice · 2023-08-15T08:26:43Z

@sheyt0 Sorry, I think I mis-understand the problem.

The certs of master and slave can be different. I have tried and it works well.

But you need to be noticed that the server connection (listen to kvrocks clients) and the client connection (connect to the master) of the slave must use the same cert, which currently is not able to split into different certs.

So internally it includes four part of SSL context:

master: server-side cert, client-side cert
slave: server-side cert, client-side cert

The two certs in master cannot be different (and also in slave), but the server-side cert in master and client-side cert in slave can be different.

PragmaTwice · 2023-08-15T08:37:22Z

@sheyt0 And, I have found why you cannot start TLS connection between master and slave.

Currently slave must have a tls-port to have TLS connection to master, this is a BUG since it is unnecessary.

I will fix it soon.

…-replica

PragmaTwice · 2023-08-15T08:40:13Z

Currently slave must have a tls-port to have TLS connection to master, this is a BUG since it is unnecessary.

It is fixed. Now I think the slave can disable the tls-port (if you want).

sheyt0 · 2023-08-29T07:18:24Z

It is fixed. Now I think the slave can disable the tls-port (if you want).

Now it works

niumcdao · 2023-11-11T09:02:50Z

I use minica generate ca file. https://github.com/jsha/minica

PragmaTwice · 2023-11-11T09:34:25Z

I use minica generate ca file. https://github.com/jsha/minica

You need to build kvrocks with cmake option ENABLE_OPENSSL=ON.

Please refer to https://github.com/apache/kvrocks#build for details.

niumcdao · 2023-12-02T04:51:34Z

tls-replication params I shout config on master node or slave node?

niumcdao · 2023-12-02T05:07:43Z

I config tls-replication on slave node,then I login on master node, info comand:

Replication slave node info is port 6666,why not 6676(master node tls port)

PragmaTwice · 2023-12-02T05:21:58Z

You need to configure the slaveof port to 6676 by yourself.

Also, you can open a discussion or question in slack channel rather than reply in this PR thread.

Support TLS for replication

4597af8

PragmaTwice requested review from git-hulk, torwig and mapleFU August 2, 2023 23:30

PragmaTwice and others added 6 commits August 3, 2023 07:30

Merge branch 'unstable' into tls-replica

c73059a

fix

3edf5d1

fix

c83d423

fix

3d08909

add test

d75f85d

more tests

02796fb

fix cmd_replication.cc

5407170

PragmaTwice and others added 3 commits August 3, 2023 20:30

fix

7afa4a9

Merge branch 'unstable' into tls-replica

81f4955

fix

5529f00

git-hulk reviewed Aug 4, 2023

View reviewed changes

src/common/io_util.cc Show resolved Hide resolved

src/common/io_util.cc Outdated Show resolved Hide resolved

src/common/io_util.cc Outdated Show resolved Hide resolved

PragmaTwice and others added 4 commits August 5, 2023 04:58

Merge branch 'unstable' into tls-replica

3e15ad7

remove header include

5a77416

Merge branch 'unstable' of https://github.com/apache/kvrocks into tls…

e1b2463

…-replica

fix conflict

68d3709

PragmaTwice mentioned this pull request Aug 9, 2023

Support rapid slot migration based on raw key value #1534

Closed

PragmaTwice added 2 commits August 9, 2023 11:14

Merge branch 'unstable' into tls-replica

b10417c

Merge branch 'unstable' into tls-replica

890b2df

git-hulk previously approved these changes Aug 12, 2023

View reviewed changes

Merge branch 'unstable' into tls-replica

83ad999

fix go test

cc5cb97

PragmaTwice dismissed git-hulk’s stale review via cc5cb97 August 15, 2023 05:27

Merge branch 'unstable' into tls-replica

35de950

Merge branch 'unstable' into tls-replica

a246df2

PragmaTwice added 2 commits August 15, 2023 16:38

fix ssl ctx init

c47e32b

Merge branch 'tls-replica' of github.com:PragmaTwice/kvrocks into tls…

d8ec37d

…-replica

PragmaTwice and others added 3 commits August 16, 2023 11:51

fix

3b6ab61

Merge branch 'unstable' into tls-replica

2914331

Merge branch 'unstable' into tls-replica

31de80f

Merge branch 'unstable' into tls-replica

312b71a

git-hulk previously approved these changes Sep 2, 2023

View reviewed changes

PragmaTwice and others added 5 commits September 3, 2023 11:43

Merge branch 'unstable' into tls-replica

352fbdb

Merge branch 'unstable' into tls-replica

05c6045

Merge branch 'unstable' into tls-replica

9f09f63

Merge branch 'unstable' into tls-replica

edf139e

fix

5f6c781

PragmaTwice dismissed git-hulk’s stale review via 5f6c781 September 11, 2023 13:31

git-hulk approved these changes Sep 11, 2023

View reviewed changes

PragmaTwice merged commit f3d796d into apache:unstable Sep 12, 2023
26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support TLS for replication #1630

Support TLS for replication #1630

PragmaTwice commented Aug 2, 2023 •

edited

Loading

gofort commented Aug 3, 2023

PragmaTwice commented Aug 3, 2023 •

edited

Loading

sheyt0 commented Aug 14, 2023

PragmaTwice commented Aug 15, 2023 •

edited

Loading

sheyt0 commented Aug 15, 2023 •

edited

Loading

PragmaTwice commented Aug 15, 2023 •

edited

Loading

sheyt0 commented Aug 15, 2023 •

edited

Loading

PragmaTwice commented Aug 15, 2023

PragmaTwice commented Aug 15, 2023 •

edited

Loading

PragmaTwice commented Aug 15, 2023

PragmaTwice commented Aug 15, 2023 •

edited

Loading

sheyt0 commented Aug 29, 2023 •

edited

Loading

niumcdao commented Nov 11, 2023

PragmaTwice commented Nov 11, 2023 •

edited

Loading

niumcdao commented Dec 2, 2023

niumcdao commented Dec 2, 2023

PragmaTwice commented Dec 2, 2023 •

edited

Loading

Support TLS for replication #1630

Support TLS for replication #1630

Conversation

PragmaTwice commented Aug 2, 2023 • edited Loading

gofort commented Aug 3, 2023

PragmaTwice commented Aug 3, 2023 • edited Loading

sheyt0 commented Aug 14, 2023

PragmaTwice commented Aug 15, 2023 • edited Loading

sheyt0 commented Aug 15, 2023 • edited Loading

PragmaTwice commented Aug 15, 2023 • edited Loading

sheyt0 commented Aug 15, 2023 • edited Loading

PragmaTwice commented Aug 15, 2023

PragmaTwice commented Aug 15, 2023 • edited Loading

PragmaTwice commented Aug 15, 2023

PragmaTwice commented Aug 15, 2023 • edited Loading

sheyt0 commented Aug 29, 2023 • edited Loading

niumcdao commented Nov 11, 2023

PragmaTwice commented Nov 11, 2023 • edited Loading

niumcdao commented Dec 2, 2023

niumcdao commented Dec 2, 2023

PragmaTwice commented Dec 2, 2023 • edited Loading

PragmaTwice commented Aug 2, 2023 •

edited

Loading

PragmaTwice commented Aug 3, 2023 •

edited

Loading

PragmaTwice commented Aug 15, 2023 •

edited

Loading

sheyt0 commented Aug 15, 2023 •

edited

Loading

PragmaTwice commented Aug 15, 2023 •

edited

Loading

sheyt0 commented Aug 15, 2023 •

edited

Loading

PragmaTwice commented Aug 15, 2023 •

edited

Loading

PragmaTwice commented Aug 15, 2023 •

edited

Loading

sheyt0 commented Aug 29, 2023 •

edited

Loading

PragmaTwice commented Nov 11, 2023 •

edited

Loading

PragmaTwice commented Dec 2, 2023 •

edited

Loading