fix: duplicated schemas on rapid elections while continuous produce of records #938

eliax1996 · 2024-08-21T16:05:47Z

coordinator rewrite

moving the coordinator in a separate thread
adding a waiting time between when the master its elected and the master can act. This has been done to avoid rapid elections of master that may produce schemas with different ids.

Example of what could happpen without the delay:

|--------------------------------------|
|Node | Node1    | Node2    | Node3    |
|Role | Master   | Follower | Follower |
|--------------------------------------|


Node1 -> Send Message A{id=max(current_ids)} to kafka

where the max(current_ids) = 10

---------------------------------------

Node1 its disconnected, the message its still in the producer queue of Node1

---------------------------------------

Node2 its elected master

|--------------------------------------|
|Node | Node1    | Node2    | Node3    |
|Role | Follower | Master   | Follower |
|--------------------------------------|

----------------------------------------


Node2 produces a message B{id=max(current_ids)} to kafka

Because the message A isn't yet delivered to Node2, the max(current_ids) returns still 10.
And we have an ID clash.

The solution its simple, each master should wait a reasonable high number of milliseconds before acting as a master.
So that all the in-flight messages are delivered to kafka + the reasonable delay of the consumer for the master node before noticing that a message has been produced

eliax1996 · 2024-08-22T16:54:52Z

github-actions · 2024-08-23T10:49:43Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/karapace
config.py
schema_reader.py					377-378
schema_registry.py					100
schema_registry_apis.py					106-107, 688-700, 1335
typing.py					111, 115
src/karapace/backup
api.py
cli.py
src/karapace/coordinator
master_coordinator.py					109, 111-112, 168
schema_coordinator.py					198, 213-214, 500
src/karapace/sentry
sentry_client.py
Project Total

_{This report was generated by python-coverage-comment-action}

jclarysse · 2024-08-26T07:28:29Z

Please also add the new config waiting_time_before_acting_as_master_ms to https://github.com/Aiven-Open/karapace/blob/main/README.rst#configuration-keys

karapace/coordinator/master_coordinator.py

karapace/coordinator/schema_coordinator.py

karapace/schema_reader.py

eliax1996 · 2024-11-13T15:54:00Z

src/karapace/schema_registry_apis.py

@@ -1307,7 +1335,7 @@ async def _forward_request_remote(
        if auth_header is not None:
            headers["Authorization"] = auth_header

-        with async_timeout.timeout(timeout):
+        async with async_timeout.timeout(timeout):


this was wrong since a while

eliax1996 · 2024-11-13T17:59:26Z

src/karapace/coordinator/schema_coordinator.py

+        LOG.info("Resetting generation status")
+        # this is called immediately after the election, we shouldn't reset this
+        # until a new node its elected aka the other path where a new node its elected
+        # otherwise this its called at each round and we keep not counting the 5 seconds
+        # required before the election.
+        # self._are_we_master = False
        self.generation = OffsetCommitRequest.DEFAULT_GENERATION_ID


this was called because we were exiting the election loop in the _async_loop of the master_coordinator.py. We must keep running the thread/algorithm otherwise we are always electing a new node since that causes a rebalance and the rebalance causes a new election (the rebalance happen because we sent the reset_generation as a side effect of closing the heartbeat task)

eliax1996 · 2024-11-14T11:21:04Z

src/karapace/coordinator/master_coordinator.py

+            # why do we need to close?
+            # we just need to keep running even when the schema registry its ready
+            # otherwise we cause a rebalance and a new election. This should run until
+            # karapace is restarted
+            # if self._sc.ready():
+            #    break


this question its mainly for @jjaakola-aiven. I inherited the initial implementation from him. I think we shouldn't exit but I wait for him to reply here

1. moving the coordinator in a separate thread 2. adding a waiting time between when the master its elected and the master can act. This has been done to avoid rapid elections of master that may produce schemas with different ids. Example of what could happpen without the delay: |--------------------------------------| |Node | Node1 | Node2 | Node3 | |Role | Master | Follower | Follower | |--------------------------------------| Node1 -> Send Message A{id=max(current_ids)} to kafka where the max(current_ids) = 10 --------------------------------------- Node1 its disconnected, the message its still in the producer queue of Node1 --------------------------------------- Node2 its elected master |--------------------------------------| |Node | Node1 | Node2 | Node3 | |Role | Follower | Master | Follower | |--------------------------------------| ---------------------------------------- Node2 produces a message B{id=max(current_ids)} to kafka Because the message A isn't yet delivered to Node2, the max(current_ids) returns still 10. And we have an ID clash. The solution its simple, each master should wait a reasonable high number of milliseconds before acting as a master. So that all the in-flight messages are delivered to kafka + the reasonable delay of the consumer for the master node before noticing that a message has been produced

keejon · 2024-11-22T08:25:49Z

src/karapace/schema_registry_apis.py

+            "/master_available",
+            callback=self.master_available,
+            # post because they cannot be cached, need to be sure always gathering the real value of that property
+            method="POST",


Not a big fan of this because it violates semantics of POST. Maybe set a Cache-Control: no-store, no-cache, must-revalidate header instead?

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch from 7b04f19 to 285c950 Compare August 22, 2024 16:25

eliax1996 changed the title ~~WIP: fix duplicated schemas on rapid elections while continuous produce of records~~ fix: duplicated schemas on rapid elections while continuous produce of records Aug 22, 2024

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch 2 times, most recently from 52fb34c to 054efb7 Compare August 22, 2024 16:50

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch 5 times, most recently from 1fa1052 to 7f657bd Compare August 23, 2024 10:14

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch from 49a9fd2 to c4e58e8 Compare August 23, 2024 14:18

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch 6 times, most recently from 8183430 to 784a2fa Compare August 29, 2024 13:34

nosahama reviewed Sep 10, 2024

View reviewed changes

karapace/coordinator/master_coordinator.py Outdated Show resolved Hide resolved

nosahama reviewed Sep 10, 2024

View reviewed changes

karapace/coordinator/schema_coordinator.py Outdated Show resolved Hide resolved

nosahama reviewed Sep 10, 2024

View reviewed changes

karapace/schema_reader.py Outdated Show resolved Hide resolved

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch from 784a2fa to 3f84dff Compare September 18, 2024 14:15

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch 7 times, most recently from 9f3d8bc to 25e6c16 Compare November 13, 2024 15:56

eliax1996 commented Nov 13, 2024

View reviewed changes

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch 2 times, most recently from a422f24 to d7cdd8a Compare November 14, 2024 07:39

eliax1996 commented Nov 14, 2024

View reviewed changes

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch 3 times, most recently from 2385470 to 0e374a7 Compare November 19, 2024 13:19

eliax1996 marked this pull request as ready for review November 19, 2024 13:19

eliax1996 requested review from a team as code owners November 19, 2024 13:19

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch 7 times, most recently from 8eb38e1 to d8067fd Compare November 21, 2024 08:55

eliax1996 force-pushed the eliax1996/make-sure-master-wait-a-while-before-being-active-and-if-new-messages-are-arriving branch from d8067fd to 5cb351b Compare November 21, 2024 10:27

keejon reviewed Nov 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: duplicated schemas on rapid elections while continuous produce of records #938

fix: duplicated schemas on rapid elections while continuous produce of records #938

eliax1996 commented Aug 21, 2024 •

edited

Loading

eliax1996 commented Aug 22, 2024

github-actions bot commented Aug 23, 2024 •

edited

Loading

jclarysse commented Aug 26, 2024

eliax1996 Nov 13, 2024

eliax1996 Nov 13, 2024

eliax1996 Nov 14, 2024

keejon Nov 22, 2024

fix: duplicated schemas on rapid elections while continuous produce of records #938

Are you sure you want to change the base?

fix: duplicated schemas on rapid elections while continuous produce of records #938

Conversation

eliax1996 commented Aug 21, 2024 • edited Loading

eliax1996 commented Aug 22, 2024

github-actions bot commented Aug 23, 2024 • edited Loading

Coverage report

jclarysse commented Aug 26, 2024

eliax1996 Nov 13, 2024

Choose a reason for hiding this comment

eliax1996 Nov 13, 2024

Choose a reason for hiding this comment

eliax1996 Nov 14, 2024

Choose a reason for hiding this comment

keejon Nov 22, 2024

Choose a reason for hiding this comment

eliax1996 commented Aug 21, 2024 •

edited

Loading

github-actions bot commented Aug 23, 2024 •

edited

Loading