Read latest configuration independently from main loop #379

hanshasselberg · 2020-01-07T10:01:08Z

Before reading the raft configuration required the main loop to work off the configurationsCh. This is problematic in different scenarios:

node is under high load
leadership is flapping

The main reason why that was the case is that the configurations attribute must not be accessed in parallel which was guaranteed because it was only ever read or written from the leaderLoop.

By adding a copy of the latest configuration (which is ultimately what was returned) and protecting it with its own mutex, it is now possible to access the latest configuration from outside of the leader loop.

Fixes #302, #356, and improves hashicorp/consul#6852.

banks · 2020-01-07T10:31:57Z

Another pretty important issue this caused for future reference is when restoring a large snapshot the leader/follower loop might be blocked for seconds or minutes resulting in Stats() calls hanging "forever".

Thanks for getting to this @i0rek.

@catsby FYI I think you were looking into this at some point so just flagging the PR in case you missed it and are interested.

banks

I think using an atomic value has lower risk for blocking raft loop as noted inline. That said, I'm open to objections if I missed something.

banks · 2020-01-07T10:37:52Z

api.go

@@ -138,6 +138,11 @@ type Raft struct {
 	// the log/snapshot.
 	configurations configurations

+	// Holds a copy of the latest configuration which can be read
+	// independently from main loop.
+	latestConfiguration     Configuration


Should we be returning the latest configuration or the latest comitted configuration? I may well be misremembering but I feel like we shouldn't be using/returning the config until it's committed right?

nm. GetConfiguration docs state:

GetConfiguration returns the latest configuration. This may not yet be currently in use. This may not yet be committed...

SO this preserves that behaviour although frankly that seems strange to me that Consul or an application would "see" a config that tis not the one actually in use and may never get committed by a quorum 🤔.

I guess that is an edge case issue we can solve separately though this preserves same behaviour while fixing the performance issue.

I was wondering about the same thing, but wanted to preserve current behaviour.

banks · 2020-01-07T10:38:00Z

api.go

-	}
+	configReq.configurations = configurations{latest: r.getLatestConfiguration()}
+	configReq.respond(nil)
+	return configReq


Does this make case future := <-r.configurationsCh: in leader/follower loop dead code now? Should we remove that?

We are still using it in our tests and adding an alternative would be more work. I decided to leave it as is so that this PR doesn't get out of hand. But it is essentially dead code now.

In theory configurations could be an atomic value, and we only ever load and store. That way I could get rid of the channel and fix the tests. But that would be quite some places to change.

🤔 hmm. Making part of the internal workings atomic and accessing it directly in tests could introduce other issues though. Atomics prevent data races but not logical ones. I think the usage here is fine as we are limiting to only a single writer and wrapping concurrent readers so they don't have direct access but making internal state atomic and then tweaking it from tests could get brittle fast if we ever update or make assumptions about whether then atomic value.

Does leaving tests using a different code path to real callers cause issues? If not I'm inclined to leave it to.

Oh we also still use it in takeSnapshot. We also rely in those places on being able to get both the latest and the committed configs which means the GetConfiguration can't do the same job.

So I say it's OK to leave it for now as it is as those call sites need to lower level and more accurate answer, this just optimises for external callers.

banks · 2020-01-07T10:44:24Z

api.go

+	// Holds a copy of the latest configuration which can be read
+	// independently from main loop.
+	latestConfiguration     Configuration
+	latestConfigurationLock sync.RWMutex


I think this would probably work well for most use cases but there is a risk that very heavy reads of GetConfiguration can starve the main loop on a config change - i.e. cause raft to block indefinitely.

For example in Consul if there were some script calling agent info on a server that accidentally got into a hot loop and called it constantly, it could bring the whole of raft grinding to a halt. While that's hopefully not super likely it feels like a pretty bad outcome.

I think in this case atomic.Value might be a better way to syncronize Load and Store of this copy. Then readers can read without blocking using latestConfiguration.Load() and high contention might slow the leaderloop a little due to cross-processor core contention on the cache line but it can't starve it indefinitely.

What do you think?

I agree that it would be better to use atomic and I will change it.

banks

Great job!

banks · 2020-01-07T13:05:45Z

api.go

@@ -138,6 +139,10 @@ type Raft struct {
 	// the log/snapshot.
 	configurations configurations

+	// Holds a copy of the latest configuration which can be read
+	// independently from main loop.
+	latestConfiguration atomic.Value


For the record.. I looked it up and atomic.Value is itself a struct which means even as a non-pointer field like this it will be aligned to 64 bit boundaries and the atomically accessed pointer within it won't have alignment issues.

hanshasselberg added 2 commits January 7, 2020 10:57

get configuration independently from main loop

8a07ad3

add comments

3b20804

hanshasselberg changed the title ~~get configuration independently from main loop~~ Read latest configuration independently from main loop Jan 7, 2020

hanshasselberg marked this pull request as ready for review January 7, 2020 10:26

hanshasselberg requested review from a team and catsby January 7, 2020 10:28

hanshasselberg self-assigned this Jan 7, 2020

hanshasselberg added the waiting-reply label Jan 7, 2020

stale bot removed the waiting-reply label Jan 7, 2020

banks requested changes Jan 7, 2020

View reviewed changes

use atomic instead of sync.

c4d0efb

banks approved these changes Jan 7, 2020

View reviewed changes

hanshasselberg merged commit 51329a7 into master Jan 8, 2020

hanshasselberg deleted the get_configuration branch January 8, 2020 11:25

hanshasselberg mentioned this pull request Jan 21, 2020

Fix autopilot test hashicorp/consul#7092

Merged

notnoop mentioned this pull request Jan 22, 2020

Handle Nomad leadership flapping (attempt 2) hashicorp/nomad#6977

Merged

notnoop mentioned this pull request Feb 14, 2020

Update Raft dependency hashicorp/nomad#7150

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Read latest configuration independently from main loop #379

Read latest configuration independently from main loop #379

hanshasselberg commented Jan 7, 2020 •

edited

Loading

banks commented Jan 7, 2020

banks left a comment

banks Jan 7, 2020

banks Jan 7, 2020

hanshasselberg Jan 7, 2020

banks Jan 7, 2020

hanshasselberg Jan 7, 2020

hanshasselberg Jan 7, 2020

banks Jan 7, 2020

hanshasselberg Jan 7, 2020

banks Jan 7, 2020 •

edited

Loading

hanshasselberg Jan 7, 2020

banks left a comment

banks Jan 7, 2020

Read latest configuration independently from main loop #379

Read latest configuration independently from main loop #379

Conversation

hanshasselberg commented Jan 7, 2020 • edited Loading

banks commented Jan 7, 2020

banks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banks Jan 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

banks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hanshasselberg commented Jan 7, 2020 •

edited

Loading

banks Jan 7, 2020 •

edited

Loading