Client RPCs are sticky to server #3738

dadgar · 2018-01-10T19:29:38Z

No description provided.

schmichael · 2018-01-19T00:57:50Z

client/client.go

+
+	// We can wait a bit and retry!
+	if time.Since(firstCheck) < c.config.RPCHoldTimeout {
+		jitter := lib.RandomStagger(5 * time.Second / nomad.JitterFraction)


Why do we use JitterFraction instead of just saying 300ms (~5s / 16)? I don't understand the point of JitterFraction.

We may want to jitter on NumNodes as a server outage could cause a stampede to the other servers. The more clients, the more we need to spread out the herd.

its meant to be a percentage (16 corresponds to 6.25%) that we want to apply as a wait time before retrying, with a random stagger added. So it will wait ~6.25% of the given overall holdout time before retrying.

also I think. this line should be lib.RandomStagger(c.config.RPCHoldTimeout/jitterFraction)

Not sure I follow the argument of using the number of clients as the fraction. That makes the percentage non deterministic, could be anywhere from 100% to a very small number.

@schmichael to add a bit more color to Preetha's response. The jitter allows you to get some retrying within an overall allowed window. So if you have 5 seconds for something to occur this jitter lets you try a few times within that window.

@preetapan Yep good catch. Both those were hardcoded to 5s before I plumbed the config and missed one. Not sure what you mean about the # of clients btw.

schmichael · 2018-01-19T01:11:15Z

client/client.go

+		if err != nil {
+			merr.Errors = append(merr.Errors, fmt.Errorf("Server at address %s failed ping: %v", addr, err))
+			continue
+		} else if !ok {


This will never be hit as ok is only false if err is non-nil.

I'd prefer only returning an error from Ping since a nil error implies ok=true and a non-file error ok=false

schmichael · 2018-01-19T01:16:18Z

client/client.go

 	}
-	if len(servers) == 0 {
+	if len(nomadServers) == 0 {
 		return fmt.Errorf("server returned no valid servers")


I never realized it before but this is a rrrreally weird error.

Improved it

schmichael

That manager.go code is my nemesis. I think I finally understand the point of the locks+atomics, but I hope we never have to touch it again as it's extremely fragile in its current state.

schmichael · 2018-01-19T22:37:48Z

client/servers/manager.go

+//
+// NOTE(sean@): We are explicitly relying on the fact that serverList will
+// be copied onto the stack.  Please keep this structure light.
+type serverList struct {


The comments are confusing and possibly incorrect. I don't believe it matters whether a serverList or *serverList is stored in atomic.Value. Even though every caller to getServerList will receive a unique copy of serverList and the slice header, the slice's backing array containing the Server pointers will be shared among all callers until a cycleServer is called which creates a new slice/backing-array or SetServers is called.

If cycleServer modified the backing area in place (which would be an equally valid looking implementation and avoid a heap allocation), it would create race conditions!

schmichael · 2018-01-19T22:38:26Z

client/servers/manager.go

+type Manager struct {
+	// listValue manages the atomic load/store of a Manager's serverList
+	listValue atomic.Value
+	listLock  sync.Mutex


It is very difficult to figure out the proper way to use these. There are no tests that exercise concurrent access to this struct, so I couldn't use -race to detect races. However, I think it all checks out? (Note I ignored AddServer and RemoverServer in my analysis/review because they're not used.)

schmichael · 2018-01-19T23:26:17Z

client/servers/manager.go

+	m.connPoolPinger = connPoolPinger // can't pass *Nomad.ConnPool: import cycle
+	m.rebalanceTimer = time.NewTimer(clientRPCMinReuseDuration)
+	m.shutdownCh = shutdownCh
+	atomic.StoreInt32(&m.offline, 1)


No need to use an atomic store in a New func since nothing can be accessing this concurrently (but as mentioned later I think this field can be removed outright).

schmichael · 2018-01-19T23:34:50Z

client/servers/manager.go

+			m.refreshServerRebalanceTimer()
+
+		case <-m.shutdownCh:
+			m.logger.Printf("[INFO] manager: shutting down")


DEBUG maybe? Seems like this won't mean anything to end users as "managers" is awfully generic.

schmichael · 2018-01-19T23:39:53Z

client/servers/manager.go

+
+func (m *Manager) SetServers(servers []*Server) {
+	// Hot Path
+	if len(servers) == 0 {


The above comment is incorrect and this code is unreachable in the current implementation as we always check for len(servers) == 0 and return an error before calling SetServers.

Enforcing an invariant that the server list can never be cleared seems ok, but we should probably mention that in a comment on the method itself in case someone changes code at the callsites without realizing this internal behavior.

schmichael · 2018-01-20T01:27:38Z

client/servers/manager.go

+}
+
+// saveServerList is a convenience method which hides the locking semantics
+// of atomic.Value from the caller.


"hides the locking semantics of atomic.Value" is a weird phrase, but whatever. getServerList hides the type assertion nicely, and saveServerList makes a good companion api.

schmichael · 2018-01-20T01:30:04Z

client/servers/manager.go

+}
+
+// NumServers takes out an internal "read lock" and returns the number of
+// servers.  numServers includes both healthy and unhealthy servers.


NumServers returns the total number of known servers whether healthy or not.

schmichael · 2018-01-20T01:30:43Z

client/servers/manager.go

+}
+
+// GetServers returns a copy of the current list of servers.
+func (m *Manager) GetServers() []*Server {


Remove or move to tests where it's used? (See NotifyFailedServer comment)

Used by client

schmichael · 2018-01-20T01:33:33Z

client/servers/manager.go

+
+func (s *Server) String() string {
+	if s.addr == "" {
+		s.addr = s.Addr.String()


Racy. Either remove memoization or add a lock.

schmichael · 2018-01-20T01:38:51Z

client/servers/manager.go

+// the receiver's serverList. Returns false if the first server does not exist
+// in the list. Newly added servers are appended to the list and other missing
+// servers are removed from the list.
+func (m *Manager) reconcileServerList(l *serverList) bool {


I do not understand this method at all. The server list we pass in is a potentially shuffled version of the one we're reconciling against here (except for a small window where SetServers could have overwritten one). I think maybe it was useful in Consul which also reconciles servers with Serf? That would make sense.

I'm fairly certain this entire method can be removed and replaced by a call directly to m.saveServerList(l) above.

schmichael

A little concerned about that mutex copy but otherwise looks 💯

schmichael · 2018-01-25T17:37:31Z

client/servers/manager.go

+	// Addr is the resolved address of the server
+	Addr net.Addr
+	addr string
+	sync.Mutex


What if we just used a NewServer func and always initialized the addr? I'm worried about what the Server copy in Manager.GetServers does to the Mutex. Alternatively GetServers could just manually copy Addr & DC into a new struct instead of copying the whole struct.

@schmichael The two mutexes were put in place originally because Serf was handling the add/remove notifications and populating/replacing the server list independently of the server lists use via the RPC pool. Basically: producer == serf/consul discovery, consumer == rpc pool. This may be out of date, but that was the history.

schmichael · 2018-01-25T17:39:55Z

client/servers/manager.go

+
+	if !foundHealthyServer {
+		m.logger.Printf("[DEBUG] manager: No healthy servers during rebalance, aborting")
+		return


Don't need this return and the word aborting in the log line is a bit misleading. We're not actually aborting anything here, we just couldn't find a healthy server while rebalancing.

github-actions · 2023-03-13T02:15:05Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

dadgar force-pushed the f-sticky-clients branch from 0f7a941 to f27a7da Compare January 11, 2018 01:04

schmichael reviewed Jan 19, 2018

View reviewed changes

schmichael requested changes Jan 20, 2018

View reviewed changes

dadgar added 6 commits January 24, 2018 21:19

use server manager

a8e4e8a

SetServer command actually returns an error if given an invalid server

a631a64

Change defaults for min use duration

f65a4f2

Plumb config

1ee3ba6

initial round of comment review

c9c1b8f

Unjankify the pkg

c55e030

dadgar force-pushed the f-sticky-clients branch from a231d3e to c55e030 Compare January 25, 2018 05:22

schmichael approved these changes Jan 25, 2018

View reviewed changes

dadgar added 5 commits January 25, 2018 09:48

Fix lint/comments

6fb4870

Fix manager tests and make testagent recover from port conflicts

b282d10

fix lint

b6946b5

Remove circular dependency

8d437cf

Use in-mem rpc

5917bd9

dadgar merged commit 47d8fc5 into f-tunnel Jan 26, 2018

dadgar deleted the f-sticky-clients branch January 26, 2018 21:30

github-actions bot locked as resolved and limited conversation to collaborators Mar 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Client RPCs are sticky to server #3738

Client RPCs are sticky to server #3738

dadgar commented Jan 10, 2018

schmichael Jan 19, 2018

preetapan Jan 23, 2018

dadgar Jan 25, 2018

schmichael Jan 19, 2018

dadgar Jan 25, 2018

schmichael Jan 19, 2018

dadgar Jan 25, 2018

schmichael left a comment

schmichael Jan 19, 2018

schmichael Jan 19, 2018

schmichael Jan 19, 2018

dadgar Jan 25, 2018

schmichael Jan 19, 2018

dadgar Jan 25, 2018

schmichael Jan 19, 2018

schmichael Jan 20, 2018

schmichael Jan 20, 2018

dadgar Jan 25, 2018

schmichael Jan 20, 2018

dadgar Jan 25, 2018

schmichael Jan 20, 2018

dadgar Jan 25, 2018

schmichael Jan 20, 2018

dadgar Jan 25, 2018

schmichael left a comment

schmichael Jan 25, 2018

sean- Jan 25, 2018

schmichael Jan 25, 2018

github-actions bot commented Mar 13, 2023

Client RPCs are sticky to server #3738

Client RPCs are sticky to server #3738

Conversation

dadgar commented Jan 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schmichael left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schmichael left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 13, 2023