Etcd reboot causes RSS to double

[davissp14]

Version: 3.1.5

I am doing some fail testing with Etcd and noticed that RSS appears to double in the event of a reboot and seems settle after 5 minutes or so. My initial assumption was a process was being forked and the RSS doubling was just due to copy-on-write. Is my assumption correct? Or is there something else at play here?

This isn't a great image i know, but the red line is RSS and orange is Cache.

My second question is what metrics should we be listening on scaling purposes? I initially thought that RSS would be good enough, but after a lot of testing it seems RSS is too spiky to be a reliable metric to listen on. I see quite a few situations where RSS quickly spikes to memory capacity then drops quickly as it sends things to SWAP.

Thanks

[gyuho]

quite a few situations where RSS quickly spikes to memory capacity

How was etcd being stressed? Any details on the workloads?
Just curious, because I've never seen such spikes (c.f. https://github.com/coreos/dbtester/tree/master/test-results/2017Q1-01-etcd-zookeeper-consul#write-1m-keys-256-byte-key-1kb-value-value-clients-1-to-1000)

[davissp14]

We are using the Etcd benchmark tool.

./benchmark --endpoints="$ENDPOINTS" --conns=100 --clients=1000 put --key-size=100 --key-space-size=10 --sequential-keys --total=2000000 --val-size=20

[davissp14]

Here's another less extreme example of RSS spiking.

[gyuho]

@davissp14 Thanks for that command. Let me try to profile that workloads.
Is that 3-node cluster?

[davissp14]

Is that 3-node cluster?

Yep!

[gyuho]

(pprof) list unsafeRange
Total: 161.04MB
ROUTINE ======================== github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend.unsafeRange in /home/gyuho/go/src/github.com/coreos/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/backend/batch_tx.go
  103.48MB   103.48MB (flat, cum) 64.26% of Total
         .          .    102:	if limit <= 0 {
         .          .    103:		limit = math.MaxInt64
         .          .    104:	}
         .          .    105:	c := bucket.Cursor()
         .          .    106:	for ck, cv := c.Seek(key); ck != nil && bytes.Compare(ck, endKey) < 0; ck, cv = c.Next() {
   51.74MB    51.74MB    107:		vs = append(vs, cv)
   51.74MB    51.74MB    108:		keys = append(keys, ck)
         .          .    109:		if limit == int64(len(keys)) {
         .          .    110:			break
         .          .    111:		}
         .          .    112:	}
         .          .    113:	return keys, vs, nil
(pprof)

Confirmed that most memory usage comes from unsafeRange when restoring boltdb kvstore (during reboot). Think it's just Go GC? After 5-min, this gets reclaimed and back to normal memory usage (while rebooting, spike up to 700MB, and back to 400MB after 5-min).

/cc @heyitsanthony @xiang90
Any thoughts?

[xiang90]

yea. it is just go GC.

[heyitsanthony]

There's a TODO in kvstore.go's restore code about loading all the keys into memory, which is probably why it's eating so much memory on boot.

[xiang90]

@heyitsanthony yea... i am not sure if this worth to fix. this is the first time someone bring it up.

[davissp14]

For additional context I am working on trying to figure out a scaling scheme for Etcd on Compose. Right now in my testing, nodes are being prematurely scaled due to the RSS spikes during reboots, which isn't ideal. If it didn't take 5 minutes to settle it wouldn't be such an issue, but right now I don't have a very good scaling story using RSS. Any other suggestions on how to deal with this?