Memberlist gossip implementation

[jacksontj]

As discussed in office hours last wee, this PR implements #7436 to swap gossip from mesh to memberlist (primarily due to scale issues with mesh -- #7427).

At this point it is functional (with some manual hacking to get the one dep PR in) -- and I have tested this on a cluster of ~800 nodes with no noticeable CPU usage increase from a ~5 node cluster.

TODO:

• AddPeer to alertmanager's cluster lib: AddPeer method to add additional peers after initial startup prometheus/alertmanager#2019 and https://github.com/jacksontj/memberlistmesh
• Wire up config for gossip through cluster spec
• Add upgrade/downgrade docs
• Update development docs on gossip
• final cleanup (mising doc strings, etc.)

cc @justinsb

[k8s-ci-robot]

Hi @jacksontj. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jacksontj]

^^ That is an implementation of the flags -- so barring the upstream PR and any review notes this should be ready to go (I'll do another round of testing on my dev cluster to verify)

[jacksontj]

Upstream AM decided that I should just fork the library, which I have done. Since I had the fork already I changed some of the deps to reduce the vendor changes in this PR as well.

[rifelpet]

/ok-to-test

[jacksontj]

I have identified most of the issues, I have another branch (based on 1.12) which is working -- I have to port those changes forward (should be able to do that monday).

In those changes I have to configure the primary+secondary gossip for both protokube and dns-controller. Unfortunately dns-controller is assuming the port for the gossip seed -- so right now I have it using the protokube gossip config to determine the port -- which means you have to not apply the change from 2 -> 1 gossip on the dns-controller until nothing is remaining on the old gossip. As of now the controller isn't auto-deploying on kops edit anyways -- so maybe this is okay? If not then I'll need to add a Seed field to the GossipConfig struct -- which would let you set the seed port separately on the dns-controller.

Thoughts?

[jacksontj]

Just updated this PR with the final changes required to get it working on my test cluster.

The upgrade path is:

1. double gossip config on master
2. double gossip config on dns-controller
3. single new gossip config on nodes (roll whole cluster)
4. single new gossip config on masters
5. single new gossip config on dns-controller

Last questions:

• dns-controller apply is still manual
• dns-controller -- seed URLs are determined from the other gossip config, so if this starts auto-applying from kops we'll need to add more options to define the seed for the controller (right now its hard-coded).

[jacksontj]

@justinsb friendly reminder bump :)

[justinsb]

/retest

At least the bazel logs were missing, indicating a bad run.

[justinsb]

OK - as we want to get this into 1.16 (I believe), going to merge!

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jacksontj, justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment