Fix for etcd client oneshot cluster member cycling

[rrati]

Backport of etcd-io/etcd#8519

[rrati]

/assign @deads2k

[deads2k]

@rrati the pick was clean?

[rrati]

@deads2k More or less. The changes to the _test file aren't included because this version of etcd didn't have the tests. The changes to the client code applied cleanly.

[deads2k]

lgtm

Do you have an issue or bug to tie to this?

[rrati]

A BZ, but no github issue

[liggitt]

remind me how we're managing picks for non-kube repos? also, an issue to track bumping to a release that includes this would be good (and an issue for kube, which is impacted the same way)

[liggitt]

/lgtm

[openshift-merge-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, rrati

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• OWNERS [liggitt]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[openshift-merge-robot]

Automatic merge from submit-queue

[sttts]

Do we have a backport of this into origin master? The upstream PR only merged into etcd 3.3+.

[liggitt]

it was backported into etcd 3.2.8 in etcd-io/etcd@15e9510 and picked up in 3.7+ in 6032f62#diff-1c6011bace39f9ad159c10fa9a674a3a

[mfojtik]

@liggitt how about 3.2.16 we have in 3.9 ?

[liggitt]

yes, >= 3.2.8

verified this fix is in master:

origin/vendor/github.com/coreos/etcd/client/client.go

Lines 375 to 398 in 1e041c8

	} else if resp.StatusCode/100 == 5 {
	switch resp.StatusCode {
	case http.StatusInternalServerError, http.StatusServiceUnavailable:
	// TODO: make sure this is a no leader response
	cerr.Errors = append(cerr.Errors, fmt.Errorf("client: etcd member %s has no leader", eps[k].String()))
	default:
	cerr.Errors = append(cerr.Errors, fmt.Errorf("client: etcd member %s returns server error [%s]", eps[k].String(), http.StatusText(resp.StatusCode)))
	}
	err = cerr.Errors[0]
	}
	if err != nil {
	if !isOneShot {
	continue
	}
	c.Lock()
	c.pinned = (k + 1) % leps
	c.Unlock()
	return nil, nil, err
	}
	if k != pinned {
	c.Lock()
	c.pinned = k
	c.Unlock()
	}

3.9:

origin/vendor/github.com/coreos/etcd/client/client.go

Lines 375 to 398 in 6d21b7d

	} else if resp.StatusCode/100 == 5 {
	switch resp.StatusCode {
	case http.StatusInternalServerError, http.StatusServiceUnavailable:
	// TODO: make sure this is a no leader response
	cerr.Errors = append(cerr.Errors, fmt.Errorf("client: etcd member %s has no leader", eps[k].String()))
	default:
	cerr.Errors = append(cerr.Errors, fmt.Errorf("client: etcd member %s returns server error [%s]", eps[k].String(), http.StatusText(resp.StatusCode)))
	}
	err = cerr.Errors[0]
	}
	if err != nil {
	if !isOneShot {
	continue
	}
	c.Lock()
	c.pinned = (k + 1) % leps
	c.Unlock()
	return nil, nil, err
	}
	if k != pinned {
	c.Lock()
	c.pinned = k
	c.Unlock()
	}

3.8:

origin/vendor/github.com/coreos/etcd/client/client.go

Lines 375 to 398 in ff72b3f

	} else if resp.StatusCode/100 == 5 {
	switch resp.StatusCode {
	case http.StatusInternalServerError, http.StatusServiceUnavailable:
	// TODO: make sure this is a no leader response
	cerr.Errors = append(cerr.Errors, fmt.Errorf("client: etcd member %s has no leader", eps[k].String()))
	default:
	cerr.Errors = append(cerr.Errors, fmt.Errorf("client: etcd member %s returns server error [%s]", eps[k].String(), http.StatusText(resp.StatusCode)))
	}
	err = cerr.Errors[0]
	}
	if err != nil {
	if !isOneShot {
	continue
	}
	c.Lock()
	c.pinned = (k + 1) % leps
	c.Unlock()
	return nil, nil, err
	}
	if k != pinned {
	c.Lock()
	c.pinned = k
	c.Unlock()
	}

3.7:

origin/vendor/github.com/coreos/etcd/client/client.go

Lines 375 to 398 in a8deba5

	} else if resp.StatusCode/100 == 5 {
	switch resp.StatusCode {
	case http.StatusInternalServerError, http.StatusServiceUnavailable:
	// TODO: make sure this is a no leader response
	cerr.Errors = append(cerr.Errors, fmt.Errorf("client: etcd member %s has no leader", eps[k].String()))
	default:
	cerr.Errors = append(cerr.Errors, fmt.Errorf("client: etcd member %s returns server error [%s]", eps[k].String(), http.StatusText(resp.StatusCode)))
	}
	err = cerr.Errors[0]
	}
	if err != nil {
	if !isOneShot {
	continue
	}
	c.Lock()
	c.pinned = (k + 1) % leps
	c.Unlock()
	return nil, nil, err
	}
	if k != pinned {
	c.Lock()
	c.pinned = k
	c.Unlock()
	}

[sttts]

@liggitt without any deeper knowledge of the client code: all your references point to the etcd2 client, not the clientv3 package.

[liggitt]

hmm, true. I wonder if there was a similar issue with the etcd v3 client

[sttts]

But even in v3, there is a 10 second context passed to the dial func. How can that block for 15 minutes? (@mfojtik noticed that)

[sttts]

With persistent grpc connections we certainly do not talk about a Dial call here.