Implement Gateway status updates #502

mangelajo · 2020-04-13T17:05:55Z

No description provided.

submariner-bot · 2020-04-13T17:06:08Z

🤖 Created branch: z_pr502/mangelajo/implement-status-updates

mangelajo · 2020-04-13T17:08:01Z

main.go

-		if err != nil {
-			klog.Fatalf("Fatal error occurred creating engine: %v", err)
-		}
+	cableEngine, err := cableengine.NewEngine(localSubnets, localCluster, localEndpoint, submarinerClient,


We move this creation ahead in time, so ... the loop reconciling the gateway status will also
work for the PASSIVE gateway.

The driver initialization does not happen until "StartEngine" though.

submariner-bot · 2020-04-13T19:39:35Z

🤖 Updated branch: z_pr502/mangelajo/implement-status-updates

submariner-bot · 2020-04-13T19:41:26Z

🤖 the base of this PR has been updated to master
please rebase this branch and remove #502 related commits

submariner-bot · 2020-04-14T06:40:15Z

🤖 Updated branch: z_pr502/mangelajo/implement-status-updates

submariner-bot · 2020-04-14T17:22:12Z

🤖 Updated branch: z_pr502/mangelajo/implement-status-updates

submariner-bot · 2020-04-15T12:42:01Z

🤖 Updated branch: z_pr502/mangelajo/implement-status-updates

submariner-bot · 2020-04-15T13:46:31Z

🤖 Updated branch: z_pr502/mangelajo/implement-status-updates

submariner-bot · 2020-04-15T14:57:50Z

🤖 Updated branch: z_pr502/mangelajo/implement-status-updates

submariner-bot · 2020-04-15T15:25:09Z

🤖 Updated branch: z_pr502/mangelajo/implement-status-updates

mangelajo · 2020-04-15T15:25:40Z

ok, removed the workaround after #506 was merged

mangelajo · 2020-04-16T16:47:56Z

E2E tests being handled here: #508

pkg/cableengine/syncgatewaystatus.go

sridhargaddam

Some small nits, otherwise it looks good. I even tried it :)
Thanks for working on this @mangelajo

pkg/cableengine/syncgatewaystatus.go

pkg/cableengine/cableengine.go

mangelajo

I handled those comments (need to update the patch) Also, I will move this gateway checking together with the redundancy part which is almost the same.

test/e2e/framework/framework.go

mangelajo · 2020-04-22T09:32:53Z

test/e2e/framework/framework.go

+
+func (f *Framework) AwaitForGatewayGone(cluster framework.ClusterIndex, name string) {
+	gwClient := SubmarinerClients[cluster].SubmarinerV1().Gateways(framework.TestContext.SubmarinerNamespace)
+	framework.AwaitUntil("await gateway gone",


test/e2e/framework/framework.go

test/e2e/gateway/gateway_reporting.go

Related-Issue: submariner-io#308

The Host field was redudant with the Gateway name, also the localEndpoint information was missing when looking at the Gateway via describe for more context. Modified as discussed over slack channel.

The active gateway handles the role of cleaning up stale gateway entries, if a gateway has not reported for x3 the reporting interval the entry will be deleted. We handle this via an annotation since k8s API does not include modification timestamp in the object metadata.

In this way, when we use some of those also as labels it's less weird since the unwritten norm in k8s is to use lowcase values in the labels instead of upcase.

submariner-bot · 2020-04-22T10:37:23Z

🤖 Updated branch: z_pr502/mangelajo/implement-status-updates

pkg/apis/submariner.io/v1/types.go

…cy tests

submariner-bot · 2020-04-22T10:57:47Z

🤖 Updated branch: z_pr502/mangelajo/implement-status-updates

mangelajo · 2020-04-22T11:56:14Z

Unrelated globalnet failure, I re-ran that test.

mangelajo · 2020-04-22T15:49:30Z

Thanks for the review @skitt

skitt · 2020-04-22T15:55:22Z

Thanks for the review @skitt

Not much of a review, but it works for me, and my concerns have been raised and addressed already!

pkg/cableengine/syncer/syncer.go

tpantelis · 2020-04-22T16:16:45Z

test/e2e/framework/framework.go

+	return gw.(*submarinerv1.Gateway)
+}
+
+func (f *Framework) AwaitForGatewayRemoved(cluster framework.ClusterIndex, name string) {


Suggested change

func (f *Framework) AwaitForGatewayRemoved(cluster framework.ClusterIndex, name string) {

func (f *Framework) AwaitGatewayRemoved(cluster framework.ClusterIndex, name string) {

tpantelis · 2020-04-22T16:19:54Z

test/e2e/redundancy/gateway_failover.go

+func testBasicGatewayReporting(f *subFramework.Framework) {
+	clusterAName := framework.TestContext.ClusterIDs[framework.ClusterA]
+
+	By(fmt.Sprintf("Ensuring that one gateway reports as active %q", clusterAName))


Suggested change

By(fmt.Sprintf("Ensuring that one gateway reports as active %q", clusterAName))

By(fmt.Sprintf("Ensuring that one gateway reports as active on %q", clusterAName))

tpantelis · 2020-04-22T16:24:34Z

test/e2e/redundancy/gateway_failover.go

+
+	By(fmt.Sprintf("Ensuring that one gateway reports as active %q", clusterAName))
+	activeGateways := f.GetGatewaysWithHAStatus(subv1.HAStatusActive)
+	Expect(activeGateways).To(HaveLen(1))


I think this should be wrapped with framework.AwaitUntil to avoid intermittent failures if the active was not yet running or written it's status. Perhaps AwaitActiveGateway.

Actually there's already AwaitGatewayWithStatus although it takes the gateway name which could be obtained by finding the active gateway node. But I realize we're actually duplicating what testEnginePodRestartScenario does so why do we need this test?

For independence, I wanted an indication that the basic reporting is working if the dataplane is broken. When we have stats then we could identify if it's the dataplane breaking, or the reporting breaking. The reporting would always affect the dataplane tests though.

I think we should do AwaitActiveGateway at least to avoid possible intermittent failures as you said.

Oh AwaitGateway does not work, because we don't know the name, but we know that we want one. I will add another function that works with any.

tpantelis · 2020-04-22T16:33:21Z

test/e2e/redundancy/gateway_failover.go

+	activeGateways := f.GetGatewaysWithHAStatus(subv1.HAStatusActive)
+	Expect(activeGateways).To(HaveLen(1))
+
+	gw := activeGateways[0]


Suggested change

gw := activeGateways[0]

activeGateway := activeGateways[0]

For consistency ...

tpantelis · 2020-04-22T16:38:23Z

test/e2e/redundancy/gateway_failover.go

+	By(fmt.Sprintf("Ensuring that the gateway reports as active %q", clusterAName))
+	activeGateways := f.GetGatewaysWithHAStatus(subv1.HAStatusActive)
+	Expect(activeGateways).To(HaveLen(1))
+


We should verify that the found active Gateway's name matches that of the gateway node found above.

Or better yet, call AwaitGatewayWithStatus.

But I want to verify there's only "1". I can't do that with Await, I can call Await just before this, which would make sense, also would remove any glitches if it's starts as passive and then moves to active.

Ok, I did it in a slightly different way, let me know what you think

tpantelis · 2020-04-22T16:41:22Z

test/e2e/redundancy/gateway_failover.go

+	gw := activeGateways[0]
+	By(fmt.Sprintf("Ensuring that the gateway %q is reporting connections", gw.Name))
+	Expect(gw.Status.Connections).NotTo(BeEmpty())
+	Expect(gw.Status.Connections[0].Status).To(Equal(subv1.Connected))


This could cause an intermittent failure if active not yet reporting connections - it seems this should call AwaitGatewayFullyConnected.

tpantelis · 2020-04-22T16:49:20Z

test/e2e/redundancy/gateway_failover.go

 	By(fmt.Sprintf("Deleting submariner engine pod %q", enginePod.Name))
 	f.DeletePod(framework.ClusterA, enginePod.Name, framework.TestContext.SubmarinerNamespace)

 	newEnginePod := f.AwaitSubmarinerEnginePod(framework.ClusterA)
 	By(fmt.Sprintf("Found new submariner engine pod %q", newEnginePod.Name))

+	By(fmt.Sprintf("Waiting for the gateway to be up and connected %q", newEnginePod.Name))
+	f.AwaitGatewayFullyConnected(framework.ClusterA, gw.Name)


This could report fully connected as the last state from the previous engine pod. Perhaps we should move this after the connectivity tests below - that way we know the gateway is re-connected and should be reporting the new state.

Oh, that is true, I was actually wanting to make sure we had connections before starting the dataplane tests.

This will work better/faster when we I implement the cleanup of state at pod exit, I didn't dig into that, but I will do right after 0.3.0-rc1
#518

yeah I was thinking that too - delete on engine shutdown.

I added a DeleteGateway function for now, right after the pod deletion, which will ensure that at least the gateway status is re-created, and that it eventually moves to "fully connected"

When I do #518 then we remove this, and probably do a different check (I guess an informer to make sure it's deleted and re-created).

tpantelis · 2020-04-22T16:51:55Z

test/e2e/redundancy/gateway_failover.go

+
+	gwActive := f.AwaitGatewayWithStatus(framework.ClusterA, initialGatewayNode.Name, subv1.HAStatusActive)
+	gwPassive := f.AwaitGatewayWithStatus(framework.ClusterA, initialNonGatewayNode.Name, subv1.HAStatusPassive)
+	Expect(gwActive.Status.Connections).ToNot(BeEmpty(), "The active gateway must have active connections")


As above, it would be safest to call AwaitGatewayFullyConnected for the active gateway.

mangelajo · 2020-04-22T17:23:06Z

@tpantelis I opened this to handle comments as a follow-up #519 , and #518 if that's ok

Currently lighthouse pushed a new ":latest" image, and this will fail when I submit any new changes.

In such way we are able to release 0.3.0-rc1 and at the same time test the changes you're proposing here (changes that make sense and I agree with).

mangelajo · 2020-04-22T17:30:16Z

Wait, may be there's no such chicken-egg, since we don't depend on lighthouse here.. ':D
testing locally and I will send a PR update if it works.

mangelajo · 2020-04-22T20:07:26Z

The technical difficulty with the lighthouse images does not exist, so it should be fine :) anyway we are talking on slack about merging this and do the follow up PR so we can cut RC1 now

tpantelis · 2020-04-22T20:13:08Z

My comments aren't critical - let's merge this and address in the follow-ups @mangelajo created.

submariner-bot · 2020-04-22T20:15:49Z

🤖 Closed branches: [z_pr502/mangelajo/implement-status-updates]

mangelajo changed the base branch from master to z_pr501/skitt/expose-status-types April 13, 2020 17:06

mangelajo requested review from tpantelis, skitt, sridhargaddam, dfarrell07 and vthapar April 13, 2020 17:06

mangelajo commented Apr 13, 2020

View reviewed changes

mangelajo force-pushed the implement-status-updates branch from 8e72eb5 to c2a26c0 Compare April 13, 2020 19:39

mangelajo mentioned this pull request Apr 13, 2020

Move connection types to the submariner.io API #501

Merged

submariner-bot changed the base branch from z_pr501/skitt/expose-status-types to master April 13, 2020 19:41

mangelajo force-pushed the implement-status-updates branch from c2a26c0 to 8fe88c4 Compare April 14, 2020 06:40

mangelajo changed the base branch from master to release-0.1 April 14, 2020 07:12

mangelajo changed the base branch from release-0.1 to master April 14, 2020 07:12

mangelajo force-pushed the implement-status-updates branch from 8fe88c4 to dde698a Compare April 14, 2020 17:21

mangelajo changed the title ~~Implement status updates~~ Implement Gateway status updates Apr 14, 2020

mangelajo force-pushed the implement-status-updates branch from 40c5a05 to d673770 Compare April 15, 2020 13:46

mangelajo force-pushed the implement-status-updates branch from e0722fa to d2fd46d Compare April 15, 2020 15:24

sridhargaddam reviewed Apr 17, 2020

View reviewed changes

pkg/cableengine/syncgatewaystatus.go Outdated Show resolved Hide resolved

pkg/cableengine/syncgatewaystatus.go Outdated Show resolved Hide resolved

sridhargaddam requested changes Apr 17, 2020

View reviewed changes

pkg/cableengine/syncgatewaystatus.go Outdated Show resolved Hide resolved

pkg/cableengine/syncgatewaystatus.go Outdated Show resolved Hide resolved

pkg/cableengine/syncgatewaystatus.go Outdated Show resolved Hide resolved

pkg/cableengine/syncgatewaystatus.go Outdated Show resolved Hide resolved

mangelajo commented Apr 21, 2020

View reviewed changes

pkg/cableengine/cableengine.go Show resolved Hide resolved

mangelajo commented Apr 21, 2020

View reviewed changes

pkg/cableengine/cableengine.go Show resolved Hide resolved

mangelajo commented Apr 22, 2020

View reviewed changes

mangelajo added 8 commits April 22, 2020 12:36

Synchronize Gateway status into the local k8s API

8d55d38

Related-Issue: submariner-io#308

Remove the Host field from Gateway and add localEndpoint

85091dc

The Host field was redudant with the Gateway name, also the localEndpoint information was missing when looking at the Gateway via describe for more context. Modified as discussed over slack channel.

Normalize API values to lowcase

808189a

In this way, when we use some of those also as labels it's less weird since the unwritten norm in k8s is to use lowcase values in the labels instead of upcase.

Apply sridhar's review comments

e3006e6

Gateway reporting API E2E testing

edbc7d1

Separate into a separate syncer object

dfe2d07

Handle comments by tpantelis

25b5afd

mangelajo force-pushed the implement-status-updates branch from d00210a to c59ecfc Compare April 22, 2020 10:37

mangelajo requested a review from tpantelis April 22, 2020 10:48

mangelajo commented Apr 22, 2020

View reviewed changes

pkg/apis/submariner.io/v1/types.go Show resolved Hide resolved

Handle tpantelis comments including the move of e2e logic to redundan…

4288cf7

…cy tests

mangelajo force-pushed the implement-status-updates branch from c59ecfc to 4288cf7 Compare April 22, 2020 10:57

skitt approved these changes Apr 22, 2020

View reviewed changes

tpantelis reviewed Apr 22, 2020

View reviewed changes

tpantelis approved these changes Apr 22, 2020

View reviewed changes

tpantelis merged commit 4bec940 into submariner-io:master Apr 22, 2020

	func (f *Framework) AwaitForGatewayRemoved(cluster framework.ClusterIndex, name string) {
	func (f *Framework) AwaitGatewayRemoved(cluster framework.ClusterIndex, name string) {

	By(fmt.Sprintf("Ensuring that one gateway reports as active %q", clusterAName))
	By(fmt.Sprintf("Ensuring that one gateway reports as active on %q", clusterAName))

Implement Gateway status updates #502

Implement Gateway status updates #502

Conversation

mangelajo commented Apr 13, 2020 • edited Loading

submariner-bot commented Apr 13, 2020

Choose a reason for hiding this comment

submariner-bot commented Apr 13, 2020

submariner-bot commented Apr 13, 2020

submariner-bot commented Apr 14, 2020

submariner-bot commented Apr 14, 2020

submariner-bot commented Apr 15, 2020

submariner-bot commented Apr 15, 2020

submariner-bot commented Apr 15, 2020

submariner-bot commented Apr 15, 2020

mangelajo commented Apr 15, 2020

mangelajo commented Apr 16, 2020

sridhargaddam left a comment

Choose a reason for hiding this comment

mangelajo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

submariner-bot commented Apr 22, 2020

submariner-bot commented Apr 22, 2020

mangelajo commented Apr 22, 2020

mangelajo commented Apr 22, 2020

skitt commented Apr 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mangelajo commented Apr 22, 2020 • edited Loading

mangelajo commented Apr 22, 2020

mangelajo commented Apr 22, 2020

tpantelis commented Apr 22, 2020

submariner-bot commented Apr 22, 2020

mangelajo commented Apr 13, 2020 •

edited

Loading

mangelajo commented Apr 22, 2020 •

edited

Loading