Contiv network is not deleted when docker network is deleted #1029

kahou82 · 2017-10-22T08:13:59Z

Steps to reproduce:

Create a docker network with docker network CLI
Ensure docker network and contiv network are created (docker network ls and netctl network ls)
Remove docker network.
Check docker network ls, docker network is deleted
Check netctl networ ls, contiv network is still test.

The root cause is due to the dockernetworkstate mapping is planned to delete after
contiv network is delete. However, the determination of contiv network deletion is
determined by the existance of the dockernetworkstate. Thefore, if the state is still
there, the contiv network will be unable to delete.

The fix is to delete the dockernetworkstate before proceed to contiv network deletion

Also refactor docker plugin update by separating the plugin and ipam driver

Signed-off-by: Kahou Lei kalei@cisco.com

Steps to reproduce: 1. Create a docker network with docker network CLI 2. Ensure docker network and contiv network are created (docker network ls and netctl network ls) 3. Remove docker network. 4. Check docker network ls, docker network is deleted 5. Check netctl networ ls, contiv network is still test. The root cause is due to the dockernetworkstate mapping is planned to delete after contiv network is delete. However, the determination of contiv network deletion is determined by the existance of the dockernetworkstate. Thefore, if the state is still there, the contiv network will be unable to delete. The fix is to delete the dockernetworkstate before proceed to contiv network deletion Signed-off-by: Kahou Lei kalei@cisco.com

kahou82 · 2017-10-22T08:15:00Z

DO NOT MERGE: I am writing unit test.

dseevr · 2017-10-22T08:24:40Z

mgmtfn/dockplugin/netDriver.go

+		// delete the dnet oper state
+		err = docknet.DeleteDockNetState(dnet.TenantName, dnet.NetworkName, dnet.ServiceName)
+		if err != nil {
+			log.Errorf("Couldn't delete docknet for nwID %s", networkID)


Log the error here to aid with debugging, please:

log.Errorf("Couldn't delete docknet for nwID %s: %s", networkID, err)

rchirakk · 2017-10-23T22:29:25Z

PTAL @gkvijay ^^

kahou82 · 2017-10-24T00:27:08Z

build PR

…ble.

kahou82 · 2017-10-24T06:25:45Z

build PR

kahou82 · 2017-10-24T07:00:45Z

build PR

chrisplo · 2017-10-25T23:05:39Z

mgmtfn/dockplugin/netDriver.go

+			log.Errorf("Couldn't delete docknet for nwID %s: %s", networkID, err.Error())
+		}
+		log.Infof("Deleted docker network mapping for %v", networkID)
+	}


shouldn't we be failing if we get err in either of these two cases? under what conditions can docker and contiv networks be out of sync (when there is not a bug) . . . err==nil or not seems too generic, can we check for "not present" vs any other err e.g. "error talking to docker"?

chrisplo · 2017-10-25T23:08:54Z

mgmtfn/dockplugin/netDriver.go

-		}
-		log.Infof("Deleted docker network mapping for %v", networkID)
-	}
+


we deleted docker network, found a contiv netowrk, but failed to delete the contiv netowrk, should we restore the docker network? (i dunno, just asking) . . L771

There is no hook for me to do that at the moment to restore the network.

Don't think we must secretly revert what the user does (docker network rm in this case). If contiv network deletion fails, I think an error to the user is good instead of reverting what the user did.

chrisplo · 2017-10-25T23:09:51Z

netmaster/docknet/docknet.go

-	log.Infof("docker v2plugin name (%s) updated to %s", netDriverName, pluginName)
-	netDriverName = pluginName
-	ipamDriverName = pluginName
+func UpdatePluginName(netdriver string, ipamDriver string) {


Surprised go vet didn't complain about that

chrisplo · 2017-10-25T23:11:09Z

netmaster/docknet/docknet.go

+	log.Infof("docker v2plugin (%s) updated to %s and ipam (%s) updated to %s",
+		netDriverName, netdriver, ipamDriverName, ipamDriver)
+	netDriverName = netdriver
+	ipamDriverName = ipamDriver


this change is missing from the commit description

chrisplo · 2017-10-26T19:55:34Z

mgmtfn/dockplugin/netDriver_test.go

+	if delErr != nil {
+		t.Fatalf("Unable to delete docker network. Error: %v", delErr)
+		t.Fail()
+	}


shouldn't we check for the network to be absent from both docker and netctl here?

chrisplo · 2017-10-26T19:59:45Z

netmaster/docknet/docknet_test.go

+func TestUpdatePluginName(t *testing.T) {
+	expectNetDriver := "bridge"
+	expectIPAMDriver := "default"
+	UpdatePluginName("bridge", "default")


no docknet prefix?

chrisplo · 2017-10-26T20:01:13Z

netmaster/docknet/docknet_test.go

+	expectIPAMDriver := "default"
+	UpdatePluginName("bridge", "default")
+
+	if expectNetDriver != netDriverName {


how is netDriverName in scope?

Since this Go test file is in the docknet package (see top of file), everything in the docknet package is available here.

This var is defined in docknet.go here: https://github.com/kahou82/netplugin/blob/37a3b829c00226b8af742f80e6c46889c20fd4d2/netmaster/docknet/docknet.go#L44

chrisplo · 2017-10-26T20:01:22Z

netmaster/docknet/docknet_test.go

+		t.Fail()
+	}
+
+	if expectIPAMDriver != ipamDriverName {


https://github.com/kahou82/netplugin/blob/37a3b829c00226b8af742f80e6c46889c20fd4d2/netmaster/docknet/docknet.go#L45

chrisplo

my first go code review, maybe my concerns are invalid

kahou82 · 2017-10-28T08:37:56Z

build PR

kahou82 · 2017-10-28T14:34:58Z

build PR

kahou82 · 2017-10-29T00:09:12Z

@chrisplo can you review? It is one of the CVD blockers.

dseevr · 2017-10-29T06:07:16Z

.idea/vcs.xml

@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="UTF-8"?>


Please git rm this file (or the whole directory). It shouldn't be included in git. You could add it to .gitignore so it's not tracked

dseevr · 2017-10-29T06:08:32Z

mgmtfn/dockplugin/netDriver_test.go

-	instInfo := core.InstanceInfo{DbURL: "etcd://127.0.0.1:2379"}
+func initFakeStateDriver() {
+	instInfo := core.InstanceInfo{}
+	d, _ := utils.NewStateDriver("fakedriver", &instInfo)


Please check the error and assert that it is nil

(you never know!)

dseevr · 2017-10-29T06:09:58Z

mgmtfn/dockplugin/netDriver_test.go

+	initFakeStateDriver()
+
+	// Delete Docker network by using helper
+	delErr := deleteNetworkHelper("non_existing_network_id")


Errors can just be named "err" unless you legitimately need multiple error variables in the same scope at the same time

dseevr · 2017-10-29T06:12:11Z

mgmtfn/dockplugin/netDriver.go

+		// delete the dnet oper state
+		err = docknet.DeleteDockNetState(dnet.TenantName, dnet.NetworkName, dnet.ServiceName)
+		if err != nil {
+			log.Errorf("Couldn't delete docknet for nwID %s: %s", networkID, err.Error())


Here, I would do something like:

msg := fmt.Sprintf("my %s", "error message") log.Error(msg) return errors.New(msg)

Return as much useful info to the caller as possible

dseevr · 2017-10-29T06:12:18Z

mgmtfn/dockplugin/netDriver.go

+		}
+		log.Infof("Deleted docker network mapping for %v", networkID)
+	} else {
+		log.Errorf("Couldn't find Docker network %s: %s", networkID, err.Error())


dseevr · 2017-10-29T06:12:43Z

mgmtfn/dockplugin/netDriver_test.go

+// initStateDriver initialize etcd state driver
+func initFakeStateDriver() {
+	instInfo := core.InstanceInfo{}
+	d, _ := utils.NewStateDriver("fakedriver", &instInfo)


Check error here

kahou82 · 2017-10-29T06:34:49Z

build PR

kahou82 · 2017-10-29T13:49:55Z

build PR

rchirakk

LGTM other than minor comments

rchirakk · 2017-10-29T22:54:20Z

mgmtfn/dockplugin/netDriver.go

+		if err != nil {
+			msg := fmt.Sprintf("Could not delete docknet for nwID %s: %s", networkID, err.Error())
+			log.Errorf(msg)
+			return errors.New(msg)


another way to simplify,

err := fmt.Errorf() log.Error(err) return err

I am going to stick with the current implementation

rchirakk · 2017-10-29T23:02:11Z

mgmtfn/dockplugin/netDriver.go

+		}
+		log.Infof("Deleted docker network mapping for %v", networkID)
+	} else {
+		msg := fmt.Sprintf("Could not find Docker network %s: %s", networkID, err.Error())


fmt package takes care of calling Error()
fmt.Sprintf("xxx %s", err)

I am going to stick with the current implementation

i understand now that due to other bad behavior (not erroring in v2plugin mode when requesting to create the network) that we can get out of sync, I'm just looking for a ticket to block the netctl when in v2plugin mode, then we can clean up this lax behavior here

rchirakk · 2017-10-29T23:20:34Z

mgmtfn/dockplugin/netDriver.go

+	} else {
+		msg := fmt.Sprintf("Could not find Docker network %s: %s", networkID, err.Error())
+		log.Errorf(msg)
+		return errors.New(msg)


don't return, log error message & proceed to delete network.

vhosakot

Nice work!

chrisplo · 2017-10-30T17:39:16Z

mgmtfn/dockplugin/netDriver_test.go

+	if err != nil {
+		t.Fatalf("Unable to delete docker network. Error: %v", err)
+		t.Fail()
+	}


checking the return is non-error is good, but that didn't actually test that the network was deleted, can you check w/ docker/netctl for current status?

Take out return statement when docker network is not there.

kahou82 · 2017-10-30T18:00:32Z

build PR

chrisplo · 2017-10-30T20:29:08Z

mgmtfn/dockplugin/netDriver.go

-			return errors.New("Failed to delete network")
+			msg := fmt.Sprintf("Failed to delete network: %s", err.Error())
+			log.Errorf(msg)
+			return errors.New(msg)
 		}
 		log.Infof("Deleted contiv network %v", networkID)
 	} else {
 		log.Infof("Could not find contiv network %v", networkID)


We should still be bubbling up an error here (like bad request to user for deleting a network that didn't exist), but happy to see this as a new ticket

At the moment, I don't think return error here will bubbling up to the docker network create command. We have to have another ticket to investigate it.

concerns addressed or discussed and understood

kahou82 · 2017-10-30T22:50:03Z

build PR

kahou82 · 2017-10-31T18:49:32Z

Going to merge as @chrisplo and @rchirakk +1

dseevr reviewed Oct 22, 2017

View reviewed changes

Print out error

d0eddc9

Add unit test and also refactor some code to make the code more testa…

0f1772f

…ble.

Add more unit test and fix typos

37a3b82

gkvijay approved these changes Oct 25, 2017

View reviewed changes

chrisplo reviewed Oct 25, 2017

View reviewed changes

chrisplo reviewed Oct 26, 2017

View reviewed changes

chrisplo previously requested changes Oct 26, 2017

View reviewed changes

kahou82 added 2 commits October 27, 2017 18:41

Add more testing

5f77471

Add key value pair in fake state

f6e8f5b

dseevr reviewed Oct 29, 2017

View reviewed changes

Refactor base on comments

5661ee9

rchirakk reviewed Oct 29, 2017

View reviewed changes

vhosakot approved these changes Oct 29, 2017

View reviewed changes

chrisplo reviewed Oct 30, 2017

View reviewed changes

Add more check in unit test.

f4208d8

Take out return statement when docker network is not there.

chrisplo reviewed Oct 30, 2017

View reviewed changes

chrisplo added squash mege please squash when merging and removed squash mege please labels Oct 31, 2017

kahou82 merged commit 4986e71 into contiv:master Oct 31, 2017

kahou82 deleted the CNTV-140 branch October 31, 2017 18:49

kahou82 mentioned this pull request Nov 1, 2017

'Docker network rm' to remove contiv network, does not remove the network completely contiv/install#176

Closed

Contiv network is not deleted when docker network is deleted #1029

Contiv network is not deleted when docker network is deleted #1029

Conversation

kahou82 commented Oct 22, 2017 • edited Loading

kahou82 commented Oct 22, 2017

Choose a reason for hiding this comment

rchirakk commented Oct 23, 2017 • edited Loading

kahou82 commented Oct 24, 2017

kahou82 commented Oct 24, 2017

kahou82 commented Oct 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisplo left a comment

Choose a reason for hiding this comment

kahou82 commented Oct 28, 2017

kahou82 commented Oct 28, 2017

kahou82 commented Oct 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kahou82 commented Oct 29, 2017

kahou82 commented Oct 29, 2017

rchirakk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vhosakot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kahou82 commented Oct 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kahou82 commented Oct 30, 2017

kahou82 commented Oct 31, 2017

kahou82 commented Oct 22, 2017 •

edited

Loading

rchirakk commented Oct 23, 2017 •

edited

Loading