phantom ip/mac vlan network after a powercycle #2295

rnataraja · 2018-11-06T01:31:04Z

I am trying to address #1743 as part of this PR.

There are two issues as part of this

When a node is power cycled unplanned or dockerd goes through a restart (unplanned), swarm scope networks are not cleaned up, Infact they are recreated when the docker daemon starts. This recreate does not work as there is already a swarmScope network has been restored and uses the same uplink.
FIX: there was an earlier PR that tackled this problem by simply deleting the network if its created with same ID. As per comments in that PR, it was preferred to re-use instead of delete. In this PR, I am trying to essentially re-use the same network ID.
Even if the swarm Scope network can be recreated there is still a problem with config-only network. As there is an additional createNetwork after the docker daemon restarts. The endpoint count on the configOnly network becomes more than what it should be. Consider the case where node has been kicked out of swarm during this unplanned reboot and added back in.
I believe a swarm scope network to its config-only network is a one to one mapping for a particular worker node. If thats not the case, then more work is required for this PR. But if that is the case, there is no real need to track the endpoint count.
FIX: Upon deleteNetwork of a swarm scope network, the correspnding configOnly Network endpoint count is zeroed out essentially making configOnly network as deleteable.

…nto issue1743

GordonTheTurtle · 2018-11-06T01:31:06Z

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "issue1743" git@github.com:rnataraja/libnetwork.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

…nto issue1743

rnataraja · 2018-11-13T00:41:14Z

@fcrisciani what are your thoughts on these changes.

swift1911 · 2018-12-26T02:07:24Z

is any progress about this PR?

Greatsamps · 2019-01-28T19:45:36Z

still present in 18.09.1

itsgk92 · 2019-02-04T11:20:05Z

+1

MalMen · 2019-02-26T21:17:42Z

still on 18.09.2

lukicsl · 2019-04-14T13:03:01Z

+1

arkodg · 2019-05-07T19:27:04Z

network.go

@@ -1051,7 +1051,13 @@ func (n *network) delete(force bool, rmLBEndpoint bool) error {

 	if n.ConfigFrom() != "" {
 		if t, err := c.getConfigNetwork(n.ConfigFrom()); err == nil {
-			if err := t.getEpCnt().DecEndpointCnt(); err != nil {


for bridge networks , multiple networks can be created from a config (--config-from), so this will break that case

@rnataraja
https://github.com/docker/libnetwork/blob/9ff9b57c344df5cd47443ad9e65702ec85c5aeb0/controller.go#L806
coupled with foundExisting should help you fix this problem

arkodg · 2019-05-07T19:29:47Z

drivers/ipvlan/ipvlan_network.go

@@ -60,10 +60,14 @@ func (d *driver) CreateNetwork(nid string, option map[string]interface{}, nInfo
 		// empty parent and --internal are handled the same. Set here to update k/v
 		config.Internal = true
 	}
-	err = d.createNetwork(config)
+	foundExisting, err := d.createNetwork(config)


for the re-create path, can you please add some integration test-cases, either in libnetwork or moby (https://github.com/moby/moby/blob/master/integration/network/service_test.go) . A SIGKILL on dockerd should be enough to reproduce this case

arkodg · 2019-06-21T21:09:40Z

@rnataraja still there ? :)

This commit carries forward the work done in moby#2295 and fixes two things 1. Allows macvlan and ipvlan to be restored properly after dockerd or the system is restarted 2. Makes sure the refcount for the configOnly network is not incremented for the above case so this network can be deleted after all the associated ConfigFrom networks are deleted Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

arkodg · 2019-07-10T04:52:51Z

thanks for driving this issue @rnataraja , I'll be taking this PR forward using #2415

This commit carries forward the work done in moby#2295 and fixes two things 1. Allows macvlan and ipvlan to be restored properly after dockerd or the system is restarted 2. Makes sure the refcount for the configOnly network is not incremented for the above case so this network can be deleted after all the associated ConfigFrom networks are deleted Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

rnataraja · 2019-07-10T21:20:34Z

thanks for driving this issue @rnataraja , I'll be taking this PR forward using #2415

Thanks @arkodg

This commit carries forward the work done in moby#2295 and fixes two things 1. Allows macvlan and ipvlan to be restored properly after dockerd or the system is restarted 2. Makes sure the refcount for the configOnly network is not incremented for the above case so this network can be deleted after all the associated ConfigFrom networks are deleted Addresses: moby#1743 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

full diff: moby/libnetwork@83d30db...09cdcc8 changes included: - moby/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling - moby/libnetwork#2411 Macvlan network handles netlabel.Internal wrong - fixes moby/libnetwork#2410 Macvlan network handles netlabel.Internal wrong - moby/libnetwork#2414 Allow network with --config-from to be --internal - fixes moby/libnetwork#2413 Network with --config-from does not honor --internal - moby/libnetwork#2351 Use fewer modprobes - relates to moby#38930 Use fewer modprobes - moby/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks - carry of moby/libnetwork#2295 phantom ip/mac vlan network after a powercycle - fixes moby/libnetwork#1743 Phantom docker network Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

full diff: moby/libnetwork@83d30db...09cdcc8 changes included: - moby/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling - moby/libnetwork#2411 Macvlan network handles netlabel.Internal wrong - fixes moby/libnetwork#2410 Macvlan network handles netlabel.Internal wrong - moby/libnetwork#2414 Allow network with --config-from to be --internal - fixes moby/libnetwork#2413 Network with --config-from does not honor --internal - moby/libnetwork#2351 Use fewer modprobes - relates to moby/moby#38930 Use fewer modprobes - moby/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks - carry of moby/libnetwork#2295 phantom ip/mac vlan network after a powercycle - fixes moby/libnetwork#1743 Phantom docker network Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: 6f234db9fef23c591d8376f96db062e7107b658f Component: engine

This commit carries forward the work done in moby#2295 and fixes two things 1. Allows macvlan and ipvlan to be restored properly after dockerd or the system is restarted 2. Makes sure the refcount for the configOnly network is not incremented for the above case so this network can be deleted after all the associated ConfigFrom networks are deleted Addresses: moby#1743 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

thaJeztah · 2019-08-22T08:52:18Z

@arkodg @selansen this was carried in #2415, which is now merged, so this one can be closed

full diff: moby/libnetwork@83d30db...09cdcc8 changes included: - moby/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling - moby/libnetwork#2411 Macvlan network handles netlabel.Internal wrong - fixes moby/libnetwork#2410 Macvlan network handles netlabel.Internal wrong - moby/libnetwork#2414 Allow network with --config-from to be --internal - fixes moby/libnetwork#2413 Network with --config-from does not honor --internal - moby/libnetwork#2351 Use fewer modprobes - relates to moby#38930 Use fewer modprobes - moby/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks - carry of moby/libnetwork#2295 phantom ip/mac vlan network after a powercycle - fixes moby/libnetwork#1743 Phantom docker network Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 6f234db) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

full diff: moby/libnetwork@83d30db...09cdcc8 changes included: - moby/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling - moby/libnetwork#2411 Macvlan network handles netlabel.Internal wrong - fixes moby/libnetwork#2410 Macvlan network handles netlabel.Internal wrong - moby/libnetwork#2414 Allow network with --config-from to be --internal - fixes moby/libnetwork#2413 Network with --config-from does not honor --internal - moby/libnetwork#2351 Use fewer modprobes - relates to moby/moby#38930 Use fewer modprobes - moby/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks - carry of moby/libnetwork#2295 phantom ip/mac vlan network after a powercycle - fixes moby/libnetwork#1743 Phantom docker network Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 6f234db9fef23c591d8376f96db062e7107b658f) Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: b6190c2713623ab455d29da4771b684e4eafc63f Component: engine

full diff: moby/libnetwork@83d30db...09cdcc8 changes included: - moby/libnetwork#2416 Fix hardcoded AF_INET for IPv6 address handling - moby/libnetwork#2411 Macvlan network handles netlabel.Internal wrong - fixes moby/libnetwork#2410 Macvlan network handles netlabel.Internal wrong - moby/libnetwork#2414 Allow network with --config-from to be --internal - fixes moby/libnetwork#2413 Network with --config-from does not honor --internal - moby/libnetwork#2351 Use fewer modprobes - relates to moby#38930 Use fewer modprobes - moby/libnetwork#2415 Support dockerd and system restarts for ipvlan and macvlan networks - carry of moby/libnetwork#2295 phantom ip/mac vlan network after a powercycle - fixes moby/libnetwork#1743 Phantom docker network Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Signed-off-by: zach <Zachary.Joyner@linux.com>

This commit carries forward the work done in moby/libnetwork#2295 and fixes two things 1. Allows macvlan and ipvlan to be restored properly after dockerd or the system is restarted 2. Makes sure the refcount for the configOnly network is not incremented for the above case so this network can be deleted after all the associated ConfigFrom networks are deleted Addresses: moby/libnetwork#1743 Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>

IGitYou · 2023-07-15T17:35:13Z

I am still seeing this Version: 24.0.4

Subscribed to watch for a fix.

thaJeztah · 2023-07-16T19:12:44Z

/cc @akerouanton

root and others added 3 commits November 5, 2018 09:09

phantom ip/mac vlan network after a powercycle

c1824b4

phantom ip/mac vlan network after a powercycle

91cab66

Merge branch 'issue1743' of https://github.com/rnataraja/libnetwork i…

bc68f38

…nto issue1743

GordonTheTurtle added the dco/no label Nov 6, 2018

rnataraja added 3 commits November 5, 2018 18:25

lint errors

766eac3

phantom ip/mac vlan network after a powercycle

d8f0804

Merge branch 'issue1743' of https://github.com/rnataraja/libnetwork i…

c994b5f

…nto issue1743

arkodg reviewed May 7, 2019

View reviewed changes

arkodg mentioned this pull request Jul 10, 2019

Support dockerd and system restarts for ipvlan and macvlan networks #2415

Merged

thaJeztah mentioned this pull request Jul 30, 2019

bump libnetwork to 09cdcc8c0eab3946c2d70e8f6225b05baf1e90d1 moby/moby#39633

Merged

thaJeztah mentioned this pull request Aug 22, 2019

[19.03 backport] bump libnetwork to 92d1fbe1eb0883cf11d283cea8e658275146411d docker-archive/engine#330

Merged

corhere added the carry-to-mobymoby label Jan 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

phantom ip/mac vlan network after a powercycle #2295

phantom ip/mac vlan network after a powercycle #2295

rnataraja commented Nov 6, 2018

GordonTheTurtle commented Nov 6, 2018

rnataraja commented Nov 13, 2018

swift1911 commented Dec 26, 2018

Greatsamps commented Jan 28, 2019

itsgk92 commented Feb 4, 2019

MalMen commented Feb 26, 2019

lukicsl commented Apr 14, 2019

arkodg May 7, 2019

arkodg May 9, 2019 •

edited

Loading

arkodg May 7, 2019

arkodg commented Jun 21, 2019

arkodg commented Jul 10, 2019

rnataraja commented Jul 10, 2019

thaJeztah commented Aug 22, 2019

IGitYou commented Jul 15, 2023

thaJeztah commented Jul 16, 2023

phantom ip/mac vlan network after a powercycle #2295

Are you sure you want to change the base?

phantom ip/mac vlan network after a powercycle #2295

Conversation

rnataraja commented Nov 6, 2018

GordonTheTurtle commented Nov 6, 2018

rnataraja commented Nov 13, 2018

swift1911 commented Dec 26, 2018

Greatsamps commented Jan 28, 2019

itsgk92 commented Feb 4, 2019

MalMen commented Feb 26, 2019

lukicsl commented Apr 14, 2019

arkodg May 7, 2019

Choose a reason for hiding this comment

arkodg May 9, 2019 • edited Loading

Choose a reason for hiding this comment

arkodg May 7, 2019

Choose a reason for hiding this comment

arkodg commented Jun 21, 2019

arkodg commented Jul 10, 2019

rnataraja commented Jul 10, 2019

thaJeztah commented Aug 22, 2019

IGitYou commented Jul 15, 2023

thaJeztah commented Jul 16, 2023

arkodg May 9, 2019 •

edited

Loading