Not master node fails to start after reboot while bond or bridge is configured on primary nic

[tsorya]

What happened:

1. Configuring bond on primary and firstSecondary interfaces
2. Bond successfully configured and have same ip as primary nic as expected
3. Restarting Node2 and it fails to connect back to kubernetes master and got another hostname (localhost.localdomain) but still has configured bond with right ip and mac.
   Restarting master node works perfectly well.

What you expected to happen:
Node2 must return and reconnect to master.

How to reproduce it (as minimally and precisely as possible):
Enable eth1:
apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
name: enable-eth1-policy
spec:
desiredState:
interfaces:
- name: eth1
type: ethernet
state: up
ipv4:
dhcp: true
enabled: true

Create bond:
apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
name: bond0-eth0-eth1-policy
spec:
desiredState:
interfaces:
- name: eth1
type: ethernet
state: up
- name: bond0
type: bond
state: up
ipv4:
dhcp: true
enabled: true
link-aggregation:
mode: balance-rr
options:
miimon: '140'
slaves:
- eth0
- eth1

Reboot node2 :
sync && sudo reboot -nf

Anything else we need to know?:

Environment:

• NodeNetworkState on affected nodes (use kubectl get nodenetworkstate <node_name> -o yaml):
  apiVersion: nmstate.io/v1alpha1
  kind: NodeNetworkState
  metadata:
  creationTimestamp: "2020-01-27T19:53:11Z"
  generation: 1
  name: node02
  ownerReferences:

  • apiVersion: v1
    kind: Node
    name: node02
    uid: 54ec6fda-5737-4f53-890d-d866ec8ab898
    resourceVersion: "1074"
    selfLink: /apis/nmstate.io/v1alpha1/nodenetworkstates/node02
    uid: 672eb79b-7140-4fb1-b691-93d07b13686c
    status:
    currentState:
    dns-resolver:
    config:
    search: []
    server: []
    running:
    search: []
    server:
    - 192.168.66.2
    - 192.168.66.2
    interfaces:
    • ipv4:
      address:
      • ip: 192.168.66.102
        prefix-length: 24
        auto-dns: true
        auto-gateway: true
        auto-routes: true
        dhcp: true
        enabled: true
        ipv6:
        autoconf: false
        dhcp: false
        enabled: false
        link-aggregation:
        mode: balance-rr
        options:
        miimon: "140"
        slaves:
      • eth1
      • eth0
        mac-address: 52:55:00:D1:55:02
        mtu: 1500
        name: bond0
        state: up
        type: bond
    • bridge:
      options:
      group-forward-mask: 0
      mac-ageing-time: 300
      multicast-snooping: true
      stp:
      enabled: false
      forward-delay: 15
      hello-time: 2
      max-age: 20
      priority: 32768
      port: []
      ipv4:
      address:
      • ip: 10.244.1.1
        prefix-length: 24
        dhcp: false
        enabled: true
        ipv6:
        address:
      • ip: fe80::2886:ebff:fed1:c07c
        prefix-length: 64
        autoconf: false
        dhcp: false
        enabled: true
        mac-address: 2A:86:EB:D1:C0:7C
        mtu: 1450
        name: cni0
        state: up
        type: linux-bridge
    • bridge:
      options:
      group-forward-mask: 0
      mac-ageing-time: 300
      multicast-snooping: true
      stp:
      enabled: false
      forward-delay: 15
      hello-time: 2
      max-age: 20
      priority: 32768
      port: []
      ipv4:
      address:
      • ip: 172.17.0.1
        prefix-length: 16
        dhcp: false
        enabled: true
        ipv6:
        autoconf: false
        dhcp: false
        enabled: false
        mac-address: 02:42:1B:DB:DD:27
        mtu: 1500
        name: docker0
        state: up
        type: linux-bridge
    • ipv4:
      dhcp: false
      enabled: false
      ipv6:
      autoconf: false
      dhcp: false
      enabled: false
      mac-address: 52:55:00:D1:55:02
      mtu: 1500
      name: eth0
      state: up
      type: ethernet
    • ipv4:
      dhcp: false
      enabled: false
      ipv6:
      autoconf: false
      dhcp: false
      enabled: false
      mac-address: 52:55:00:D1:55:02
      mtu: 1500
      name: eth1
      state: up
      type: ethernet
    • ipv4:
      address:
      • ip: 192.168.66.129
        prefix-length: 24
        auto-dns: true
        auto-gateway: true
        auto-routes: true
        dhcp: true
        enabled: true
        ipv6:
        address:
      • ip: fe80::a027:749b:f5ac:e7a
        prefix-length: 64
        auto-dns: true
        auto-gateway: true
        auto-routes: true
        autoconf: true
        dhcp: true
        enabled: true
        mac-address: 52:55:00:D1:56:03
        mtu: 1500
        name: eth2
        state: up
        type: ethernet
    • ipv4:
      enabled: false
      ipv6:
      enabled: false
      mac-address: 6E:5C:28:99:EC:FE
      mtu: 1450
      name: flannel.1
      state: down
      type: vxlan
      vxlan:
      base-iface: eth0
      destination-port: 8472
      id: 1
      remote: ""
    • ipv4:
      enabled: false
      ipv6:
      enabled: false
      mtu: 65536
      name: lo
      state: down
      type: unknown
      route-rules:
      config: []
      routes:
      config: []
      running:
      • destination: 0.0.0.0/0
        metric: 300
        next-hop-address: 192.168.66.2
        next-hop-interface: bond0
        table-id: 254
      • destination: 192.168.66.0/24
        metric: 300
        next-hop-address: ""
        next-hop-interface: bond0
        table-id: 254
      • destination: 10.244.1.0/24
        metric: 0
        next-hop-address: ""
        next-hop-interface: cni0
        table-id: 254
      • destination: 172.17.0.0/16
        metric: 0
        next-hop-address: ""
        next-hop-interface: docker0
        table-id: 254
      • destination: 0.0.0.0/0
        metric: 102
        next-hop-address: 192.168.66.2
        next-hop-interface: eth2
        table-id: 254
      • destination: 192.168.66.0/24
        metric: 102
        next-hop-address: ""
        next-hop-interface: eth2
        table-id: 254
      • destination: fe80::/64
        metric: 256
        next-hop-address: ""
        next-hop-interface: cni0
        table-id: 254
      • destination: fe80::/64
        metric: 102
        next-hop-address: ""
        next-hop-interface: eth2
        table-id: 254
      • destination: ff00::/8
        metric: 256
        next-hop-address: ""
        next-hop-interface: cni0
        table-id: 255
      • destination: ff00::/8
        metric: 256
        next-hop-address: ""
        next-hop-interface: eth2
        table-id: 255

• Problematic NodeNetworkConfigurationPolicy:

• kubernetes-nmstate image (use kubectl get pods --all-namespaces -l app=kubernetes-nmstate -o jsonpath='{.items[0].spec.containers[0].image}'):
  registry:5000/nmstate/kubernetes-nmstate-handler

• NetworkManager version (use nmcli --version)
  nmcli tool, version 1.20.11-23922.ee7bbddb6f.el7

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

• OS (e.g. from /etc/os-release):
  PRETTY_NAME="CentOS Linux 7 (Core)"

• Others:

[tsorya]

@qinqon @phoracek

[phoracek]

Thanks @tsorya!

The new hostname appears on the host or only in kubectl get nodes?

[tsorya]

@phoracek on the host that's why it fails to connect to master(at least it seems so).
on kubectl get nodes -> node is in state NotReady

[phoracek]

I wonder who configures the hostname, it could be either be given by DHCP server on set locally. Does it survive reboots without bond configured?

[tsorya]

Yap

[phoracek]

And with a bridge without bonding? I'd like to make sure it is indeed caused by the bonding, then we can dig deeper to see if it is a bug in nmstate or NetworkManager

[tsorya]

Didn't check. will try

[tsorya]

@phoracek same problem with bridge

[qinqon]

We have to implement this use case with copy-mac-from

[qinqon]

Let's adapt the default interface mac bonding test and examples to use two interfaces and do the mac cloning with https://github.com/nmstate/nmstate/blob/2599d3afa48a507d6631ce6924e3c3564dd81630/libnmstate/schema.py#L34