Cleanup multus conf on exit

[adrianchiris]

• if cleanup-config-on-exit is set delete generated multus
  config file on exit.

• add an option to skip watch for master cni config and kubeconfig
  as cleanup-config-on-exit with multus-conf-file=auto also triggered
  the watch for cases when deletion of multus config is desired but watch isnt

• setup signal handling to allow config file cleanup on exit

[adrianchiris]

This is a proposed alternative to : #1298

as we used to clean up multus conf file in preStop hook, however since the container no longer has shell we cannot run simple sh commands for cleanup.

[coveralls]

coverage: 62.736% (-0.4%) from 63.116%
when pulling 4198a76 on adrianchiris:cleanup-multus-conf
into aff99fc on k8snetworkplumbingwg:master.

[coveralls]

coverage: 62.667% (-0.4%) from 63.116%
when pulling 273c175 on adrianchiris:cleanup-multus-conf
into 41013e7 on k8snetworkplumbingwg:master.

[coveralls]

coverage: 62.667% (-0.4%) from 63.116%
when pulling ecc67c7 on adrianchiris:cleanup-multus-conf
into 41013e7 on k8snetworkplumbingwg:master.

[dougbtv]

Let's add some notes about the flags @ https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#entrypoint-script-parameters

[coveralls]

coverage: 62.667% (-0.7%) from 63.364%
when pulling 6ade0ce on adrianchiris:cleanup-multus-conf
into f54693f on k8snetworkplumbingwg:master.

[adrianchiris]

@dougbtv docs update.

additional feedback/review appreciated :)

[dougbtv]

Appreciate the updates for the docs!

[seastco]

@dougbtv @adrianchiris ty for landing this.

is disabling the watch functionality necessary if CleanupConfigOnExit=true and MultusConfFile=auto because the file may be recreated before we're able to completely exit? does the cleanupMultusConf being in the defer block not ensure this? I'm not seeing how the file could be recreated after this is called.

if I'm understanding correctly, CleanupConfigOnExit and the watch functionality are basically incompatible. happy to attempt improving this once I clarify my understanding.

[adrianchiris]

@seastco there is a select statement on signal handler context. cleanup config on exit should work both if watching or not.

the "skip-config-watch" flag was added to preserve backward compatibility since cleanup config on exit originally triggered the watch for some reason (imo it shouldnt and is counter intuitive).