Integrate NIReconciler with zedrouter

[milan-zededa]

This commit untangles the close coupling between zedrouter and Linux by replacing all remaining netlink and system calls with the calls to NIReconciler (which has generic interface and replaceable implementations).

Moreover, this commit contains a substantial cleanup of the zedrouter code-base - some of the source files had too many lines, some functions were way overcomplicated, had strange names, obsolete comments, used strings instead of net.IP/net.HardwareAddr, etc.

Lastly, VLANs for switch network instances are now configured by zedrouter and not domainmgr - where it does not belong.

This is the last and arguably the most difficult part of zedrouter refactoring. It was unfortunately not feasible to split this into more smaller PRs. For review, it is probably best to start with the updated documentation.

[uncleDecart]

LGTM, does it make sense to try to test zedrouter?

[milan-zededa]

Rebased to resolve merge conflicts with #3203

[eriknordmark]

There is a lot of stuff to review here.
Are there any protobuf changes apart from the deprecation/removal of bridgeIPSets?
I think it makes sense doing putting that removal in a separate PR to get it out of the way (which also reduces the conflicts you're likely to have on the generated files).

For the other files, do you have a way to describe or otherwise facilitate seeing what is functionality which has moved to the reconciler but where the logic is unchanged as opposed to new logic and functionality?

A different way to ask the same question, what are the things you'd want the reviewers to look at more closely?

[milan-zededa]

There is a lot of stuff to review here. Are there any protobuf changes apart from the deprecation/removal of bridgeIPSets? I think it makes sense doing putting that removal in a separate PR to get it out of the way (which also reduces the conflicts you're likely to have on the generated files).

For the other files, do you have a way to describe or otherwise facilitate seeing what is functionality which has moved to the reconciler but where the logic is unchanged as opposed to new logic and functionality?

A different way to ask the same question, what are the things you'd want the reviewers to look at more closely?

I have split the API change into a separate PR: #3225
Once it is merged I will rebase this one and fix merge conflicts (edit: Done).

Since refactoring leads to many changes and the resulting diff can be confusing, it is probably better to look at the refactored zedrouter in my fork.
I would recommend to start with the updated and extended documentation.

Here we initialize all components of zedrouter and pubsub channels.

Here is the main event loop of zedrouter.
In the main event loop we process pubsub messages, run config reconciliation from, handle detected IP assignments (and update published info), publish flow logs, etc.

This new file contains all methods used to validate config for network instances.

This new file contains all methods related to IP allocation (aka IPAM).

This new file is where all zedrouter's pubsub handlers are placed.
Those pubsub handlers call various methods for managing network instances (create, activate, deactivate, etc.) and methods for managing application networks (VIFs).

In the refactored zedrouter you will not find any netlink calls - this is now all in LinuxNIReconciler (as one and currently the only implementation of NIReconciler interface). Package cmd/zedrouter contains only generic handling of network instances, not tight to any particular network stack.

[eriknordmark]

I note that there are two unrelated yetus complains against code not modified by this PR which we can ignore.

[eriknordmark]

Since @uncleDecart has reviewed this I think we can proceed with testing.
Let me know if we should add some more code reviewers.

[milan-zededa]

From the testing results I can see that ZFS takes too much time to fit into 6 hours - soon this will be addressed by splitting eden workflow into multiple smaller ones, run in parallel.
Also we hit the infamous kernel crash that we get often and I still cannot figure out its source. Maybe with more powerful runners this will go away.
Other than that, tests haven't detected any issues introduced by this PR.