Refactor endpoint handling and reconcilliation #21
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As part of the Go 1.20 release it seems like the default for `CGO_ENABLED` is no longer carried over from the tools. This leads to linking issues on systems that use different versions of glibc from what the base image uses. See golang/go#58550 for more details. This change should fix cosmonic-labs#16
Use `scratch` as a base image since we're generating a static binary anyway. Also be more explicity about the platform and target OS during the build.
DRAFT With the release of Nomad 1.6 it's possible to get the network address of the allocation from Nomad. The change to enable this is only in the client library and does not require updating the Nomad server. The IP was sent back by older Nomad versions, it just wasn't available in the client. This enables refactoring the endpoint reconcilliation to make use of the IP address to identify the endpoint within Cilium. There is no longer a dependency on Consul for policies. Additional, endpoints are now labelled with the task group and task information as services can be created at those levels.
Remove unused flags from the readme and command line and refactor the code to allow for testing.
✅ Deploy Preview for netreap canceled.
|
protochron
reviewed
Jul 31, 2023
There was a problem hiding this comment.
@deverton-godaddy Great work here! This is almost ready to merge, I just had one more follow up question around some code that was removed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Feature or Problem
With the release of Nomad 1.6 it's possible to get the network address of the allocation from Nomad. The change to enable this is only in the client library and does not require updating the Nomad server. The IP was sent back by older Nomad versions, it just wasn't available in the client.
This enables refactoring the endpoint reconcilliation to make use of the IP address to identify the endpoint within Cilium. There is no longer a dependency on Consul for policy reconcilliaton. Additionally, endpoints are now labelled with the task group and task information as services can be created at those levels.
On startup of netreap the reconciliation process is as follows:
There's no need to delete endpoints (as far as I can tell) as Nomad should be deleting the endpoints as part of its integration with the CNI plugin.
For event processing netreap will now monitor the Allocation topic as the
AllocationUpdatedevent from that topic contains the network details associated with the allocation. The process is now:AllocationUpdatedevent.networkconfiguration won't) ignore it.This should be fairly scalable as the number of calls to the Nomad API will be one per-endpoint allocation.
One other change is patch the endpoint with an endpoint change request event. This triggers Cilium to remove the
reserved:initlabel from the endpoints as it considers the endpoints labelled properly at that point. This change is based on how the Cilium Docker network plugin works.Note that it is no longer possible to filter out services by tags. I'm not sure that feature makes sense with the refactored code as you would always want to apply labels to Cilium endpoints.
Related Issues
Potentially fixes #20 and #9