Skip to content

Commit

Permalink
Refactor endpoint handling and reconcilliation (#21)
Browse files Browse the repository at this point in the history
* Disable cgo

As part of the Go 1.20 release it seems like the default for `CGO_ENABLED` is no longer carried over from the tools. This leads to linking issues on systems that use different versions of glibc from what the base image uses. See golang/go#58550 for more details.

This change should fix #16

* Slim down the final image.

Use `scratch` as a base image since we're generating a static binary
anyway. Also be more explicity about the platform and target OS during
the build.

* Refactor endpoint handling and reconcilliation.

DRAFT

With the release of Nomad 1.6 it's possible to get the network address
of the allocation from Nomad. The change to enable this is only in the
client library and does not require updating the Nomad server. The IP
was sent back by older Nomad versions, it just wasn't available in the
client.

This enables refactoring the endpoint reconcilliation to make use of the
IP address to identify the endpoint within Cilium.

There is no longer a dependency on Consul for policies.

Additional, endpoints are now labelled with the task group and task
information as services can be created at those levels.

* Tidy

* Update the readme and add some basic tests.

Remove unused flags from the readme and command line and refactor the
code to allow for testing.

* Slightly better logging of labels

* Always update labels on reconcile.

* Fixes and align with upstream Dockerfile

* Missed one

* Reduce Dockerfile diff further

* Restore periodic reconcilliation of endpoints

* Remove stray return
  • Loading branch information
deverton-godaddy authored Aug 2, 2023
1 parent 345e3f9 commit db0802d
Show file tree
Hide file tree
Showing 9 changed files with 384 additions and 365 deletions.
6 changes: 4 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
FROM golang:1.20-bullseye as builder
WORKDIR /netreap
COPY . /netreap
COPY go.mod go.sum /netreap/
RUN go mod download
COPY . /netreap/
ARG VERSION
RUN go build -ldflags "-s -w -X 'main.Version=$VERSION'"

FROM gcr.io/distroless/base-debian11
WORKDIR /
COPY --from=builder /netreap/netreap /usr/bin/netreap
ENTRYPOINT ["/usr/bin/netreap"]
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@ docker:

ci:
docker buildx build --platform $(platforms) --tag $(repo):$(VERSION) --push .

test:
go test -v ./...
13 changes: 4 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,10 +186,6 @@ job "netreap" {
task "netreap" {
driver = "docker"
env {
NETREAP_CILIUM_CIDR = "172.16.0.0/16"
}
config {
image = "ghcr.io/cosmonic/netreap:0.1.2"
network_mode = "host"
Expand Down Expand Up @@ -235,15 +231,14 @@ clients are available to Netreap.

| Flag | Env Var | Default | Description |
| ---------------------- | --------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------- |
| `--cilium-cidr`, `-c`, | `NETREAP_CILIUM_CIDR` | None, this is a required flag | The CIDR block of the address space used by Cilium. This allows netreap to identify if a job is a Cilium one. |
| `--debug` | `NETREAP_DEBUG` | `false` | Turns on debug logging |
| `--policy-key` | `NETREAP_POLICY_KEY` | `netreap.io/policy` | Consul key that Netreap watches for changes to the Cilium policy JSON value |
| `--exclude-tags` | `NETREAP_EXCLUDE_TAG` | None | List of Consul service tags to use as a filter to exclude from Netreap |

Please note that to configure the Nomad and Consul clients that Netreap uses,
Please note that to configure the Nomad, Consul and Cilium clients that Netreap uses,
we leverage the well defined environment variables for
[Nomad](https://www.nomadproject.io/docs/commands#environment-variables) and
[Consul](https://www.consul.io/commands#environment-variables).
[Nomad](https://www.nomadproject.io/docs/commands#environment-variables),
[Consul](https://www.consul.io/commands#environment-variables) and
[Cilium](https://pkg.go.dev/github.com/cilium/cilium/pkg/client#DefaultSockPath).

Right now we only allow connecting to the local Unix socket endpoint for the
Cilium agent. As we determine how we are going to set things up with Cilium, we
Expand Down
8 changes: 4 additions & 4 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,10 @@ go 1.20

require (
github.com/cilium/cilium v1.13.2
github.com/google/uuid v1.3.0
github.com/hashicorp/consul/api v1.18.0
github.com/hashicorp/nomad/api v0.0.0-20230420161604-206236039ccd
github.com/hashicorp/nomad/api v0.0.0-20230719205936-8d2894699319
github.com/urfave/cli/v2 v2.11.2
go.uber.org/zap v1.23.0
golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a
)

require (
Expand Down Expand Up @@ -42,8 +40,9 @@ require (
github.com/golang/protobuf v1.5.2 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/hashicorp/cronexpr v1.1.1 // indirect
github.com/hashicorp/cronexpr v1.1.2 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-hclog v1.4.0 // indirect
Expand Down Expand Up @@ -110,6 +109,7 @@ require (
go.opentelemetry.io/otel/trace v1.11.2 // indirect
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
golang.org/x/exp v0.0.0-20230713183714-613f0c0eb8a1 // indirect
golang.org/x/net v0.7.0 // indirect
golang.org/x/oauth2 v0.4.0 // indirect
golang.org/x/sync v0.1.0 // indirect
Expand Down
17 changes: 9 additions & 8 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5Kwzbycv
github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w=
github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk=
github.com/felixge/httpsnoop v1.0.3 h1:s/nj+GCswXYzN5v2DpNMuMQYe+0DDwt5WVCU6CWBdXk=
github.com/frankban/quicktest v1.14.3 h1:FJKSZTDHjyhriyC81FLQ0LY93eSai0ZyR/ZIkd3ZUKE=
github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
Expand Down Expand Up @@ -278,8 +279,8 @@ github.com/hashicorp/consul/api v1.18.0 h1:R7PPNzTCeN6VuQNDwwhZWJvzCtGSrNpJqfb22
github.com/hashicorp/consul/api v1.18.0/go.mod h1:owRRGJ9M5xReDC5nfT8FTJrNAPbT4NM6p/k+d03q2v4=
github.com/hashicorp/consul/sdk v0.13.0 h1:lce3nFlpv8humJL8rNrrGHYSKc3q+Kxfeg3Ii1m6ZWU=
github.com/hashicorp/consul/sdk v0.13.0/go.mod h1:0hs/l5fOVhJy/VdcoaNqUSi2AUs95eF5WKtv+EYIQqE=
github.com/hashicorp/cronexpr v1.1.1 h1:NJZDd87hGXjoZBdvyCF9mX4DCq5Wy7+A/w+A7q0wn6c=
github.com/hashicorp/cronexpr v1.1.1/go.mod h1:P4wA0KBl9C5q2hABiMO7cp6jcIg96CDh1Efb3g1PWA4=
github.com/hashicorp/cronexpr v1.1.2 h1:wG/ZYIKT+RT3QkOdgYc+xsKWVRgnxJ1OJtjjy84fJ9A=
github.com/hashicorp/cronexpr v1.1.2/go.mod h1:P4wA0KBl9C5q2hABiMO7cp6jcIg96CDh1Efb3g1PWA4=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
Expand Down Expand Up @@ -325,8 +326,8 @@ github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO
github.com/hashicorp/mdns v1.0.4/go.mod h1:mtBihi+LeNXGtG8L9dX59gAEa12BDtBQSp4v/YAJqrc=
github.com/hashicorp/memberlist v0.5.0 h1:EtYPN8DpAURiapus508I4n9CzHs2W+8NZGbmmR/prTM=
github.com/hashicorp/memberlist v0.5.0/go.mod h1:yvyXLpo0QaGE59Y7hDTsTzDD25JYBZ4mHgHUZ8lrOI0=
github.com/hashicorp/nomad/api v0.0.0-20230420161604-206236039ccd h1:jnx5Zofpo0xmn7agq1E1buUTFEGs25vEoOU8t7kENBc=
github.com/hashicorp/nomad/api v0.0.0-20230420161604-206236039ccd/go.mod h1:2TCrNvonL09r7EiQ6M2rNt+Cmjbn1QbzchFoTWJFpj4=
github.com/hashicorp/nomad/api v0.0.0-20230719205936-8d2894699319 h1:o7qzfGTTqk9QlCDQnvcrOe6sNltH4NbcgyWD7aXK2Uw=
github.com/hashicorp/nomad/api v0.0.0-20230719205936-8d2894699319/go.mod h1:O23qLAZuCx4htdY9zBaO4cJPXgleSFEdq6D/sezGgYE=
github.com/hashicorp/serf v0.10.1 h1:Z1H2J60yRKvfDYAOZLd2MU0ND4AH/WDz7xYHDWQsIPY=
github.com/hashicorp/serf v0.10.1/go.mod h1:yL2t6BqATOLGc5HF7qbFkTfXoPIY0WZdWHfEvMqbG+4=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
Expand Down Expand Up @@ -492,7 +493,7 @@ github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529 h1:nn5Wsu0esKSJiIVhscUt
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/shirou/gopsutil/v3 v3.23.1 h1:a9KKO+kGLKEvcPIs4W62v0nu3sciVDOOOPUD0Hz7z/4=
github.com/shirou/gopsutil/v3 v3.23.1/go.mod h1:NN6mnm5/0k8jw4cBfCnJtr5L7ErOTg18tMNpgFkn0hA=
github.com/shoenig/test v0.6.3 h1:GVXWJFk9PiOjN0KoJ7VrJGH6uLPnqxR7/fe3HUPfE0c=
github.com/shoenig/test v0.6.6 h1:Oe8TPH9wAbv++YPNDKJWUnI8Q4PPWCx3UbOfH+FxiMU=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
Expand Down Expand Up @@ -615,8 +616,8 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a h1:tlXy25amD5A7gOfbXdqCGN5k8ESEed/Ee1E5RcrYnqU=
golang.org/x/exp v0.0.0-20230108222341-4b8118a2686a/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc=
golang.org/x/exp v0.0.0-20230713183714-613f0c0eb8a1 h1:MGwJjxBy0HJshjDNfLsYO8xppfqWlA5ZT9OhtUUhTNw=
golang.org/x/exp v0.0.0-20230713183714-613f0c0eb8a1/go.mod h1:FXUEEKJgO7OQYeo8N01OfiKP8RXMtf6e8aTskBGqWdc=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
Expand All @@ -642,7 +643,7 @@ golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.7.0 h1:LapD9S96VoQRhi/GrNTqeBJFrUjs5UHCAtTlgwA5oZA=
golang.org/x/mod v0.11.0 h1:bUO06HqtnRcc/7l71XBe4WcqTZ+3AH1J59zWDDwLKgU=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
Expand Down
39 changes: 10 additions & 29 deletions main.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ import (
"context"
"fmt"
"log"
"net"
"os"
"os/signal"

cilium_client "github.com/cilium/cilium/pkg/client"
consul_api "github.com/hashicorp/consul/api"
nomad_api "github.com/hashicorp/nomad/api"
"github.com/urfave/cli/v2"
Expand All @@ -20,9 +20,7 @@ import (
var Version = "unreleased"

type config struct {
net string
policyKey string
excludeTags []string
policyKey string
}

func main() {
Expand All @@ -35,8 +33,6 @@ func main() {
}
defer logger.Sync()

excludeTags := cli.StringSlice{}

app := &cli.App{
Name: "netreap",
Usage: "A custom monitor and reaper for cleaning up Cilium endpoints and nodes",
Expand All @@ -48,14 +44,6 @@ func main() {
EnvVars: []string{"NETREAP_DEBUG"},
Destination: &debug,
},
&cli.StringFlag{
Name: "cilium-cidr",
Aliases: []string{"c"},
Usage: "The CIDR block that Cilium addresses belong to. This is used for checking if a service is a Cilium service or not",
EnvVars: []string{"NETREAP_CILIUM_CIDR"},
Destination: &conf.net,
Required: true,
},
&cli.StringFlag{
Name: "policy-key",
Aliases: []string{"k"},
Expand All @@ -64,13 +52,6 @@ func main() {
EnvVars: []string{"NETREAP_POLICY_KEY"},
Destination: &conf.policyKey,
},
&cli.StringSliceFlag{
Name: "exclude-tag",
Aliases: []string{"e"},
Usage: "Consul service tags to skip when checking for Cilium-enabled jobs",
EnvVars: []string{"NETREAP_EXCLUDE_TAG"},
Destination: &excludeTags,
},
},
Before: func(ctx *cli.Context) error {
if debug {
Expand All @@ -84,7 +65,6 @@ func main() {
return nil
},
Action: func(c *cli.Context) error {
conf.excludeTags = excludeTags.Value()
return run(conf)
},
Version: Version,
Expand All @@ -96,17 +76,18 @@ func main() {
}

func run(conf config) error {
_, net, err := net.ParseCIDR(conf.net)
if err != nil {
return fmt.Errorf("unable to parse Cilium CIDR block: %s", err)
}

// Step 0: Construct clients
consul_client, err := consul_api.NewClient(consul_api.DefaultConfig())
if err != nil {
return fmt.Errorf("unable to connect to Consul: %s", err)
}

// Looks for the default Cilium socket path or uses the value from CILIUM_SOCK
cilium_client, err := cilium_client.NewDefaultClient()
if err != nil {
return fmt.Errorf("error when connecting to cilium agent: %s", err)
}

// DefaultConfig fetches configuration data from well-known nomad variables (e.g. NOMAD_ADDR,
// NOMAD_CACERT), so we'll just leverage that for now.
nomad_client, err := nomad_api.NewClient(nomad_api.DefaultConfig())
Expand All @@ -133,12 +114,12 @@ func run(conf config) error {
}

zap.S().Debug("Starting endpoint reaper")
endpoint_reaper, err := reapers.NewEndpointReaper(ctx, nomad_client, consul_client, net, conf.excludeTags)
endpoint_reaper, err := reapers.NewEndpointReaper(cilium_client, nomad_client.Allocations(), nomad_client.EventStream())
if err != nil {
return err
}

endpointFailChan, err := endpoint_reaper.Run()
endpointFailChan, err := endpoint_reaper.Run(ctx)
if err != nil {
return fmt.Errorf("unable to start endpoint reaper: %s", err)
}
Expand Down
34 changes: 34 additions & 0 deletions reapers/api.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
package reapers

import (
"context"

"github.com/cilium/cilium/api/v1/models"
nomad_api "github.com/hashicorp/nomad/api"
)

type AllocationInfo interface {
Info(allocID string, q *nomad_api.QueryOptions) (*nomad_api.Allocation, *nomad_api.QueryMeta, error)
}

type EventStreamer interface {
Stream(ctx context.Context, topics map[nomad_api.Topic][]string, index uint64, q *nomad_api.QueryOptions) (<-chan *nomad_api.Events, error)
}

type EndpointLister interface {
EndpointList() ([]*models.Endpoint, error)
}

type EndpointGetter interface {
EndpointGet(id string) (*models.Endpoint, error)
}

type EndpointPatcher interface {
EndpointPatch(id string, ep *models.EndpointChangeRequest) error
}

type EndpointUpdater interface {
EndpointLister
EndpointGetter
EndpointPatcher
}
Loading

0 comments on commit db0802d

Please sign in to comment.