Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create istio exclusion for CSI Driver in case of codeModules or public registry #3343

Closed
wants to merge 22 commits into from
Closed
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
f9d2522
Create istio exclusion for CSI Driver in case of codeModules or publi…
waodim Jun 20, 2024
a4ea79f
Merge branch 'main' into feature/csi-istio-exclusion
waodim Jun 20, 2024
c5f59e5
Remove unnecessary check and complete mock interface
waodim Jun 20, 2024
9a2ba7b
Merge branch 'feature/csi-istio-exclusion' of github.com:Dynatrace/dy…
waodim Jun 20, 2024
8bd069e
Fix linting
waodim Jun 20, 2024
b5566bd
Move istio reconciliation and use CodeModulesImage()
waodim Jun 20, 2024
8d971bd
Fix linting by removing cyclomatic complexity
waodim Jun 20, 2024
c92b4c3
Use url library to properly parse url
waodim Jun 20, 2024
8075a65
Add unit tests for parseCodeModuleUrl
waodim Jun 20, 2024
eeb60eb
Move csi istio reconciliation to already present istio check
waodim Jun 20, 2024
1564bbe
Fix linting
waodim Jun 20, 2024
81d0714
Update pkg/controllers/dynakube/istio/config.go
waodim Jun 24, 2024
0105021
Fix linting
waodim Jun 24, 2024
fdc1c92
Move CSI Driver reconciliation into comm hosts reconciliation
waodim Jun 25, 2024
a7c72c4
Update pkg/controllers/dynakube/istio/reconciler_test.go
waodim Jul 1, 2024
619e50a
Extend tests and apply feedback from review
waodim Jul 1, 2024
ea80b58
Set docker.io as default host for image url
waodim Jul 1, 2024
379c22d
Fix docker host
waodim Jul 3, 2024
ad97487
Merge branch 'main' into feature/csi-istio-exclusion
waodim Jul 4, 2024
d23af95
Fix handling of docker case
waodim Jul 5, 2024
9cd9a5d
Fix linting
waodim Jul 5, 2024
6bfb710
Further linting fix
waodim Jul 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion pkg/controllers/dynakube/controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
"github.com/Dynatrace/dynatrace-operator/pkg/controllers"
"github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/activegate"
"github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/connectioninfo"
"github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/connectioninfo/oneagent"
oaconnectioninfo "github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/connectioninfo/oneagent"
"github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/injection"
"github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/istio"
"github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/oneagent"
Expand Down Expand Up @@ -595,6 +595,19 @@ func TestSetupIstio(t *testing.T) {
virtualService, err := fakeIstio.NetworkingV1beta1().VirtualServices(dynakube.GetNamespace()).Get(ctx, expectedName, metav1.GetOptions{})
require.NoError(t, err)
assert.NotNil(t, virtualService)

err = istioReconciler.ReconcileCSIDriver(ctx, dynakube)

require.NoError(t, err)

expectedName = istio.BuildNameForFQDNServiceEntry(dynakube.GetName(), istio.CSIDiverComponent)
serviceEntry, err = fakeIstio.NetworkingV1beta1().ServiceEntries(dynakube.GetNamespace()).Get(ctx, expectedName, metav1.GetOptions{})
require.NoError(t, err)
assert.NotNil(t, serviceEntry)

virtualService, err = fakeIstio.NetworkingV1beta1().VirtualServices(dynakube.GetNamespace()).Get(ctx, expectedName, metav1.GetOptions{})
require.NoError(t, err)
assert.NotNil(t, virtualService)
})
}

Expand Down
14 changes: 14 additions & 0 deletions pkg/controllers/dynakube/injection/reconciler.go
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,11 @@ func (r *reconciler) Reconcile(ctx context.Context) error {
if err != nil {
log.Error(err, "error reconciling istio configuration for codemodules")
}

err = r.reconcileIstioForCSIDriver(ctx)
if err != nil {
return err
}
}

if !r.dynakube.NeedAppInjection() {
Expand Down Expand Up @@ -120,6 +125,15 @@ func (r *reconciler) Reconcile(ctx context.Context) error {
return nil
}

func (r *reconciler) reconcileIstioForCSIDriver(ctx context.Context) error {
err := r.istioReconciler.ReconcileCSIDriver(ctx, r.dynakube)
if err != nil {
return errors.WithMessage(err, "failed to reconcile istio objects for CSI Driver")
}

return nil
}

func (r *reconciler) removeAppInjection(ctx context.Context) (err error) {
namespaces, err := mapper.GetNamespacesForDynakube(ctx, r.apiReader, r.dynakube.Name)
if err != nil {
Expand Down
1 change: 1 addition & 0 deletions pkg/controllers/dynakube/istio/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ var (
)

const (
CSIDiverComponent = "csi-driver"
waodim marked this conversation as resolved.
Show resolved Hide resolved
OperatorComponent = "operator"
OneAgentComponent = "oneagent"
CodeModuleComponent = "CodeModule"
Expand Down
56 changes: 55 additions & 1 deletion pkg/controllers/dynakube/istio/reconciler.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@ import (
"context"
goerrors "errors"
"net"
"net/url"
"strings"

dynatracev1beta2 "github.com/Dynatrace/dynatrace-operator/pkg/api/v1beta2/dynakube"
dtclient "github.com/Dynatrace/dynatrace-operator/pkg/clients/dynatrace"
"github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/connectioninfo/activegate"
"github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/connectioninfo/oneagent"
oaconnectioninfo "github.com/Dynatrace/dynatrace-operator/pkg/controllers/dynakube/connectioninfo/oneagent"
"github.com/Dynatrace/dynatrace-operator/pkg/util/conditions"
"github.com/Dynatrace/dynatrace-operator/pkg/util/kubeobjects/labels"
"github.com/Dynatrace/dynatrace-operator/pkg/util/timeprovider"
Expand All @@ -22,6 +23,7 @@ type Reconciler interface {
ReconcileAPIUrl(ctx context.Context, dynakube *dynatracev1beta2.DynaKube) error
ReconcileCodeModuleCommunicationHosts(ctx context.Context, dynakube *dynatracev1beta2.DynaKube) error
ReconcileActiveGateCommunicationHosts(ctx context.Context, dynakube *dynatracev1beta2.DynaKube) error
ReconcileCSIDriver(ctx context.Context, dynakube *dynatracev1beta2.DynaKube) error
0sewa0 marked this conversation as resolved.
Show resolved Hide resolved
}

type reconciler struct {
Expand All @@ -38,6 +40,58 @@ func NewReconciler(istio *Client) Reconciler {
}
}

func (r *reconciler) ReconcileCSIDriver(ctx context.Context, dynakube *dynatracev1beta2.DynaKube) error {
log.Info("reconciling istio components for the CSI driver")

if dynakube == nil {
return errors.New("can't reconcile csi driver of nil dynakube")
}

codeModulesURL := dynakube.CodeModulesImage()

parsedCodeModulesURL, err := parseCodeModulesImageURL(codeModulesURL)
if err != nil {
return err
}

codeModulesHost, err := dtclient.ParseEndpoint(parsedCodeModulesURL)
if err != nil {
return err
}

err = r.reconcileCommunicationHosts(ctx, []dtclient.CommunicationHost{codeModulesHost}, CSIDiverComponent)
if err != nil {
return errors.WithMessage(err, "error reconciling config for codeModulesImage")
}

log.Info("reconciled istio objects for CSI driver")

return nil
}

func parseCodeModulesImageURL(rawUrl string) (string, error) {
parsedURL, err := url.Parse(rawUrl)
if err != nil {
return "", errors.New("can't parse the codeModules image URL")
}

if parsedURL.Scheme == "" {
parsedURL.Scheme = "https"

if !strings.HasPrefix(rawUrl, "//") {
// if no scheme at all is set we want to add this prefix
rawUrl = "//" + rawUrl
}
0sewa0 marked this conversation as resolved.
Show resolved Hide resolved

parsedURL, err = url.Parse(parsedURL.Scheme + ":" + rawUrl)
if err != nil {
return "", errors.New("can't parse the codeModules image URL")
}
}

return parsedURL.String(), nil
}

func (r *reconciler) ReconcileAPIUrl(ctx context.Context, dynakube *dynatracev1beta2.DynaKube) error {
log.Info("reconciling istio components for the Dynatrace API url")

Expand Down
109 changes: 109 additions & 0 deletions pkg/controllers/dynakube/istio/reconciler_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"strings"
"testing"

"github.com/Dynatrace/dynatrace-operator/pkg/api/status"
dynatracev1beta2 "github.com/Dynatrace/dynatrace-operator/pkg/api/v1beta2/dynakube"
dtclient "github.com/Dynatrace/dynatrace-operator/pkg/clients/dynatrace"
"github.com/stretchr/testify/assert"
Expand Down Expand Up @@ -212,6 +213,62 @@ func TestReconcileAPIUrl(t *testing.T) {
})
}

func TestReconcileCSIDriver(t *testing.T) {
ctx := context.Background()
dynakube := createTestDynaKube()

t.Run("nil => error", func(t *testing.T) {
istioClient := newTestingClient(nil, dynakube.GetNamespace())
reconciler := NewReconciler(istioClient)

err := reconciler.ReconcileCSIDriver(ctx, nil)
require.Error(t, err)
})
t.Run("malformed image uri (no protocol prefix) => still no error", func(t *testing.T) {
dynakube := createTestDynaKube()
dynakube.Status = dynatracev1beta2.DynaKubeStatus{
CodeModules: dynatracev1beta2.CodeModulesStatus{
VersionStatus: status.VersionStatus{
ImageID: "something-random",
},
},
}
fakeClient := fakeistio.NewSimpleClientset()
istioClient := newTestingClient(fakeClient, dynakube.GetNamespace())
reconciler := NewReconciler(istioClient)

err := reconciler.ReconcileCSIDriver(ctx, dynakube)
require.NoError(t, err)
})
t.Run("success", func(t *testing.T) {
fakeClient := fakeistio.NewSimpleClientset()
istioClient := newTestingClient(fakeClient, dynakube.GetNamespace())
reconciler := NewReconciler(istioClient)

err := reconciler.ReconcileCSIDriver(ctx, dynakube)
require.NoError(t, err)

expectedName := BuildNameForFQDNServiceEntry(dynakube.GetName(), CSIDiverComponent)
serviceEntry, err := fakeClient.NetworkingV1beta1().ServiceEntries(dynakube.GetNamespace()).Get(ctx, expectedName, metav1.GetOptions{})
require.NoError(t, err)
assert.NotNil(t, serviceEntry)

virtualService, err := fakeClient.NetworkingV1beta1().VirtualServices(dynakube.GetNamespace()).Get(ctx, expectedName, metav1.GetOptions{})
require.NoError(t, err)
assert.NotNil(t, virtualService)
})
t.Run("unknown k8s client error => error", func(t *testing.T) {
fakeClient := fakeistio.NewSimpleClientset()
fakeClient.PrependReactor("*", "*", boomReaction)

istioClient := newTestingClient(fakeClient, dynakube.GetNamespace())
reconciler := NewReconciler(istioClient)

err := reconciler.ReconcileCSIDriver(ctx, dynakube)
require.Error(t, err)
})
}

func TestReconcileOneAgentCommunicationHosts(t *testing.T) {
ctx := context.Background()

Expand Down Expand Up @@ -301,6 +358,7 @@ func TestReconcileOneAgentCommunicationHosts(t *testing.T) {

err = r.ReconcileCodeModuleCommunicationHosts(ctx, dynakube)
require.NoError(t, err)

statusCondition = meta.FindStatusCondition(*dynakube.Conditions(), "IstioForCodeModule")
require.Nil(t, statusCondition)

Expand Down Expand Up @@ -450,6 +508,57 @@ func TestReconcileActiveGateCommunicationHosts(t *testing.T) {
})
}

func TestParseCodeModulesImageURL(t *testing.T) {
Copy link
Contributor

@0sewa0 0sewa0 Jul 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is a special case that was missed:

  • when the image used is in dockerhub: codeModulesImage: dynatrace/dynatrace-codemodules:1.283.139.20240209-194956

the resulting ServiceEntry:

apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  annotations: ...
  labels: ...
  name: test-fqdn-csi-driver
  namespace: dynatrace
  ownerReferences: ...
spec:
  hosts:
  - dynatrace
  ports:
  - name: https-443
    number: 443
    protocol: HTTPS
  resolution: DNS

which doesn't help the CSI-driver:

{"level":"info","ts":"2024-07-01T07:55:00.825Z","logger":"oneagent-image","msg":"failed to extract agent binaries from image via proxy","image":"dynatrace/dynatrace-codemodules:1.283.139.20240209-194956","imageCacheDir":"/data/cache","err":"getting image \"dynatrace/dynatrace-codemodules:1.283.139.20240209-194956\": Get \"https://index.docker.io/v2/\": read tcp 10.108.3.147:42910->3.219.239.5:443: read: connection reset by peer","errVerbose":"Get \"https://index.docker.io/v2/\": read tcp 10.108.3.147:42910->3.219.239.5:443: read: connection reset by peer\ngetting image \"dynatrace/dynatrace-codemodules:1.283.139.20240209-194956\""}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how to figure out if the URL is incomplete because its referencing a dockerhub image?

I have no clue really 😬

How the image library does it: https://github.com/google/go-containerregistry/blob/main/pkg/name/repository.go#L80-L87

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice catch, did not consider it. I will have a look 👍

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did it now that way, this ONLY covers docker so I am not sure if we should go for it, pls let me know.
ea80b58

Copy link
Contributor

@0sewa0 0sewa0 Jul 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still doesn't work: 😢 (probably need index.docker.io)

{"error":"getting image \"dynatrace/dynatrace-codemodules:1.293.133.20240618-095559\": Get \"https://index.docker.io/v2/\": read tcp 10.12.1.11:42066-\u003e34.226.69.105:443: read: connection reset by peer","level":"info","logger":"oneagent-image","msg":"pullImageInfo","stacktrace":"Get \"https://index.docker.io/v2/\": read tcp 10.12.1.11:42066-\u003e34.226.69.105:443: read: connection reset by peer
provisioner getting image \"dynatrace/dynatrace-codemodules:1.293.133.20240618-095559\"","ts":"2024-07-03T12:20:54.687Z"}
provisioner {"level":"info","ts":"2024-07-03T12:20:54.687Z","logger":"oneagent-image","msg":"failed to extract agent binaries from image via proxy","image":"dynatrace/dynatrace-codemodules:1.293.133.20240618-095559","imageCacheDir":"/data/cache","err":"getting image \"dynatrace/dynatrace-codemodules:1.293.133.20240618-095559\": Get \"https://index.docker.io/v2/\": read tcp 10.12.1.11:42066->34.226.69.105:443: read: connection reset by peer","errVerbose":"Get \"https://index.docker.io/v2/\": read tcp 10.12.1.11:42066->34.226.69.105:443: read: connection reset by peer\ngetting image \"dynatrace/dynatrace-codemodules:1.293.133.20240618-095559\""}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

provisioner {"level":"info","ts":"2024-07-03T12:46:47.397Z","logger":"oneagent-image","msg":"installing agent from image"}                                                                                         │
│ provisioner {"level":"info","ts":"2024-07-03T12:46:47.397Z","logger":"oneagent-image","msg":"installing agent","target dir":"/data/codemodules/ZHluYXRyYWNlL2R5bmF0cmFjZS1jb2RlbW9kdWxlczoxLjI5My4xMzMuMjAyNDA2MTg │
│ provisioner {"level":"info","ts":"2024-07-03T12:46:47.885Z","logger":"oneagent-image","msg":"pullOciImage","ref_identifier":"1.293.133.20240618-095559","ref.Name":"index.docker.io/dynatrace/dynatrace-codemodule │
│ provisioner {"level":"info","ts":"2024-07-03T12:46:57.345Z","logger":"oneagent-image","msg":"unpackOciImage","sourcePath":"/data/cache/ZHluYXRyYWNlL2R5bmF0cmFjZS1jb2RlbW9kdWxlczoxLjI5My4xMzMuMjAyNDA2MTgtMDk1NTU │
│ provisioner {"level":"info","ts":"2024-07-03T12:46:57.345Z","logger":"oneagent-zip","msg":"extracting tar gzip","source":"/data/cache/ZHluYXRyYWNlL2R5bmF0cmFjZS1jb2RlbW9kdWxlczoxLjI5My4xMzMuMjAyNDA2MTgtMDk1NTU5 │
│ provisioner {"level":"info","ts":"2024-07-03T12:47:07.814Z","logger":"oneagent-zip","msg":"moving unpacked archive to target","targetDir":"/data/codemodules/ZHluYXRyYWNlL2R5bmF0cmFjZS1jb2RlbW9kdWxlczoxLjI5My4xM │
│ provisioner {"level":"info","ts":"2024-07-03T12:47:07.815Z","logger":"oneagent-image","msg":"unpackOciImage","targetDir":"/data/codemodules/ZHluYXRyYWNlL2R5bmF0cmFjZS1jb2RlbW9kdWxlczoxLjI5My4xMzMuMjAyNDA2MTgtMD │
│ provisioner {"level":"info","ts":"2024-07-03T12:47:07.897Z","logger":"oneagent-symlink","msg":"found version","version":"1.293.133.20240618-095559"}                                       

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Is your csi-driver injected by istio?
  2. Is your istio configured to be in restricted mode?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After configuring my istio as you suggested:

istioctl install --set meshConfig.outboundTrafficPolicy.mode=REGISTRY_ONLY

I can confirm that the csi driver is injected with the istio containers.

dynatrace-oneagent-csi-driver-5htgs 5/5 Running 0 5m55s

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are going to love this (IMO, I don't even want to do anything like this, because this shows how unpredictable this whole scenario is)

It still does not work:

provisioner {"error":"getting image \"dynatrace/dynatrace-codemodules:1.293.133.20240618-095559\": Get \"https://auth.docker.io/token?scope=repository%3Adynatrace%2Fdynatrace-codemodules%3Apull\u0026service=registry.docker.io\": read tcp 10.12.3.29:43420-\u003e54.196.99.49:443: read: connection reset by peer","level":"info","logger":"oneagent-image","msg":"pullImageInfo","stacktrace":"Get \"https://auth.docker.io/token?scope=repository%3Adynatrace%2Fdynatrace-codemodules%3Apull\u0026service=registry.docker.io\": read tcp 10.12.3.29:43420-\u003e54.196.99.49:443: read: connection reset by peer"}

so just add auth.docker.io, right?
nope, then you will get:

provisioner {"level":"info","ts":"2024-07-04T12:23:46.322Z","logger":"oneagent-image","msg":"saving v1.Image img as an OCI Image Layout at path","/data/cache":"Get \"https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/64/644c3f0771f461096ccb2c5ce4132447032b833b466cfd845c1a89fd4e5eb762/data?verify=1720098825-yv65IR1rXZ98%2FyHhfEAM%2FMx6cJY%3D\": read tcp 10.12.3.29:55420->104.16.99.215:443: read: connection reset by peer"}

so you have to add production.cloudflare.docker.com as well

Sidenote:

  • using a wildcard like *.docker.io or *.docker.com is not that simple (you have to mess with the DNS within your application if I understand it correctly)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in d23af95 although i do not like this solution at all tbh. Seems to constructed for this case...

tests := []struct {
input string
output string
parsedCorreclty bool
}{
{
input: "some.url.com/test",
output: "https://some.url.com/test",
parsedCorreclty: true,
},
{
input: "http://some.url.com/test",
output: "http://some.url.com/test",
parsedCorreclty: true,
},
{
input: "https://some.url.com/test",
output: "https://some.url.com/test",
parsedCorreclty: true,
},
{
input: ":example.com/test",
output: "",
parsedCorreclty: false,
},
{
input: "//some.url.com/test",
output: "https://some.url.com/test",
parsedCorreclty: true,
},
}
0sewa0 marked this conversation as resolved.
Show resolved Hide resolved

for _, test := range tests {
output, err := parseCodeModulesImageURL(test.input)
if !test.parsedCorreclty {
if err == nil {
t.Errorf("Expected error for input %s, but did not get any", test.input)
}
} else {
if err != nil {
t.Errorf("Did not expect error for input %s, but got one: %v", test.input, err)
}

if output != test.output {
t.Errorf("For input %s, expected output %s, but got this %s", test.input, test.output, output)
}
}
}
waodim marked this conversation as resolved.
Show resolved Hide resolved
}

func createTestIPCommunicationHost() dtclient.CommunicationHost {
return dtclient.CommunicationHost{
Protocol: "http",
Expand Down
47 changes: 47 additions & 0 deletions test/mocks/pkg/controllers/dynakube/istio/reconciler.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading