operator: Use cluster monitoring alertmanager by default on openshift clusters #7272

aminesnow · 2022-09-27T16:55:10Z

What this PR does / why we need it:
This PR enables the use of cluster monitoring alertmanager by default on Openshift clusters when applicable. The configuration can however be overridden by the user config.

Special notes for your reviewer:

Checklist

Reviewed the CONTRIBUTING.md guide
Documentation added
Tests updated
CHANGELOG.md updated
Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

grafanabot · 2022-09-27T17:01:01Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

-           ingester	-0.1%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0.6%

grafanabot · 2022-09-28T16:21:22Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

periklis

In general it looks good to me. Only a couple of code separation improvements suggested to keep openshift things in a single package.

periklis · 2022-09-28T16:28:30Z

operator/internal/handlers/lokistack_create_or_update.go

+	if stack.Spec.Tenants != nil && opts.Stack.Tenants.Mode == lokiv1.OpenshiftLogging {
+		var svc corev1.Service
+		key := client.ObjectKey{Name: manifests.MonitoringSVCOperated, Namespace: manifests.MonitoringNS}
+
+		err = k.Get(ctx, key, &svc)
+		if err != nil && !apierrors.IsNotFound(err) {
+			return kverrors.Wrap(err, "failed to lookup alertmanager service", "name", key)
+		}
+
+		if err == nil {
+			opts.Ruler.OCPAlertManagerEnabled = true
+		}
+	}


I suggest to push this code block in a separate package internal to handlers package. This handler is already too big and needs some uncluttering, let's start with new additions like this one.

periklis · 2022-09-28T16:31:42Z

operator/internal/manifests/config.go

+	if opt.Stack.Tenants != nil && opt.Ruler.OCPAlertManagerEnabled && opt.Stack.Tenants.Mode == lokiv1.OpenshiftLogging {
 		rulerEnabled = true

+		if opt.Ruler.Spec == nil {
+			opt.Ruler.Spec = &lokiv1beta1.RulerConfigSpec{
+				AlertManagerSpec: &lokiv1beta1.AlertManagerSpec{},
+			}
+		}
+
+		if opt.Ruler.Spec.AlertManagerSpec == nil || len(opt.Ruler.Spec.AlertManagerSpec.Endpoints) == 0 {
+			ams := &lokiv1beta1.AlertManagerSpec{
+				Endpoints: []string{"https://_web._tcp.alertmanager-operated.openshift-monitoring.svc"},
+				EnableV2:  true,
+				DiscoverySpec: &lokiv1beta1.AlertManagerDiscoverySpec{
+					EnableSRV:       true,
+					RefreshInterval: "1m",
+				},
+			}
+
+			if err := mergo.Merge(opt.Ruler.Spec.AlertManagerSpec, ams); err != nil {
+				return config.Options{}, kverrors.Wrap(err, "failed merging RulerSpec options")
+			}
+		}
+
+	}


The general pattern to apply openshift partial patches for automation looks like this:

Create a configure method in the openshift package

Apply the configure after ConfigOptions() returns and handle any error

See ConfigureDeploymentForTenantMode.

periklis · 2022-09-28T16:35:28Z

operator/internal/manifests/var.go

+	monitoringSVCMain = "alertmanager-main"
+	// MonitoringNS is the namespace containing cluster monitoring objects such as alertmanager.
+	MonitoringNS = "openshift-monitoring"
+	// MonitoringSVCOperated is the name of the alertmanager service used for alerts.
+	MonitoringSVCOperated = "alertmanager-operated"


All these belongs to the openshift package.

periklis · 2022-09-28T16:37:11Z

operator/internal/manifests/config_test.go

+			wantOptions: nil,
+		},
+		{
+			desc: "openshift-logging mode",


We have a fourth mode to test nowadays openshift-network

operator/internal/manifests/config_test.go

grafanabot · 2022-09-29T14:53:31Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
- querier/queryrange	-0.1%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-09-29T15:43:51Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

periklis · 2022-09-29T16:55:32Z

operator/internal/manifests/config.go

+	if opt.OpenShiftOptions.BuildOpts.OCPAlertManagerEnabled {
+		if err = ConfigureOptionsForMode(&cfg, opt.Stack.Tenants.Mode); err != nil {
+			return nil, "", err
+		}
+	}


Can you flip the outer if block inside openshift.ConfigureOptions(cfg). This way we limit exposing OpenShiftOptions outside of the openshift package.

periklis · 2022-09-29T16:57:03Z

operator/internal/handlers/internal/rules/rules.go

+// OCPAlertManagerEnabled returns true if the Openshift AlertManager is present in the cluster.
+func OCPAlertManagerEnabled(ctx context.Context, opts manifests.Options, k k8s.Client) (bool, error) {
+	if opts.Stack.Tenants != nil && opts.Stack.Tenants.Mode == lokiv1.OpenshiftLogging {
+		var svc corev1.Service
+		key := client.ObjectKey{Name: manifests.MonitoringSVCOperated, Namespace: manifests.MonitoringNS}
+
+		err := k.Get(ctx, key, &svc)
+		if err != nil && !apierrors.IsNotFound(err) {
+			return false, kverrors.Wrap(err, "failed to lookup alertmanager service", "name", key)
+		}
+
+		return err == nil, nil
+	}
+
+	return false, nil
+}


I suggest we create a separate openshift package and place the code in a file alertmanager.go, naming the function as openshift.alertManagerSvcExists

periklis · 2022-09-29T16:57:44Z

operator/internal/manifests/config.go

+func ConfigureOptionsForMode(cfg *config.Options, mode lokiv1.ModeType) error {
+	switch mode {
+	case lokiv1.Static, lokiv1.Dynamic:
+		return nil // nothing to configure
+	case lokiv1.OpenshiftLogging, lokiv1.OpenshiftNetwork:
+		return openshift.ConfigureOptions(cfg)
+	}
+
+	return nil
+}


Please move this to the gateway_tenants.go file where all the other configure function live.

why gateway_tenants? it has nothing to do with the gateway..

Actually the filename is wrong nowadays it should be more tenants.go

ok I see, I will rename it then to avoid confusion

Let's rename in a separate PR, because this is a crucial file that will break many PRs right now.

periklis · 2022-09-29T16:58:40Z

operator/internal/manifests/openshift/options.go

+	GatewaySvcTargetPort   string
+	RulerName              string
+	Labels                 map[string]string
+	OCPAlertManagerEnabled bool


You can skip the OCP prefix here, as the field is accessible via openshift.BuiltOptions.AlertManagerEnabled

grafanabot · 2022-09-30T09:15:52Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

periklis

Great work!

periklis · 2022-09-30T17:51:24Z

operator/internal/handlers/internal/openshift/alertmanager.go

+	if opts.Stack.Tenants != nil && opts.Stack.Tenants.Mode == lokiv1.OpenshiftLogging {
+		var svc corev1.Service
+		key := client.ObjectKey{Name: openshift.MonitoringSVCOperated, Namespace: openshift.MonitoringNS}
+
+		err := k.Get(ctx, key, &svc)
+		if err != nil && !apierrors.IsNotFound(err) {
+			return false, kverrors.Wrap(err, "failed to lookup alertmanager service", "name", key)
+		}
+
+		return err == nil, nil
+	}
+
+	return false, nil


Suggested change

if opts.Stack.Tenants != nil && opts.Stack.Tenants.Mode == lokiv1.OpenshiftLogging {

var svc corev1.Service

key := client.ObjectKey{Name: openshift.MonitoringSVCOperated, Namespace: openshift.MonitoringNS}

err := k.Get(ctx, key, &svc)

if err != nil && !apierrors.IsNotFound(err) {

return false, kverrors.Wrap(err, "failed to lookup alertmanager service", "name", key)

}

return err == nil, nil

}

return false, nil

if opts.Stack.Tenants == nil || opts.Stack.Tenants.Mode != lokiv1.OpenshiftLogging {

return false, nil

}

var svc corev1.Service

key := client.ObjectKey{Name: openshift.MonitoringSVCOperated, Namespace: openshift.MonitoringNS}

err := k.Get(ctx, key, &svc)

if err != nil && !apierrors.IsNotFound(err) {

return false, kverrors.Wrap(err, "failed to lookup alertmanager service", "name", key)

}

return err == nil, nil

grafanabot · 2022-10-03T08:32:45Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

grafanabot · 2022-10-04T07:54:16Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

periklis

Two minor bits and we are set to go.

periklis · 2022-10-04T07:54:46Z

operator/internal/handlers/lokistack_create_or_update.go

+	enabled, err := openshift.AlertManagerSVCExists(ctx, opts, k)
+	if err != nil {
+		ll.Error(err, "failed to check OCP AlertManager")
+		return err
+	}
+
+	opts.OpenShiftOptions.BuildOpts.AlertManagerEnabled = enabled


Can we move this up to the the if-branch on Line 173 to make this call optional only if rules are enabled.

periklis · 2022-10-04T07:56:01Z

operator/internal/manifests/config.go

@@ -45,7 +55,7 @@ func LokiConfigMap(opt Options) (*corev1.ConfigMap, string, error) {
 }

 // ConfigOptions converts Options to config.Options
-func ConfigOptions(opt Options) config.Options {
+func ConfigOptions(opt Options) (config.Options, error) {


Nothing is throwing an error here anymore. Let's remove the error return type.

…o operator_alertmanager

grafanabot · 2022-10-04T13:05:48Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

… clusters (grafana#7272)

Mohamed-Amine Bouqsimi added 4 commits September 27, 2022 18:35

wip

bd0aeda

add dns resolution and comments

3d5fa0f

fixes

4ea2316

add ut

7aa7a01

pull-request-size bot added the size/L label Sep 27, 2022

github-actions bot added the sig/operator label Sep 27, 2022

aminesnow marked this pull request as ready for review September 28, 2022 16:15

aminesnow requested a review from a team as a code owner September 28, 2022 16:15

Mohamed-Amine Bouqsimi added 2 commits September 28, 2022 18:15

refactor

6c253b4

changelog

aca5da8

periklis requested changes Sep 28, 2022

View reviewed changes

pull-request-size bot added size/XL and removed size/L labels Sep 29, 2022

refactor

43bb609

more refactoring

80f6747

periklis requested changes Sep 29, 2022

View reviewed changes

refactor

9cada6c

periklis reviewed Sep 30, 2022

View reviewed changes

refactor

bd7943b

Merge branch 'main' into operator_alertmanager

fe1c860

periklis reviewed Oct 4, 2022

View reviewed changes

Mohamed-Amine Bouqsimi added 2 commits October 4, 2022 14:59

refactor

d38b85a

Merge branch 'operator_alertmanager' of github.com:aminesnow/loki int…

90b4b84

…o operator_alertmanager

periklis approved these changes Oct 4, 2022

View reviewed changes

periklis merged commit 4103b32 into grafana:main Oct 4, 2022

lxwzy pushed a commit to lxwzy/loki that referenced this pull request Nov 7, 2022

operator: Use cluster monitoring alertmanager by default on openshift…

cef696b

… clusters (grafana#7272)

changhyuni pushed a commit to changhyuni/loki that referenced this pull request Nov 8, 2022

operator: Use cluster monitoring alertmanager by default on openshift…

5e84fdb

… clusters (grafana#7272)

Abuelodelanada pushed a commit to canonical/loki that referenced this pull request Dec 1, 2022

operator: Use cluster monitoring alertmanager by default on openshift…

ea5c4d1

… clusters (grafana#7272)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

operator: Use cluster monitoring alertmanager by default on openshift clusters #7272

operator: Use cluster monitoring alertmanager by default on openshift clusters #7272

aminesnow commented Sep 27, 2022 •

edited

Loading

grafanabot commented Sep 27, 2022

grafanabot commented Sep 28, 2022

periklis left a comment

periklis Sep 28, 2022

periklis Sep 28, 2022

periklis Sep 28, 2022

periklis Sep 28, 2022

grafanabot commented Sep 29, 2022

grafanabot commented Sep 29, 2022

periklis Sep 29, 2022

periklis Sep 29, 2022

periklis Sep 29, 2022

aminesnow Sep 30, 2022

periklis Sep 30, 2022

aminesnow Sep 30, 2022

periklis Sep 30, 2022

aminesnow Sep 30, 2022

periklis Sep 29, 2022

grafanabot commented Sep 30, 2022

periklis left a comment

periklis Sep 30, 2022

grafanabot commented Oct 3, 2022

grafanabot commented Oct 4, 2022

periklis left a comment

periklis Oct 4, 2022

periklis Oct 4, 2022

grafanabot commented Oct 4, 2022

operator: Use cluster monitoring alertmanager by default on openshift clusters #7272

operator: Use cluster monitoring alertmanager by default on openshift clusters #7272

Conversation

aminesnow commented Sep 27, 2022 • edited Loading

grafanabot commented Sep 27, 2022

grafanabot commented Sep 28, 2022

periklis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grafanabot commented Sep 29, 2022

grafanabot commented Sep 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grafanabot commented Sep 30, 2022

periklis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grafanabot commented Oct 3, 2022

grafanabot commented Oct 4, 2022

periklis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grafanabot commented Oct 4, 2022

aminesnow commented Sep 27, 2022 •

edited

Loading