Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CECO-2][DatadogMonitor] Configure force sync period #1404

Merged
merged 2 commits into from
Oct 8, 2024
Merged

[CECO-2][DatadogMonitor] Configure force sync period #1404

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4a1ce82
add env var to configure sync period + update doc
tbavelier Sep 11, 2024
6a54ec4
Merge branch 'main' into tbavelier/parameterize_force_sync_monitor
tbavelier Oct 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/datadog_monitor.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,11 @@ To deploy a `DatadogMonitor` with the Datadog Operator, use the [`datadog-operat
```

This automatically creates a new monitor in Datadog. You can find it on the [Manage Monitors][7] page of your Datadog account.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding this break line so the note stands out properly instead of being on the same line:
image

*Note*: All monitors created from `DatadogMonitor` are automatically tagged with `generated:kubernetes`.

By default, the Operator ensures that the API monitor definition stays in sync with the DatadogMonitor resource every **60** minutes (per monitor). This interval can be adjusted using the environment variable `DD_MONITOR_FORCE_SYNC_PERIOD`, which specifies the number of minutes. For example, setting this variable to `"30"` changes the interval to 30 minutes.

## Cleanup

The following commands delete the monitor from your Datadog account and all the Kubernetes resources created by the above instructions:
Expand Down
24 changes: 18 additions & 6 deletions internal/controller/datadogmonitor/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,9 @@ package datadogmonitor
import (
"context"
"fmt"
"os"
"sort"
"strconv"
"strings"
"time"

Expand All @@ -35,10 +37,11 @@ import (
)

const (
defaultRequeuePeriod = 60 * time.Second
defaultErrRequeuePeriod = 5 * time.Second
defaultForceSyncPeriod = 60 * time.Minute
maxTriggeredStateGroups = 10
defaultRequeuePeriod = 60 * time.Second
defaultErrRequeuePeriod = 5 * time.Second
defaultForceSyncPeriod = 60 * time.Minute
maxTriggeredStateGroups = 10
DDMonitorForceSyncPeriodEnvVar = "DD_MONITOR_FORCE_SYNC_PERIOD"
)

var supportedMonitorTypes = map[string]bool{
Expand Down Expand Up @@ -89,11 +92,20 @@ func (r *Reconciler) internalReconcile(ctx context.Context, req reconcile.Reques
logger := r.log.WithValues("datadogmonitor", req.NamespacedName)
logger.Info("Reconciling DatadogMonitor")
now := metav1.NewTime(time.Now())
forceSyncPeriod := defaultForceSyncPeriod

forceSyncPeriodInt, err := strconv.Atoi(os.Getenv(DDMonitorForceSyncPeriodEnvVar))
if err != nil {
logger.Error(err, "Invalid value for monitor force sync period. Defaulting to 60 minutes.")
} else {
logger.V(1).Info("Setting monitor force sync period", "minutes", forceSyncPeriodInt)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid spamming the used sync period, this log is set to appear only in debug with logger.V(1)

forceSyncPeriod = time.Duration(forceSyncPeriodInt) * time.Minute
}

// Get instance
instance := &datadoghqv1alpha1.DatadogMonitor{}
var result ctrl.Result
err := r.client.Get(ctx, req.NamespacedName, instance)
err = r.client.Get(ctx, req.NamespacedName, instance)
if err != nil {
if apierrors.IsNotFound(err) {
// Request object not found, could have been deleted after reconcile request.
Expand Down Expand Up @@ -140,7 +152,7 @@ func (r *Reconciler) internalReconcile(ctx context.Context, req reconcile.Reques
// Custom resource manifest has changed, need to update the API
logger.V(1).Info("DatadogMonitor manifest has changed")
shouldUpdate = true
} else if instance.Status.MonitorLastForceSyncTime == nil || (defaultForceSyncPeriod-now.Sub(instance.Status.MonitorLastForceSyncTime.Time)) <= 0 {
} else if instance.Status.MonitorLastForceSyncTime == nil || (forceSyncPeriod-now.Sub(instance.Status.MonitorLastForceSyncTime.Time)) <= 0 {
// Periodically force a sync with the API monitor to ensure parity
// Get monitor to make sure it exists before trying any updates. If it doesn't, set shouldCreate
m, err = r.get(instance, newStatus)
Expand Down
Loading