Redis enterprise (#2433)

* initial create of redis enterprise agent check

* initial add of enterprise tile

* added codeowners

* added CODEOWNERS, set disply=true, synced conf.yaml.example

* fix dashhboard names

* manifest passes validation

* update namespace

* review readme, example config

* addressed comments, fixed lint errors

* fixed validation sync issue

* tiny lint fixes

* fix import lint issues

* last lint fixes

* last lint error only fixed by ruff --config ../pyproject.toml --fix .

* corrected unit and sub-unit declarations

* fixed description, remove non-v2 dashboards, minor edits

* added svg logo

* yet another svg logo

* delete unused logos

* change logo url in overview

* replace conf.yaml.example

* fix lint errors on metrics

* fixed manifest, validate all passes

* linter, again (line too long)

* linter, again (missing commas)

* linter, again (delete empty line)

* unit & integration tests, corrected metrics, etc.

* linter fixes

* one tiny linter fix

* import block fixes

* attempt to reduce sleeps so that duration < 360

* commented integration tests - they take longer than 360 seconds to run

* comment unused method import used by integration

* fix comment for linter

* remove commented metric

* commented unit test that fails in github but passes locally

* removed all docker actions from conftest as they time out

* return instances from dd_environment but do not run setup

* empty e2e test

* linter says add a space

* correct some metrics and change a dashboard key

* addresses recent reviewers comments

* fix remaining

* updated README, added images, etc.

* add another image

* README tailored for enterprise with more explicit usage guidelines

* revamped overview, removed all integration test support

* use new redis logo and replace three images with new ones

* remove unused dashboard, add description

* modify manifest for removed dashboards

* add images to media, remove commented code

* deleted eleventh column in metadata

* Array of Dictionaries

* Array of Dictionaries II

* Array of Dictionaries III, trailing comma

* fix media entries, move images, interval to integer

* fix metadata

* remove per_unit_name values

* adjust svg icon for aspect ratio

* two dashboards

* typo from pretty print

* logo placement adjustment

* logo placement adjustment

* logo placement adjustment

* logo placement adjustment

* remove v2 from name

* re-sync after manifest edit

* re-sync after spec.yaml edit

* name identifier can't be changed at this time

* rename dashboards so they will be unique

* reset source type name to include V2

* sync

* change service name, too

* change name of overview dashboard to include dash

* clarify configuration example

* Update CODEOWNERS

added @DataDog/logs-backend to tailscale

* corrected metadata typing

* sync fixes

* sync fixes

* remove unnecessary check() implementation

* remove extra line

* add extra line

* Revert "sync fixes"

This reverts commit 949a4ae.

* Revert "Update CODEOWNERS"

This reverts commit 105a35d.

* set deprecation of previous version to sept 1, 2024

* change support email

* replace raises with try/catch

* lint fixes on top of lint workarounds

* removed broken instance down test

* corrected overview graphic link, edited redis enterprise database dashboard's description

* Update metadata.csv

fixed copy pasta

* reset conf example

* set integration to redis_enterprise_v2 in metadata.csv

* edit metadata.csv

* edit metadata.csv(2)

* edit metadata.csv(3)

* set banner_img url, modified config template to include necessary params

* trigger GitHub actions

* add excludes to spec.yaml, code support in check

* remove unused variable

* use renamed parameter in tests

* fix addition of extra_metrics

* desperately trying to fix something that is broken - running ddev validate config produces no changes

* desperately trying to fix something that is broken - running ddev validate config produces no changes #2

* manually updating instance,py

* regenerated conf.yaml.example, added test for exclude metrics

* re-validated with ddev 9.10

* fix linter issues

* yet another validation run

* fixed typo

* remove queried data type from manifest

* remove namespace from spec.yaml as it shouldn't be changed

* run validate models

redis_enterprise/README.md

@@ -4,7 +4,7 @@
 
 Redis is a fast, versatile data store that supports a variety of data structures, including strings, hashes, lists, sets, streams, and more. It also offers programmability, extensibility, persistence, clustering, and high availability. The community edition of Redis adds additional data models and capabilities, which include vector search, probabilistic data structures, JSON support, and full-text search.
 
-This integration works with on-premsises and private cloud installations of [Redis Enterprise][1].
+This integration works with on-premises and private cloud installations of [Redis Enterprise][1].
 The integration provides metrics for three critical cluster components: databases, nodes, and shards. This allows you to monitor database throughput, memory utilization, CPU usage, connection counts, replication health, and a variety of additional metrics within Datadog.
 You can use this information to understand the overall health of your Redis Enterprise clusters, diagnose application performance issues, and prevent downtime.
 

redis_enterprise/assets/configuration/spec.yaml

@@ -7,4 +7,16 @@ files:
     - template: init_config/default
   - template: instances
     options:
-    - template: instances/default
+    - template: instances/openmetrics
+      overrides:
+        openmetrics_endpoint.value.example: https://<host>:8070/metrics
+        openmetrics_endpoint.description: |
+          Insert Description here for rdse OpenMetrics endpoint or link to docs
+        openmetrics_endpoint.display_priority: 3
+        tls_verify.value.example: false
+        tls_verify.enabled: true
+        tls_verify.display_priority: 1
+        extra_metrics.value.example: [ foo, bar ]
+        extra_metrics.enabled: false
+        exclude_metrics.value.example: [ foo, bar ]
+        exclude_metrics.enabled: false

redis_enterprise/assets/dashboards/redis_enterprise_database.json

@@ -1,6 +1,6 @@
 {
   "title": "Redis Enterprise - Database",
-  "description": "[[Redis Enterprise - Overview]]",
+  "description": "Redis Enterprise Databases are real-time, partitioned databases that use modules to provide additional functionality",
   "widgets": [
     {
       "id": 8148371123349408,

redis_enterprise/assets/dashboards/redis_enterprise_overview.json

@@ -6,7 +6,7 @@
       "id": 325090501093728,
       "definition": {
         "title": "About Redis Enterprise",
-        "banner_img": "https://raw.githubusercontent.com/DataDog/integrations-extras/58e0302ab52d7e1425cdcc5eb14dd16b9898db9e/redis_enterprise/assets/logos/logo.svg",
+        "banner_img": "https://raw.githubusercontent.com/DataDog/integrations-extras/master/redis_enterprise/assets/logos/logo.svg",
         "show_title": false,
         "type": "group",
         "layout_type": "ordered",

redis_enterprise/datadog_checks/redis_enterprise/check.py

@@ -23,16 +23,26 @@ def _parse_config(self):
         metrics = self.get_default_config()
 
         additional = []
-        groups = self.instance.get('metric_groups', [])
+        groups = self.instance.get('extra_metrics', [])
         for g in groups:
-            if g not in ADDITIONAL_METRICS:
-                raise ConfigurationError(f'invalid metric in config: {g}')
+            if g not in ADDITIONAL_METRICS.keys():
+                raise ConfigurationError(f'invalid group in extra_metrics: {g}')
             additional.append(ADDITIONAL_METRICS[g])
 
         if len(additional) > 0:
             self.service_check("more_groups", AgentCheck.OK)
             metrics += additional
 
+        excludes = self.instance.get('exclude_metrics', [])
+        for m in excludes:
+            found = False
+            for mg in metrics:
+                if m in mg.keys():
+                    mg.pop(m)
+                    found = True
+            if not found:
+                raise ConfigurationError(f'invalid metric in excludes: {m}')
+
         config = {
             'openmetrics_endpoint': metrics_endpoint,
             'namespace': self.__NAMESPACE__,

redis_enterprise/datadog_checks/redis_enterprise/config_models/defaults.py

@@ -4,6 +4,30 @@
 #     ddev -x validate models -s <INTEGRATION_NAME>
 
 
+def instance_allow_redirects():
+    return True
+
+
+def instance_auth_type():
+    return 'basic'
+
+
+def instance_cache_metric_wildcards():
+    return True
+
+
+def instance_cache_shared_labels():
+    return True
+
+
+def instance_collect_counters_with_distributions():
+    return False
+
+
+def instance_collect_histogram_buckets():
+    return True
+
+
 def instance_disable_generic_tags():
     return False
 
@@ -12,5 +36,85 @@ def instance_empty_default_hostname():
     return False
 
 
+def instance_enable_health_service_check():
+    return True
+
+
+def instance_histogram_buckets_as_distributions():
+    return False
+
+
+def instance_ignore_connection_errors():
+    return False
+
+
+def instance_kerberos_auth():
+    return 'disabled'
+
+
+def instance_kerberos_delegate():
+    return False
+
+
+def instance_kerberos_force_initiate():
+    return False
+
+
+def instance_log_requests():
+    return False
+
+
 def instance_min_collection_interval():
     return 15
+
+
+def instance_non_cumulative_histogram_buckets():
+    return False
+
+
+def instance_persist_connections():
+    return False
+
+
+def instance_request_size():
+    return 16
+
+
+def instance_skip_proxy():
+    return False
+
+
+def instance_tag_by_endpoint():
+    return True
+
+
+def instance_telemetry():
+    return False
+
+
+def instance_timeout():
+    return 10
+
+
+def instance_tls_ignore_warning():
+    return False
+
+
+def instance_tls_use_host_header():
+    return False
+
+
+def instance_tls_verify():
+    return False
+
+
+def instance_use_latest_spec():
+    return False
+
+
+def instance_use_legacy_auth_encoding():
+    return True
+
+
+def instance_use_process_start_time():
+    return False

redis_enterprise/datadog_checks/redis_enterprise/config_models/instance.py

@@ -5,16 +5,36 @@
 
 from __future__ import annotations
 
-from typing import Optional
+from types import MappingProxyType
+from typing import Any, Optional, Union
 
-from pydantic import BaseModel, ConfigDict, field_validator, model_validator
+from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
 
 from datadog_checks.base.utils.functions import identity
 from datadog_checks.base.utils.models import validation
 
 from . import defaults, validators
 
 
+class AuthToken(BaseModel):
+    model_config = ConfigDict(
+        arbitrary_types_allowed=True,
+        frozen=True,
+    )
+    reader: Optional[MappingProxyType[str, Any]] = None
+    writer: Optional[MappingProxyType[str, Any]] = None
+
+
+class ExtraMetrics(BaseModel):
+    model_config = ConfigDict(
+        arbitrary_types_allowed=True,
+        extra='allow',
+        frozen=True,
+    )
+    name: Optional[str] = None
+    type: Optional[str] = None
+
+
 class MetricPatterns(BaseModel):
     model_config = ConfigDict(
         arbitrary_types_allowed=True,
@@ -24,18 +44,108 @@ class MetricPatterns(BaseModel):
     include: Optional[tuple[str, ...]] = None
 
 
+class Metrics(BaseModel):
+    model_config = ConfigDict(
+        arbitrary_types_allowed=True,
+        extra='allow',
+        frozen=True,
+    )
+    name: Optional[str] = None
+    type: Optional[str] = None
+
+
+class Proxy(BaseModel):
+    model_config = ConfigDict(
+        arbitrary_types_allowed=True,
+        frozen=True,
+    )
+    http: Optional[str] = None
+    https: Optional[str] = None
+    no_proxy: Optional[tuple[str, ...]] = None
+
+
+class ShareLabels(BaseModel):
+    model_config = ConfigDict(
+        arbitrary_types_allowed=True,
+        frozen=True,
+    )
+    labels: Optional[tuple[str, ...]] = None
+    match: Optional[tuple[str, ...]] = None
+
+
 class InstanceConfig(BaseModel):
     model_config = ConfigDict(
         validate_default=True,
         arbitrary_types_allowed=True,
         frozen=True,
     )
+    allow_redirects: Optional[bool] = None
+    auth_token: Optional[AuthToken] = None
+    auth_type: Optional[str] = None
+    aws_host: Optional[str] = None
+    aws_region: Optional[str] = None
+    aws_service: Optional[str] = None
+    cache_metric_wildcards: Optional[bool] = None
+    cache_shared_labels: Optional[bool] = None
+    collect_counters_with_distributions: Optional[bool] = None
+    collect_histogram_buckets: Optional[bool] = None
+    connect_timeout: Optional[float] = None
     disable_generic_tags: Optional[bool] = None
     empty_default_hostname: Optional[bool] = None
+    enable_health_service_check: Optional[bool] = None
+    exclude_labels: Optional[tuple[str, ...]] = None
+    exclude_metrics: Optional[tuple[str, ...]] = None
+    exclude_metrics_by_labels: Optional[MappingProxyType[str, Union[bool, tuple[str, ...]]]] = None
+    extra_headers: Optional[MappingProxyType[str, Any]] = None
+    extra_metrics: Optional[tuple[Union[str, MappingProxyType[str, Union[str, ExtraMetrics]]], ...]] = None
+    headers: Optional[MappingProxyType[str, Any]] = None
+    histogram_buckets_as_distributions: Optional[bool] = None
+    hostname_format: Optional[str] = None
+    hostname_label: Optional[str] = None
+    ignore_connection_errors: Optional[bool] = None
+    ignore_tags: Optional[tuple[str, ...]] = None
+    include_labels: Optional[tuple[str, ...]] = None
+    kerberos_auth: Optional[str] = None
+    kerberos_cache: Optional[str] = None
+    kerberos_delegate: Optional[bool] = None
+    kerberos_force_initiate: Optional[bool] = None
+    kerberos_hostname: Optional[str] = None
+    kerberos_keytab: Optional[str] = None
+    kerberos_principal: Optional[str] = None
+    log_requests: Optional[bool] = None
     metric_patterns: Optional[MetricPatterns] = None
+    metrics: Optional[tuple[Union[str, MappingProxyType[str, Union[str, Metrics]]], ...]] = None
     min_collection_interval: Optional[float] = None
+    namespace: Optional[str] = Field(None, pattern='\\w*')
+    non_cumulative_histogram_buckets: Optional[bool] = None
+    ntlm_domain: Optional[str] = None
+    openmetrics_endpoint: str
+    password: Optional[str] = None
+    persist_connections: Optional[bool] = None
+    proxy: Optional[Proxy] = None
+    raw_line_filters: Optional[tuple[str, ...]] = None
+    raw_metric_prefix: Optional[str] = None
+    read_timeout: Optional[float] = None
+    rename_labels: Optional[MappingProxyType[str, Any]] = None
+    request_size: Optional[float] = None
     service: Optional[str] = None
+    share_labels: Optional[MappingProxyType[str, Union[bool, ShareLabels]]] = None
+    skip_proxy: Optional[bool] = None
+    tag_by_endpoint: Optional[bool] = None
     tags: Optional[tuple[str, ...]] = None
+    telemetry: Optional[bool] = None
+    timeout: Optional[float] = None
+    tls_ca_cert: Optional[str] = None
+    tls_cert: Optional[str] = None
+    tls_ignore_warning: Optional[bool] = None
+    tls_private_key: Optional[str] = None
+    tls_protocols_allowed: Optional[tuple[str, ...]] = None
+    tls_use_host_header: Optional[bool] = None
+    tls_verify: Optional[bool] = None
+    use_latest_spec: Optional[bool] = None
+    use_legacy_auth_encoding: Optional[bool] = None
+    use_process_start_time: Optional[bool] = None
+    username: Optional[str] = None
 
     @model_validator(mode='before')
     def _initial_validation(cls, values):