Skip to content

Service Monitoring Descriptor Language Reference

Jump to bottom
Endre Major edited this page Apr 28, 2021 · 29 revisions

The Service Monitoring Definition Descriptor Language (MDL) is used in the service.mdl file to describe monitoring-related information for the CSD service to Cloudera Manager. It is worth first looking at the CSD primer to get an idea of how all the CSD components fit together.

Below is a description of each section of the MDL in more detail. There are some structures that are shared between sections. For example, the metrics under service are the same structures as the metrics under roles. Those structures will be described at the end.

Note that that the structure described here is evolving. There is no guarantee that there will be backwards support for them. This caveat will be removed in a future release.

Also note that this feature is not yet useful for third-party CSD developers. While it is now possible to define metrics in a CSD, it is not yet possible to publish metrics to Cloudera Manager. This functionality will be included in a future release.

Service

This is the top level section that describes the service as a whole. For metricDefinitions use the "roles" section for pure CSD projects.

Example

{
  "name" : "ECHO",
  "nameForCrossEntityAggregateMetrics" : "echos",
  "version" : 1,
  "metricDefinitions" : [
    {
      "name" : "echo_service_metric_one",
      "label" : "Echo Service Metric One",
      "description" : "This is the Echo Service's first metric.",
      "numeratorUnit" : "bytes",
      "type" : "gauge"
    },
    {
      "name" : "echo_service_metric_two",
      "label" : "Echo Service Metric Two",
      "description" : "This is the Echo Service's second metric.",
      "numeratorUnit" : "echos",
      "type" : "counter"
    },
    {
      "name" : "echo_common_metric",
      "label" : "Echo Service Common Metric",
      "description" : "This is the Echo Service's common metric.",
      "numeratorUnit" : "seconds",
      "type" : "gauge"
    }
  ],
  "metricEntityAttributeDefinitions" : [
    {
      "name" : "echoServiceAttributeOne",
      "label" : "Echo Service Attribute One",
      "description" : "The Echo Service's First Entity Attribute",
      "valueCaseSensitive": false
    },
    {
      "name" : "echoServiceAttributeTwo",
      "label" : "Echo Service Attribute Two",
      "description" : "The Echo Service's Second Entity Attribute",
      "valueCaseSensitive": true
    }
  ],
  "metricEntityTypeDefinitions" : [
    {
      "name" : "ECHO_ENTITY_TYPE_ONE",
      "nameForCrossEntityAggregateMetrics" : "echo_entity_type_ones",
      "entityNameFormat" :  [
         "serviceName",
         "echoServiceAttributeOne"
      ],
      "label" : "Echo Service Entity Type One",
      "labelPlural" : "Echo Service Entity Type Ones",
      "description" : "The Echo Service's First Entity Type",
      "immutableAttributeNames" : [
         "serviceName",
         "echoServiceAttributeOne"
      ],
      "mutableAttributeNames" : [
        "echoServiceAttributeTwo"
      ],
      "parentMetricEntityTypeNames" : [
         "ECHO"
      ],
      "metricDefinitions" : [
        {
          "name" : "echo_entity_type_one_metric_one",
          "label" : "Echo Service Entity Type One Metric One",
          "description" : "This is the Echo Service's Entity Type One's first metric.",
          "numeratorUnit" : "bytes",
          "type" : "gauge"
        },
        {
          "name" : "echo_entity_type_one_metric_two",
          "label" : "Echo Service Entity Type One Metric Two",
          "description" : "This is the Echo Service's Entity Type Two's first metric.",
          "numeratorUnit" : "echos",
          "type" : "counter"
        },
        {
          "name" : "echo_common_metric",
          "label" : "Echo Service Common Metric",
          "description" : "This is the Echo Service's common metric.",
          "numeratorUnit" : "seconds",
          "type" : "gauge"
        }
      ]
    },
    {
      "name" : "ECHO_ENTITY_TYPE_TWO",
      "nameForCrossEntityAggregateMetrics" : "echo_entity_type_twos",
      "entityNameFormat" :  [
         "roleName",
         "echoServiceAttributeOne"
      ],
      "label" : "Echo Service Entity Type Two",
      "labelPlural" : "Echo Service Entity Type Twos",
      "description" : "The Echo Service's Second Entity Type",
      "immutableAttributeNames" : [
         "roleName",
         "echoServiceAttributeOne"
      ],
      "parentMetricEntityTypeNames" : [
         "ECHO-ECHO_WEBSERVER",
         "ECHO_ENTITY_TYPE_ONE"
      ],
      "metricDefinitions" : [
        {
          "name" : "echo_entity_type_two_metric_one",
          "label" : "Echo Service Entity Type Two Metric One",
          "description" : "This is the Echo Service's Entity Type Two's first metric.",
          "numeratorUnit" : "bytes",
          "type" : "gauge"
        },
        {
          "name" : "echo_entity_type_two_metric_two",
          "label" : "Echo Service Entity Type Two Metric Two",
          "description" : "This is the Echo Service's Entity Type Two's second metric.",
          "numeratorUnit" : "echos",
          "type" : "counter"
        },
        {
          "name" : "echo_common_metric",
          "label" : "Echo Service Common Metric",
          "description" : "This is the Echo Service's common metric.",
          "numeratorUnit" : "seconds",
          "type" : "gauge"
        }
      ]
    }
  ],
  "roles" : [
   {
      "name" : "ECHO_MASTER_SERVER",
       "metricDefinitions" : [
         {
          "name" : "echo_master_server_metric_one",
          "label" : "Echo Master Server Metric One",
          "description" : "This is the Echo Service's Master Server's first metric.",
          "numeratorUnit" : "bytes",
          "type" : "gauge"
         },
         {
          "name" : "echo_master_server_metric_two",
          "label" : "Echo Master Server Metric Two",
          "description" : "This is the Echo Service's Master Server's second metric.",
          "numeratorUnit" : "echos",
          "type" : "counter"
         },
         {
          "name" : "echo_common_metric",
          "label" : "Echo Service Common Metric",
          "description" : "This is the Echo Service's common metric.",
          "numeratorUnit" : "seconds",
          "type" : "gauge"
         }
       ]
    },
    {
       "name" : "ECHO_WEBSERVER",
       "metricDefinitions" : [
         {
          "name" : "echo_web_server_metric_one",
          "label" : "Echo Web Server Metric One",
          "description" : "This is the Echo Service's Web Server's first metric.",
          "numeratorUnit" : "bytes",
          "type" : "gauge"
         },
         {
          "name" : "echo_web_server_metric_two",
          "label" : "Echo Web Server Metric Two",
          "description" : "This is the Echo Service's Web Server's second metric.",
          "numeratorUnit" : "echos",
          "type" : "counter"
         },
         {
          "name" : "echo_common_metric",
          "label" : "Echo Service Common Metric",
          "description" : "This is the Echo Service's common metric.",
          "numeratorUnit" : "seconds",
          "type" : "gauge"
         }
       ]
    }
  ]
}

name

This is the logical name of the service. This must match the name in the service.sdl.

Required?: Yes

{
  "name" : "ECHO"
}

nameForCrossEntityAggregateMetrics

The string to use to pluralize the name of the service for cross entity aggregate metrics. See nameForCrossEntityAggregateMetrics.

version

The version of the CSD. This must match the name in the service.sdl.

Required?: Yes

{  
  "version" : 1
}

compatibility

Describes compatibility requirements for this CSD. This must match the compatibility section in the service.sdl.

Required?: No, by default compatibility checks are not enforced.

{
  "compatibility" : {
    "generation" : 2,
    "cdhVersion" : {
      "min" : 4,
      "max" : 5
    }
  }
}
Key Description Required?
generation An integer used to communicate compatibility between different CSDs of the same name. Compatibility is intentionally independent of CSD versions. yes
cdhVersion The range of CDH cluster versions compatible with this CSD. Defaults to all versions. no

metric definitions

A list of metrics for the service. See Metrics. For metricDefinitions use the "roles" section for pure CSD projects.

Required?: No

{
  "metricDefinitions" : [
    {
      "name" : "echo_service_metric_one",
      "label" : "Echo Service Metric One",
      "description" : "This is the Echo Service's first metric.",
      "numeratorUnit" : "bytes",
      "type" : "gauge"
    },
    {
      "name" : "echo_service_metric_two",
      "label" : "Echo Service Metric Two",
      "description" : "This is the Echo Service's second metric.",
      "numeratorUnit" : "echos",
      "type" : "counter"
    },
    {
      "name" : "echo_common_metric",
      "label" : "Echo Service Common Metric",
      "description" : "This is the Echo Service's common metric.",
      "numeratorUnit" : "seconds",
      "type" : "gauge"
    }
  ]
}

metric entity attribute definitions

A list of metric entity attribute definititions that can be used in the metric entity type definitions in addition to the built in metric entity attributes.

Required?: No

{
  "metricEntityAttributeDefinitions" : [
    {
      "name" : "echoServiceAttributeOne",
      "label" : "Echo Service Attribute One",
      "description" : "The Echo Service's First Entity Attribute",
      "valueCaseSensitive": false
    },
    {
      "name" : "echoServiceAttributeTwo",
      "label" : "Echo Service Attribute Two",
      "description" : "The Echo Service's Second Entity Attribute",
      "valueCaseSensitive": true
    }
  ]
}
Key Description Required?
name The name of the attribute. This name uniquely identifies this attribute and is used to reference the metric in the Cloudera Manager API and charting features. yes
label The display name of the attribute. yes
description The description of the attribute. yes
valueCaseSensitive Whether to treat attribute values as case-sensitive. yes

metric entity type definitions

A list of additional metric entity types to define in addition to the implicitly defined service and role types. A metric entity type defines the properties of a class of entities to which metrics can be associated.

Required?: No

{
  "metricEntityTypeDefinitions" : [
    {
      "name" : "ECHO_ENTITY_TYPE_ONE",
      "nameForCrossEntityAggregateMetrics" : "echo_entity_type_ones",
      "entityNameFormat" :  [
         "serviceName",
         "echoServiceAttributeOne"
      ],
      "label" : "Echo Service Entity Type One",
      "labelPlural" : "Echo Service Entity Type Ones",
      "description" : "The Echo Service's First Entity Type",
      "immutableAttributeNames" : [
         "serviceName",
         "echoServiceAttributeOne"
      ],
      "mutableAttributeNames" : [
        "echoServiceAttributeTwo"
      ],
      "parentMetricEntityTypeNames" : [
         "ECHO"
      ],
      "metricDefinitions" : [
        {
          "name" : "echo_entity_type_one_metric_one",
          "label" : "Echo Service Entity Type One Metric One",
          "description" : "This is the Echo Service's Entity Type One's first metric.",
          "numeratorUnit" : "bytes",
          "type" : "gauge"
        },
        {
          "name" : "echo_entity_type_one_metric_two",
          "label" : "Echo Service Entity Type One Metric Two",
          "description" : "This is the Echo Service's Entity Type Two's first metric.",
          "numeratorUnit" : "echos",
          "type" : "counter"
        },
        {
          "name" : "echo_common_metric",
          "label" : "Echo Service Common Metric",
          "description" : "This is the Echo Service's common metric.",
          "numeratorUnit" : "seconds",
          "type" : "gauge"
        }
      ]
    },
    {
      "name" : "ECHO_ENTITY_TYPE_TWO",
      "nameForCrossEntityAggregateMetrics" : "echo_entity_type_twos",
      "entityNameFormat" :  [
         "roleName",
         "echoServiceAttributeOne"
      ],
      "label" : "Echo Service Entity Type Two",
      "labelPlural" : "Echo Service Entity Type Twos",
      "description" : "The Echo Service's Second Entity Type",
      "immutableAttributeNames" : [
         "roleName",
         "echoServiceAttributeOne"
      ],
      "parentMetricEntityTypeNames" : [
         "ECHO-ECHO_WEBSERVER",
         "ECHO_ENTITY_TYPE_ONE"
      ],
      "metricDefinitions" : [
        {
          "name" : "echo_entity_type_two_metric_one",
          "label" : "Echo Service Entity Type Two Metric One",
          "description" : "This is the Echo Service's Entity Type Two's first metric.",
          "numeratorUnit" : "bytes",
          "type" : "gauge"
        },
        {
          "name" : "echo_entity_type_two_metric_two",
          "label" : "Echo Service Entity Type Two Metric Two",
          "description" : "This is the Echo Service's Entity Type Two's second metric.",
          "numeratorUnit" : "echos",
          "type" : "counter"
        },
        {
          "name" : "echo_common_metric",
          "label" : "Echo Service Common Metric",
          "description" : "This is the Echo Service's common metric.",
          "numeratorUnit" : "seconds",
          "type" : "gauge"
        }
      ]
    }
  ]
}
Key Description Required?
name The name of the entity type. This name uniquely identifies this entity type and is used to reference the type in the Cloudera Manager API and charting features. yes
nameForCrossEntityAggregateMetrics The string to use to pluralize the name of the entity type for cross entity aggregate metrics. See nameForCrossEntityAggregateMetrics. no
label The display name of the entity type. yes
labelPlural The pluralized display name of the entity type. yes
description The description of the entity type. yes
immutableAttributeNames The list immutable attributes for this entity type. Immutable attributes values for an entity may not change over its lifetime. yes
mutableAttributeNames The list mutable attributes for this entity type. Mutable attributes for an entity may change over its lifetime. yes
entityNameFormat The list of attribute names that will be used to construct entity names for entities of this type. The attributes named here must be immutable attributes of this type or a parent type. yes
entityLabelFormat A format string that will be used to construct the display name of entities of this type. If this returns null the entity name would be used as the display name. The entity attribute values are used to replace $attribute name portions of this format string. For example, an entity with roleType "DATANODE" and hostname "foo.com" will have a display name "DATANODE (foo.com)" if the format is "$roleType ($hostname)". no
parentMetricEntityTypeNames A list of metric entity type names which are parents of this metric entity type. A metric entity type inherits the attributes of its ancestors. For example a role metric entity type has its service as a parent. A service metric entity type has a cluster as a parent. The role type inherits its cluster name attribute through its service parent. Only parent ancestors should be returned here. In the example given, only the service metric entity type should be specified in the parent list. yes
metricDefinitions A list of metrics for the entity type. See Metrics. no

Roles

Monitoring definitions for the services roles. It is not required that all the service's roles be in this list.

Required?: No

{
  "roles" : [
   {
      "name" : "ECHO_MASTER_SERVER",
       "metricDefinitions" : [
         {
          "name" : "echo_master_server_metric_one",
          "label" : "Echo Master Server Metric One",
          "description" : "This is the Echo Service's Master Server's first metric.",
          "numeratorUnit" : "bytes",
          "type" : "gauge"
         },
         {
          "name" : "echo_master_server_metric_two",
          "label" : "Echo Master Server Metric Two",
          "description" : "This is the Echo Service's Master Server's second metric.",
          "numeratorUnit" : "echos",
          "type" : "counter"
         },
         {
          "name" : "echo_common_metric",
          "label" : "Echo Service Common Metric",
          "description" : "This is the Echo Service's common metric.",
          "numeratorUnit" : "seconds",
          "type" : "gauge"
         }
       ]
    },
    {
       "name" : "ECHO_WEBSERVER",
       "metricDefinitions" : [
         {
          "name" : "echo_web_server_metric_one",
          "label" : "Echo Web Server Metric One",
          "description" : "This is the Echo Service's Web Server's first metric.",
          "numeratorUnit" : "bytes",
          "type" : "gauge"
         },
         {
          "name" : "echo_web_server_metric_two",
          "label" : "Echo Web Server Metric Two",
          "description" : "This is the Echo Service's Web Server's second metric.",
          "numeratorUnit" : "echos",
          "type" : "counter"
         },
         {
          "name" : "echo_common_metric",
          "label" : "Echo Service Common Metric",
          "description" : "This is the Echo Service's common metric.",
          "numeratorUnit" : "seconds",
          "type" : "gauge"
         }
       ]
    }
  ]
}
Key Description Required?
name The role name. This must match a role name in the service.sdl. yes
nameForCrossEntityAggregateMetrics The string to use to pluralize the name of the role for cross entity aggregate metrics. See nameForCrossEntityAggregateMetrics. no
metricDefinitions A list of metrics for the role. See Metrics. no

Name For Cross Entity Aggregates

The string to use to pluralize the name of the service, role or entity for cross entity aggregate metrics. For example, for a service named "DUCKY" this should be "duckies". Cross entity aggregate metric names will be composed using this to generate metrics named like 'alerts_across_duckies'. The string must be prefixed with the service name and be unique within this descriptor. If this is not specified the name will be constructed from the servicename and an "s".

Required?: No

{
  "nameForCrossEntityAggregateMetrics" : "echos"
}

Metrics

A list of metrics that will be registered for the service, role or entity type.

Required?: No

{
  "metricDefinitions" : [
    {
      "name" : "echo_service_metric_one",
      "label" : "Echo Service Metric One",
      "description" : "This is the Echo Service's first metric.",
      "numeratorUnit" : "bytes",
      "type" : "gauge"
    },
    {
      "name" : "echo_service_metric_two",
      "label" : "Echo Service Metric Two",
      "description" : "This is the Echo Service's second metric.",
      "numeratorUnit" : "echos",
      "type" : "counter"
    },
    {
      "name" : "echo_common_metric",
      "label" : "Echo Service Common Metric",
      "description" : "This is the Echo Service's common metric.",
      "numeratorUnit" : "seconds",
      "type" : "gauge"
    }
  ]
}
Key Description Required?
name The name of the metric. This name uniquely identifies this metric and is used to reference the metric in the Cloudera Manager API and charting features. yes
label The display name of the metric. yes
description The description of the metric. yes
numeratorUnit The numerator unit for the metric. no
denominatorUnit The denominator unit for the metric. no
type Type of the the metric. Valid values are "gauge", "counter", or "status". Counter-based metrics are treated specially within Cloudera Manager. They are in general converted to and exposed as rates. Status metrics are non-numeric, and can be one of the following OK/WARNING/FAILURE/UNKNOWN. They are useful for modeling canary type of health tests. yes
weightingMetricName The name of the "weighting metric", if any, associated with this metric. A "weighting metric" is one metric in a metric pair used to track both the count of something as well as an average of values associated with those counted somethings. For example, one metric might count the number of file upload operations processed while a second metric counts the average time taken by those operations. In this case, the average time would have the operation counter as the weighting metric. The weighting metric is used by Cloudera Manager when aggregating a metric over time or across metric. no
context An opaque context for the metric. This context will be available for the metric in the metric schema. no

The context field describes how the metric is extracted from the roles metric page. See Service Descriptor Language Reference: metricsSource

'::' is used as a separator in the context fields.

SIMPLE format: The context is a list of section names which lead to the desired metric.

{
"context": "data::system::memory::heapUsed",
"counter": false,
"description": "JVM heap used memory.",
"label": "JVM Heap Used Memory",
"name": "atlas_server_memory_heap_used",
"numeratorUnit": "bytes"
},

JMX format: The context starts with a regex which identifies the bean containing the requested metric. It is followed by a list of section names which lead to the desired metric. The regex is between REGEX$$ and the '::' divider. Since release 7.1.6 text values (e.g.: status) can be converted to numeric values using the ENUM$$ expression in the last part. The enum expression is separated by commas and contains name=value pairs. There is a special name DEFAULT$$ for a default value, it is optional. The values can be integers or floating points.

An example input:

{
  "beans" : [ {
      "name" : "testMetricOne",
      "value" : 123
    }, 
    {
      "name" : "Catalina:type=RequestProcessor,worker=http-1600",
      "value" : 456
    }, 
    {
      "name" : "id1=123,id2=456",
      "value" : 789
    },
    {
      "name": "java.lang:type=Memory",
      "HeapMemoryUsage": {
        "committed": 410517504,
        "init": 428077632,
        "max": 954728448,
        "used": 125101192
      }
    },
    {
      "name": "EchoServer",
      "Status": {
        "Health": "GOOD",
        "HAStatus": "Active"
      }
    }
  ]
}

Examples for the context field:

"""{
      "context" : "REGEX$$testMetricOne::value"
...
      "context" : "REGEX$$Catalina:type=RequestProcessor,worker=::value"
...
      "context" : "REGEX$$id1=[0-9]+,id2=[0-9]+::value"
...
      "context" : "REGEX$$java.lang:type=Memory::HeapMemoryUsage::used"
...
      "context" : "REGEX$$EchoServer::Status::Health::ENUM$$,GOOD=10,WARN=5.5,BAD=2.2,CRITICAL=0.0"
...
      "context" : "REGEX$$EchoServer::Status::HAStatus::ENUM$$,Active=100,Standby=0,DEFAULT$$=-1"
Clone this wiki locally