Skip to content

Latest commit

 

History

History
141 lines (116 loc) · 7.98 KB

README.md

File metadata and controls

141 lines (116 loc) · 7.98 KB

ECS Prometheus Exporter Auto Discovery

Overview

This module provides the Prometheus exporter auto discovery functionality based on the ECS task metadata.

There are two modes to discover the Prometheus targets based on the customer config:

  • Mode 1: Docker label Based: Customers add the docker labels to the containers to indicate the port and metric path of the Prometheus metrics. Customers configure the CWAgent to discover the Prometheus targets with the matching docker label and container port.
  • Mode 2: ECS Task Definition ARN based: Customers configure the CWAgent to discover the Prometheus targets when its ECS task definition arn matches the configured regex and it matches the configured container ports.

Two modes can be enabled together and CWAgent will de-dup the discovered targets based on: {private_ip}:{port}/{metrics_path}

Service Discovery Workflow

  1. List the current running ECS task ARNs for the specific ECS cluster by ECS:ListTasks paginated call
  2. Describe the ECS tasks based on the ListTasks response ECS:DescribeTasks batch call
  3. Get the ECS Task Definition from LRU cache, if there is none in cache, call ECS:DescribeTaskDefinition and cache. LRU cache size (2000) based on ECS service quota
  4. Check the Container Docker Label if there is Docker Label based Service Discovery config
  5. Check the Task Definition ARN if there is Task Definition ARN Regex config
  6. Filter the ECS tasks that match the above two checking for further processing
  7. Get the containerInstance/ec2 instance info from LRU cache if the tasks is running on EC2 launch type. LRU cache size (2000) based on ECS service quota
  8. Call ECS: DescribeContainerInstances and EC2:DescribeInstances for the instances have not been cached. Batching Call with batch size = 100.
  9. Export the ECS Prometheus targets into file configured by sd_result_file

Configuration Options

Overall Configuration Options

Configuration Field Description
sd_frequency Mandatory frequency to discover the prometheus exporters
sd_target_cluster Mandatory target ECS cluster name for service discovery
sd_cluster_region Mandatory the target ECS clusters' AWS region name
sd_result_file Mandatory path of the yaml file for the Prometheus target results
docker_label Optional docker label based service discovery configurations. If this structure is nil, docker label based service discovery is disabled
task_definition_list Optional ECS task definition based service discovery configurations slice. If this slice is empty, task definition based service discovery is disabled

Service Endpoint Based Auto Discovery

Configuration Field Description
sd_service_name_pattern Mandatory ECS service name regex pattern
sd_metrics_ports Mandatory semicolon separated containerPort for Prometheus metrics.
sd_container_name_pattern Optional ECS task container name regex pattern
sd_metrics_path Optional Prometheus metric path. If not specified, the default path /metrics is assumed
sd_job_name Optional Prometheus scrape job name. If not specified, the job name in prometheus.yaml is used

Docker Label Based Auto Discovery Configuration

Configuration Field Description
sd_port_label Mandatory Container's docker label name that specify the containerPort for Prometheus metrics. Only one value is allowed
sd_metrics_path_label Optional Container's docker label name that specify the Prometheus metric path. If not specified, the default path /metrics is assumed.
sd_job_name_label Optional Container's docker label name that specify the Prometheus scrape job name. If not specified, the job name in prometheus.yaml is used.

Task Definition Based Auto Discovery

Configuration Field Description
sd_task_definition_arn_pattern Mandatory ECS task definition arn regex pattern
sd_metrics_ports Mandatory semicolon separated containerPort for Prometheus metrics.
sd_container_name_pattern Optional ECS task container name regex pattern
sd_metrics_path Optional Prometheus metric path. If not specified, the default path /metrics is assumed
sd_job_name Optional Prometheus scrape job name. If not specified, the job name in prometheus.yaml is used

Configuration Example

Sample Configuration in TOML format:

    [inputs.prometheus.ecs_service_discovery]
      sd_cluster_region = "us-east-2"
      sd_frequency = "15s"
      sd_result_file = "/opt/aws/amazon-cloudwatch-agent/etc/ecs_sd_targets.yaml"
      sd_target_cluster = "EC2-Justin-Testing"
      [inputs.prometheus.ecs_service_discovery.docker_label]
        sd_job_name_label = "ECS_PROMETHEUS_JOB_NAME"
        sd_metrics_path_label = "ECS_PROMETHEUS_METRICS_PATH"
        sd_port_label = "ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_A"

      [[inputs.prometheus.ecs_service_discovery.task_definition_list]]
        sd_job_name = "task_def_1"
        sd_metrics_path = "/stats/metrics"
        sd_metrics_ports = "9901;9404;9406"
        sd_task_definition_arn_pattern = ".*:task-definition/bugbash-java-fargate-awsvpc-task-def-only:[0-9]+"

      [[inputs.prometheus.ecs_service_discovery.task_definition_list]]
        sd_container_name_pattern = "^bugbash-jar.*$"
        sd_metrics_ports = "9902"
        sd_task_definition_arn_pattern = ".*:task-definition/nginx:[0-9]+"

Permission

ECS Task Role needs to be granted the following permission so CWAgent can query the ECS/EC2 frontend to get the task meteData.

  • ECS Policy
ECS:ListTasks,
ECS:DescribeContainerInstances,
ECS:DescribeTasks,
ECS:DescribeTaskDefinition
EC2:DescribeInstances

Example Result

- targets:
  - 10.6.1.95:32785
  labels:
    __metrics_path__: /metrics
    ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_B: "9406"
    ECS_PROMETHEUS_JOB_NAME: demo-jar-ec2-bridge-subset-b-dynamic
    ECS_PROMETHEUS_METRICS_PATH: /metrics
    InstanceType: t3.medium
    LaunchType: EC2
    SubnetId: subnet-0347624eeea6c5969
    TaskDefinitionFamily: demo-jar-ec2-bridge-dynamic-port-subset-b
    TaskGroup: family:demo-jar-ec2-bridge-dynamic-port-subset-b
    TaskRevision: "7"
    VpcId: vpc-033b021cd7ecbcedb
    container_name: demo-jar-ec2-bridge-dynamic-port-subset-b
    job: task_def_2
- targets:
  - 10.6.1.95:32783
  labels:
    __metrics_path__: /metrics
    ECS_PROMETHEUS_EXPORTER_PORT_SUBSET_B: "9406"
    ECS_PROMETHEUS_JOB_NAME: demo-jar-ec2-bridge-subset-b-dynamic
    ECS_PROMETHEUS_METRICS_PATH: /metrics
    InstanceType: t3.medium
    LaunchType: EC2
    SubnetId: subnet-0347624eeea6c5969
    TaskDefinitionFamily: demo-jar-ec2-bridge-dynamic-port-subset-b
    TaskGroup: family:demo-jar-ec2-bridge-dynamic-port-subset-b
    TaskRevision: "7"
    VpcId: vpc-033b021cd7ecbcedb
    container_name: demo-jar-ec2-bridge-dynamic-port-subset-b
    job: task_def_2