Skip to content

External controller to remove terminated EC2 instances from AWS Cloud Map service

License

Notifications You must be signed in to change notification settings

spreaker/aws-cloud-unmap

Repository files navigation

AWS Cloud (un)Map

External controller to remove terminated EC2 instances from AWS Cloud Map service.

How it works

This application scans - at a regular interval - the instances registered to a Cloud Map service and match them with the EC2 instances running in 1+ region: it will then deregister any instance registered in the service which doesn't match a running EC2 instance.

Requisites:

  • The instance must be registered in the Cloud Map service with Cloud Map instance id equal to the EC2 instance id
  • The instance must be registered in the Cloud Map service with AWS_INSTANCE_IPV4 attribute (can be the private or public IP address)

How the matching is done:

  • A registered instance is considered valid if both the instance id and the AWS_INSTANCE_IPV4 address match a running EC2 instance
  • A registered instance is skipped (left untouched) if registered without AWS_INSTANCE_IPV4 attribute

Safety countermeasures:

  • The application logs a warning and do not deregister the unmatching instances, in case that would leave the service without registered instance
  • The application handles graceful shutdown on SIGINT and SIGTERM. If such signals are received during a reconciling, it would complete the on-going reconcile before exiting

How to run it

You have two options to run it:

  1. Manually install and run the aws-cloud-unmap Python package

    pip3 install aws-cloud-unmap
    
    aws-cloud-unmap --service-id srv-12345 --service-region us-east-1 --instances-region us-east-1
    
  2. Use the Docker image available on Docker hub

    docker run --env AWS_ACCESS_KEY_ID="id" --env AWS_SECRET_ACCESS_KEY="secret" spreaker/aws-cloud-unmap --service-id srv-12345 --service-region us-east-1 --instances-region us-east-1
    

The cli supports the following arguments:

Argument Required Description
--service-id ID yes AWS CloudMap service ID
--service-region REGION yes AWS CloudMap service region
--instances-region REGION [REGION ...] yes AWS regions where EC2 instances should be checked
--frequency N How frequently the service should be reconciled (in seconds). Defaults to 300 sec
--single-run Run a single reconcile and then exit
--enable-prometheus Enable the Prometheus exporter. Disabled by default
--prometheus-host The host at which the Prometheus exporter should listen to. Defaults to 127.0.0.1
--prometheus-port The port at which the Prometheus exporter should listen to. Defaults to 9100
--log-level LOG_LEVEL Minimum log level. Accepted values are: DEBUG, INFO, WARNING, ERROR, CRITICAL. Defaults to INFO

Exported metrics

The application features an integrated Prometheus exporter. The following metrics are exported:

Metric name Labels Description
aws_cloud_unmap_up service_id Always 1: can be used to check if it's running
aws_cloud_unmap_last_reconcile_success_timestamp_seconds service_id The timestamp (in seconds) of the last successful reconciliation

Required IAM privileges

In order to successfully run, this application requires the following IAM privileges:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid":      "DescribeEC2Instances",
      "Effect":   "Allow",
      "Action":   [ "ec2:DescribeInstances" ],
      "Resource": "*"
    },{
      "Sid":      "ListAndDeregisterServiceInstances",
      "Effect":   "Allow",
      "Action":   [
        "servicediscovery:ListInstances",
        "servicediscovery:DeregisterInstance",
        "route53:GetHealthCheck",
        "route53:DeleteHealthCheck",
        "route53:UpdateHealthCheck"
      ],
      "Resource": "*"
    },{
      "Sid":      "UpdateDnsWhileDeregisteringServiceInstances",
      "Effect":   "Allow",
      "Action":   [ "route53:ChangeResourceRecordSets" ],
      "Resource": [
        "ARN-OF-YOUR-ROUTE53-HOSTEDZONE"
      ]
    }
  ]
}

Development

Run the development environment:

docker-compose build dev && docker-compose run --rm dev

Run tests in the dev environment:

python3 -m unittest

License

This software is released under the MIT license.

About

External controller to remove terminated EC2 instances from AWS Cloud Map service

Resources

License

Stars

Watchers

Forks

Packages

No packages published