Failed to dispense plugin threshold:

[sizief]

Hi!

I think this is a simple config error somewhere, but I couldn't find it. So the Autoscaler agent is working fine if I don't use Threshold strategy. As soon as I add this strategy the following error would appear (This plugin is installed by default, right?)

2021-10-15T22:08:10.731Z [ERROR] agent: failed to start agent: error="failed to setup plugins: 1 error occurred:
	* failed to dispense plugin threshold: failed to instantiate plugin threshold client: fork/exec /plugins/threshold: no such file or directory

And here is my agent template:

job "autoscaler" {
  type = "service"

  datacenters = ["preproduction"]

  group "autoscaler" {
    count = 1

    network {
      port "http" {}
    }

    task "autoscaler" {
      driver = "exec"

      config {
        command = "/usr/local/bin/nomad-autoscaler"
        args    = [
          "agent",
          "-config",
          "${NOMAD_TASK_DIR}/config.hcl",
          "-policy-dir",
          "$${NOMAD_TASK_DIR}/policies/"
        ]
      }

      artifact {
        source      = "https://releases.hashicorp.com/nomad-autoscaler/0.3.0/nomad-autoscaler_0.3.0_linux_amd64.zip"
        destination = "/usr/local/bin"
      }

      template {
        data = <<EOF
          policy_eval {
            ack_timeout    = "10m"
            delivery_limit = 4

            workers = {
              cluster    = 2
              horizontal = 2
            }
          }

          nomad {
            address = "http://{{env "attr.unique.network.ip-address" }}:4646"
          }

          apm "datadog" {
            driver = "datadog"

            config = {
              dd_api_key = "KEY"
              dd_app_key = "KEY"
            }
          }
          
          strategy "threshold" {
            driver = "threshold"
          }
 
          target "aws-asg" {
            driver = "aws-asg"
            config = {
              aws_region = "eu-central"
            }
          }
          EOF

        destination = "${NOMAD_TASK_DIR}/config.hcl"
      }

      template {
        data = <<EOF
          scaling "cluster_policy" {
            min     = 2
            max     = 5
            enabled = true

            policy {
              cooldown            = "30s"
              evaluation_interval = "60s"

              check "free_memory" {
                source = "datadog"
                query  = "max:system.mem.free{host:nomad-*,env:preproduction}"

                strategy "threshold" {
                  upper_bound = 8000000000
                  lower_bound = 3500000000
                }
              }

              target "aws-asg" {
                ...
              }
            }
          }
        EOF
        destination = "${NOMAD_TASK_DIR}/policies/hashistack.hcl"
      }

      resources {
        cpu    = "1000"
        memory = "250"
      }

      service {
        name = "autoscaler"
        port = "http"

        check {
          type     = "http"
          path     = "/v1/health"
          interval = "3s"
          timeout  = "1s"
        }
      }
    }
  }
}

This is the use case: I want Autoscaler to monitor the free memory of each node and make sure there is enough memory for future allocations, also kill nodes if too much memory is available. So I chose threshold strategy for this with these two conditions: (data is provided by Datadog)

• The maximum free memory in any node is less than X -> more resource needed, the max free memory on any node is not enough for a new allocation.
• Or maximum free memory on any node is more than Y -> there are nodes with full free memory, so kill them.

If you think I can handle this by other strategies, please guide. Thanks so much for this product and your help :)

[vthiery]

Hello @sizief

Which version are you running? I just tried with v0.3.3 (which released the "threshold" strategy) and it works fine. As you did, I configured the agent using this block:

strategy "threshold" {
  driver = "threshold"
}

Nevertheless, I lost quite some time before thinking about adding this block. I already configured some autoscaling with the "target-value" strategy without having to configure it in the agent and naively thought other official strategies documented here would work too.

Apparently, the only strategy enabled by default is "target-value":

// Default is used to generate a new default agent configuration.
func Default() (*Agent, error) {
	[...]

	return &Agent{
		[...]

		Strategies: []*Plugin{{Name: plugins.InternalStrategyTargetValue, Driver: plugins.InternalStrategyTargetValue}},
	}, nil
}

The documentation could be clearer about that and/or other strategies ("threshold", "pass-through", "fixed-value") should be enabled in the default agent config too.

[jrasell]

Hi @sizief, as pointed out the threshold plugin needs to be configured within the Nomad Autoscaler agent for running currently. I'll mark this issue to improve the documentation.

The suggestion of adding this to the default list of launched plugins is also a fair comment and something we can discuss internally.