AWS EBS CSI Volume is still being allocated to the job that already dead/purged

[habibiefaried]

Nomad version

Nomad v1.0.0 (cfca640)

Operating system and Environment details

OS: Amazon Linux machine

Issue

AWS EBS CSI Volume is still being allocated to the job that already dead/purged. Please note that, this issue is intermittent, maybe you cannot reproduce this case with single hit.

Reproduction steps

1. Run a job with AWS EBS CSI
2. Stop the job
3. Nomad CSI still tell you that, the volume is still being allocated

Job file (if appropriate)

job "elasticsearch" {
  datacenters = ["dc1"]

  group "demo" {
    count = 1

    volume "elasticsearch" {
      type      = "csi"
      read_only = false
      source    = "elasticsearch"
    }

    network {
      port "elasticsearch_9200" { 
        to = 9200
        static = 9200
      }

    }

    task "server" {
      template {
        destination = "local/env"
        env         = true
        data        = <<-EOH
        discovery.type = "single-node"
        ES_JAVA_OPTS = "-Xms2g -Xmx6g"
        EOH
      }

      driver = "docker"

      volume_mount {
        volume      = "elasticsearch"
        destination = "/usr/share/elasticsearch/data"
        read_only   = false
      }

      config {
        image = "docker.elastic.co/elasticsearch/elasticsearch:7.10.2"
        ports = ["elasticsearch_9200"]
      }

      resources {
        cpu    = 512
        memory = 3072
      }
    }

    service {
      name = "elasticsearch"

      check {
        port        = "elasticsearch_9200"
        type        = "tcp"
        interval    = "15s"
        timeout     = "14s"
      }

    }
  }
}

Nomad Client logs (if appropriate)

2021-02-02T23:33:32.117Z [ERROR] http: request failed: method=DELETE path=/v1/volume/csi/elasticsearch?force=false error="rpc error: volume in use: elasticsearch" code=500
2021-02-02T23:33:32.117Z [DEBUG] http: request complete: method=DELETE path=/v1/volume/csi/elasticsearch?force=false duration=4.707841ms
2021-02-02T23:33:32.189Z [ERROR] nomad.fsm: CSIVolumeDeregister failed: error="volume in use: elasticsearch"

Nomad Server logs (if appropriate)

nomad.fsm: CSIVolumeDeregister failed: error="volume in use: elasticsearch"

[tgross]

This should be definitively fixed by #11890 which shipped in 1.2.5. Going to treat this as a duplicate of #10927 so if folks run into this issue again after 1.2.5, please comment over there instead of reopening this issue. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.