[CSI, hcloud] Plugin status returns 404

[mkrueger-sabio]

I am running the node plugin of the hcloud-csi-driver:

job "node" {
  datacenters = ["dc1"]

  # you can run node plugins as service jobs as well, but running
  # as a system job ensures all nodes in the DC have a copy.
  type = "system"

  group "node" {
    task "plugin" {
      driver = "docker"

      config {
        image = "hetznercloud/hcloud-csi-driver:1.2.3"
        privileged = true
      }

      env {
        CSI_ENDPOINT="unix://tmp/csi.sock"    
        HCLOUD_TOKEN="token"
        HCLOUD_SERVER_ID="1234567"
      }

      csi_plugin {
        id        = "csi.hetzner.cloud"
        type      = "node"
        mount_dir = "/csi"
      }
    }
  }
}

Logs of the plugin:

level=debug ts=2020-04-20T07:19:23.999878729Z msg="using server id from HCLOUD_SERVER_ID env var" server-id=1234567
level=debug ts=2020-04-20T07:19:23.99991031Z msg="fetching server"
level=info ts=2020-04-20T07:19:24.545894377Z msg="fetched server" server-name=postgres-test01.nomadsupport-internal.hetz.sabio.de
level=debug ts=2020-04-20T07:19:24.54711923Z component=metrics-service msg="registering metrics with registry"
level=debug ts=2020-04-20T07:19:24.547356108Z component=metrics-service msg="registered metrics"
level=debug ts=2020-04-20T07:19:24.550126782Z component=metrics-service msg="starting prometheus http server" addr=:9189

But when getting plugin status I get

$ ./nomad plugin status    
Container Storage Interface
Error querying CSI plugins: Unexpected response code: 404 ()

The same when I create a volume. There a no additional logs, either in the plugin logs nor in the server logs.

Can you tell me how to debug this? Which method of the plugin is called when getting plugin status?

[tgross]

There a no additional logs, either in the plugin logs nor in the server logs.

The Nomad client logs should have some more information (logs like "CSI Plugin registration failed" and then a more specific error message).

But I think I see the problem in your jobspec (or one problem 😀). The mount_dir is the directory inside the task where Nomad will expect to find the csi.sock socket file. The CSI_ENDPOINT you've provided is in the directory /tmp. Try something like this instead:

job "node" {
  datacenters = ["dc1"]

  type = "system"

  group "node" {
    task "plugin" {
      driver = "docker"

      config {
        image = "hetznercloud/hcloud-csi-driver:1.2.3"
        privileged = true
      }

      env {
        CSI_ENDPOINT="unix:///csi/csi.sock"    
        HCLOUD_TOKEN="token"
        HCLOUD_SERVER_ID="1234567"
      }

      csi_plugin {
        id        = "csi.hetzner.cloud"
        type      = "node"
        mount_dir = "/csi"
      }
    }
  }
}

[mkrueger-sabio]

Thx for support. I changed the config but get the same results.

[tgross]

Ok, did you take a look at the Nomad client logs?

[angrycub]

Playing on my own hetzner node. Looks like this plugin is a monolith type. That could be part of the issue. Here is my working configuration, but this does require the fix in #7754 :

node.nomad

job "node" {
  datacenters = ["dc1"]
  type = "system"

  group "node" {
    task "plugin" {
      driver = "docker"

      config {
        image = "hetznercloud/hcloud-csi-driver:1.2.3"
        privileged = true
      }

      env {
        CSI_ENDPOINT="unix:///csi/csi.sock"    
        HCLOUD_TOKEN="«your token»"
      }

      csi_plugin {
        id        = "csi.hetzner.cloud"
        type      = "monolith"
        mount_dir = "/csi"
      }
    }
  }
}

volume.hcl

# volume registration
type = "csi"
id = "block1"
name = "block1"
external_id = "«volume id as listed in the Hetzner UI Volumes page»"
access_mode = "single-node-writer"
attachment_mode = "file-system"
plugin_id = "csi.hetzner.cloud"

the original job workload from earlier in the ticket - job.nomad

job "httpd1" {
  datacenters = ["dc1"]

  group "httpd" {
    restart {
      attempts = 10
      interval = "5m"
      delay    = "25s"
      mode     = "delay"
    }
    volume "block1" {
      type      = "csi"
      read_only = false
      source    = "block1"
    }

    task "httpd" {
      driver = "docker"

      volume_mount {
        volume      = "block1"
        destination = "/srv"
        read_only   = false
      }

      config {
        image = "httpd:2.4.41-alpine"
      }
    }
  }
}

You can work around the issue fixed by #7754 by setting the id in your volume spec (in my case, volume.hcl) to match the external_id, and then using that ID in your job "httpd1" » group "httpd" » volume "block1" » source argument

[mkrueger-sabio]

Ok, the problem was on my side. I started the nomad server via vagrant, but did not change the version to 0.11.0 although I am 100% sure that I have done that.

Thx for support.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.