Error: Not Found Allocation of id: [object Object] was not found. #16890

icyleaf · 2023-04-14T15:41:25Z

Nomad version

Nomad v1.5.3
BuildDate 2023-04-04T20:09:50Z
Revision 434f7a1

Operating system and Environment details

Debian 12
Ubuntu 12.04

Issue

I ran nomad clusters with 4 nodes, one is metal-barel as main server node, the others is one server node and two client node created with ubuntu vm in proxmox.

Run a job with one service provide by nomad but it returns 3 services of Allocations

Reproduction steps

Run, change jobspec, repeat, and repeat

Expected Result

One count = 1 service, then it will show one service url in traefik.

Actual Result

Three service of Allocations, and three url in traefik

Job file (if appropriate)

resource "nomad_job" "jobs" {
      ~ allocation_ids          = [
          - "8d4ea305-5281-ab02-a8d6-a8cdcf195377",
        ] -> (known after apply)
        id                      = "media-radarr"
      ~ jobspec                 = <<-EOT
            job "media-radarr" {
              type        = "service"
              datacenters = ["bunker"]
              
          +   update {
          +     stagger      = "10s"
          +     max_parallel = 1
          +     auto_revert  = true
          +     auto_promote = true
          +     canary       = 1
          +   }
          + 
              group "radarr" {
          +     count = 1
          + 
                network {
                  port "web" {
                    to = 7878
                  }
                }
            
                service {
                  name = "radarr"
                  port = "web"
                  provider = "nomad"
            
                  tags = [
                    "traefik.enable=true",
                    "traefik.http.routers.radarr.entrypoints=web, websecure",
                    "traefik.http.routers.radarr.rule=Host(`radarr.example.com`)",
                  ]
                }
            
                task "radarr" {
                  driver = "docker"
                  config {
                    image = "lscr.io/linuxserver/radarr:latest"
                    ports = ["web"]
                    labels = {
                      "homepage.group"        = "Media"
                      "homepage.name"         = "radarr"
                      "homepage.icon"         = "radarr.png"
                      "homepage.href"         = "http://radarr.example.com"
                      "homepage.description"  = "电源资源刮削器"
                    }
                  }
            
                  env {
                    TZ      = "Asia/Shanghai"
                    PUID    = "1026"
                    PGID    = "100"
                  }
            
                  resources {
                    cpu = 200
                    memory = 256
                  }
            
                  volume_mount {
                    volume      = "radarr-config"
                    destination = "/config"
                    read_only   = false
                  }
                        
                  volume_mount {
                    volume      = "media-volume"
                    destination = "/video"
                    read_only   = false
                  }
                }
            
                volume "media-volume" {
                  type            = "csi"
                  read_only       = false
            
                  source          = "pve-media"
                  access_mode     = "multi-node-multi-writer"
                  attachment_mode = "file-system"
                }
            
                volume "radarr-config" {
                  type            = "csi"
                  read_only       = false
            
                  source          = "radarr-config"
                  access_mode     = "single-node-writer"
                  attachment_mode = "file-system"
                }
                      }
            }
        EOT
        name                    = "media-radarr"
    }

The text was updated successfully, but these errors were encountered:

lgfa29 · 2023-04-18T02:08:35Z

Hi @icyleaf 👋

Thanks for the report. Where do you see the error message you mentioned in the title?

Error: Not Found Allocation of id: [object Object] was not found

Is this showing up in some log? And if so, which log is that?

Any extra information that can provide would be great. Thanks!

icyleaf · 2023-04-18T10:44:51Z

Thanks for reply, "[object Object] was not found" error wase raise by nomad ui, I think this is the result of my daily task "nomad gc".

Where can i to fetch the nomad logs? jourctl or node monistor form nomad ui?

This is another case:

I update postgres job, the previous was marked COMPLETE but it still publish the service.

postgres services return two services by 1babe023 and c6fe6b74, 1babe023 is running allocation, c6fe6b74 was killed by new update allocation.

ubuntu@nomad-bird-server-0:~$ nomad service info postgres
Job ID             Address              Tags                                                                                                                    Node ID   Alloc ID
database-postgres  192.168.16.125:5432  [traefik.enable=true,traefik.tcp.routers.postgres.rule=HostSNI(`*`),traefik.tcp.routers.postgres.entrypoints=postgres]  2fcffc42  1babe023
database-postgres  192.168.16.120:5432  [traefik.enable=true,traefik.tcp.routers.postgres.rule=HostSNI(`*`),traefik.tcp.routers.postgres.entrypoints=postgres]  76a8edbd  c6fe6b74


ubuntu@nomad-bird-server-0:~$ nomad alloc status 1babe023
ID                  = 1babe023-47d6-2710-6222-82a0b48e4e7c
Eval ID             = fc3dff73
Name                = database-postgres.postgres[0]
Node ID             = 2fcffc42
Node Name           = nomad-bird-client-0
Job ID              = database-postgres
Job Version         = 0
Client Status       = running
Client Description  = Tasks are running
Desired Status      = run
Desired Description = <none>
Created             = 2m44s ago
Modified            = 2m18s ago
Deployment ID       = 780c3a03
Deployment Health   = healthy

Allocation Addresses:
Label      Dynamic  Address
*postgres  yes      192.168.16.125:5432 -> 5432

Nomad Service Checks:
Service   Task     Name                       Mode         Status
postgres  (group)  service: "postgres" check  healthiness  success

Task "postgres" is "running"
Task Resources:
CPU        Memory          Disk     Addresses
0/512 MHz  18 MiB/500 MiB  300 MiB

CSI Volumes:
ID             Read Only
postgres-data  false

Task Events:
Started At     = 2023-04-18T10:34:03Z
Finished At    = N/A
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type        Description
2023-04-18T18:34:03+08:00  Started     Task started by client
2023-04-18T18:34:01+08:00  Task Setup  Building Task Directory
2023-04-18T18:34:01+08:00  Received    Task received by client



ubuntu@nomad-bird-server-0:~$ nomad alloc status c6fe6b74
ID                   = c6fe6b74-a7bc-289c-1b9a-ca81585ba024
Eval ID              = f4304f50
Name                 = database-postgres.postgres[0]
Node ID              = 76a8edbd
Node Name            = nomad-bird-server-0
Job ID               = database-postgres
Job Version          = 1
Client Status        = complete
Client Description   = All tasks have completed
Desired Status       = stop
Desired Description  = alloc is being updated due to job update
Created              = 1d6h ago
Modified             = 28m45s ago
Deployment ID        = 9390140a
Deployment Health    = healthy
Replacement Alloc ID = 72983f88

Allocation Addresses:
Label      Dynamic  Address
*postgres  yes      192.168.16.120:5432 -> 5432

Task "postgres" is "dead"
Task Resources:
CPU        Memory          Disk     Addresses
0/512 MHz  21 MiB/200 MiB  300 MiB

CSI Volumes:
ID             Read Only
postgres-data  false

Task Events:
Started At     = 2023-04-18T10:00:12Z
Finished At    = 2023-04-18T10:08:03Z
Total Restarts = 1
Last Restart   = 2023-04-18T18:00:10+08:00

Recent Events:
Time                       Type              Description
2023-04-18T18:08:03+08:00  Killed            Task successfully killed
2023-04-18T18:08:03+08:00  Terminated        Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2023-04-18T18:01:20+08:00  Killing           Sent interrupt. Waiting 5s before force killing
2023-04-18T18:00:12+08:00  Started           Task started by client
2023-04-18T18:00:10+08:00  Restarting        Task restarting in 0s
2023-04-18T18:00:10+08:00  Terminated        Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2023-04-18T17:59:57+08:00  Restart Signaled  User requested running tasks to restart
2023-04-17T12:27:45+08:00  Started           Task started by client
2023-04-17T12:27:44+08:00  Task Setup        Building Task Directory
2023-04-17T12:26:09+08:00  Received          Task received by client

icyleaf · 2023-04-18T10:53:32Z

postgres case log on c6fe6b74 alloc of nomad node which it was killed:

Apr 18 18:00:33 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:00:33.869+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Started msg="Task started by client" failed=false
Apr 18 18:01:20 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:01:20.658+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
Apr 18 18:01:39 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:01:39.812+0800 [ERROR] client.driver_mgr.docker: failed to stop container: container_id=21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152 driver=docker error="API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:01:39 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:01:39.812+0800 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres backoff=5s error="Failed to stop container 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:02:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:02:03.959+0800 [ERROR] client.driver_mgr.docker: failed to stop container: container_id=21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152 driver=docker error="API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:02:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:02:03.960+0800 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres backoff=20s error="Failed to stop container 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:02:43 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:02:43.106+0800 [ERROR] client.driver_mgr.docker: failed to stop container: container_id=21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152 driver=docker error="API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:02:43 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:02:43.107+0800 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres backoff=1m20s error="Failed to stop container 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:03:27 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:27.451+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Terminated msg="Exit Code: 2, Exit Message: \"Docker container exited with non-zero exit code: 2\"" failed=false
Apr 18 18:03:27 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:27.456+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=17505
Apr 18 18:03:27 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:27.545+0800 [INFO]  client.alloc_runner.task_runner: restarting task: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup reason="Restart within policy" delay=16.438698371s
Apr 18 18:03:27 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:27.545+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Restarting msg="Task restarting in 16.438698371s" failed=false
Apr 18 18:03:27 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:27.674+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=f6c6a319-b814-bdff-1208-e1614320372f task=gitea type=Terminated msg="Exit Code: 0" failed=false
Apr 18 18:03:27 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:27.677+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=9828
Apr 18 18:03:27 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:27.870+0800 [INFO]  client.alloc_runner.task_runner: restarting task: alloc_id=f6c6a319-b814-bdff-1208-e1614320372f task=gitea reason="Restart within policy" delay=15.258673291s
Apr 18 18:03:27 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:27.870+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=f6c6a319-b814-bdff-1208-e1614320372f task=gitea type=Restarting msg="Task restarting in 15.258673291s" failed=false
Apr 18 18:03:44 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:44.015+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Driver msg="Downloading image" failed=false
Apr 18 18:03:47 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:47.594+0800 [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=a0a10a28f68cfd6a02174833ec39478aa44f8d2a63289a9e28ab6691b0aec11f
Apr 18 18:03:48 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:48.388+0800 [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=a0a10a28f68cfd6a02174833ec39478aa44f8d2a63289a9e28ab6691b0aec11f
Apr 18 18:03:48 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:03:48.466+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Started msg="Task started by client" failed=false
Apr 18 18:04:22 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:04:22.272+0800 [ERROR] client.driver_mgr.docker: failed to stop container: container_id=21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152 driver=docker error="API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:04:22 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:04:22.272+0800 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres backoff=2m0s error="Failed to stop container 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:04:59 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:04:59.668+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Terminated msg="Exit Code: 2, Exit Message: \"Docker container exited with non-zero exit code: 2\"" failed=false
Apr 18 18:04:59 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:04:59.672+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=161251
Apr 18 18:04:59 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:04:59.740+0800 [INFO]  client.alloc_runner.task_runner: restarting task: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup reason="Restart within policy" delay=17.774720275s
Apr 18 18:04:59 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:04:59.741+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Restarting msg="Task restarting in 17.774720275s" failed=false
Apr 18 18:05:14 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:05:14.122+0800 [ERROR] nomad.csi_plugin: csi raft apply failed: error="plugin in use" method=delete
Apr 18 18:05:14 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:05:14.144+0800 [ERROR] nomad.csi_plugin: csi raft apply failed: error="plugin in use" method=delete
Apr 18 18:05:17 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:05:17.540+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Driver msg="Downloading image" failed=false
Apr 18 18:05:21 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:05:21.097+0800 [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=1b00d439c83827cd8e58b11668b4211fc2d55782d3d10b3a77ac3484a000dfed
Apr 18 18:05:21 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:05:21.819+0800 [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=1b00d439c83827cd8e58b11668b4211fc2d55782d3d10b3a77ac3484a000dfed
Apr 18 18:05:21 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:05:21.888+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Started msg="Task started by client" failed=false
Apr 18 18:06:30 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:30.241+0800 [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=4b3432d0a25dec880946bc7c6c4ac59ec6fa510f5307abbad69f2fefc8c8cf04
Apr 18 18:06:30 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:30.978+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Terminated msg="Exit Code: 2, Exit Message: \"Docker container exited with non-zero exit code: 2\"" failed=false
Apr 18 18:06:30 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:30.982+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=229306
Apr 18 18:06:31 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:31.060+0800 [INFO]  client.alloc_runner.task_runner: restarting task: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup reason="Restart within policy" delay=17.743238601s
Apr 18 18:06:31 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:31.060+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Restarting msg="Task restarting in 17.743238601s" failed=false
Apr 18 18:06:31 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:31.458+0800 [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=4b3432d0a25dec880946bc7c6c4ac59ec6fa510f5307abbad69f2fefc8c8cf04
Apr 18 18:06:31 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:31.567+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=f6c6a319-b814-bdff-1208-e1614320372f task=gitea type=Started msg="Task started by client" failed=false
Apr 18 18:06:41 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:41.426+0800 [ERROR] client.driver_mgr.docker: failed to stop container: container_id=21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152 driver=docker error="API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:06:41 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:41.426+0800 [ERROR] client.alloc_runner.task_runner: failed to kill task: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres backoff=2m0s error="Failed to stop container 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: API error (500): cannot stop container: 21baba8e609d5386c07494c2695ce8c8ce7c6d4c68328b871030d2392763d152: tried to kill container, but did not receive an exit event"
Apr 18 18:06:48 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:48.831+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Driver msg="Downloading image" failed=false
Apr 18 18:06:57 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:57.317+0800 [INFO]  client.driver_mgr.docker: created container: driver=docker container_id=d6788ec83720c1611783fe4fef4ce26f02fab47a0803de5a10a90116f0c986c2
Apr 18 18:06:58 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:58.092+0800 [INFO]  client.driver_mgr.docker: started container: driver=docker container_id=d6788ec83720c1611783fe4fef4ce26f02fab47a0803de5a10a90116f0c986c2
Apr 18 18:06:58 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:06:58.165+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Started msg="Task started by client" failed=false
Apr 18 18:07:14 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:07:14.098+0800 [INFO]  client.rocketduck-csi-plugin-nfs: finished client unary call: grpc.code=DeadlineExceeded duration=2m0.000428636s grpc.service=csi.v1.Node grpc.method=NodeUnpublishVolume
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:08:02.271+0800 [ERROR] nomad.volumes_watcher: error releasing volume claims: namespace=default volume_id=postgres-data
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:   error=
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:   | 1 error occurred:
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:   | \t* could not detach from node: node detach volume: CSI.NodeDetachVolume: 2 errors occurred:
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:   | \t* rpc error: code = DeadlineExceeded desc = context deadline exceeded
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:   | \t* remove /var/nomad/data/client/csi/node/rocketduck-csi-plugin-nfs/per-alloc/c6fe6b74-a7bc-289c-1b9a-ca81585ba024/postgres-data/rw-file-system-single-node-writer: device or resource busy
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:   |
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:   |
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:   |
Apr 18 18:08:02 nomad-bird-server-0 nomad[374975]:
Apr 18 18:08:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:08:03.573+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres type=Terminated msg="Exit Code: 137, Exit Message: \"Docker container exited with non-zero exit code: 137\"" failed=false
Apr 18 18:08:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:08:03.586+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=9673
Apr 18 18:08:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:08:03.692+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres type=Killed msg="Task successfully killed" failed=false
Apr 18 18:08:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:08:03.781+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Terminated msg="Exit Code: 2, Exit Message: \"Docker container exited with non-zero exit code: 2\"" failed=false
Apr 18 18:08:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:08:03.791+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=299759
Apr 18 18:08:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:08:03.801+0800 [INFO]  client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres path=/usr/bin/nomad pid=379712
Apr 18 18:08:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:08:03.802+0800 [INFO]  agent: (runner) stopping
Apr 18 18:08:03 nomad-bird-server-0 nomad[374975]:     2023-04-18T18:08:03.803+0800 [INFO]  client.gc: marking allocation for GC: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024

when i reboot the node, the nomad log show it try to delete non-existent service registration

Apr 18 18:47:32 nomad-bird-server-0 systemd[1]: Started Nomad.
Apr 18 18:47:33 nomad-bird-server-0 nomad[412]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Apr 18 18:47:33 nomad-bird-server-0 nomad[412]: ==> Loaded configuration from /etc/nomad.d/client.hcl, /etc/nomad.d/consul.hcl, /etc/nomad.d/nomad.hcl, /etc/nomad.d/server.hcl
Apr 18 18:47:33 nomad-bird-server-0 nomad[412]: ==> Starting Nomad agent...
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]: ==> Nomad agent configuration:
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:        Advertise Addrs: HTTP: 192.168.16.120:4646; RPC: 192.168.16.120:4647; Serf: 192.168.16.120:4648
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:             Bind Addrs: HTTP: [0.0.0.0:4646]; RPC: 0.0.0.0:4647; Serf: 0.0.0.0:4648
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:                 Client: true
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:              Log Level: INFO
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:                 Region: global (DC: dc1)
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:                 Server: true
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:                Version: 1.5.3
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]: ==> Nomad agent started! Log data will stream in below:
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:33.642+0800 [INFO]  nomad: setting up raft bolt store: no_freelist_sync=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:33.751+0800 [INFO]  nomad.raft: starting restore from snapshot: id=7-172867-1681597027680 last-index=172867 last-term=7 size-in-bytes=439344
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:33.927+0800 [INFO]  nomad.raft: snapshot restore progress: id=7-172867-1681597027680 last-index=172867 last-term=7 size-in-bytes=439344 read-bytes=439344 percent-complete="100.00%"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:33.927+0800 [INFO]  nomad.raft: restored from snapshot: id=7-172867-1681597027680 last-index=172867 last-term=7 size-in-bytes=439344
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.582+0800 [INFO]  nomad.raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:a319526f-13f1-130c-c531-4cba7d975285 Address:192.168.16.120:4647}]"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.582+0800 [INFO]  nomad.raft: entering follower state: follower="Node at 192.168.16.120:4647 [Follower]" leader-address= leader-id=
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.590+0800 [INFO]  nomad: serf: EventMemberJoin: nomad-bird-server-0.global 192.168.16.120
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.590+0800 [INFO]  nomad: starting scheduling worker(s): num_workers=2 schedulers=["batch", "system", "sysbatch", "service", "_core"]
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.591+0800 [WARN]  nomad: serf: Failed to re-join any previously known node
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.591+0800 [INFO]  nomad: started scheduling worker(s): num_workers=2 schedulers=["batch", "system", "sysbatch", "service", "_core"]
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.592+0800 [WARN]  agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/var/nomad/data/plugins
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.594+0800 [INFO]  agent: detected plugin: name=java type=driver plugin_version=0.1.0
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.594+0800 [INFO]  agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.594+0800 [INFO]  agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.594+0800 [INFO]  agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.594+0800 [INFO]  agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.603+0800 [INFO]  nomad: adding server: server="nomad-bird-server-0.global (Addr: 192.168.16.120:4647) (DC: dc1)"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.603+0800 [ERROR] agent.plugin_loader.docker: failed to list pause containers: plugin_dir=/var/nomad/data/plugins error=<nil>
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.603+0800 [ERROR] agent.plugin_loader.docker: failed to list pause containers: plugin_dir=/var/nomad/data/plugins error=<nil>
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.625+0800 [INFO]  client: using state directory: state_dir=/var/nomad/data/client
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.628+0800 [INFO]  client: using alloc directory: alloc_dir=/var/nomad/data/alloc
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.628+0800 [INFO]  client: using dynamic ports: min=20000 max=32000 reserved=""
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.768+0800 [INFO]  client.fingerprint_mgr.cgroup: cgroups are available
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.783+0800 [INFO]  client.fingerprint_mgr.consul: consul agent is available
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.783+0800 [WARN]  client.fingerprint_mgr.landlock: failed to fingerprint kernel landlock feature: error="operation not supported"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.807+0800 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=eth0
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.809+0800 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=lo
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:35.816+0800 [WARN]  client.fingerprint_mgr.network: unable to parse speed: path=/usr/sbin/ethtool device=eth0
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:36.860+0800 [WARN]  nomad.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:36.860+0800 [INFO]  nomad.raft: entering candidate state: node="Node at 192.168.16.120:4647 [Candidate]" term=9
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.030+0800 [INFO]  nomad.raft: election won: term=9 tally=1
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.030+0800 [INFO]  nomad.raft: entering leader state: leader="Node at 192.168.16.120:4647 [Leader]"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.032+0800 [INFO]  nomad: cluster leadership acquired
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.367+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.393+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.394+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.395+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.395+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.395+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.398+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.403+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.403+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.412+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.601+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.602+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.603+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.603+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.604+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.604+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.604+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.604+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.604+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.605+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.606+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.606+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.606+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.609+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.609+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.615+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.626+0800 [ERROR] nomad.fsm: DeleteServiceRegistrationByID failed: error="service registration not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.626+0800 [ERROR] nomad.fsm: DeleteServiceRegistrationByID failed: error="service registration not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.659+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.660+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.660+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.660+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.780+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.780+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.782+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.782+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.782+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.782+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.782+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.782+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.782+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.784+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.784+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.784+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.784+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.784+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.787+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.788+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.789+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.790+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.794+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.794+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.796+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.798+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.799+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.803+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.806+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.809+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.809+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.812+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.812+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.812+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.819+0800 [ERROR] nomad.state_store: AllocByID failed to find alloc: alloc_id=782ab1fb-a3a4-f138-3d01-ffec83257890
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.819+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="allocation missing: 782ab1fb-a3a4-f138-3d01-ffec83257890"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.955+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.956+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.957+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.959+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.959+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.959+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.959+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.959+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.969+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.970+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.977+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.979+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.980+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume max claims reached"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.985+0800 [ERROR] nomad.fsm: CSIVolumeClaim failed: error="volume is currently unschedulable"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.986+0800 [INFO]  nomad: eval broker status modified: paused=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:37.986+0800 [INFO]  nomad: blocked evals status modified: paused=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.321+0800 [INFO]  client.plugin: starting plugin manager: plugin-type=csi
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.322+0800 [INFO]  client.plugin: starting plugin manager: plugin-type=driver
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.322+0800 [INFO]  client.plugin: starting plugin manager: plugin-type=device
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.371+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=02203698-ac58-252a-aef3-f5e453d280f7 task=plugin type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.397+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.402+0800 [INFO]  client: node registration complete
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.430+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=12da4752-e4f5-aad7-bcbf-5e853f9a10f1 task=plugin type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.434+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1000
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.438+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.469+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=19848ab6-17c2-8284-fe42-874630563eed task=backup type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.471+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=29df7573-d29b-0e20-0c4b-80aeadb2768e task=plugin type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.473+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.477+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1009
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.514+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=4bc162dc-e119-fb0c-aa8c-bffb52d08f25 task=postgres type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.516+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1019
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.516+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=5cf7bacc-3811-b92c-0bd3-604b95e4eee7 task=plugin type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.521+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.555+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=632da9da-428e-bc26-7649-b2fbfb8e3e94 task=redis type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.559+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1028
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.567+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.598+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=66927535-67bb-261d-7f4b-0b3a6b7bb7b3 task=docker-socket-proxy type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.601+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1037
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.621+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.653+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=85275cde-0974-eaa4-3bf7-d85a9b5467cf task=excalidraw type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.665+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1046
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.670+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.697+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=87cf2d2b-c437-962e-124b-e51431592c0d task=backup type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.699+0800 [WARN]  client: found an alloc without any local state, skipping restore: alloc_id=974974cc-305c-cecb-5516-939c3132ed94
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.700+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=a98a03a0-93ca-28d0-cd30-2e5b6acfd531 task=postgres type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.700+0800 [WARN]  client: found an alloc without any local state, skipping restore: alloc_id=af46beb3-501a-6d41-cb47-859d5c63474c
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.701+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=b1cbabb5-fa00-e115-59ea-8cc7782bfbf0 task=homepage type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.704+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1055
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.708+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.734+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024 task=postgres type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.735+0800 [WARN]  client: found an alloc without any local state, skipping restore: alloc_id=dba06dae-ec16-12ce-6ebf-bd892957800c
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.736+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=e8abf0f0-6b84-cbdc-7ad2-05ca88517493 task=zealot type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.737+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=e8abf0f0-6b84-cbdc-7ad2-05ca88517493 task=init-database type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.739+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1063
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.741+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.771+0800 [WARN]  client: found an alloc without any local state, skipping restore: alloc_id=ec2ccebe-2232-8209-c5a5-4224bb47f56e
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.772+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=ed2b9600-cb97-69ac-e0c4-e8e6e835fd98 task=watchtower type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.774+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1072
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.787+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.816+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=f281fe6c-9160-3dc3-ccab-1cb1352f6a9b task=uptrace type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.816+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=f281fe6c-9160-3dc3-ccab-1cb1352f6a9b task=init-database type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.818+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=f38c0dac-84ff-8232-1c4c-d6c385818a90 task=openwrt type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.820+0800 [WARN]  client.driver_mgr.docker: failed to reattach to docker logger process: driver=docker error="failed to reattach to docker logger process: Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.822+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1081
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.850+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=f6c6a319-b814-bdff-1208-e1614320372f task=gitea type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.850+0800 [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=f6c6a319-b814-bdff-1208-e1614320372f task=init-database type=Received msg="Task received by client" failed=false
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.852+0800 [INFO]  client: started client: node_id=76a8edbd-6212-0f6b-675b-5fe81344ff69
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.856+0800 [INFO]  client.gc: marking allocation for GC: alloc_id=4bc162dc-e119-fb0c-aa8c-bffb52d08f25
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.856+0800 [INFO]  client.gc: marking allocation for GC: alloc_id=87cf2d2b-c437-962e-124b-e51431592c0d
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.856+0800 [INFO]  client.gc: marking allocation for GC: alloc_id=c6fe6b74-a7bc-289c-1b9a-ca81585ba024
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.856+0800 [INFO]  client.gc: marking allocation for GC: alloc_id=19848ab6-17c2-8284-fe42-874630563eed
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.856+0800 [INFO]  client.gc: marking allocation for GC: alloc_id=f281fe6c-9160-3dc3-ccab-1cb1352f6a9b
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.860+0800 [WARN]  agent: using deprecated retry_join fields. Upgrade configuration to use server_join
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.864+0800 [INFO]  agent.joiner: starting retry join: servers=192.168.16.120
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.869+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=29df7573-d29b-0e20-0c4b-80aeadb2768e task=plugin error="Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.869+0800 [INFO]  client.gc: marking allocation for GC: alloc_id=f6c6a319-b814-bdff-1208-e1614320372f
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.883+0800 [INFO]  client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker path=/usr/bin/nomad pid=1090
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.887+0800 [INFO]  agent.joiner: retry join completed: initial_servers=1 agent_mode=server
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.894+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=02203698-ac58-252a-aef3-f5e453d280f7 task=plugin error="Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.898+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=5cf7bacc-3811-b92c-0bd3-604b95e4eee7 task=plugin error="Reattachment process not found"
Apr 18 18:47:47 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:47.924+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=12da4752-e4f5-aad7-bcbf-5e853f9a10f1 task=plugin error="Reattachment process not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.386+0800 [INFO]  client.gc: marking allocation for GC: alloc_id=a98a03a0-93ca-28d0-cd30-2e5b6acfd531
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.442+0800 [ERROR] nomad.fsm: DeleteServiceRegistrationByID failed: error="service registration not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.442+0800 [ERROR] nomad.fsm: DeleteServiceRegistrationByID failed: error="service registration not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.442+0800 [ERROR] client.rpc: error performing RPC to server: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.442+0800 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.442+0800 [INFO]  client.service_registration.nomad: attempted to delete non-existent service registration: service_id=_nomad-task-4bc162dc-e119-fb0c-aa8c-bffb52d08f25-group-postgres-postgres-postgres namespace=default
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.442+0800 [ERROR] nomad.fsm: DeleteServiceRegistrationByID failed: error="service registration not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.442+0800 [ERROR] nomad.fsm: DeleteServiceRegistrationByID failed: error="service registration not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.443+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=ed2b9600-cb97-69ac-e0c4-e8e6e835fd98 task=watchtower error="Reattachment process not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.443+0800 [ERROR] client.rpc: error performing RPC to server: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.443+0800 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.443+0800 [INFO]  client.service_registration.nomad: attempted to delete non-existent service registration: service_id=_nomad-task-f6c6a319-b814-bdff-1208-e1614320372f-group-gitea-gitea-web-web namespace=default
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.443+0800 [ERROR] client.rpc: error performing RPC to server: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.444+0800 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.444+0800 [INFO]  client.service_registration.nomad: attempted to delete non-existent service registration: service_id=_nomad-task-f6c6a319-b814-bdff-1208-e1614320372f-group-gitea-gitea-ssh-ssh namespace=default
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.444+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=f38c0dac-84ff-8232-1c4c-d6c385818a90 task=openwrt error="Reattachment process not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.444+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=b1cbabb5-fa00-e115-59ea-8cc7782bfbf0 task=homepage error="Reattachment process not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.445+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=632da9da-428e-bc26-7649-b2fbfb8e3e94 task=redis error="Reattachment process not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.445+0800 [ERROR] client.rpc: error performing RPC to server: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.445+0800 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.445+0800 [INFO]  client.service_registration.nomad: attempted to delete non-existent service registration: service_id=_nomad-task-f281fe6c-9160-3dc3-ccab-1cb1352f6a9b-group-uptrace-uptrace-web namespace=default
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.446+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=66927535-67bb-261d-7f4b-0b3a6b7bb7b3 task=docker-socket-proxy error="Reattachment process not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.449+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=e8abf0f0-6b84-cbdc-7ad2-05ca88517493 task=zealot error="Reattachment process not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.459+0800 [WARN]  client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=85275cde-0974-eaa4-3bf7-d85a9b5467cf task=excalidraw error="Reattachment process not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.540+0800 [ERROR] nomad.fsm: DeleteServiceRegistrationByID failed: error="service registration not found"
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.540+0800 [ERROR] client.rpc: error performing RPC to server: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.540+0800 [ERROR] client.rpc: error performing RPC to server which is not safe to automatically retry: error="rpc error: service registration not found" rpc=ServiceRegistration.DeleteByID server=192.168.16.120:4647
Apr 18 18:47:48 nomad-bird-server-0 nomad[412]:     2023-04-18T18:47:48.540+0800 [INFO]  client.service_registration.nomad: attempted to delete non-existent service registration: service_id=_nomad-task-a98a03a0-93ca-28d0-cd30-2e5b6acfd531-group-postgres-postgres-postgres namespace=default

lgfa29 · 2023-04-18T20:44:39Z

Ahh I see it now, so the problem is that the service instance for allocations that are not running anymore are still being returned by Nomad service discovery, is that right?

If this is the case, then I think this is a duplicate of #16616 which is still being investigated.

Those error: service registration not found messages may be a red herring and are also still being investigated (#16286).

I'm going to close this as a duplicate of #16616. Feel free to add any additional context you may have in that issue 🙂

icyleaf · 2023-04-19T02:47:38Z

I'll continue to discuss in that issue.

BTW, If you have time, would you follow up on the issues and PRs of terraform-provider-nomad, it's a bit out of step with the latest version of nomad, as you are also the maintainer there, thanks.

icyleaf · 2023-07-30T07:24:24Z

For now, i written a script to clean all invalid services.

icyleaf added the type/bug label Apr 14, 2023

lgfa29 added the stage/needs-investigation label Apr 18, 2023

lgfa29 self-assigned this Apr 18, 2023

lgfa29 added theme/networking stage/duplicate theme/service-discovery/nomad and removed stage/needs-investigation labels Apr 18, 2023

lgfa29 removed their assignment Apr 18, 2023

lgfa29 closed this as not planned Won't fix, can't repro, duplicate, stale Apr 18, 2023

icyleaf mentioned this issue Apr 19, 2023

Services not unregistered #16616

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error: Not Found Allocation of id: [object Object] was not found. #16890

Error: Not Found Allocation of id: [object Object] was not found. #16890

icyleaf commented Apr 14, 2023 •

edited

Loading

lgfa29 commented Apr 18, 2023

icyleaf commented Apr 18, 2023

icyleaf commented Apr 18, 2023 •

edited

Loading

lgfa29 commented Apr 18, 2023

icyleaf commented Apr 19, 2023 •

edited

Loading

icyleaf commented Jul 30, 2023

Error: Not Found Allocation of id: [object Object] was not found. #16890

Error: Not Found Allocation of id: [object Object] was not found. #16890

Comments

icyleaf commented Apr 14, 2023 • edited Loading

Nomad version

Operating system and Environment details

Issue

Reproduction steps

Expected Result

Actual Result

Job file (if appropriate)

lgfa29 commented Apr 18, 2023

icyleaf commented Apr 18, 2023

icyleaf commented Apr 18, 2023 • edited Loading

lgfa29 commented Apr 18, 2023

icyleaf commented Apr 19, 2023 • edited Loading

icyleaf commented Jul 30, 2023

icyleaf commented Apr 14, 2023 •

edited

Loading

icyleaf commented Apr 18, 2023 •

edited

Loading

icyleaf commented Apr 19, 2023 •

edited

Loading