Scheduler Fails to Allocate Pod Due to Unrecognized Available GPU

[Clasyc]

So I just freshly installed TrueNAS Scale and tried to deploy Immich app on k3s.

I'm using Ryzen 4650G CPU, I see during deployment that my integrated AMD GPU is correctly recognized.

I see available amd gpu on my node:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                290m (2%)   18 (150%)
  memory             570Mi (1%)  35938Mi (114%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
  amd.com/gpu        1           1
Events:              <none>

pod container requests gpu:

Containers:
  immich:
    Image:      altran1502/immich-server:v1.100.0
    Port:       32003/TCP
    Host Port:  0/TCP
    Command:
      /bin/sh
    Args:
      -c
      /usr/src/app/start-microservices.sh
    Limits:
      amd.com/gpu:  1
      cpu:          4
      memory:       8Gi
    Requests:
      amd.com/gpu:  1
      cpu:          10m
      memory:       50Mi

but I still see in the scheduler errors:

Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  29m                  default-scheduler  0/1 nodes are available: 1 Insufficient amd.com/gpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..
  Warning  FailedScheduling  9m18s (x3 over 24m)  default-scheduler  0/1 nodes are available: 1 Insufficient amd.com/gpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..

k3s service logs:

Apr 03 21:06:19 truenas k3s[7743]: E0403 11:06:19.479080    7743 event_broadcaster.go:253] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"immich-microservices-77c66b674f-ks6ww.17c2d711b8df5452", GenerateName:"", Namespace:"ix-immich", SelfLink:"", UID:"32dfb47c-e6a4-4f0f-a08e-9ccb67699728", ResourceVersion:"6273", Generation:0, CreationTimestamp:time.Date(2024, time.April, 3, 11, 1, 19, 0, time.Local), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"k3s", Operation:"Update", APIVersion:"events.k8s.io/v1", Time:time.Date(2024, time.April, 3, 11, 1, 19, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc00cafe078), Subresource:""}}}, EventTime:time.Date(2024, time.April, 3, 11, 1, 19, 470263000, time.Local), Series:(*v1.EventSeries)(0xc005f568e0), ReportingController:"default-scheduler", ReportingInstance:"default-scheduler-truenas", Action:"Scheduling", Reason:"FailedScheduling", Regarding:v1.ObjectReference{Kind:"Pod", Namespace:"ix-immich", Name:"immich-microservices-77c66b674f-ks6ww", UID:"a92e5d34-0d52-4577-be81-353eaf8c1df3", APIVersion:"v1", ResourceVersion:"5607", FieldPath:""}, Related:(*v1.ObjectReference)(nil), Note:"0/1 nodes are available: 1 Insufficient amd.com/gpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..", Type:"Warning", DeprecatedSource:v1.EventSource{Component:"", Host:""}, DeprecatedFirstTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeprecatedLastTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeprecatedCount:0}': 'Event "immich-microservices-77c66b674f-ks6ww.17c2d711b8df5452" is invalid: series.count: Invalid value: "": should be at least 2' (will not retry!)

I noticed similar issue across multiple places with no solution, why this would happen:

https://www.truenas.com/community/threads/immich-ix-apps-with-amd-gpu-0-1-nodes-are-available-1-insufficient-amd-com-gpu.116472/
https://stackoverflow.com/questions/77217560/k3s-nvidia-container-runtime-support-not-detecting-gpu

Can someone guide me to the right direction what should I do?

k3s version v1.26.6+k3s-e18037a7-dirty (e18037a7)
go version go1.19.8

[brandond]

According to your output, your node has 1 GPU, and you already have 1 GPU allocated. You cannot run another GPU pod until the first pod goes away and releases its allocation.

[Clasyc]

Thanks for the answer!
Ah you are right, it's a bummer, I would like to share this GPU, but thats probably amd gpu plugin or TrueNAS implementation scope?

kube-system   amdgpu-device-plugin-daemonset-vwgwn      1/1     Running   0          125m

[brandond]

I would defer to the gpu plugin docs.