feat(sycl): Add support for Intel GPUs with sycl (#1647)

[mudler]

part of #1647

based on: ggerganov/llama.cpp#2690

Note on exposing the GPU with docker: ggerganov/llama.cpp#2690 (comment)

Testing with:

$ docker build --build-arg BUILD_TYPE=sycl_f32 --build-arg IMAGE_TYPE=core -t local-ai .
# Build only llama.cpp
$ docker build --build-arg BUILD_TYPE=sycl_f32 --build-arg IMAGE_TYPE=core --build-arg GRPC_BACKENDS=backend-assets/grpc/llama-cpp -t local-ai .
$ docker run -ti -p 8080:8080 -v $PWD/models:/build/models \
   --rm -e DEBUG=1 -e GGML_SYCL_DEVICE=0 \
   -e BUILD_TYPE=sycl_f32 --device /dev/dri/renderD128:/dev/dri/renderD128 \
   --device /dev/dri/card1:/dev/dri/card1 -t \
   local-ai https://gist.githubusercontent.com/mudler/55805db5a3675ddb877630a47d25bef3/raw/5a3a33447efb5b422793ae1ff130020d0c7876c9/llava-1.5

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	fd13312
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/65bbcaf48a8eff0008c7005f
😎 Deploy Preview	https://deploy-preview-1660--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[mudler]

I'm expecting some fun here as this is going to be interactive

[mudler]

it fails here:

11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr The program was built for 1 devices                                                                            
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stdout GGML_SYCL_DEBUG=0                                                                                              
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Build program log for '12th Gen Intel(R) Core(TM) i7-1280P':                                                   
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Compilation started                                                                                            
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Compilation done                                                                                               
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Linking started                                                                                                
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Linking done                                                                                                   
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Device build started                                                                                           
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Options used by backend compiler:                                                                              
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Failed to build device program                                                                                 
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr CompilerException Failed to lookup symbol _ZTSZZL13norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0
_clES7_EUlNS3_7nd_itemILi3EEEE_                                                                                                                                                           
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr JIT session error: Symbols not found: [ _Z11fmax_commonDv32_fS_S_ ]                                            
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Failed to materialize symbols: { (main, { _ZTSZZL17soft_max_f32_syclPKfS0_PfiiifPN4sycl3_V15queueEENKUlRNS3_7ha
ndlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZGVdN32uuuuuuu__ZTSZZL17soft_max_f32_syclPKfS0_PfiiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZTSZZL17rms_norm_f32
_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZTSZZL19group_norm_f32_syclPKfPfiiiPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_itemILi3E
EEE_, _ZTSZZL19group_norm_f32_syclPKfPfiiiPN4sycl3_V15queueEENKUlRNS3_7handlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZTSZZL17rms_norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0
_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZGVdN32uuuuuu__ZTSZZL19group_norm_f32_syclPKfPfiiiPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZTSZL17sum_rows_f32_syclPKfPf
iiPN4sycl3_V15queueEEUlNS3_7nd_itemILi3EEEE_, _ZTSZZL13norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZGVdN32uuuuuu__ZTSZZL17rms_norm_f32_sy
clPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZGVdN32uuuuuu__ZTSZZL13norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_it
emILi3EEEE_, _ZTSZZL13norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_itemILi3EEEE_ }) }                                                                    
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr                                                                                                                
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr  -11 (PI_ERROR_BUILD_PROGRAM_FAILURE)Exception caught at file:/build/backend/cpp/llama/llama.cpp/ggml-sycl.cpp,
 line:12644                                                                    

friendly ping @abhilash1910 @NeoZhangJianyu , chances you can help here? any pointers would be appreciated! thanks!

It looks something wrt linking, but I thought the installation steps should be general enough to apply to all the binaries in the example folder?

[mudler]

it fails here:

11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr The program was built for 1 devices                                                                            
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stdout GGML_SYCL_DEBUG=0                                                                                              
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Build program log for '12th Gen Intel(R) Core(TM) i7-1280P':                                                   
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Compilation started                                                                                            
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Compilation done                                                                                               
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Linking started                                                                                                
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Linking done                                                                                                   
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Device build started                                                                                           
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Options used by backend compiler:                                                                              
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Failed to build device program                                                                                 
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr CompilerException Failed to lookup symbol _ZTSZZL13norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0
_clES7_EUlNS3_7nd_itemILi3EEEE_                                                                                                                                                           
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr JIT session error: Symbols not found: [ _Z11fmax_commonDv32_fS_S_ ]                                            
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr Failed to materialize symbols: { (main, { _ZTSZZL17soft_max_f32_syclPKfS0_PfiiifPN4sycl3_V15queueEENKUlRNS3_7ha
ndlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZGVdN32uuuuuuu__ZTSZZL17soft_max_f32_syclPKfS0_PfiiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZTSZZL17rms_norm_f32
_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZTSZZL19group_norm_f32_syclPKfPfiiiPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_itemILi3E
EEE_, _ZTSZZL19group_norm_f32_syclPKfPfiiiPN4sycl3_V15queueEENKUlRNS3_7handlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZTSZZL17rms_norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0
_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZGVdN32uuuuuu__ZTSZZL19group_norm_f32_syclPKfPfiiiPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZTSZL17sum_rows_f32_syclPKfPf
iiPN4sycl3_V15queueEEUlNS3_7nd_itemILi3EEEE_, _ZTSZZL13norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZGVdN32uuuuuu__ZTSZZL17rms_norm_f32_sy
clPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_itemILi3EEEE_, _ZGVdN32uuuuuu__ZTSZZL13norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_it
emILi3EEEE_, _ZTSZZL13norm_f32_syclPKfPfiifPN4sycl3_V15queueEENKUlRNS3_7handlerEE0_clES7_EUlNS3_7nd_itemILi3EEEE_ }) }                                                                    
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr                                                                                                                
11:31PM DBG GRPC(c0c3c83d0ec33ffe925657a56b06771b-127.0.0.1:33397): stderr  -11 (PI_ERROR_BUILD_PROGRAM_FAILURE)Exception caught at file:/build/backend/cpp/llama/llama.cpp/ggml-sycl.cpp,
 line:12644                                                                    

friendly ping @abhilash1910 @NeoZhangJianyu , chances you can help here? any pointers would be appreciated! thanks!

It looks something wrt linking, but I thought the installation steps should be general enough to apply to all the binaries in the example folder?

nevermind, turns out I'm somehow not capable to select the GPU and hence that's what happens (the error, even if cryptic is not regarding linking issues in build-time, but rather ops not supported by the device that was selected)

[NeoZhangJianyu]

sion error:

Could you share whole log?
I guess your device id is wrong, you set non-gpu device id.

[mudler]

sion error:

Could you share whole log? I guess your device id is wrong, you set non-gpu device id.

it looks that somehow I cannot see the GPU at all, I was confused that's why I picked up the first one (listed as acc):

$ ./bin/ls-sycl-device 
found 2 SYCL devices:
  Device 0: 12th Gen Intel(R) Core(TM) i7-1280P,        compute capability 3.0,
        max compute_units 20,   max work group size 8192,       max sub group size 64,  global mem size 67084083200
  Device 1: Intel(R) FPGA Emulation Device,     compute capability 1.2,
        max compute_units 20,   max work group size 67108864,   max sub group size 64,  global mem size 67084083200
$ sycl-ls
[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2  [2023.16.12.0.12_195853.xmain-hotfix]
[opencl:cpu:1] Intel(R) OpenCL, 12th Gen Intel(R) Core(TM) i7-1280P OpenCL 3.0 (Build 0) [2023.16.12.0.12_195853.xmain-hotfix]
$ clinfo -l                          
Platform #0: Intel(R) OpenCL
 `-- Device #0: 12th Gen Intel(R) Core(TM) i7-1280P
Platform #1: Intel(R) FPGA Emulation Platform for OpenCL(TM)
 `-- Device #0: Intel(R) FPGA Emulation Device
$ hwinfo --display
32: PCI 02.0: 0300 VGA compatible controller (VGA)              
  [Created at pci.386]
  Unique ID: _Znp.usB9nIk3U2E
  SysFS ID: /devices/pci0000:00/0000:00:02.0
  SysFS BusID: 0000:00:02.0
  Hardware Class: graphics card
  Model: "Intel VGA compatible controller"
  Vendor: pci 0x8086 "Intel Corporation"
  Device: pci 0x46a6 
  SubVendor: pci 0x1028 "Dell"
  SubDevice: pci 0x0b08 
  Revision: 0x0c
  Driver: "i915"
  Driver Modules: "i915"
  Memory Range: 0x6054000000-0x6054ffffff (rw,non-prefetchable)
  Memory Range: 0x4000000000-0x400fffffff (ro,non-prefetchable)
  I/O Ports: 0x3000-0x303f (rw)
  Memory Range: 0x000c0000-0x000dffff (rw,non-prefetchable,disabled)
  IRQ: 187 (44937037 events)
  Module Alias: "pci:v00008086d000046A6sv00001028sd00000B08bc03sc00i00"
  Driver Info #0:
    Driver Status: i915 is active
    Driver Activation Cmd: "modprobe i915"
  Config Status: cfg=new, avail=yes, need=no, active=unknown

Primary display adapter: #32
$ groups
mudler vboxusers video render

must be related to my drivers, and likely the ones I have in my openSUSE box don't work out of the box with sycl

[mudler]

ok, going to follow-up from images built from master. I think the changes here are correct as I see sycl output all over, the problem is I cannot select my GPU device

[NeoZhangJianyu]

ok, going to follow-up from images built from master. I think the changes here are correct as I see sycl output all over, the problem is I cannot select my GPU device

You can choose GPU by environment variable: GGML_SYCL_DEVICE

[mudler]

ok, going to follow-up from images built from master. I think the changes here are correct as I see sycl output all over, the problem is I cannot select my GPU device

You can choose GPU by environment variable: GGML_SYCL_DEVICE

Correct, I've tried that, but in my case see #1660 (comment) , there is no iGPU detected

[NeoZhangJianyu]

ok, going to follow-up from images built from master. I think the changes here are correct as I see sycl output all over, the problem is I cannot select my GPU device

You can choose GPU by environment variable: GGML_SYCL_DEVICE

Correct, I've tried that, but in my case see #1660 (comment) , there is no iGPU detected

Currently, GGML SYCL backend only support GPU. If there is no GPU, it can't work well. If you want to run on Intel GPU, you could use GGML oneMKL backend.