Building Dockerfile with CUDA support and smaller image.

[jflachman]

I had two issues trying to build a file with CUDA support.

1. I could not build with llama-cpp-pytyon:2.82. I had to revert to 2.77. I also reverted to CUDA 12.1.1.
2. The docker image size is a little more than 13GB. I wanted a smaller docker image.

After much try and error, I solved 1 by reverting to build 2.77. I used the approach for @Hansson0728 had for issue #1119. #1119. However, that still did not work completely for me. It turns out I needed to set LD_LIBRARY_PATH for the CUDA libraries. The path in the devel image had also changed since he created that solution.

I have attached the working Dockerfile below.

[jflachman]

Solved

I finally solved the problem. I rolled back to the version of llama-cpp-python/docker/cuda-simple/Dockerfile from before update #1530 made 3 weeks ago. This used a base image of nvidia/cuda/12.1.1-devel-ubuntu22.04 and I used llama-cpp-python==0.2.77 with the associate whl file.

[jflachman]

After getting a working CUDA build (13GB), I updated the Dockerfile for multistep build. This reduced the build to 5GB.

Here is the Dockerfile that works. (5GB image)

Dockerfile.txt

You may also download the docker container here: Docker Hub: jflachman/llama-cpp-python. This also includes a brief guide for its use. Using this container will allow you to provide a server.config file on startup to support server configuration and multiple model setup.