Version 0.1.60 does not install correctly

[Barafu]

Short

In the Oobabooga installation, on Windows11, the package stops working when upgrading from 0.1.57 to 0.1.60. Reverting back to 0.1.57 fixes the problem. Complains about Shared library with base name 'llama' not found. There is no llama.dll anywhere in the folder, only llama.lib . I do not see any compilation errors.

Logs.

Installation log:

 .\cmd_windows.bat
(F:\oobabooga_windows\installer_files\env) F:\oobabooga_windows>pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir
Collecting llama-cpp-python
  Downloading llama-cpp-python-0.1.60.tar.gz (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 516.0 kB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting typing-extensions>=4.5.0
  Downloading typing_extensions-4.6.3-py3-none-any.whl (31 kB)
Collecting numpy>=1.20.0
  Downloading numpy-1.24.3-cp310-cp310-win_amd64.whl (14.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.8/14.8 MB 1.2 MB/s eta 0:00:00
Collecting diskcache>=5.6.1
  Downloading diskcache-5.6.1-py3-none-any.whl (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.6/45.6 kB 2.2 MB/s eta 0:00:00
Building wheels for collected packages: llama-cpp-python
  Building wheel for llama-cpp-python (pyproject.toml) ... done
  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.60-cp310-cp310-win_amd64.whl size=68815 sha256=3963c2f660e18df2a5e72cf2295cffd4d0c632accea654a06b31ee96cf6c5c52
  Stored in directory: F:\oobabooga_windows\installer_files\pip-ephem-wheel-cache-1a8td5kq\wheels\eb\a7\7e\e2f6aaef90347cd520e86d37bf5c613d1b96eeab4512dc080a
Successfully built llama-cpp-python
Installing collected packages: typing-extensions, numpy, diskcache, llama-cpp-python
  Attempting uninstall: typing-extensions
    Found existing installation: typing_extensions 4.5.0
    Uninstalling typing_extensions-4.5.0:
      Successfully uninstalled typing_extensions-4.5.0
  Attempting uninstall: numpy
    Found existing installation: numpy 1.24.3
    Uninstalling numpy-1.24.3:
      Successfully uninstalled numpy-1.24.3
  Attempting uninstall: diskcache
    Found existing installation: diskcache 5.6.1
    Uninstalling diskcache-5.6.1:
      Successfully uninstalled diskcache-5.6.1
  Attempting uninstall: llama-cpp-python
    Found existing installation: llama-cpp-python 0.1.57
    Uninstalling llama-cpp-python-0.1.57:
      Successfully uninstalled llama-cpp-python-0.1.57
Successfully installed diskcache-5.6.1 llama-cpp-python-0.1.60 numpy-1.24.3 typing-extensions-4.6.3

Launch log

PS F:\oobabooga_windows> .\start_windows.bat
bin F:\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll
INFO:Loading settings from settings.yaml...
The following models are available:

1. facebook_opt-350m
2. gpt4-x-alpaca-native-13B-ggml-q5_1.bin
3. guanaco-13B.ggmlv3.q5_1.bin
4. Manticore-13B-Chat-Pyg-Guanaco-GGML-q4_0.bin
5. Manticore-13B.ggmlv3.q5_1.bin
6. Manticore-13B.ggmlv3.q5_K_M.bin
7. pygmalion-13b-ggml-q5_1.bin
8. Wizard-Vicuna-30B-Uncensored.ggmlv3.q4_1.bin
9. WizardLM-13B-1.0.ggmlv3.q5_1.bin
10. WizardLM-Uncensored-SuperCOT-Storytelling.ggmlv3.q4_0.bin

Which one do you want to load? 1-10

3

INFO:Loading guanaco-13B.ggmlv3.q5_1.bin...
Traceback (most recent call last):
  File "F:\oobabooga_windows\text-generation-webui\server.py", line 1079, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "F:\oobabooga_windows\text-generation-webui\modules\models.py", line 94, in load_model
    output = load_func(model_name)
  File "F:\oobabooga_windows\text-generation-webui\modules\models.py", line 262, in llamacpp_loader
    from modules.llamacpp_model import LlamaCppModel
  File "F:\oobabooga_windows\text-generation-webui\modules\llamacpp_model.py", line 11, in <module>
    from llama_cpp import Llama, LlamaCache
  File "F:\oobabooga_windows\installer_files\env\lib\site-packages\llama_cpp\__init__.py", line 1, in <module>
    from .llama_cpp import *
  File "F:\oobabooga_windows\installer_files\env\lib\site-packages\llama_cpp\llama_cpp.py", line 77, in <module>
    _lib = _load_shared_library(_lib_base_name)
  File "F:\oobabooga_windows\installer_files\env\lib\site-packages\llama_cpp\llama_cpp.py", line 68, in _load_shared_library
    raise FileNotFoundError(
FileNotFoundError: Shared library with base name 'llama' not found

Done!
Press any key to continue . . .
PS F:\oobabooga_windows>

[Barafu]

I just tested 0.1.59 - it works.

[abetlen]

@Barafu thanks for reporting, likely due to #350, it installed correctly in the actions environment so I assumed it would be fine. I'll work on a fix.

Since you're on windows, is it installed from source or via the wheels from the release page? You should be able to tell by doing calling pip freeze.

[Barafu]

pip freeze only says llama-cpp-python==0.1.60. I do have various compilers installed, if that is relevant.

[abetlen]

Thanks! Do you mind running pip install llama-cpp-python --force-reinstall --upgrade --no-cache-dir --verbose and pasting the output here.

[Barafu]

Sure! Here

[abetlen]

Thank you!

Hmm, it looks like the .dll should be getting installed according to line 241

  -- Installing: F:/oobabooga_windows/installer_files/pip-install-r9o9syij/llama-cpp-python_e8ccf4a8b92f4361952dae89106d5f78/llama_cpp/llama.lib
  -- Installing: F:/oobabooga_windows/installer_files/pip-install-r9o9syij/llama-cpp-python_e8ccf4a8b92f4361952dae89106d5f78/llama_cpp/llama.dll
  *** scikit-build-core 0.4.4 using CMake 3.26.4

I'll set up a github action workflow to install from pypi / run tests.

[Barafu]

No llama.dll exists anywhere, I checked with search. Could it get a wrong name?

With 1.59 it goes to F:\oobabooga_windows\installer_files\env\Lib\site-packages\llama_cpp\llama.dll

[minemo]

Having the same problem. Maybe something to do with
scikit_build_core - WARNING - Can't find a Python library, got libdir=None, ldlibrary=None, multiarch=None, masd=None ?

[Celppu]

I have same problem
...
fatal: not a git repository: /tmp/pip-install-dly1xgll/llama-cpp-python_670d21f762724dc986670f5f60a6d74d/vendor/llama.cpp/../../.git/modules/vendor/llama.cpp
fatal: not a git repository: /tmp/pip-install-dly1xgll/llama-cpp-python_670d21f762724dc986670f5f60a6d74d/vendor/llama.cpp/../../.git/modules/vendor/llama.cpp
...

reverting back to 0.1.59 works

[stillmatic]

adding +1 to 1.60 failing on MacOS with the same error.

  Running command Building wheel for llama-cpp-python (pyproject.toml)
  loading initial cache file /var/folders/wn/8cd29xbx4vq78v3wqm91rdlm0000gn/T/tmp6xdla7wb/build/CMakeInit.txt
  -- The C compiler identification is AppleClang 14.0.3.14030022
  -- The CXX compiler identification is AppleClang 14.0.3.14030022
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- Configuring done (1.0s)
  -- Generating done (0.0s)
  -- Build files have been written to: /var/folders/wn/8cd29xbx4vq78v3wqm91rdlm0000gn/T/tmp6xdla7wb/build
  [1/1] cd /private/var/folders/wn/8cd29xbx4vq78v3wqm91rdlm0000gn/T/pip-install-x3i2pshd/llama-cpp-python_1e42829dafad4a18b695aa2c0af0f617/vendor/llama.cpp && make libllama.so
  I llama.cpp build info:
  I UNAME_S:  Darwin
  I UNAME_P:  arm
  I UNAME_M:  arm64
  I CFLAGS:   -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -DGGML_USE_K_QUANTS -DGGML_USE_ACCELERATE
  I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread
  I LDFLAGS:   -framework Accelerate
  I CC:       Apple clang version 14.0.3 (clang-1403.0.22.14.1)
  I CXX:      Apple clang version 14.0.3 (clang-1403.0.22.14.1)

  c++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -c llama.cpp -o llama.o
  cc  -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -DGGML_USE_K_QUANTS -DGGML_USE_ACCELERATE   -c ggml.c -o ggml.o
  cc -I.              -O3 -std=c11   -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -pthread -DGGML_USE_K_QUANTS -DGGML_USE_ACCELERATE   -c -o k_quants.o k_quants.c
  c++ -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -shared -fPIC -o libllama.so llama.o ggml.o k_quants.o  -framework Accelerate
  -- Install configuration: "Release"
  -- Installing: /private/var/folders/wn/8cd29xbx4vq78v3wqm91rdlm0000gn/T/pip-install-x3i2pshd/llama-cpp-python_1e42829dafad4a18b695aa2c0af0f617/llama_cpp/libllama.so
  *** scikit-build-core 0.4.4 using CMake 3.26.4
  *** Configurating CMake...
  *** Building project with Ninja...
  *** Installing project into wheel...
  *** Making wheel...
  Building wheel for llama-cpp-python (pyproject.toml) ... done
  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.60-cp310-cp310-macosx_13_0_arm64.whl size=28396 sha256=ba91c2b560e7d5ffd171e2631c5ecf471e4e71d1cda2f3fa3d9a1871d84b20ee
  Stored in directory: /private/var/folders/wn/8cd29xbx4vq78v3wqm91rdlm0000gn/T/pip-ephem-wheel-cache-9myv_y5t/wheels/eb/a7/7e/e2f6aaef90347cd520e86d37bf5c613d1b96eeab4512dc080a
Successfully built llama-cpp-python

AFAICT the build suceeeds but it's still unable to load.

[abetlen]

Yeah it looks like 0.1.60 is completely broken when installing from pip, sorry about that. I'll work on a fix and yank the broken release. Should have this fixed shortly.

[DocShotgun]

I've been trying to build 0.1.60 from source all day wondering if it was just me not getting a llama.dll lol. Looking forward to trying out k-quants with the Windows GPU offloading fixes.

[abetlen]

@Barafu @minemo @Celppu @stillmatic @DocShotgun pushed v0.1.61 to PyPI, reverted the build system but kept the recent updates / fixes to llama.cpp (metal support still missing until our PR is merged), let me know if everything works now, cheers!

[Barafu]

Works for me.

[minemo]

Same on my end. Thanks ^^

[Celppu]

@Barafu @minemo @Celppu @stillmatic @DocShotgun pushed v0.1.61 to PyPI, reverted the build system but kept the recent updates / fixes to llama.cpp (metal support still missing until our PR is merged), let me know if everything works now, cheers!

Thank you, 0.1.61 works!