llamafile : improve sgemm.cpp #6796
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Re-enable by default - Fix issue described in ggerganov#6716 - Make code more abstract, elegant, and maintainable - Faster handling of weirdly shaped `m` an `n` edge cases
|
📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for Expand details for performance related PR only
More---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 465 iterations"
y-axis "llamacpp:prompt_tokens_seconds"
x-axis "llamacpp:prompt_tokens_seconds" 1713727596 --> 1713728232
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 903.41, 903.41, 903.41, 903.41, 903.41, 655.86, 655.86, 655.86, 655.86, 655.86, 683.83, 683.83, 683.83, 683.83, 683.83, 703.98, 703.98, 703.98, 703.98, 703.98, 725.35, 725.35, 725.35, 725.35, 725.35, 754.87, 754.87, 754.87, 754.87, 754.87, 750.82, 750.82, 750.82, 750.82, 750.82, 760.52, 760.52, 760.52, 760.52, 760.52, 768.01, 768.01, 768.01, 768.01, 768.01, 764.59, 764.59, 764.59, 764.59, 764.59, 771.48, 771.48, 771.48, 771.48, 771.48, 766.39, 766.39, 766.39, 766.39, 766.39, 769.0, 769.0, 769.0, 769.0, 769.0, 785.47, 785.47, 785.47, 785.47, 785.47, 783.08, 783.08, 783.08, 783.08, 783.08, 750.46, 750.46, 750.46, 750.46, 750.46, 646.23, 646.23, 646.23, 646.23, 646.23, 646.67, 646.67, 646.67, 646.67, 646.67, 650.24, 650.24, 650.24, 650.24, 650.24, 650.22, 650.22, 650.22, 650.22, 650.22, 655.8, 655.8, 655.8, 655.8, 655.8, 663.75, 663.75, 663.75, 663.75, 663.75, 661.86, 661.86, 661.86, 661.86, 661.86, 662.94, 662.94, 662.94, 662.94, 662.94, 666.12, 666.12, 666.12, 666.12, 666.12, 666.5, 666.5, 666.5, 666.5, 666.5, 669.18, 669.18, 669.18, 669.18, 669.18, 670.52, 670.52, 670.52, 670.52, 670.52, 643.8, 643.8, 643.8, 643.8, 643.8, 647.81, 647.81, 647.81, 647.81, 647.81, 648.55, 648.55, 648.55, 648.55, 648.55, 647.84, 647.84, 647.84, 647.84, 647.84, 646.89, 646.89, 646.89, 646.89, 646.89, 648.43, 648.43, 648.43, 648.43, 648.43, 648.49, 648.49, 648.49, 648.49, 648.49, 651.7, 651.7, 651.7, 651.7, 651.7, 654.74, 654.74, 654.74, 654.74, 654.74, 654.25, 654.25, 654.25, 654.25, 654.25, 655.42, 655.42, 655.42, 655.42, 655.42, 658.0, 658.0, 658.0, 658.0, 658.0, 666.24, 666.24, 666.24, 666.24, 666.24, 670.04, 670.04, 670.04, 670.04, 670.04, 667.2, 667.2, 667.2, 667.2, 667.2, 666.66, 666.66, 666.66, 666.66, 666.66, 665.98, 665.98, 665.98, 665.98, 665.98, 666.02, 666.02, 666.02, 666.02, 666.02, 668.94, 668.94, 668.94, 668.94, 668.94, 671.72, 671.72, 671.72, 671.72, 671.72, 680.15, 680.15, 680.15, 680.15, 680.15, 673.62, 673.62, 673.62, 673.62, 673.62, 655.87, 655.87, 655.87, 655.87, 655.87, 654.24, 654.24, 654.24, 654.24, 654.24, 653.49, 653.49, 653.49, 653.49, 653.49, 652.47, 652.47, 652.47, 652.47, 652.47, 650.04, 650.04, 650.04, 650.04, 650.04, 652.53, 652.53, 652.53, 652.53, 652.53, 652.77, 652.77, 652.77, 652.77, 652.77, 655.36, 655.36, 655.36, 655.36, 655.36, 657.45, 657.45, 657.45, 657.45, 657.45, 658.24, 658.24, 658.24, 658.24, 658.24, 657.69, 657.69, 657.69, 657.69, 657.69, 657.69, 657.69, 657.69, 657.69]
More---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 465 iterations"
y-axis "llamacpp:predicted_tokens_seconds"
x-axis "llamacpp:predicted_tokens_seconds" 1713727596 --> 1713728232
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 36.3, 36.3, 36.3, 36.3, 36.3, 22.22, 22.22, 22.22, 22.22, 22.22, 22.9, 22.9, 22.9, 22.9, 22.9, 24.05, 24.05, 24.05, 24.05, 24.05, 24.57, 24.57, 24.57, 24.57, 24.57, 24.7, 24.7, 24.7, 24.7, 24.7, 26.32, 26.32, 26.32, 26.32, 26.32, 26.81, 26.81, 26.81, 26.81, 26.81, 27.0, 27.0, 27.0, 27.0, 27.0, 26.82, 26.82, 26.82, 26.82, 26.82, 26.41, 26.41, 26.41, 26.41, 26.41, 26.1, 26.1, 26.1, 26.1, 26.1, 25.1, 25.1, 25.1, 25.1, 25.1, 24.45, 24.45, 24.45, 24.45, 24.45, 24.38, 24.38, 24.38, 24.38, 24.38, 23.83, 23.83, 23.83, 23.83, 23.83, 23.23, 23.23, 23.23, 23.23, 23.23, 22.94, 22.94, 22.94, 22.94, 22.94, 22.93, 22.93, 22.93, 22.93, 22.93, 22.99, 22.99, 22.99, 22.99, 22.99, 22.74, 22.74, 22.74, 22.74, 22.74, 22.5, 22.5, 22.5, 22.5, 22.5, 22.35, 22.35, 22.35, 22.35, 22.35, 22.06, 22.06, 22.06, 22.06, 22.06, 21.89, 21.89, 21.89, 21.89, 21.89, 21.92, 21.92, 21.92, 21.92, 21.92, 22.05, 22.05, 22.05, 22.05, 22.05, 22.1, 22.1, 22.1, 22.1, 22.1, 22.27, 22.27, 22.27, 22.27, 22.27, 22.43, 22.43, 22.43, 22.43, 22.43, 22.54, 22.54, 22.54, 22.54, 22.54, 22.36, 22.36, 22.36, 22.36, 22.36, 22.36, 22.36, 22.36, 22.36, 22.36, 22.38, 22.38, 22.38, 22.38, 22.38, 22.57, 22.57, 22.57, 22.57, 22.57, 22.63, 22.63, 22.63, 22.63, 22.63, 22.72, 22.72, 22.72, 22.72, 22.72, 22.92, 22.92, 22.92, 22.92, 22.92, 22.98, 22.98, 22.98, 22.98, 22.98, 22.99, 22.99, 22.99, 22.99, 22.99, 22.95, 22.95, 22.95, 22.95, 22.95, 22.87, 22.87, 22.87, 22.87, 22.87, 22.78, 22.78, 22.78, 22.78, 22.78, 22.72, 22.72, 22.72, 22.72, 22.72, 22.74, 22.74, 22.74, 22.74, 22.74, 22.75, 22.75, 22.75, 22.75, 22.75, 22.93, 22.93, 22.93, 22.93, 22.93, 22.97, 22.97, 22.97, 22.97, 22.97, 22.94, 22.94, 22.94, 22.94, 22.94, 22.6, 22.6, 22.6, 22.6, 22.6, 22.42, 22.42, 22.42, 22.42, 22.42, 22.34, 22.34, 22.34, 22.34, 22.34, 22.05, 22.05, 22.05, 22.05, 22.05, 21.71, 21.71, 21.71, 21.71, 21.71, 21.58, 21.58, 21.58, 21.58, 21.58, 21.61, 21.61, 21.61, 21.61, 21.61, 21.7, 21.7, 21.7, 21.7, 21.7, 21.71, 21.71, 21.71, 21.71, 21.71, 21.79, 21.79, 21.79, 21.79, 21.79, 21.84, 21.84, 21.84, 21.84, 21.84, 21.88, 21.88, 21.88, 21.88]
Details
More---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 465 iterations"
y-axis "llamacpp:kv_cache_usage_ratio"
x-axis "llamacpp:kv_cache_usage_ratio" 1713727596 --> 1713728232
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.01, 0.01, 0.01, 0.01, 0.01, 0.33, 0.33, 0.33, 0.33, 0.33, 0.23, 0.23, 0.23, 0.23, 0.23, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.11, 0.11, 0.11, 0.11, 0.11, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.17, 0.17, 0.17, 0.17, 0.17, 0.2, 0.2, 0.2, 0.2, 0.2, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.21, 0.25, 0.25, 0.25, 0.25, 0.25, 0.18, 0.18, 0.18, 0.18, 0.18, 0.35, 0.35, 0.35, 0.35, 0.35, 0.28, 0.28, 0.28, 0.28, 0.28, 0.14, 0.14, 0.14, 0.14, 0.14, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.17, 0.17, 0.17, 0.17, 0.17, 0.26, 0.26, 0.26, 0.26, 0.26, 0.25, 0.25, 0.25, 0.25, 0.25, 0.29, 0.29, 0.29, 0.29, 0.29, 0.21, 0.21, 0.21, 0.21, 0.21, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.18, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.11, 0.11, 0.11, 0.11, 0.11, 0.15, 0.15, 0.15, 0.15, 0.15, 0.26, 0.26, 0.26, 0.26, 0.26, 0.13, 0.13, 0.13, 0.13, 0.13, 0.17, 0.17, 0.17, 0.17, 0.17, 0.15, 0.15, 0.15, 0.15, 0.15, 0.1, 0.1, 0.1, 0.1, 0.1, 0.15, 0.15, 0.15, 0.15, 0.15, 0.08, 0.08, 0.08, 0.08, 0.08, 0.14, 0.14, 0.14, 0.14, 0.14, 0.13, 0.13, 0.13, 0.13, 0.13, 0.21, 0.21, 0.21, 0.21, 0.21, 0.2, 0.2, 0.2, 0.2, 0.2, 0.29, 0.29, 0.29, 0.29, 0.29, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.11, 0.11, 0.11, 0.11, 0.11, 0.3, 0.3, 0.3, 0.3, 0.3, 0.42, 0.42, 0.42, 0.42, 0.42, 0.51, 0.51, 0.51, 0.51, 0.51, 0.4, 0.4, 0.4, 0.4, 0.4, 0.36, 0.36, 0.36, 0.36, 0.36, 0.39, 0.39, 0.39, 0.39, 0.39, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.14, 0.14, 0.14, 0.14, 0.13, 0.13, 0.13, 0.13, 0.13, 0.19, 0.19, 0.19, 0.19, 0.19, 0.14, 0.14, 0.14, 0.14, 0.14, 0.1, 0.1, 0.1, 0.1, 0.1, 0.19, 0.19, 0.19, 0.19, 0.19, 0.27, 0.27, 0.27, 0.27]
More---
config:
xyChart:
titleFontSize: 12
width: 900
height: 600
themeVariables:
xyChart:
titleColor: "#000000"
---
xychart-beta
title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
duration=10m 465 iterations"
y-axis "llamacpp:requests_processing"
x-axis "llamacpp:requests_processing" 1713727596 --> 1713728232
line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 2.0, 2.0, 2.0, 2.0, 2.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0]
|
ggerganov
reviewed
Apr 21, 2024
There was a problem hiding this comment.
The build fails on Mac:
$ make -j
I llama.cpp build info:
I UNAME_S: Darwin
I UNAME_P: arm
I UNAME_M: arm64
I CFLAGS: -I. -Icommon -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -DNDEBUG -DGGML_USE_ACCELERATE -DACCELERATE_NEW_LAPACK -DACCELERATE_LAPACK_ILP64 -DGGML_USE_LLAMAFILE -DGGML_USE_METAL -DGGML_METAL_EMBED_LIBRARY -std=c11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -Werror=implicit-int -Werror=implicit-function-declaration -pthread -Wunreachable-code-break -Wunreachable-code-return -Wdouble-promotion
I CXXFLAGS: -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread -Wunreachable-code-break -Wunreachable-code-return -Wmissing-prototypes -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -DNDEBUG -DGGML_USE_ACCELERATE -DACCELERATE_NEW_LAPACK -DACCELERATE_LAPACK_ILP64 -DGGML_USE_LLAMAFILE -DGGML_USE_METAL -DGGML_METAL_EMBED_LIBRARY
I NVCCFLAGS: -std=c++11 -O3
I LDFLAGS: -framework Accelerate -framework Foundation -framework Metal -framework MetalKit
I CC: Apple clang version 15.0.0 (clang-1500.1.0.2.5)
I CXX: Apple clang version 15.0.0 (clang-1500.1.0.2.5)
c++ -std=c++11 -fPIC -O3 -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wmissing-declarations -Wmissing-noreturn -pthread -Wunreachable-code-break -Wunreachable-code-return -Wmissing-prototypes -Wextra-semi -I. -Icommon -D_XOPEN_SOURCE=600 -D_DARWIN_C_SOURCE -DNDEBUG -DGGML_USE_ACCELERATE -DACCELERATE_NEW_LAPACK -DACCELERATE_LAPACK_ILP64 -DGGML_USE_LLAMAFILE -DGGML_USE_METAL -DGGML_METAL_EMBED_LIBRARY -c sgemm.cpp -o sgemm.o
sgemm.cpp:515:13: error: unknown type name 'D'
D Cv[RN][RM] = {};
^
sgemm.cpp:516:41: error: use of undeclared identifier 'KN'
for (int l = 0; l < k; l += KN)
^
2 errors generated.
make: *** [sgemm.o] Error 1Also, apply the following patch to enable LLAMAFILE with CMake builds:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f134a153..58a1805b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -43,17 +43,11 @@ else()
set(LLAMA_METAL_DEFAULT OFF)
endif()
-# TODO: fix this for Android CI
-# https://github.com/ggerganov/llama.cpp/pull/6716#issuecomment-2061509191
-#if (CMAKE_SYSTEM_NAME MATCHES "ANDROID")
-# set(LLAMA_LLAMAFILE_DEFAULT OFF)
-#else()
-# set(LLAMA_LLAMAFILE_DEFAULT ON)
-#endif()
-
-# TODO: temporary disable until MoE is fixed
-# https://github.com/ggerganov/llama.cpp/pull/6716
-set(LLAMA_LLAMAFILE_DEFAULT OFF)
+if (CMAKE_SYSTEM_NAME MATCHES "ANDROID")
+ set(LLAMA_LLAMAFILE_DEFAULT OFF)
+else()
+ set(LLAMA_LLAMAFILE_DEFAULT ON)
+endif()
# general
option(BUILD_SHARED_LIBS "build shared libraries" OFF)
|
@ggerganov Fixed. PTAL Please give Do you know if there's a better way to fix the Android issue? IIUC it's due to an instruction not being available on 32-bit ARM. Is there a way we could solve that with an |
|
@ggerganov Could you advise me on how I might bring the benefits of |
|
On make clean && LLAMA_NO_METAL=1 make -j && ./llama-bench -m models/mistral-7b-v0.2/ggml-model-fp16.gguf -m models/mistral-7b-v0.2/ggml-model-q8_0.gguf -m models/mistral-7b-v0.2/ggml-model-q4_0.gguf -ngl 0 -n 0
build: 8960fe8 (2713) With this PR without make clean && LLAMA_NO_ACCELERATE=1 LLAMA_NO_METAL=1 make -j && ./llama-bench -m models/mistral-7b-v0.2/ggml-model-fp16.gguf -m models/mistral-7b-v0.2/ggml-model-q8_0.gguf -m models/mistral-7b-v0.2/ggml-model-q4_0.gguf -ngl 0 -n 0
build: 6b220dc (2704) So for me, F16 is faster now, Q8_0 is the same and Q4_0 is slower. Btw, I've looked some more, and I think the proper call in diff --git a/ggml.c b/ggml.c
index e3356bdb..086db96a 100644
--- a/ggml.c
+++ b/ggml.c
@@ -10878,15 +10878,13 @@ UseGgmlGemm1:;
const size_t row_size = ggml_row_size(vec_dot_type, ne10);
#if GGML_USE_LLAMAFILE
- if (src1_cont) {
+ if (src1->type != vec_dot_type) {
for (int64_t i13 = 0; i13 < ne13; i13++)
for (int64_t i12 = 0; i12 < ne12; i12++)
if (!llamafile_sgemm(ne01, ne11, ne00/ggml_blck_size(src0->type),
(const char *)src0->data + i12/r2*nb02 + i13/r3*nb03,
nb01/ggml_type_size(src0->type),
- (const char *)wdata + ggml_row_size(vec_dot_type,
- nb12/ggml_type_size(src1->type)*i12 +
- nb13/ggml_type_size(src1->type)*i13),
+ (const char *)wdata + (i12*ne11 + i13*ne12*ne11)*row_size,
row_size/ggml_type_size(vec_dot_type),
(char *)dst->data + i12*nb2 + i13*nb3,
nb1/ggml_type_size(dst->type),Regarding Android: in Lines 291 to 307 in e931888 We should not repeat the same implementation twice - we have to see if we can reuse it. I also don't have setup Android builds and it takes me some time to get the build running. So for now, let's focus on fixing the SGEMM and later we can think about improving Android support Let me think some time about |
|
Review comments addressed. PTAL. Agree on Android 32-bit.
I studied the code for hours and managed to figure it out. I've got a |
ggerganov
approved these changes
Apr 22, 2024
|
These changes cause failed assertions when running Cohere's Command R+ model: When I reverted commit 192090b the problem disappeared. Log files attached. |
|
Here are instruction to trigger this assert: # convert to GGUF
python3 convert-hf-to-gguf.py ~/Data/huggingface/c4ai-command-r-plus/ --outfile models/command-r-plus/ggml-model-f16.gguf --outtype f16
# quantize to Q8_0 + F16 token embeddings
make -j
./quantize --token-embedding-type f16 ./models/command-r-plus/ggml-model-f16.gguf ./models/command-r-plus/ggml-model-q8_0.gguf q8_0
# build in DEBUG and run
make clean
LLAMA_DEBUG=1 LLAMA_NO_METAL=1 LLAMA_NO_ACCELERATE=1 make -j
./main -m ./models/command-r-plus/ggml-model-q8_0.gguf |
|
I just tried a naive solution and replaced all ints in sgemm.cpp and sgemm.h with int64_t, and the resulting code works fine without any performance penalty (at least on my Epyc Genoa). Also there are no more crashes due to int overflow in pointer calculations when using Command R+. By the way, @jart thank you for these changes, they improved the prompt eval time on my system by 65% on llama-3 70B Q8! |
|
Thanks for the inbox bump. Making this my top priority now. Expect a PR shortly. |
This was referenced Apr 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
mannedge cases