Eval bug: DeepScaleR-1.5B-Preview produces random tokens on AArch64 with fp16fml #11920
Labels
Comments
|
This patch shows that scaling values can fix the problem, but ideally the scale would be applied at load time rather than during computation. diff --git a/ggml/src/ggml-cpu/ggml-cpu.c b/ggml/src/ggml-cpu/ggml-cpu.c
index dbef5df2..326ebcf2 100644
--- a/ggml/src/ggml-cpu/ggml-cpu.c
+++ b/ggml/src/ggml-cpu/ggml-cpu.c
@@ -1469,9 +1469,13 @@ static void ggml_vec_dot_f16(int n, float * restrict s, size_t bs, ggml_fp16_t *
GGML_F16_VEC ax[GGML_F16_ARR];
GGML_F16_VEC ay[GGML_F16_ARR];
+ ggml_float scaleup = 8.0;
+
+ GGML_F16_VEC scaledown = GGML_F16_VEC_SET1(1.0 / scaleup);
+
for (int i = 0; i < np; i += GGML_F16_STEP) {
for (int j = 0; j < GGML_F16_ARR; j++) {
- ax[j] = GGML_F16_VEC_LOAD(x + i + j*GGML_F16_EPR, j);
+ ax[j] = GGML_F16_VEC_MUL(GGML_F16_VEC_LOAD(x + i + j*GGML_F16_EPR, j), scaledown);
ay[j] = GGML_F16_VEC_LOAD(y + i + j*GGML_F16_EPR, j);
sum[j] = GGML_F16_VEC_FMA(sum[j], ax[j], ay[j]);
@@ -1481,6 +1485,8 @@ static void ggml_vec_dot_f16(int n, float * restrict s, size_t bs, ggml_fp16_t *
// reduce sum0..sum3 to sum0
GGML_F16_VEC_REDUCE(sumf, sum);
+ sumf *= scaleup;
+
// leftovers
for (int i = np; i < n; ++i) {
sumf += (ggml_float)(GGML_FP16_TO_FP32(x[i])*GGML_FP16_TO_FP32(y[i]));
diff --git a/ggml/src/ggml-cpu/llamafile/sgemm.cpp b/ggml/src/ggml-cpu/llamafile/sgemm.cpp
index e0482c59..6b7c8785 100644
--- a/ggml/src/ggml-cpu/llamafile/sgemm.cpp
+++ b/ggml/src/ggml-cpu/llamafile/sgemm.cpp
@@ -2446,6 +2446,7 @@ bool llamafile_sgemm(const struct ggml_compute_params * params, int64_t m, int64
return false;
}
case GGML_TYPE_F16: {
+ return false;
#if defined(__AVX512F__)
if (Btype == GGML_TYPE_F16) {
tinyBLAS<16, __m512, __m512, ggml_fp16_t, ggml_fp16_t, float> tb{ params, k, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Name and Version
version: 4733 (faaa9b93)
built with cc (Ubuntu 14.2.0-4ubuntu2) 14.2.0 for aarch64-linux-gnu
Operating systems
Linux
GGML backends
CPU
Hardware
Tested on Snapdragon X Elite and Cortex-A76.
Models
https://huggingface.co/bartowski/agentica-org_DeepScaleR-1.5B-Preview-GGUF/blob/main/agentica-org_DeepScaleR-1.5B-Preview-IQ4_NL.gguf
Other formats are broken as well, for example Q4_0 and F16.
Problem description & steps to reproduce
When llama.cpp is compiled with the
fp16fmlCPU flag (-DGGML_NATIVE=0 -DGGML_CPU_ARM_ARCH=armv8.2-a+fp16fml), the "DeepScaleR" model can start outputting random tokens, for example:I've found that
ggml_vec_dot_f16can return values as high as 321000 for this model. That's okay withoutfp16fml, since single-precision accumulators are used.But with
fp16fml, the intermediate values are summed in fp16, where the maximum normalized number is 65504. So the values are about five times larger than what can be represented. Some of the time it still works, because accumulation is done into thesumvector, and so while the returnedsumfwould overflow a half-precision variable, the value is spread out over multiple elements of the vector. But other times, a single one of the 32 accumulator elements overflows, and soggml_vec_dot_f16returnsinf.(The code in
sgemm.cppexhibits similar issues, but for this testing I've completely disabled the FP16 case there.)What are possible solutions?
I guess what would make the most sense is to use a scale factor somewhere, for example dividing by eight or sixteen for this model.
There are a few other possibilities I can think of:
isfiniteand handle overflow when it happens somehowGGML_F16_STEPand makeGGML_F16x8_REDUCEdo everything in single-precisionFirst Bad Commit
No response
Relevant log output
The text was updated successfully, but these errors were encountered: