Code

4-bit Integer quantisation#27

Code

Merged

ggerganov merged 38 commits intomasterggerganov/ggml:masterfrom gqggerganov/ggml:gqCopy head branch name to clipboardMar 29, 2023

+6,928-1,306

Commits

Commits on Mar 29, 2023

• gq : attempt at n-bit quantization

  ggerganov

  committed
  8f45628

  Copy full SHA for 8f45628

• gq : add amax based method 3

  ggerganov

  committed
  b0a46fd

  Copy full SHA for b0a46fd

• gq : progress on method 2

  ggerganov

  committed
  da2de94

  Copy full SHA for da2de94

• gq : method 4 (AVX2)

  ggerganov

  committed
  aa5506c

  Copy full SHA for aa5506c

• gq : method 4 (ARM)

  ggerganov

  committed
  1fc11de

  Copy full SHA for 1fc11de

• gq : method 4 (AVX2 attempt) + method 5 (no min)

  ggerganov

  committed
  dae323c

  Copy full SHA for dae323c

• gq : method 5 (ARM)

  ggerganov

  committed
  349e917

  Copy full SHA for 349e917

• gpt-2 : model conversion for Q4_0 quantization

  ggerganov

  committed
  ff4c653

  Copy full SHA for ff4c653

• ggml : Q4_0 quantization support (ggml_get_rows())

  ggerganov

  committed
  21514b7

  Copy full SHA for 21514b7

• gpt-2 : loading Q4_0 quantized model

  ggerganov

  committed
  ff54fda

  Copy full SHA for ff54fda

• ggml : q4_0 quantization support

  ggerganov

  committed
  2219c11

  Copy full SHA for 2219c11

• ggml : q4_1 quantization support (seems to work for bigger models)

  ggerganov

  committed
  5bdfce2

  Copy full SHA for 5bdfce2

• gpt-2 : add gpt-2-quantize tool for quantizing f32 GPT-2 models

  ggerganov

  committed
  b0cab89

  Copy full SHA for b0cab89

• ggml : 4-bit quantization works (only scalar for now)

  ggerganov

  committed
  2f11888

  Copy full SHA for 2f11888

• gq : add method 6 (ARM)

  ggerganov

  committed
  b82c27f

  Copy full SHA for b82c27f

• ggml : vectorized mad q4_0 (ARM)

  ggerganov

  committed
  2e75d8f

  Copy full SHA for 2e75d8f

• ggml : vectorized quantize_row_q4_0 (ARM)

  ggerganov

  committed
  b0c22a4

  Copy full SHA for b0c22a4

• ggml : simplify mad q4_0 (ARM)

  ggerganov

  committed
  3c757a4

  Copy full SHA for 3c757a4

• ggml : minor indentations

  ggerganov

  committed
  e3ad879

  Copy full SHA for e3ad879

• gpt-j : support for 4-bit quantized model inference

  ggerganov

  committed
  8abcab4

  Copy full SHA for 8abcab4

• ggml : GGML_ASSERT() instead of assert() where appropriate

  ggerganov

  committed
  c21972c

  Copy full SHA for c21972c

• gpt : avoid ggml_transpose on model tensors (new models!)

  ggerganov

  committed
  4a56c5b

  Copy full SHA for 4a56c5b

• gpt-2 : minor

  ggerganov

  committed
  904605c

  Copy full SHA for 904605c

• gpt-j : fix conversion for FP16 models (such as GPT-JT-6B)

  ggerganov

  committed
  441a38f

  Copy full SHA for 441a38f

• ggml : add ggml_compute_forward_rope_f16()

  ggerganov

  committed
  5336828

  Copy full SHA for 5336828

• gpt : fix memory usage computation

  ggerganov

  committed
  99af48e

  Copy full SHA for 99af48e

• ggml : fix ggml_is_contiguous() to take into account blck size

  ggerganov

  committed
  6aae09e

  Copy full SHA for 6aae09e

• whisper : add whisper-qunatize tool

  ggerganov

  committed
  d0ac5eb

  Copy full SHA for d0ac5eb

• whisper : add support for quantized models

  ggerganov

  committed
  37d427d

  Copy full SHA for 37d427d

• whisper : mem usage based on model format type

  ggerganov

  committed
  e904a58

  Copy full SHA for e904a58

• gpt : seems not worth to use FP16 for KV cache

  ggerganov

  committed
  63a8f62

  Copy full SHA for 63a8f62

• gpt : support quantisation of f16 models files

  ggerganov

  committed
  519ce47

  Copy full SHA for 519ce47

• ggml : fixes for rpi4

  ggerganov

  committed
  9881c2b

  Copy full SHA for 9881c2b

• whisper : add Q4_1 model sizes

  ggerganov

  committed
  a85bc0f

  Copy full SHA for a85bc0f

• ggml : add WASM SIMD for Q4_0

  ggerganov

  committed
  c4f1403

  Copy full SHA for c4f1403

• utils : print quantization histograms

  ggerganov

  committed
  331a862

  Copy full SHA for 331a862

• ggml : sync all changes from llama.cpp and whisper.cpp

  ggerganov

  committed
  154fcc3

  Copy full SHA for 154fcc3

• ggml : finalize the Q4_1 quantization for ARM_NEON

  ggerganov

  committed
  724c45d

  Copy full SHA for 724c45d