Code

IQ4_XS: a 4.25 bpw quantization#5747

Code

Merged

ikawrakow merged 11 commits intomasterggerganov/llama.cpp:masterfrom ik/iq4_nl_xsggerganov/llama.cpp:ik/iq4_nl_xsCopy head branch name to clipboardFeb 27, 2024

+668-38

Commits

Commits on Feb 26, 2024

• Try IQ4_NL with blocks of 64 - does not look good

  Kawrakow

  committed
  67264b3

  Copy full SHA for 67264b3

• iq4_xs: go to super-blocks of 256 and 6-bit scales for blocks of 32

  Kawrakow

  committed
  2b21d37

  Copy full SHA for 2b21d37

• iq4_xs: CUDA works - 133.2 t/s

  Kawrakow

  committed
  fddbfe8

  Copy full SHA for fddbfe8

• iq4_xs: AVX2 dot product

  Kawrakow

  committed
  061a16f

  Copy full SHA for 061a16f

• iq4_xs: ARM_NEON dot product

  Kawrakow

  committed
  a37980c

  Copy full SHA for a37980c

• iq4_nl: Metal implementation

  Show description for ad40ae6
  Kawrakow

  committed
  ad40ae6

  Copy full SHA for ad40ae6

Commits on Feb 27, 2024

• iq3_xs: minor fix

  Kawrakow

  committed
  6c2b233

  Copy full SHA for 6c2b233

• iq4_xs: shrink by using IQ3_S for attn_k and attn_q

  Kawrakow

  committed
  5c2b230

  Copy full SHA for 5c2b230

• iq4_xs: revert using IQ3_S for attn_k and attn_v

  Show description for f162fca
  Kawrakow

  committed
  f162fca

  Copy full SHA for f162fca

• Fix CI

  Kawrakow

  committed
  801f998

  Copy full SHA for 801f998

• iq4_xs: Added forgotten check for 256 divisibility

  Kawrakow

  committed
  d7bb4b6

  Copy full SHA for d7bb4b6