support Avx2

[yuchengliu1]

Type of Change

feature
No API changed

Run LLM on client CPU with AVX2 (without avx512)

detail description
JIRA ticket: xxx

Expected Behavior & Potential Risk

CPU: i7-9850H@2.6GHz
memory: single channel 32GB@2666MHz (memory bandwidth 21.3GB/s)
compute_type is FP32

 

	first token	next token
gptj-6B_q4j_b128	382.57 ms (63.76 ms per token)	220.18 ms per token
llama-7B_q4j_b128	515.35 ms (73.62 ms per token)	250.71 ms per token
llama2-7B_q4j_b128	518.14 ms (74.02 ms per token)	253.23 ms per token

 

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

[airMeng]

I think you already remove all warnings in this PR?

[airMeng]

test on i7-9850H@2.6Hz

 

  first token next token
gptj-6B_q4j_b128 382.57 ms (63.76 ms per token) 220.18 ms per token
llama-7B_q4j_b128 515.35 ms (73.62 ms per token) 250.71 ms per token
llama2-7B_q4j_b128 518.14 ms (74.02 ms per token) 253.23 ms per token

Can you provide memory bandwidth comparison between clients and SPR? It's useful to judge whether the gap is meaningful.

[a32543254]

could we auto detect the machine's isa to add follow compile args without manually ? some consumers may not sure about their machine's isa

[yuchengliu1]

fix in PR511

[DDEle]

I don't think so. If I remember correctly, there was a similar discussion for the deprecated executor, and the result is that it is not possible to find the hardware info of the runtime match during compile time on the machine that compiles the executable.
cc @luoyu-intel

[yuchengliu1]

I think you already remove all warnings in this PR?

Yes, there is no warning now.

[a32543254]

LGTM

[a32543254]

Better we also give the same models's performance between ITREX AVX2 and GGML AVX2 to find our benefit or gap.

[yuchengliu1]

Better we also give the same models's performance between ITREX AVX2 and GGML AVX2 to find our benefit or gap.

llama2-7B	first token	next token
llama.cpp	494.15 ms (61.77 ms per token)	141.69 ms per token
ITREX q4_0	743.89ms (106.27ms per token)	136.37ms per token
ITREX q4_j	518.14 ms (74.02 ms per token)	253.23 ms per token

the profiling of ITREX AVX2 (profile on q4_j without ffn fusion)

perf_total_per_op_us[ ADD] = 0.184 ms
perf_total_per_op_us[ MUL] = 0.327 ms
perf_total_per_op_us[ SILU] = 3.488 ms
perf_total_per_op_us[ RMS_NORM] = 0.396 ms
perf_total_per_op_us[ MUL_MAT] = 2.497 ms
perf_total_per_op_us[ SCALE] = 0.032 ms
perf_total_per_op_us[ CPY] = 1.177 ms
perf_total_per_op_us[ RESHAPE] = 0.037 ms
perf_total_per_op_us[ VIEW] = 0.066 ms
perf_total_per_op_us[ PERMUTE] = 0.029 ms
perf_total_per_op_us[ TRANSPOSE] = 0.006 ms
perf_total_per_op_us[ GET_ROWS] = 0.008 ms
perf_total_per_op_us[ DIAG_MASK_INF] = 0.025 ms
perf_total_per_op_us[ SOFT_MAX] = 0.166 ms
perf_total_per_op_us[ ROPE] = 2.940 ms
perf_total_per_op_us[ INNER PRODUCT] = 254.974 ms

the profiling of GGML AVX2 (profile on q4_0)

perf_total_per_op_us[ ADD] = 0.119 ms
perf_total_per_op_us[ MUL] = 0.244 ms
perf_total_per_op_us[ SILU] = 3.395 ms
perf_total_per_op_us[ RMS_NORM] = 0.345 ms
perf_total_per_op_us[ MUL_MAT] = 2.281 ms
perf_total_per_op_us[ SCALE] = 0.024 ms
perf_total_per_op_us[ CPY] = 0.976 ms
perf_total_per_op_us[ RESHAPE] = 0.032 ms
perf_total_per_op_us[ VIEW] = 0.063 ms
perf_total_per_op_us[ PERMUTE] = 0.023 ms
perf_total_per_op_us[ TRANSPOSE] = 0.017 ms
perf_total_per_op_us[ GET_ROWS] = 0.004 ms
perf_total_per_op_us[ DIAG_MASK_INF] = 0.015 ms
perf_total_per_op_us[ SOFT_MAX] = 0.117 ms
perf_total_per_op_us[ ROPE] = 0.271 ms
perf_total_per_op_us[ INNER PRODUCT] = 123.976 ms