-
|
Is Fused Attention as described here also planned to be implemented for other ISAs, like AVX_VNNI? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Thank you for your interest of the Fused Attention optimization. We are planning on this. However, it will take some time as Fused Attention works only on activation which are more prone to quantization in terms of accuracy. In addition, we need to be more careful in terms of performance as there is no zero-cost weight quantization. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds good, thanks for your reply! |
Beta Was this translation helpful? Give feedback.
Thank you for your interest of the Fused Attention optimization. We are planning on this. However, it will take some time as Fused Attention works only on activation which are more prone to quantization in terms of accuracy. In addition, we need to be more careful in terms of performance as there is no zero-cost weight quantization.