add sq4 quantizer and simd operators #41

ShawnShawnYou · 2024-09-25T04:06:10Z

Add sq4 quantizer and simd operators.
For sq4-uniform, support simd (avx512, avx2, sse).
For sq4, only support generic distance computation.

jiaweizone

Part-I

src/quantization/sq4_quantizer.h

jiaweizone · 2024-09-25T06:56:14Z

src/quantization/sq4_quantizer.h

+    uint32_t vector_encoded_size_;
+};
+
+template <MetricType Metric>


Metric -> metric

src/quantization/sq4_quantizer.h

jiaweizone · 2024-09-26T04:06:51Z

src/quantization/sq4_quantizer.h

+    return true;
+}
+
+template <MetricType Metric>


Metric -> metric

jiaweizone · 2024-09-26T04:14:06Z

src/quantization/sq4_quantizer.h

+SQ4Quantizer<Metric>::DecodeOneImpl(const uint8_t* codes, DataType* data) {
+    for (uint32_t d = 0; d < this->dim_; d++) {
+        if (d & 1) {
+            data[d] = ((codes[d / 2] & 0xf0) >> 4) / 15.0 * diff_[d] + lower_bound_[d];


d / 2 can rewrite to d >> 1 ?

jiaweizone · 2024-09-26T16:09:06Z

src/quantization/sq4_quantizer.h

+    } else if (Metric == MetricType::METRIC_TYPE_IP) {
+        return SQ4ComputeCodesIP(codes1, codes2, lower_bound_.data(), diff_.data(), this->dim_);
+    } else if (Metric == MetricType::METRIC_TYPE_COSINE) {
+        return SQ4ComputeCodesIP(codes1, codes2, lower_bound_.data(), diff_.data(), this->dim_);


There are not implement the Cosine Metric ?

yes, for now, we use IP as a temporary implement for cosine.

jiaweizone · 2024-09-26T16:12:50Z

src/quantization/sq4_quantizer.h

+            SQ4ComputeCodesIP(computer.buf_, codes, lower_bound_.data(), diff_.data(), this->dim_);
+    } else if (Metric == MetricType::METRIC_TYPE_COSINE) {
+        dists[0] =
+            SQ4ComputeCodesIP(computer.buf_, codes, lower_bound_.data(), diff_.data(), this->dim_);


jiaweizone · 2024-09-26T16:13:23Z

src/quantization/sq4_quantizer.h

+        dists[0] =
+            SQ4ComputeCodesIP(computer.buf_, codes, lower_bound_.data(), diff_.data(), this->dim_);
+    } else {
+        dists[0] = 0;


Here need print any error log ?

jiaweizone · 2024-09-26T16:46:19Z

src/simd/generic.cpp

+float
+SQ4ComputeCodesIP(const uint8_t* codes1,
+                  const uint8_t* codes2,
+                  const float* lowerBound,


lowerBound -> lower_bound

jiaweizone · 2024-09-26T16:47:26Z

src/simd/generic.cpp

+            x_hi = ((codes1[d / 2] & 0xf0) >> 4) * 15.0 / diff[d + 1] + lowerBound[d + 1];
+            y_hi = ((codes2[d / 2] & 0xf0) >> 4) * 15.0 / diff[d + 1] + lowerBound[d + 1];


jiaweizone · 2024-09-26T16:48:06Z

src/simd/sq4_simd.h

+float
+SQ4ComputeIP(const float* query,
+             const uint8_t* codes,
+             const float* lowerBound,


jiaweizone · 2024-09-26T16:49:08Z

src/simd/sse.cpp

+    __m128i sum = _mm_setzero_si128();
+    __m128i mask = _mm_set1_epi8(0xf);
+    for (; d + 31 < dim; d += 32) {
+        auto xx = _mm_loadu_si128((__m128i*)(codes1 + d / 2));


d / 2 -> d >> 1 ?

jiaweizone · 2024-09-26T16:50:29Z

tests/fixtures/fixtures.cpp

+            if (delta < 0) {
+                delta = 0;
+            }
+            if (delta > 0.999) {


missing change ?

src/quantization/sq4_quantizer.h

src/quantization/sq4_uniform_quantizer.h

src/simd/generic.cpp

src/simd/sq4_simd.h

jiaweizone

style: change all lowerBound to lower_bound

src/quantization/sq4_quantizer.h

src/quantization/sq4_uniform_quantizer.h

jiaweizone · 2024-10-02T14:08:58Z

tests/fixtures/fixtures.cpp

+            if (delta < 0) {
+                delta = 0;
+            }
+            if (delta > 0.999) {


missing change ?

jiaweizone · 2024-10-02T14:09:18Z

src/simd/sse.cpp

@@ -511,6 +514,76 @@ SQ8ComputeCodesL2Sqr(const uint8_t* codes1,
    return Generic::SQ8ComputeCodesIP(codes1, codes2, lowerBound, diff, dim);


lowerBound -> lower_bound

this issue is belongs to SQ8, it should be fixed in another PR?

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

src/quantization/sq4_uniform_quantizer.h

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

wxyucs

lgtm

jiaweizone

LGTM

jiaweizone · 2024-10-14T06:14:32Z

src/simd/sse.cpp

@@ -511,6 +514,76 @@ SQ8ComputeCodesL2Sqr(const uint8_t* codes1,
    return Generic::SQ8ComputeCodesIP(codes1, codes2, lowerBound, diff, dim);


ShawnShawnYou requested review from jiaweizone, wxyucs, inabao and LHT129 September 25, 2024 04:06

ShawnShawnYou force-pushed the add-sq4-quantizer-and-simd-operators branch from 5a799da to b33c714 Compare September 25, 2024 04:08

wxyucs added the kind/feature New feature or request label Sep 25, 2024

wxyucs self-assigned this Sep 25, 2024

LHT129 mentioned this pull request Sep 25, 2024

Add a new Layer "DataCell" to manage data access #40

Closed

jiaweizone reviewed Sep 25, 2024

View reviewed changes

jiaweizone reviewed Sep 26, 2024

View reviewed changes

pull-request-size bot added the size/XXL label Sep 27, 2024

ShawnShawnYou force-pushed the add-sq4-quantizer-and-simd-operators branch from e15dcaa to de0ef9c Compare September 27, 2024 08:36

wxyucs requested changes Sep 27, 2024

View reviewed changes

jiaweizone reviewed Sep 27, 2024

View reviewed changes

LHT129 reviewed Sep 29, 2024

View reviewed changes

src/quantization/sq4_quantizer.h Show resolved Hide resolved

LHT129 requested changes Sep 29, 2024

View reviewed changes

jiaweizone reviewed Oct 2, 2024

View reviewed changes

LHT129 and others added 11 commits October 9, 2024 16:11

add sq4 quantizer(include uniform)

300dc45

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

update compiler issues

a634d1c

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

reconstruct code

c75223a

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

reconstruct operators

6bbc644

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

unify variable names, add log for error, simplify codes

45aa090

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

remove unordered map in offsets

3e5de57

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

rename variables

dab4caa

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

rename variables

70d383c

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

rename variables

6c09f1d

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

fix bug of lowest and add constexpr

ed7ddd5

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

add test cases

ddc5865

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

ShawnShawnYou force-pushed the add-sq4-quantizer-and-simd-operators branch from 9af72b7 to ddc5865 Compare October 9, 2024 08:31

fix asan issues

d9cb712

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

revert fixture modification

e891c7a

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

LHT129 requested changes Oct 11, 2024

View reviewed changes

src/quantization/sq4_uniform_quantizer.h Show resolved Hide resolved

add todo

aa6dad0

Signed-off-by: zhongxiaoyao.zxy <zhongxiaoyao.zxy@antgroup.com>

ShawnShawnYou requested review from LHT129, wxyucs and jiaweizone October 11, 2024 03:53

wxyucs approved these changes Oct 12, 2024

View reviewed changes

LHT129 approved these changes Oct 12, 2024

View reviewed changes

ShawnShawnYou changed the title ~~Add sq4 quantizer and simd operators~~ add sq4 quantizer and simd operators Oct 12, 2024

jiaweizone approved these changes Oct 14, 2024

View reviewed changes

ShawnShawnYou merged commit b391616 into main Oct 14, 2024
2 checks passed

ShawnShawnYou deleted the add-sq4-quantizer-and-simd-operators branch October 14, 2024 06:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add sq4 quantizer and simd operators #41

add sq4 quantizer and simd operators #41

ShawnShawnYou commented Sep 25, 2024

jiaweizone left a comment

jiaweizone Sep 25, 2024

ShawnShawnYou Sep 27, 2024

jiaweizone Sep 26, 2024

ShawnShawnYou Sep 27, 2024

jiaweizone Sep 26, 2024

ShawnShawnYou Sep 27, 2024

jiaweizone Sep 26, 2024

ShawnShawnYou Sep 27, 2024

jiaweizone Sep 26, 2024

ShawnShawnYou Sep 27, 2024

jiaweizone Sep 26, 2024

ShawnShawnYou Sep 27, 2024

jiaweizone Sep 26, 2024

jiaweizone Sep 26, 2024

jiaweizone Sep 26, 2024

jiaweizone Sep 26, 2024

ShawnShawnYou Sep 27, 2024

jiaweizone Sep 26, 2024

ShawnShawnYou Sep 27, 2024

jiaweizone Oct 2, 2024

jiaweizone left a comment

jiaweizone Oct 2, 2024

jiaweizone Oct 2, 2024

ShawnShawnYou Oct 7, 2024

jiaweizone Oct 14, 2024

wxyucs left a comment

jiaweizone left a comment

jiaweizone Oct 14, 2024

		x_hi = ((codes1[d / 2] & 0xf0) >> 4) * 15.0 / diff[d + 1] + lowerBound[d + 1];
		y_hi = ((codes2[d / 2] & 0xf0) >> 4) * 15.0 / diff[d + 1] + lowerBound[d + 1];

		@@ -511,6 +514,76 @@ SQ8ComputeCodesL2Sqr(const uint8_t* codes1,
		return Generic::SQ8ComputeCodesIP(codes1, codes2, lowerBound, diff, dim);

add sq4 quantizer and simd operators #41

add sq4 quantizer and simd operators #41

Conversation

ShawnShawnYou commented Sep 25, 2024

jiaweizone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiaweizone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wxyucs left a comment

Choose a reason for hiding this comment

jiaweizone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment