Skip to content

Commit

Permalink
Updating Ip fields to use doc_values to search (opensearch-project#11508
Browse files Browse the repository at this point in the history
)

* Updating Ip fields to use doc_values to search

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Fix IP tests

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Fix skip to allow yaml test to pass on main

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Update tests to use existing test file

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Changing skip version to match bwc

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Using exact match instead of range

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Spotless

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Fix IP field tests

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Fix spotless + precommit failure

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Get point out of query and into value

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Fix term tests

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

* Add skip test logic to only doc_values test

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>

---------

Signed-off-by: Harsha Vamsi Kalluri <harshavamsi096@gmail.com>
  • Loading branch information
harshavamsi authored and rayshrey committed Mar 18, 2024
1 parent 7f0c542 commit f32b965
Show file tree
Hide file tree
Showing 4 changed files with 323 additions and 53 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Ensure Jackson default maximums introduced in 2.16.0 do not conflict with OpenSearch settings ([#11890](https://github.com/opensearch-project/OpenSearch/pull/11890))
- Extract cluster management for integration tests into JUnit test rule out of OpenSearchIntegTestCase ([#11877](https://github.com/opensearch-project/OpenSearch/pull/11877)), ([#12000](https://github.com/opensearch-project/OpenSearch/pull/12000))
- Workaround for https://bugs.openjdk.org/browse/JDK-8323659 regression, introduced in JDK-21.0.2 ([#11968](https://github.com/opensearch-project/OpenSearch/pull/11968))
- Updates IpField to be searchable when only `doc_values` are enabled ([#11508](https://github.com/opensearch-project/OpenSearch/pull/11508))

### Deprecated

Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,3 @@
setup:
- skip:
features: [ "headers" ]
version: " - 2.11.99"
reason: "searching with only doc_values was added in 2.12.0"
---
"search on fields with both index and doc_values enabled":
- do:
Expand Down Expand Up @@ -47,18 +42,22 @@ setup:
type: unsigned_long
index: true
doc_values: true
ip_field:
type: ip
index: true
doc_values: true

- do:
bulk:
index: test-iodvq
refresh: true
body:
- '{"index": {"_index": "test-iodvq", "_id": "1" }}'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800 }'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800, "ip_field": "192.168.0.1" }'
- '{ "index": { "_index": "test-iodvq", "_id": "2" }}'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801 }'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801, "ip_field": "192.168.0.2" }'
- '{ "index": { "_index": "test-iodvq", "_id": "3" } }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802 }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802, "ip_field": "192.168.0.3" }'

- do:
search:
Expand Down Expand Up @@ -162,7 +161,6 @@ setup:

- match: { hits.total: 1 }


- do:
search:
rest_total_hits_as_int: true
Expand All @@ -174,6 +172,16 @@ setup:

- match: { hits.total: 1 }

- do:
search:
rest_total_hits_as_int: true
index: test-iodvq
body:
query:
term:
ip_field: "192.168.0.1"

- match: {hits.total: 1}

- do:
search:
Expand All @@ -186,7 +194,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -264,6 +271,17 @@ setup:

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-iodvq
body:
query:
terms:
ip_field: ["192.168.0.1", "192.168.0.2"]

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -384,6 +402,19 @@ setup:

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-iodvq
body:
query:
range:
ip_field:
gte: "192.168.0.1"
lte: "192.168.0.2"

- match: { hits.total: 2 }

---
"search on fields with only index enabled":
- do:
Expand Down Expand Up @@ -428,18 +459,22 @@ setup:
type: unsigned_long
index: true
doc_values: false
ip_field:
type: ip
index: true
doc_values: false

- do:
bulk:
index: test-index
refresh: true
body:
- '{"index": {"_index": "test-index", "_id": "1" }}'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800 }'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800, "ip_field": "192.168.0.1" }'
- '{ "index": { "_index": "test-index", "_id": "2" }}'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801 }'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801, "ip_field": "192.168.0.2" }'
- '{ "index": { "_index": "test-index", "_id": "3" } }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802 }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802, "ip_field": "192.168.0.3" }'

- do:
search:
Expand All @@ -465,7 +500,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -555,6 +589,16 @@ setup:

- match: { hits.total: 1 }

- do:
search:
rest_total_hits_as_int: true
index: test-index
body:
query:
term:
ip_field: "192.168.0.1"

- match: {hits.total: 1}

- do:
search:
Expand All @@ -567,7 +611,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -645,6 +688,17 @@ setup:

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-index
body:
query:
terms:
ip_field: ["192.168.0.1", "192.168.0.2"]

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -765,8 +819,24 @@ setup:

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-index
body:
query:
range:
ip_field:
gte: "192.168.0.1"
lte: "192.168.0.2"

- match: { hits.total: 2 }
---
"search on fields with only doc_values enabled":
- skip:
features: [ "headers" ]
version: " - 2.99.99"
reason: "searching with only doc_values was added in 3.0.0"
- do:
indices.create:
index: test-doc-values
Expand Down Expand Up @@ -809,18 +879,22 @@ setup:
type: unsigned_long
index: false
doc_values: true
ip_field:
type: ip
index: false
doc_values: true

- do:
bulk:
index: test-doc-values
refresh: true
body:
- '{"index": {"_index": "test-doc-values", "_id": "1" }}'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800 }'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800, "ip_field": "192.168.0.1" }'
- '{ "index": { "_index": "test-doc-values", "_id": "2" }}'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801 }'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801, "ip_field": "192.168.0.2" }'
- '{ "index": { "_index": "test-doc-values", "_id": "3" } }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802 }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802, "ip_field": "192.168.0.3" }'

- do:
search:
Expand All @@ -846,7 +920,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -924,7 +997,6 @@ setup:

- match: { hits.total: 1 }


- do:
search:
rest_total_hits_as_int: true
Expand All @@ -936,6 +1008,16 @@ setup:

- match: { hits.total: 1 }

- do:
search:
rest_total_hits_as_int: true
index: test-doc-values
body:
query:
term:
ip_field: "192.168.0.3"

- match: { hits.total: 1 }

- do:
search:
Expand All @@ -948,7 +1030,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -1145,3 +1226,16 @@ setup:
}

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-doc-values
body:
query:
range:
ip_field:
gte: "192.168.0.1"
lte: "192.168.0.2"

- match: { hits.total: 2 }
Loading

0 comments on commit f32b965

Please sign in to comment.