Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] Release version 2.13.0 #1473

Closed
23 tasks
opensearch-trigger-bot bot opened this issue Feb 9, 2024 · 2 comments
Closed
23 tasks

[RELEASE] Release version 2.13.0 #1473

opensearch-trigger-bot bot opened this issue Feb 9, 2024 · 2 comments
Assignees
@vibrantvarun
Labels
v2.13.0

Comments

@opensearch-trigger-bot
Copy link
Contributor

This is a component issue for 2.13.0.
Coming from opensearch-project/opensearch-build#4433. Please follow the following checklist.
Please refer to the DATES in that post.

How to use this issue

This Component Release Issue

This issue captures the state of the OpenSearch release, on component/plugin level; its assignee is responsible for driving the release. Please contact them or @mention them on this issue for help.
Any release related work can be linked to this issue or added as comments to create visiblity into the release status.

Release Steps

There are several steps to the release process; these steps are completed as the whole component release and components that are behind present risk to the release. The component owner resolves the tasks in this issue and communicate with the overall release owner to make sure each component are moving along as expected.

Steps have completion dates for coordinating efforts between the components of a release; components can start as soon as they are ready far in advance of a future release. The most current set of dates is on the overall release issue linked at the top of this issue.

The Overall Release Issue

Linked at the top of this issue, the overall release issue captures the state of the entire OpenSearch release including references to this issue, the release owner which is the assignee is responsible for communicating the release status broadly. Please contact them or @mention them on that issue for help.

What should I do if my plugin isn't making any changes?

If including changes in this release, increment the version on 2.x branch to 2.13.0 for Min/Core, and 2.13.0.0 for components. Otherwise, keep the version number unchanged for both.

Preparation

  • Assign this issue to a release owner.
  • Finalize scope and feature set and update the Public Roadmap.
  • All the tasks in this issue have been reviewed by the release owner.
  • Create, update, triage and label all features and issues targeted for this release with v2.13.0.
  • Finalize the code and create the the release branch 2.13 from the 2.x branch.

CI/CD

  • All code changes for 2.13.0 are complete.
  • Ensure working and passing CI.
  • Check that this repo is included in the distribution manifest.

Pre-Release

  • Increment the version on the parent branch to the next development iteration.
  • Gather, review and publish release notes following the rules and back port it to the release branch.git-release-notes may be used to generate release notes from your commit history.
  • Confirm that all changes for 2.13.0 have been merged.
  • Add this repo to the manifest for the next developer iteration.

Release Testing

  • Find/fix bugs using latest tarball and docker image provided in parent release issue and update the release notes if necessary.
  • Code Complete: Test within the distribution, ensuring integration, backwards compatibility, and performance tests pass.
  • Sanity Testing: Sanity testing and fixing of critical issues found.
  • File issues for all intermittent test failures.

Release

  • Complete documentation.
  • Verify all issued labeled for this release are closed or labelled for the next release.
  • Verify the release date mentioned in release notes is correct and matches actual release date.

Post Release

  • Prepare for an eventual security fix development iteration by incrementing the version on the release branch to the next eventual patch version.
  • Add this repo to the manifest of the next eventual security patch version.
  • Suggest improvements to this template.
  • Conduct a retrospective, and publish its results.
@gaiksaya gaiksaya mentioned this issue Mar 13, 2024
Closed
72 tasks
@vibrantvarun vibrantvarun self-assigned this Mar 19, 2024
@vibrantvarun
Copy link
Member

vibrantvarun commented Mar 23, 2024 •
edited by ryanbogan
Loading

Configurations

Data Sets

Dataset Train size Dimensions Distance Format K
sift 1,000,000 128 L2 hdf5 100

Total Queries: 10,000

Cluster Configuration

Config item Value
Leader Nodes 3
Leader Node Type c5.xlarge
Leader Node Disk Space 20
Data Nodes 3
Data Node Type r5.4xlarge
Data Node Disk Space 500
Primary Shard Count 24
Replica Count 1
AZ us-east-1

Client Configuration

Config item Value
OS Amazon Linux
Instance Type c5.9xlarge
Count 1
EBS Size 800G
AZ us-east-1

Ingest
ingest_took_total2 12 02 13 0

Query
2 12 0and2 13 0

Full Data

{
    "metadata": {
      "test_name": "Faiss HNSW Nested Field Test",
      "test_id": "Faiss HNSW Nested Field Test",
      "date": "03/22/2024 19:40:46",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "528420864 (used) / 72527462400 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 278974.2554366667,
      "delete_index_took_total": 327.3632179999974,
      "create_index_took_total": 346.21404566663233,
      "ingest_nested_field_took_total": 108566.26303633326,
      "refresh_index_store_kb_total": 2866762.9322916665,
      "refresh_index_took_total": 27448.181433999856,
      "force_merge_took_total": 93201.88210200012,
      "warmup_operation_took_total": 429.0182673335039,
      "query_nested_field_took_total": 48655.333333333336,
      "query_nested_field_took_p50": 5.0,
      "query_nested_field_took_p90": 5.0,
      "query_nested_field_took_p99": 6.0,
      "query_nested_field_took_p99.9": 9.666666666666666,
      "query_nested_field_took_p100": 603.0,
      "query_nested_field_client_time_total": 80019.66666666667,
      "query_nested_field_client_time_p50": 7.666666666666667,
      "query_nested_field_client_time_p90": 8.666666666666666,
      "query_nested_field_client_time_p99": 9.0,
      "query_nested_field_client_time_p99.9": 51.0,
      "query_nested_field_client_time_p100": 608.3333333333334,
      "query_nested_field_memory_kb_total": 1297460.0,
      "query_nested_field_recall@K_total": 0.999208,
      "query_nested_field_recall@1_total": 1.0
    }
}


{
    "metadata": {
      "test_name": "Faiss HNSW Relaxed Filter Test",
      "test_id": "Faiss HNSW Relaxed Filter Test",
      "date": "03/22/2024 19:09:45",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "553975808 (used) / 72502603776 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 267354.1889273332,
      "delete_index_took_total": 238.96889299995414,
      "create_index_took_total": 393.0780260000499,
      "ingest_multi_field_took_total": 101615.68468333331,
      "refresh_index_store_kb_total": 3029222.724283854,
      "refresh_index_took_total": 25158.732593666587,
      "force_merge_took_total": 92977.77527533336,
      "warmup_operation_took_total": 540.9494559999833,
      "query_with_filter_took_total": 46429.0,
      "query_with_filter_took_p50": 4.333333333333333,
      "query_with_filter_took_p90": 5.333333333333333,
      "query_with_filter_took_p99": 6.333333333333333,
      "query_with_filter_took_p99.9": 28.0,
      "query_with_filter_took_p100": 632.6666666666666,
      "query_with_filter_client_time_total": 80415.0,
      "query_with_filter_client_time_p50": 7.666666666666667,
      "query_with_filter_client_time_p90": 8.666666666666666,
      "query_with_filter_client_time_p99": 10.333333333333334,
      "query_with_filter_client_time_p99.9": 69.33333333333333,
      "query_with_filter_client_time_p100": 641.3333333333334,
      "query_with_filter_memory_kb_total": 1297436.0,
      "query_with_filter_recall@K_total": 0.9990853333333334,
      "query_with_filter_recall@1_total": 1.0
    }
}

{
    "metadata": {
      "test_name": "Faiss HNSW Restrictive Filter Test",
      "test_id": "Faiss HNSW Restrictive Filter Test",
      "date": "03/22/2024 19:23:30",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "554053632 (used) / 72502550528 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 220338.53631366685,
      "delete_index_took_total": 305.5797240000781,
      "create_index_took_total": 361.53933133338495,
      "ingest_multi_field_took_total": 63978.18258300004,
      "refresh_index_store_kb_total": 3040378.332356771,
      "refresh_index_took_total": 20059.380936666606,
      "force_merge_took_total": 93823.10412966674,
      "warmup_operation_took_total": 530.7496090000162,
      "query_with_filter_took_total": 41280.0,
      "query_with_filter_took_p50": 4.0,
      "query_with_filter_took_p90": 4.0,
      "query_with_filter_took_p99": 5.0,
      "query_with_filter_took_p99.9": 12.0,
      "query_with_filter_took_p100": 776.6666666666666,
      "query_with_filter_client_time_total": 71297.66666666667,
      "query_with_filter_client_time_p50": 7.0,
      "query_with_filter_client_time_p90": 7.0,
      "query_with_filter_client_time_p99": 8.0,
      "query_with_filter_client_time_p99.9": 58.0,
      "query_with_filter_client_time_p100": 781.3333333333334,
      "query_with_filter_memory_kb_total": 0.0,
      "query_with_filter_recall@K_total": 1.0,
      "query_with_filter_recall@1_total": 1.0
    }
}

{
    "metadata": {
      "test_name": "Faiss HNSW Test",
      "test_id": "Faiss HNSW Test",
      "date": "03/22/2024 19:56:05",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "486506496 (used) / 72568807424 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 248053.9378346668,
      "delete_index_took_total": 325.39014400026645,
      "create_index_took_total": 354.86718366670783,
      "ingest_took_total": 98083.23917666667,
      "refresh_index_store_kb_total": 2691479.8154296875,
      "refresh_index_took_total": 24435.437595666677,
      "force_merge_took_total": 91777.94906866651,
      "warmup_operation_took_total": 526.0546660000122,
      "query_took_total": 32551.0,
      "query_took_p50": 3.0,
      "query_took_p90": 4.0,
      "query_took_p99": 4.0,
      "query_took_p99.9": 6.0,
      "query_took_p100": 451.6666666666667,
      "query_client_time_total": 65535.333333333336,
      "query_client_time_p50": 6.666666666666667,
      "query_client_time_p90": 7.0,
      "query_client_time_p99": 7.666666666666667,
      "query_client_time_p99.9": 45.0,
      "query_client_time_p100": 461.0,
      "query_memory_kb_total": 1297436.0,
      "query_recall@K_total": 0.9992179999999999,
      "query_recall@1_total": 1.0
    }
}

{
    "metadata": {
      "test_name": "Faiss HNSW PQ Test",
      "test_id": "Faiss HNSW PQ Test",
      "date": "03/22/2024 20:08:02",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "489451520 (used) / 72565866496 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 186011.18727066592,
      "delete_model_took_total": 63.07463833339474,
      "delete_index_took_total": 297.0598450001489,
      "train_model_took_total": 14966.93951266631,
      "create_index_took_total": 348.64231533326046,
      "ingest_took_total": 61706.95492699996,
      "refresh_index_store_kb_total": 2003455.4677734375,
      "refresh_index_took_total": 11998.065110666781,
      "force_merge_took_total": 65666.07642666652,
      "warmup_operation_took_total": 157.37449499950648,
      "query_took_total": 30807.0,
      "query_took_p50": 3.0,
      "query_took_p90": 3.0,
      "query_took_p99": 3.3333333333333335,
      "query_took_p99.9": 4.666666666666667,
      "query_took_p100": 731.0,
      "query_client_time_total": 58801.333333333336,
      "query_client_time_p50": 6.0,
      "query_client_time_p90": 6.0,
      "query_client_time_p99": 7.0,
      "query_client_time_p99.9": 40.333333333333336,
      "query_client_time_p100": 735.3333333333334,
      "query_memory_kb_total": 334828.0,
      "query_recall@K_total": 0.629135,
      "query_recall@1_total": 0.9947333333333334
    }
}  

{
    "metadata": {
      "test_name": "Faiss IVF Relaxed Filter Test",
      "test_id": "Faiss IVF Relaxed Filter Test",
      "date": "03/22/2024 20:16:40",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "547930112 (used) / 72507330560 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 115523.00870833265,
      "delete_model_took_total": 82.10371066616062,
      "delete_index_took_total": 320.8874476664884,
      "train_model_took_total": 1395.1433833332583,
      "create_index_took_total": 351.684021999669,
      "ingest_multi_field_took_total": 62713.67995866664,
      "refresh_index_store_kb_total": 2804340.246419271,
      "refresh_index_took_total": 624.4898216667328,
      "force_merge_took_total": 7544.4038103335815,
      "warmup_operation_took_total": 422.94988733344024,
      "query_with_filter_took_total": 42067.666666666664,
      "query_with_filter_took_p50": 4.0,
      "query_with_filter_took_p90": 5.0,
      "query_with_filter_took_p99": 15.0,
      "query_with_filter_took_p99.9": 21.0,
      "query_with_filter_took_p100": 675.6666666666666,
      "query_with_filter_client_time_total": 72610.33333333333,
      "query_with_filter_client_time_p50": 7.0,
      "query_with_filter_client_time_p90": 8.0,
      "query_with_filter_client_time_p99": 17.666666666666668,
      "query_with_filter_client_time_p99.9": 54.0,
      "query_with_filter_client_time_p100": 681.0,
      "query_with_filter_memory_kb_total": 1050030.0,
      "query_with_filter_recall@K_total": 0.47706133333333334,
      "query_with_filter_recall@1_total": 0.9998999999999999
    }
}

{
    "metadata": {
      "test_name": "Faiss IVF restrictive Filter Test",
      "test_id": "Faiss IVF restrictive Filter Test",
      "date": "03/22/2024 20:25:27",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "551743488 (used) / 72503545856 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 114583.26888566643,
      "delete_model_took_total": 83.2594853336559,
      "delete_index_took_total": 291.20860300008644,
      "train_model_took_total": 1391.3259386666443,
      "create_index_took_total": 385.2109436666069,
      "ingest_multi_field_took_total": 63053.62369200005,
      "refresh_index_store_kb_total": 2007931.6881510417,
      "refresh_index_took_total": 602.5838849994519,
      "force_merge_took_total": 7647.179390333501,
      "warmup_operation_took_total": 378.210280999762,
      "query_with_filter_took_total": 40750.666666666664,
      "query_with_filter_took_p50": 4.0,
      "query_with_filter_took_p90": 4.0,
      "query_with_filter_took_p99": 5.0,
      "query_with_filter_took_p99.9": 11.333333333333334,
      "query_with_filter_took_p100": 504.3333333333333,
      "query_with_filter_client_time_total": 76583.33333333333,
      "query_with_filter_client_time_p50": 7.0,
      "query_with_filter_client_time_p90": 8.0,
      "query_with_filter_client_time_p99": 8.666666666666666,
      "query_with_filter_client_time_p99.9": 53.666666666666664,
      "query_with_filter_client_time_p100": 570.0,
      "query_with_filter_memory_kb_total": 0.0,
      "query_with_filter_recall@K_total": 1.0,
      "query_with_filter_recall@1_total": 1.0
    }
}

{
    "metadata": {
      "test_name": "Faiss IVF",
      "test_id": "Faiss IVF",
      "date": "03/22/2024 20:33:15",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "464769024 (used) / 72590467072 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 100801.46728533326,
      "delete_model_took_total": 79.57467233336502,
      "delete_index_took_total": 342.79723299975257,
      "train_model_took_total": 1403.1763176668999,
      "create_index_took_total": 367.9308480001661,
      "ingest_took_total": 60465.3513593333,
      "refresh_index_store_kb_total": 2771269.298828125,
      "refresh_index_took_total": 535.5319493331384,
      "force_merge_took_total": 6739.5040946666995,
      "warmup_operation_took_total": 374.93414433326205,
      "query_took_total": 30492.666666666668,
      "query_took_p50": 3.0,
      "query_took_p90": 3.0,
      "query_took_p99": 3.0,
      "query_took_p99.9": 4.333333333333333,
      "query_took_p100": 392.0,
      "query_client_time_total": 60161.666666666664,
      "query_client_time_p50": 6.0,
      "query_client_time_p90": 6.333333333333333,
      "query_client_time_p99": 7.0,
      "query_client_time_p99.9": 42.0,
      "query_client_time_p100": 442.3333333333333,
      "query_memory_kb_total": 1050030.0,
      "query_recall@K_total": 0.947137,
      "query_recall@1_total": 1.0
    }
}

{
    "metadata": {
      "test_name": "Faiss IVF PQ Test",
      "test_id": "Faiss IVF PQ Test",
      "date": "03/22/2024 20:41:37",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "464478208 (used) / 72590757888 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 113663.25179566594,
      "delete_model_took_total": 79.23492866636177,
      "delete_index_took_total": 343.33748433346045,
      "train_model_took_total": 14566.67772899982,
      "create_index_took_total": 378.37760133334086,
      "ingest_took_total": 60447.43759466655,
      "refresh_index_store_kb_total": 1590987.9153645833,
      "refresh_index_took_total": 804.2725333334602,
      "force_merge_took_total": 7689.416466666444,
      "warmup_operation_took_total": 55.49745766650934,
      "query_took_total": 29299.0,
      "query_took_p50": 3.0,
      "query_took_p90": 3.0,
      "query_took_p99": 3.0,
      "query_took_p99.9": 4.0,
      "query_took_p100": 969.3333333333334,
      "query_client_time_total": 58303.333333333336,
      "query_client_time_p50": 5.333333333333333,
      "query_client_time_p90": 6.0,
      "query_client_time_p99": 6.333333333333333,
      "query_client_time_p99.9": 50.0,
      "query_client_time_p100": 974.0,
      "query_memory_kb_total": 87428.0,
      "query_recall@K_total": 0.613616,
      "query_recall@1_total": 0.9945
    }
}

{
    "metadata": {
      "test_name": "Lucene HNSW Nested Field Test",
      "test_id": "Lucene HNSW Nested Field Test",
      "date": "03/22/2024 21:53:51",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "529608704 (used) / 72525553664 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 439976.37768933526,
      "delete_index_took_total": 361.8264496678118,
      "create_index_took_total": 337.1748640001897,
      "ingest_nested_field_took_total": 213726.7893416665,
      "refresh_index_store_kb_total": 2820220.583658854,
      "refresh_index_took_total": 657.611715333284,
      "force_merge_took_total": 172968.04455000043,
      "warmup_operation_took_total": 7.597435333688433,
      "query_nested_field_took_total": 51917.333333333336,
      "query_nested_field_took_p50": 5.0,
      "query_nested_field_took_p90": 6.0,
      "query_nested_field_took_p99": 6.0,
      "query_nested_field_took_p99.9": 8.666666666666666,
      "query_nested_field_took_p100": 1004.3333333333334,
      "query_nested_field_client_time_total": 86614.66666666667,
      "query_nested_field_client_time_p50": 8.666666666666666,
      "query_nested_field_client_time_p90": 9.0,
      "query_nested_field_client_time_p99": 9.666666666666666,
      "query_nested_field_client_time_p99.9": 48.0,
      "query_nested_field_client_time_p100": 1010.6666666666666,
      "query_nested_field_memory_kb_total": 0.0,
      "query_nested_field_recall@K_total": 0.999173,
      "query_nested_field_recall@1_total": 1.0
    }
}

{
    "metadata": {
      "test_name": "Lucene HNSW Relaxed Filter Test",
      "test_id": "Lucene HNSW Relaxed Filter Test",
      "date": "03/22/2024 21:05:41",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "555601920 (used) / 72499601408 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 419157.3318789997,
      "delete_index_took_total": 324.86718699995737,
      "create_index_took_total": 326.03889233329636,
      "ingest_multi_field_took_total": 151421.82257333296,
      "refresh_index_store_kb_total": 2279283.857421875,
      "refresh_index_took_total": 666.895949000415,
      "force_merge_took_total": 191496.70727733304,
      "query_with_filter_took_total": 74921.0,
      "query_with_filter_took_p50": 7.0,
      "query_with_filter_took_p90": 8.0,
      "query_with_filter_took_p99": 9.0,
      "query_with_filter_took_p99.9": 13.333333333333334,
      "query_with_filter_took_p100": 1066.3333333333333,
      "query_with_filter_client_time_total": 111971.0,
      "query_with_filter_client_time_p50": 10.666666666666666,
      "query_with_filter_client_time_p90": 12.0,
      "query_with_filter_client_time_p99": 13.0,
      "query_with_filter_client_time_p99.9": 179.0,
      "query_with_filter_client_time_p100": 1070.6666666666667,
      "query_with_filter_memory_kb_total": 0.0,
      "query_with_filter_recall@K_total": 0.9993639999999999,
      "query_with_filter_recall@1_total": 1.0
    }
}

{
    "metadata": {
      "test_name": "Lucene HNSW Restrictive Filter Test",
      "test_id": "Lucene HNSW Restrictive Filter Test",
      "date": "03/22/2024 21:29:01",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "554020864 (used) / 72501149696 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 407285.07108066714,
      "delete_index_took_total": 370.9832243330311,
      "create_index_took_total": 294.4558279996272,
      "ingest_multi_field_took_total": 147183.66533966703,
      "refresh_index_store_kb_total": 2398017.0442708335,
      "refresh_index_took_total": 611.7198016672774,
      "force_merge_took_total": 201107.58022033345,
      "query_with_filter_took_total": 57716.666666666664,
      "query_with_filter_took_p50": 5.0,
      "query_with_filter_took_p90": 6.0,
      "query_with_filter_took_p99": 6.0,
      "query_with_filter_took_p99.9": 10.0,
      "query_with_filter_took_p100": 1078.6666666666667,
      "query_with_filter_client_time_total": 94527.0,
      "query_with_filter_client_time_p50": 8.666666666666666,
      "query_with_filter_client_time_p90": 9.333333333333334,
      "query_with_filter_client_time_p99": 9.666666666666666,
      "query_with_filter_client_time_p99.9": 167.0,
      "query_with_filter_client_time_p100": 1085.0,
      "query_with_filter_memory_kb_total": 0.0,
      "query_with_filter_recall@K_total": 1.0,
      "query_with_filter_recall@1_total": 1.0
    }
}

{
    "metadata": {
      "test_name": "Lucene HNSW",
      "test_id": "Lucene HNSW",
      "date": "03/22/2024 22:17:04",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "466878464 (used) / 72588222464 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 408757.2952156674,
      "delete_index_took_total": 374.09130500054744,
      "create_index_took_total": 301.59278133396583,
      "ingest_took_total": 138725.7337593328,
      "refresh_index_store_kb_total": 2355100.9420572915,
      "refresh_index_took_total": 541.4932136670055,
      "force_merge_took_total": 219579.7174896664,
      "query_took_total": 49234.666666666664,
      "query_took_p50": 4.0,
      "query_took_p90": 5.0,
      "query_took_p99": 5.0,
      "query_took_p99.9": 6.333333333333333,
      "query_took_p100": 1083.6666666666667,
      "query_client_time_total": 81284.0,
      "query_client_time_p50": 7.333333333333333,
      "query_client_time_p90": 8.0,
      "query_client_time_p99": 8.333333333333334,
      "query_client_time_p99.9": 131.0,
      "query_client_time_p100": 1089.6666666666667,
      "query_memory_kb_total": 0.0,
      "query_recall@K_total": 0.9990963333333333,
      "query_recall@1_total": 1.0
    }
}

{
    "metadata": {
      "test_name": "Nmslib HNSW Test",
      "test_id": "Nmslib HNSW Test",
      "date": "03/22/2024 22:30:28",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "488157184 (used) / 72566968320 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 210980.38798300023,
      "delete_index_took_total": 402.6293436660732,
      "create_index_took_total": 322.2179433335744,
      "ingest_took_total": 60366.392843666596,
      "refresh_index_store_kb_total": 2803392.19140625,
      "refresh_index_took_total": 21639.264739333765,
      "force_merge_took_total": 96619.33123933342,
      "warmup_operation_took_total": 439.88520700016426,
      "query_took_total": 31190.666666666668,
      "query_took_p50": 3.0,
      "query_took_p90": 3.0,
      "query_took_p99": 4.0,
      "query_took_p99.9": 4.333333333333333,
      "query_took_p100": 859.0,
      "query_client_time_total": 64023.666666666664,
      "query_client_time_p50": 6.0,
      "query_client_time_p90": 6.666666666666667,
      "query_client_time_p99": 7.0,
      "query_client_time_p99.9": 45.333333333333336,
      "query_client_time_p100": 864.0,
      "query_memory_kb_total": 1305864.0,
      "query_recall@K_total": 0.9988786666666667,
      "query_recall@1_total": 1.0
    }
}

@vibrantvarun
Copy link
Member

RC5

SQFP16
SQFP16_Faiss_HNSW-RC5
SQFP16_RC5_INDEX

Regular

Regular_Faiss_HNSW_Index-RC5
RegularFaiss-HNSW-RC5

{
  "metadata": {
    "test_name": "Faiss HNSW Nested Field Test",
    "test_id": "Faiss HNSW Nested Field Test",
    "date": "03/29/2024 04:37:26",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "555577344 (used) / 72496943104 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 183005.63083899987,
    "delete_index_took_total": 283.8712903333847,
    "create_index_took_total": 199.14458733425514,
    "ingest_nested_field_took_total": 68590.83262700005,
    "refresh_index_store_kb_total": 2669788.76953125,
    "refresh_index_took_total": 14642.767411000628,
    "force_merge_took_total": 62421.237708332534,
    "warmup_operation_took_total": 271.77721499901963,
    "query_nested_field_took_total": 36596.0,
    "query_nested_field_took_p50": 4.0,
    "query_nested_field_took_p90": 4.0,
    "query_nested_field_took_p99": 4.0,
    "query_nested_field_took_p99.9": 6.333333333333333,
    "query_nested_field_took_p100": 834.3333333333334,
    "query_nested_field_client_time_total": 71472.66666666667,
    "query_nested_field_client_time_p50": 6.666666666666667,
    "query_nested_field_client_time_p90": 7.666666666666667,
    "query_nested_field_client_time_p99": 8.0,
    "query_nested_field_client_time_p99.9": 46.666666666666664,
    "query_nested_field_client_time_p100": 840.0,
    "query_nested_field_memory_kb_total": 1297460.0,
    "query_nested_field_recall@K_total": 0.9991866666666667,
    "query_nested_field_recall@1_total": 1.0
  }
}

{
  "metadata": {
    "test_name": "Faiss HNSW Relaxed Filter Test",
    "test_id": "Faiss HNSW Relaxed Filter Test",
    "date": "03/29/2024 04:13:54",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "580988928 (used) / 72471597056 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 181439.7556373328,
    "delete_index_took_total": 237.41673633412574,
    "create_index_took_total": 187.49395333240196,
    "ingest_multi_field_took_total": 61053.773959999184,
    "refresh_index_store_kb_total": 2916217.783528646,
    "refresh_index_took_total": 12922.04480866591,
    "force_merge_took_total": 62905.33926433394,
    "warmup_operation_took_total": 321.6869146672252,
    "query_with_filter_took_total": 43812.0,
    "query_with_filter_took_p50": 4.0,
    "query_with_filter_took_p90": 5.666666666666667,
    "query_with_filter_took_p99": 6.0,
    "query_with_filter_took_p99.9": 10.333333333333334,
    "query_with_filter_took_p100": 623.0,
    "query_with_filter_client_time_total": 76832.66666666667,
    "query_with_filter_client_time_p50": 7.333333333333333,
    "query_with_filter_client_time_p90": 8.333333333333334,
    "query_with_filter_client_time_p99": 9.333333333333334,
    "query_with_filter_client_time_p99.9": 65.33333333333333,
    "query_with_filter_client_time_p100": 628.0,
    "query_with_filter_memory_kb_total": 1297436.0,
    "query_with_filter_recall@K_total": 0.9990886666666667,
    "query_with_filter_recall@1_total": 1.0
  }
}

{
  "metadata": {
    "test_name": "Faiss HNSW Relaxed Filter Test",
    "test_id": "Faiss HNSW Relaxed Filter Test",
    "date": "03/29/2024 04:13:54",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "580988928 (used) / 72471597056 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 181439.7556373328,
    "delete_index_took_total": 237.41673633412574,
    "create_index_took_total": 187.49395333240196,
    "ingest_multi_field_took_total": 61053.773959999184,
    "refresh_index_store_kb_total": 2916217.783528646,
    "refresh_index_took_total": 12922.04480866591,
    "force_merge_took_total": 62905.33926433394,
    "warmup_operation_took_total": 321.6869146672252,
    "query_with_filter_took_total": 43812.0,
    "query_with_filter_took_p50": 4.0,
    "query_with_filter_took_p90": 5.666666666666667,
    "query_with_filter_took_p99": 6.0,
    "query_with_filter_took_p99.9": 10.333333333333334,
    "query_with_filter_took_p100": 623.0,
    "query_with_filter_client_time_total": 76832.66666666667,
    "query_with_filter_client_time_p50": 7.333333333333333,
    "query_with_filter_client_time_p90": 8.333333333333334,
    "query_with_filter_client_time_p99": 9.333333333333334,
    "query_with_filter_client_time_p99.9": 65.33333333333333,
    "query_with_filter_client_time_p100": 628.0,
    "query_with_filter_memory_kb_total": 1297436.0,
    "query_with_filter_recall@K_total": 0.9990886666666667,
    "query_with_filter_recall@1_total": 1.0
  }
}

{
    "metadata": {
      "test_name": "Faiss HNSW SQFP16 Test",
      "test_id": "Faiss HNSW SQFP16 Test",
      "date": "03/29/2024 05:08:52",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "497115136 (used) / 72555409408 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 150036.56717433155,
      "delete_index_took_total": 295.09022900068277,
      "create_index_took_total": 189.57153366621546,
      "ingest_took_total": 57898.91524933167,
      "refresh_index_store_kb_total": 2500929.152018229,
      "refresh_index_took_total": 8961.463047665651,
      "force_merge_took_total": 54494.2714676666,
      "warmup_operation_took_total": 199.2556470007306,
      "query_took_total": 27998.0,
      "query_took_p50": 3.0,
      "query_took_p90": 3.0,
      "query_took_p99": 3.0,
      "query_took_p99.9": 7.0,
      "query_took_p100": 585.3333333333334,
      "query_client_time_total": 58664.666666666664,
      "query_client_time_p50": 5.333333333333333,
      "query_client_time_p90": 6.333333333333333,
      "query_client_time_p99": 6.333333333333333,
      "query_client_time_p99.9": 53.333333333333336,
      "query_client_time_p100": 611.3333333333334,
      "query_memory_kb_total": 797432.0,
      "query_recall@K_total": 0.9992083333333334,
      "query_recall@1_total": 1.0
    }
  }


{
  "metadata": {
    "test_name": "Faiss HNSW Test",
    "test_id": "Faiss HNSW Test",
    "date": "03/29/2024 04:48:23",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "514105344 (used) / 72538411008 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 161066.14950200068,
    "delete_index_took_total": 310.505525667395,
    "create_index_took_total": 195.48973666678648,
    "ingest_took_total": 58209.53189433203,
    "refresh_index_store_kb_total": 3001221.70703125,
    "refresh_index_took_total": 11868.007703001544,
    "force_merge_took_total": 62221.7749896663,
    "warmup_operation_took_total": 260.83965266661835,
    "query_took_total": 28000.0,
    "query_took_p50": 3.0,
    "query_took_p90": 3.0,
    "query_took_p99": 3.0,
    "query_took_p99.9": 6.666666666666667,
    "query_took_p100": 575.6666666666666,
    "query_client_time_total": 61069.333333333336,
    "query_client_time_p50": 5.666666666666667,
    "query_client_time_p90": 6.666666666666667,
    "query_client_time_p99": 6.666666666666667,
    "query_client_time_p99.9": 47.666666666666664,
    "query_client_time_p100": 593.6666666666666,
    "query_memory_kb_total": 1297436.0,
    "query_recall@K_total": 0.9991796666666666,
    "query_recall@1_total": 1.0
  }
}

{
  "metadata": {
    "test_name": "Faiss HNSW PQ Test",
    "test_id": "Faiss HNSW PQ Test",
    "date": "03/29/2024 04:58:36",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "487882752 (used) / 72564629504 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 142841.14558600186,
    "delete_model_took_total": 68.39159066657885,
    "delete_index_took_total": 249.16221333357194,
    "train_model_took_total": 14670.138996333359,
    "create_index_took_total": 188.1881853332743,
    "ingest_took_total": 57395.092572335365,
    "refresh_index_store_kb_total": 2067629.0940755208,
    "refresh_index_took_total": 7093.225442666153,
    "force_merge_took_total": 41504.44009700004,
    "warmup_operation_took_total": 95.83982166683806,
    "query_took_total": 21576.666666666668,
    "query_took_p50": 2.0,
    "query_took_p90": 2.0,
    "query_took_p99": 3.0,
    "query_took_p99.9": 4.333333333333333,
    "query_took_p100": 823.3333333333334,
    "query_client_time_total": 58978.333333333336,
    "query_client_time_p50": 5.666666666666667,
    "query_client_time_p90": 6.0,
    "query_client_time_p99": 6.666666666666667,
    "query_client_time_p99.9": 46.0,
    "query_client_time_p100": 829.0,
    "query_memory_kb_total": 334828.0,
    "query_recall@K_total": 0.6291573333333332,
    "query_recall@1_total": 0.9947333333333334
  }
}

{
  "metadata": {
    "test_name": "Faiss IVF Relaxed Filter Test",
    "test_id": "Faiss IVF Relaxed Filter Test",
    "date": "03/29/2024 05:17:20",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "575700992 (used) / 72476807168 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 106557.8035796673,
    "delete_model_took_total": 73.31820566711637,
    "delete_index_took_total": 270.99780199932866,
    "train_model_took_total": 1310.8843510005197,
    "create_index_took_total": 186.34712800000366,
    "ingest_multi_field_took_total": 58877.08494733306,
    "refresh_index_store_kb_total": 2814368.2099609375,
    "refresh_index_took_total": 360.7177390003926,
    "force_merge_took_total": 5657.1674739995315,
    "warmup_operation_took_total": 175.2859326673691,
    "query_with_filter_took_total": 39646.0,
    "query_with_filter_took_p50": 4.0,
    "query_with_filter_took_p90": 5.0,
    "query_with_filter_took_p99": 15.0,
    "query_with_filter_took_p99.9": 23.333333333333332,
    "query_with_filter_took_p100": 447.6666666666667,
    "query_with_filter_client_time_total": 78028.66666666667,
    "query_with_filter_client_time_p50": 7.666666666666667,
    "query_with_filter_client_time_p90": 8.666666666666666,
    "query_with_filter_client_time_p99": 18.666666666666668,
    "query_with_filter_client_time_p99.9": 66.66666666666667,
    "query_with_filter_client_time_p100": 558.6666666666666,
    "query_with_filter_memory_kb_total": 1050030.0,
    "query_with_filter_recall@K_total": 0.47706166666666666,
    "query_with_filter_recall@1_total": 0.9998999999999999
  }
}

{
  "metadata": {
    "test_name": "Faiss IVF restrictive Filter Test",
    "test_id": "Faiss IVF restrictive Filter Test",
    "date": "03/29/2024 05:25:16",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "585535488 (used) / 72466984960 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 98709.81626166306,
    "delete_model_took_total": 76.88634966568013,
    "delete_index_took_total": 223.4011179995529,
    "train_model_took_total": 1321.8403006649169,
    "create_index_took_total": 197.4836386665023,
    "ingest_multi_field_took_total": 58964.86060833316,
    "refresh_index_store_kb_total": 2812205.152669271,
    "refresh_index_took_total": 357.7993353331597,
    "force_merge_took_total": 5386.6543693341855,
    "warmup_operation_took_total": 259.890541665906,
    "query_with_filter_took_total": 31921.0,
    "query_with_filter_took_p50": 3.0,
    "query_with_filter_took_p90": 4.0,
    "query_with_filter_took_p99": 4.0,
    "query_with_filter_took_p99.9": 10.0,
    "query_with_filter_took_p100": 498.0,
    "query_with_filter_client_time_total": 66575.0,
    "query_with_filter_client_time_p50": 6.333333333333333,
    "query_with_filter_client_time_p90": 7.0,
    "query_with_filter_client_time_p99": 7.333333333333333,
    "query_with_filter_client_time_p99.9": 57.666666666666664,
    "query_with_filter_client_time_p100": 527.0,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 1.0,
    "query_with_filter_recall@1_total": 1.0
  }
}

{
    "metadata": {
      "test_name": "Faiss IVF SQFP16",
      "test_id": "Faiss IVF SQFP16",
      "date": "03/29/2024 05:47:40",
      "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
      "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
      "processor": "x86_64, 36 cores",
      "memory": "483520512 (used) / 72569024512 (available) / 73743400960 (total)"
    },
    "results": {
      "test_took": 85296.29697899711,
      "delete_model_took_total": 73.8530586665244,
      "delete_index_took_total": 292.9958699993828,
      "train_model_took_total": 1398.5867749991787,
      "create_index_took_total": 179.2626433331558,
      "ingest_took_total": 56388.50180499988,
      "refresh_index_store_kb_total": 1921099.2154947917,
      "refresh_index_took_total": 291.31272866531316,
      "force_merge_took_total": 4558.796591666275,
      "warmup_operation_took_total": 158.65417333407095,
      "query_took_total": 21954.333333333332,
      "query_took_p50": 2.0,
      "query_took_p90": 2.6666666666666665,
      "query_took_p99": 3.0,
      "query_took_p99.9": 5.0,
      "query_took_p100": 788.3333333333334,
      "query_client_time_total": 58348.333333333336,
      "query_client_time_p50": 5.666666666666667,
      "query_client_time_p90": 6.0,
      "query_client_time_p99": 6.666666666666667,
      "query_client_time_p99.9": 46.666666666666664,
      "query_client_time_p100": 793.6666666666666,
      "query_memory_kb_total": 550028.0,
      "query_recall@K_total": 0.9470713333333333,
      "query_recall@1_total": 1.0
    }
  }

{
  "metadata": {
    "test_name": "Faiss IVF",
    "test_id": "Faiss IVF",
    "date": "03/29/2024 05:32:29",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "489844736 (used) / 72562679808 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 85373.69480733504,
    "delete_model_took_total": 62.06680499963113,
    "delete_index_took_total": 267.83839400013676,
    "train_model_took_total": 1291.170227334078,
    "create_index_took_total": 196.66118066743365,
    "ingest_took_total": 54729.30807099814,
    "refresh_index_store_kb_total": 2779912.1640625,
    "refresh_index_took_total": 319.33463533399237,
    "force_merge_took_total": 4712.753515001775,
    "warmup_operation_took_total": 302.2286456665218,
    "query_took_total": 23492.333333333332,
    "query_took_p50": 2.0,
    "query_took_p90": 3.0,
    "query_took_p99": 3.0,
    "query_took_p99.9": 6.333333333333333,
    "query_took_p100": 579.6666666666666,
    "query_client_time_total": 57121.333333333336,
    "query_client_time_p50": 5.333333333333333,
    "query_client_time_p90": 6.333333333333333,
    "query_client_time_p99": 6.333333333333333,
    "query_client_time_p99.9": 46.0,
    "query_client_time_p100": 584.3333333333334,
    "query_memory_kb_total": 1050030.0,
    "query_recall@K_total": 0.9471383333333333,
    "query_recall@1_total": 1.0
  }
}

{
  "metadata": {
    "test_name": "Faiss IVF PQ Test",
    "test_id": "Faiss IVF PQ Test",
    "date": "03/29/2024 05:40:18",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "486281216 (used) / 72566267904 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 97949.69035800213,
    "delete_model_took_total": 72.73381200138829,
    "delete_index_took_total": 275.0276703336567,
    "train_model_took_total": 15112.031108333516,
    "create_index_took_total": 189.19837299957484,
    "ingest_took_total": 55120.644610666204,
    "refresh_index_store_kb_total": 1852061.3658854167,
    "refresh_index_took_total": 510.32411533378763,
    "force_merge_took_total": 5594.21530800076,
    "warmup_operation_took_total": 132.5153603332486,
    "query_took_total": 20943.0,
    "query_took_p50": 2.0,
    "query_took_p90": 2.0,
    "query_took_p99": 3.0,
    "query_took_p99.9": 5.0,
    "query_took_p100": 570.3333333333334,
    "query_client_time_total": 54886.666666666664,
    "query_client_time_p50": 5.333333333333333,
    "query_client_time_p90": 5.666666666666667,
    "query_client_time_p99": 6.333333333333333,
    "query_client_time_p99.9": 45.0,
    "query_client_time_p100": 575.6666666666666,
    "query_memory_kb_total": 87428.0,
    "query_recall@K_total": 0.613616,
    "query_recall@1_total": 0.9945
  }
}

{
  "metadata": {
    "test_name": "Lucene HNSW Nested Field Test",
    "test_id": "Lucene HNSW Nested Field Test",
    "date": "03/29/2024 06:47:12",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "564654080 (used) / 72487841792 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 358769.5830583276,
    "delete_index_took_total": 271.28407833030604,
    "create_index_took_total": 190.85240900070252,
    "ingest_nested_field_took_total": 197509.43715133084,
    "refresh_index_store_kb_total": 2819974.2041015625,
    "refresh_index_took_total": 447.341603001405,
    "force_merge_took_total": 117918.97705366621,
    "warmup_operation_took_total": 8.690762998109372,
    "query_nested_field_took_total": 42423.0,
    "query_nested_field_took_p50": 4.0,
    "query_nested_field_took_p90": 5.0,
    "query_nested_field_took_p99": 5.0,
    "query_nested_field_took_p99.9": 6.0,
    "query_nested_field_took_p100": 1089.6666666666667,
    "query_nested_field_client_time_total": 75061.0,
    "query_nested_field_client_time_p50": 7.333333333333333,
    "query_nested_field_client_time_p90": 8.0,
    "query_nested_field_client_time_p99": 8.333333333333334,
    "query_nested_field_client_time_p99.9": 49.666666666666664,
    "query_nested_field_client_time_p100": 1095.0,
    "query_nested_field_memory_kb_total": 0.0,
    "query_nested_field_recall@K_total": 0.999138,
    "query_nested_field_recall@1_total": 1.0
  }
}

{
  "metadata": {
    "test_name": "Lucene HNSW Relaxed Filter Test",
    "test_id": "Lucene HNSW Relaxed Filter Test",
    "date": "03/29/2024 06:07:54",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "577318912 (used) / 72475226112 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 345616.1840136675,
    "delete_index_took_total": 301.6134626668645,
    "create_index_took_total": 187.80955466718297,
    "ingest_multi_field_took_total": 127657.70617433374,
    "refresh_index_store_kb_total": 2395127.566080729,
    "refresh_index_took_total": 442.93943533314933,
    "force_merge_took_total": 150900.11538666658,
    "query_with_filter_took_total": 66126.0,
    "query_with_filter_took_p50": 6.0,
    "query_with_filter_took_p90": 7.0,
    "query_with_filter_took_p99": 8.0,
    "query_with_filter_took_p99.9": 13.333333333333334,
    "query_with_filter_took_p100": 1071.0,
    "query_with_filter_client_time_total": 101229.66666666667,
    "query_with_filter_client_time_p50": 9.333333333333334,
    "query_with_filter_client_time_p90": 10.333333333333334,
    "query_with_filter_client_time_p99": 11.333333333333334,
    "query_with_filter_client_time_p99.9": 215.66666666666666,
    "query_with_filter_client_time_p100": 1075.6666666666667,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 0.9993383333333333,
    "query_with_filter_recall@1_total": 1.0
  }
}

{
  "metadata": {
    "test_name": "Lucene HNSW Restrictive Filter Test",
    "test_id": "Lucene HNSW Restrictive Filter Test",
    "date": "03/29/2024 06:26:28",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "579338240 (used) / 72473206784 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 314497.0733886578,
    "delete_index_took_total": 249.81054866293562,
    "create_index_took_total": 207.95316666529592,
    "ingest_multi_field_took_total": 134805.3741499995,
    "refresh_index_store_kb_total": 2398412.3723958335,
    "refresh_index_took_total": 454.2230179977196,
    "force_merge_took_total": 130931.37917199905,
    "query_with_filter_took_total": 47848.333333333336,
    "query_with_filter_took_p50": 4.0,
    "query_with_filter_took_p90": 5.0,
    "query_with_filter_took_p99": 5.0,
    "query_with_filter_took_p99.9": 10.0,
    "query_with_filter_took_p100": 1069.3333333333333,
    "query_with_filter_client_time_total": 79326.0,
    "query_with_filter_client_time_p50": 7.0,
    "query_with_filter_client_time_p90": 8.0,
    "query_with_filter_client_time_p99": 8.0,
    "query_with_filter_client_time_p99.9": 216.0,
    "query_with_filter_client_time_p100": 1074.0,
    "query_with_filter_memory_kb_total": 0.0,
    "query_with_filter_recall@K_total": 1.0,
    "query_with_filter_recall@1_total": 1.0
  }
}

{
  "metadata": {
    "test_name": "Lucene HNSW",
    "test_id": "Lucene HNSW",
    "date": "03/29/2024 07:05:55",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "492167168 (used) / 72560340992 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 320623.4305363389,
    "delete_index_took_total": 277.90426800250617,
    "create_index_took_total": 202.59882299918294,
    "ingest_took_total": 122428.10709233406,
    "refresh_index_store_kb_total": 2343940.275716146,
    "refresh_index_took_total": 383.97838266731316,
    "force_merge_took_total": 154095.17530366915,
    "query_took_total": 43235.666666666664,
    "query_took_p50": 4.0,
    "query_took_p90": 4.0,
    "query_took_p99": 4.0,
    "query_took_p99.9": 11.666666666666666,
    "query_took_p100": 1046.3333333333333,
    "query_client_time_total": 73334.33333333333,
    "query_client_time_p50": 7.0,
    "query_client_time_p90": 7.0,
    "query_client_time_p99": 8.0,
    "query_client_time_p99.9": 193.0,
    "query_client_time_p100": 1050.3333333333333,
    "query_memory_kb_total": 0.0,
    "query_recall@K_total": 0.9991033333333332,
    "query_recall@1_total": 1.0
  }
}


{
  "metadata": {
    "test_name": "Nmslib HNSW Test",
    "test_id": "Nmslib HNSW Test",
    "date": "03/29/2024 07:16:43",
    "python_version": "3.9.18 (main, Sep 11 2023, 13:41:44) \n[GCC 11.2.0]",
    "os_version": "Linux-4.14.326-245.539.amzn2.x86_64-x86_64-with-glibc2.26",
    "processor": "x86_64, 36 cores",
    "memory": "515203072 (used) / 72537300992 (available) / 73743400960 (total)"
  },
  "results": {
    "test_took": 160361.9497846673,
    "delete_index_took_total": 305.41585766574525,
    "create_index_took_total": 204.99163833543813,
    "ingest_took_total": 57003.40605100064,
    "refresh_index_store_kb_total": 2825292.8359375,
    "refresh_index_took_total": 13676.234865333148,
    "force_merge_took_total": 62267.475193667146,
    "warmup_operation_took_total": 258.75951199850533,
    "query_took_total": 26645.666666666668,
    "query_took_p50": 3.0,
    "query_took_p90": 3.0,
    "query_took_p99": 3.0,
    "query_took_p99.9": 7.0,
    "query_took_p100": 640.0,
    "query_client_time_total": 57391.0,
    "query_client_time_p50": 5.333333333333333,
    "query_client_time_p90": 6.333333333333333,
    "query_client_time_p99": 6.333333333333333,
    "query_client_time_p99.9": 46.333333333333336,
    "query_client_time_p100": 673.6666666666666,
    "query_memory_kb_total": 1305860.3333333333,
    "query_recall@K_total": 0.9988790000000001,
    "query_recall@1_total": 1.0
  }
}

@navneet1v navneet1v mentioned this issue Apr 4, 2024
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vibrantvarun vibrantvarun

Labels
v2.13.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jmazanec15 @vibrantvarun