Remove resource usages object from search response headers

[ansjcy]

Description

This PR (#13172) introduced shard-level resource usage tracking by adding task resource usage objects to the node response headers. However, by default, all headers from the thread context are added to the response header: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/http/DefaultRestChannel.java#L156

This behavior could lead to excessive headers being sent to the client, potentially impacting latency. To address this, we now explicitly remove task-related headers before sending the response to the client.

Testing

before the changes, run an opensearch cluster and insert some document. Do search and get response headers.

➜  OpenSearch git:(main) ✗ curl -v -X GET "localhost:9201/my-index-*/_search?size=1000&pretty" -H 'Content-Type: application/json' -d '{}'
* Host localhost:9201 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9201...
* Connected to localhost (::1) port 9201
> GET /my-index-*/_search?size=1000&pretty HTTP/1.1
> Host: localhost:9201
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 2
>
* upload completely sent off: 2 bytes
< HTTP/1.1 200 OK
< TASK_RESOURCE_USAGE: {"action":"indices:data/read/search[phase/query]","taskId":898,"parentTaskId":865,"nodeId":"Ld-7uLOxQPOnDjnCnSuFRQ","taskResourceUsage":{"cpu_time_in_nanos":22979000,"memory_in_bytes":2000624}}
< X-OpenSearch-Version: OpenSearch/3.0.0-SNAPSHOT (opensearch)
< content-type: application/json; charset=UTF-8
< content-length: 563
<
{
  "took" : 2064,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-index-0",
        "_id" : "HnDr45IBtfKEb7mfm5L1",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2099-11-15T13:12:00",
          "message" : "this is document 0",
          "user" : {
            "id" : "cyji"
          }
        }
      }
    ]
  }
}

With the change, run an opensearch cluster and insert some document. Do search and get response headers.

➜  OpenSearch git:(main) ✗ curl -v -X GET "localhost:9201/my-index-*/_search?size=1000&pretty" -H 'Content-Type: application/json' -d '{}'
* Host localhost:9201 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9201...
* Connected to localhost (::1) port 9201
> GET /my-index-*/_search?size=1000&pretty HTTP/1.1
> Host: localhost:9201
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 2
>
* upload completely sent off: 2 bytes
< HTTP/1.1 200 OK
< X-OpenSearch-Version: OpenSearch/3.0.0-SNAPSHOT (opensearch)
< content-type: application/json; charset=UTF-8
< content-length: 564
<
{
  "took" : 44872,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-index-0",
        "_id" : "v7z_45IBOxouwpmRaIKI",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2099-11-15T13:12:00",
          "message" : "this is document 0",
          "user" : {
            "id" : "cyji"
          }
        }
      }
    ]
  }
}

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[sgup432]

Should we also consider adding an explicit dynamic setting for query/coordinator level resource stats?
This way we can continue to use shard level resource stats independently if we face any issue with coordinator level stats.

[github-actions]

❌ Gradle check result for 93f2012: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[msfroh]

@ansjcy Can you please add a changelog entry for this?

[github-actions]

❕ Gradle check result for beac88d: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.10%. Comparing base (f57b889) to head (beac88d).
Report is 2 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16532      +/-   ##
============================================
+ Coverage     72.07%   72.10%   +0.02%     
- Complexity    65048    65108      +60     
============================================
  Files          5313     5313              
  Lines        303442   303449       +7     
  Branches      43910    43910              
============================================
+ Hits         218719   218814      +95     
+ Misses        66786    66744      -42     
+ Partials      17937    17891      -46     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.