Merge branch 'main' into version/deobfuscate-node-metadata

.github/CODEOWNERS

@@ -22,3 +22,6 @@ x-pack/plugin/core/src/main/resources/fleet-* @elastic/fleet
 
 # Kibana Security
 x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java @elastic/kibana-security
+
+# APM
+x-pack/plugin/apm-data @elastic/apm-server

REST_API_COMPATIBILITY.md

@@ -1,6 +1,6 @@
 #  REST API compatibility developers guide
 
-REST API compatibility is intended to minimize the impact for breaking changes applied to the REST API. Compatibility is implemented in a best effort to strike a balance between preserving the REST API contract across major versions while still allowing for breaking changes. URL paths, parameters, HTTP verbs, request bodies and response bodies are all covered by REST API compatibility.
+REST API compatibility is intended to minimize the impact for breaking changes applied to the REST API. Compatibility is implemented to strike a balance between preserving the REST API contract across major versions while still allowing for breaking changes. URL paths, parameters, HTTP verbs, request bodies and response bodies are all covered by REST API compatibility.
 
 ### Example use case
 
@@ -10,7 +10,7 @@ For example, assume a REST request requires a consumer to send a "limit" paramet
 
 ### Workflow
 
-REST API compatibility is opt-in per request using a specialized value for the `Accept` or `Content-Type` HTTP header.  The intent is that this header may be sent prior to a major version upgrade resulting in no differences with the standard header values when the client and server match versions. For example, assume the consumer/client using the REST API for Elasticsearch version 7.  If the client sends the specialized header value(s) for compatibility with version 7, then it will have no effect when talking with Elasticsearch version 7. However, once Elasticsearch is upgraded to version 8, and the compatibility with version 7 headers are sent, then Elasticsearch version 8 will do a best effort honor the version 7 REST API contract. This allows Elasticsearch to be upgraded to version 8 while the clients can generally remain on version 7. Since compatibility is best effort, it is not always guaranteed to fix all upgrade issues but should provide an additional level of confidence during/post upgrade.
+REST API compatibility is opt-in per request using a specialized value for the `Accept` or `Content-Type` HTTP header.  The intent is that this header may be sent prior to a major version upgrade resulting in no differences with the standard header values when the client and server match versions. For example, assume the consumer/client using the REST API for Elasticsearch version 7.  If the client sends the specialized header value(s) for compatibility with version 7, then it will have no effect when talking with Elasticsearch version 7. However, once Elasticsearch is upgraded to version 8, and the compatibility with version 7 headers are sent, then Elasticsearch version 8 attempts to honor the version 7 REST API contract. This allows Elasticsearch to be upgraded to version 8 while the clients can generally remain on version 7. It is not always guaranteed to fix all upgrade issues but should provide an additional level of confidence during/post upgrade.
 
 Breaking changes always follow a life-cycle of deprecation (documentation and warnings) in the current version before implementing the breaking change in the next major version. This give users an opportunity to adopt the non-deprecated variants in the current version. It is the still recommended approach to stop the usage of all deprecated functionality prior to a major version upgrade. However, in practice this can be a difficult, error prone, and a time consuming task. So it also recommended that consumers/clients send the specialized header to enable REST API compatibility before an upgrade to help catch any missed usages of deprecated functionality.
 

benchmarks/src/main/java/org/elasticsearch/benchmark/script/ScriptScoreBenchmark.java

@@ -78,7 +78,7 @@ public class ScriptScoreBenchmark {
         null,
         Path.of(System.getProperty("plugins.dir"))
     );
-    private final ScriptModule scriptModule = new ScriptModule(Settings.EMPTY, pluginsService.filterPlugins(ScriptPlugin.class));
+    private final ScriptModule scriptModule = new ScriptModule(Settings.EMPTY, pluginsService.filterPlugins(ScriptPlugin.class).toList());
 
     private final Map<String, MappedFieldType> fieldTypes = Map.ofEntries(
         Map.entry("n", new NumberFieldType("n", NumberType.LONG, false, false, true, true, null, Map.of(), null, false, null, null))

client/rest/src/test/java/org/elasticsearch/client/RestClientMultipleHostsIntegTests.java

@@ -258,7 +258,6 @@ public void onFailure(Exception exception) {
         }
     }
 
-    @Ignore("https://github.com/elastic/elasticsearch/issues/45577")
     public void testCancelAsyncRequests() throws Exception {
         int numRequests = randomIntBetween(5, 20);
         final List<Response> responses = new CopyOnWriteArrayList<>();

docs/changelog/100368.yaml

@@ -0,0 +1,5 @@
+pr: 100368
+summary: "Status codes for Aggregation errors, part 2"
+area: Aggregations
+type: enhancement
+issues: []

docs/changelog/101066.yaml

@@ -0,0 +1,6 @@
+pr: 101066
+summary: Log errors in `RestResponse` regardless of `error_trace` parameter
+area: "Infra/Core"
+type: enhancement
+issues:
+ - 100884

docs/changelog/101585.yaml

@@ -0,0 +1,6 @@
+pr: 101585
+summary: Reroute on shard snapshot completion
+area: Snapshot/Restore
+type: bug
+issues:
+ - 101514

docs/changelog/101607.yaml

@@ -0,0 +1,5 @@
+pr: 101607
+summary: Log stacktrace together with log message in order to help debugging
+area: Transform
+type: bug
+issues: []

docs/reference/data-management.asciidoc

@@ -9,15 +9,15 @@ The data you store in {es} generally falls into one of two categories:
 * Content: a collection of items you want to search, such as a catalog of products
 * Time series data: a stream of continuously-generated timestamped data, such as log entries
 
-Content might be frequently updated, 
-but the value of the content remains relatively constant over time. 
-You want to be able to retrieve items quickly regardless of how old they are. 
-
-Time series data keeps accumulating over time, so you need strategies for 
-balancing the value of the data against the cost of storing it. 
-As it ages, it tends to become less important and less-frequently accessed, 
-so you can move it to less expensive, less performant hardware. 
-For your oldest data, what matters is that you have access to the data. 
+Content might be frequently updated,
+but the value of the content remains relatively constant over time.
+You want to be able to retrieve items quickly regardless of how old they are.
+
+Time series data keeps accumulating over time, so you need strategies for
+balancing the value of the data against the cost of storing it.
+As it ages, it tends to become less important and less-frequently accessed,
+so you can move it to less expensive, less performant hardware.
+For your oldest data, what matters is that you have access to the data.
 It's ok if queries take longer to complete.
 
 To help you manage your data, {es} offers you:
@@ -26,16 +26,16 @@ To help you manage your data, {es} offers you:
 * <<data-stream-lifecycle, Data stream lifecycle>> which is the built-in lifecycle of data streams and addresses the most
 common lifecycle management needs.
 
-preview::["The built-in data stream lifecycle is in technical preview and may be changed or removed in a future release. Elastic will apply best effort to fix any issues, but this feature is not subject to the support SLA of official GA features."]
+preview::["The built-in data stream lifecycle is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but this feature is not subject to the support SLA of official GA features."]
 
 **{ilm-init}** can be used to manage both indices and data streams and it allows you to:
 
 * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.
 Data older than this period can be deleted by {es}.
 * Define <<data-tiers, multiple tiers>> of data nodes with different performance characteristics.
 * Automatically transition indices through the data tiers according to your performance needs and retention policies.
-* Leverage <<searchable-snapshots, searchable snapshots>> stored in a remote repository to provide resiliency 
-for your older indices while reducing operating costs and maintaining search performance.  
+* Leverage <<searchable-snapshots, searchable snapshots>> stored in a remote repository to provide resiliency
+for your older indices while reducing operating costs and maintaining search performance.
 * Perform <<async-search-intro, asynchronous searches>> of data stored on less-performant hardware.
 
 **Data stream lifecycle** is less feature rich but is focused on simplicity, so it allows you to easily:

docs/reference/esql/esql-kibana.asciidoc

@@ -6,7 +6,7 @@
 ++++
 
 You can use {esql} in {kib} to query and aggregate your data, create
-visualizations, and set up alerts. 
+visualizations, and set up alerts.
 
 This guide shows you how to use {esql} in Kibana. To follow along with the
 queries, load the "Sample web logs" sample data set by clicking *Try sample
@@ -72,7 +72,7 @@ following query is identical to the previous one:
 
 [source,esql]
 ----
-FROM kibana_sample_data_logs 
+FROM kibana_sample_data_logs
 | LIMIT 10
 ----
 
@@ -82,7 +82,7 @@ bar:
 
 image::images/esql/esql-expanded-query-bar.png[align="center"]
 
-To return to a compact query bar, click the minimize editor button 
+To return to a compact query bar, click the minimize editor button
 (image:images/esql/esql-icon-minimize-query-bar.svg[]).
 
 [discrete]
@@ -253,4 +253,9 @@ quick statistics for that field.
 of rows that are retrieved by the query and displayed in Discover. Any query or
 aggregation runs on the full data set.
 * Discover shows no more than 50 columns. If a query returns
-more than 50 columns, Discover only shows the first 50.
+more than 50 columns, Discover only shows the first 50.
+* Querying many many indices at once without any filters can cause an error in
+kibana which looks like `[esql] > Unexpected error from Elasticsearch: The
+content length (536885793) is bigger than the maximum allowed string (536870888)`.
+The response from {esql} is too long. Use <<esql-drop>> or <<esql-keep>> to limit the number
+of fields returned.

docs/reference/ilm/actions/ilm-forcemerge.asciidoc

@@ -4,33 +4,32 @@
 
 Phases allowed: hot, warm.
 
-<<indices-forcemerge,Force merges>> the index into 
+<<indices-forcemerge,Force merges>> the index into
 the specified maximum number of <<indices-segments,segments>>.
 
 [NOTE]
-The `forcemerge` action is best effort. It might happen that some of the
-shards are relocating, in which case they will not be merged.
+Shards that are relocating during a `forcemerge` will not be merged.
 
 To use the `forcemerge` action in the `hot` phase, the `rollover` action *must* be present.
-If no rollover action is configured, {ilm-init} will reject the policy. 
+If no rollover action is configured, {ilm-init} will reject the policy.
 
 [[ilm-forcemerge-performance]]
 .Performance considerations
 ****
-Force merge is a resource-intensive operation. 
-If too many force merges are triggered at once, it can negatively impact your cluster. 
-This can happen when you apply an {ilm-init} policy that includes a force merge action 
+Force merge is a resource-intensive operation.
+If too many force merges are triggered at once, it can negatively impact your cluster.
+This can happen when you apply an {ilm-init} policy that includes a force merge action
 to existing indices.
-If they meet the `min_age` criteria, they can immediately proceed through multiple phases. 
-You can prevent this by increasing the `min_age` or setting `index.lifecycle.origination_date` 
-to change how the index age is calculated. 
+If they meet the `min_age` criteria, they can immediately proceed through multiple phases.
+You can prevent this by increasing the `min_age` or setting `index.lifecycle.origination_date`
+to change how the index age is calculated.
 
 If you experience a force merge task queue backlog,
-you might need to increase the size of the force merge threadpool so  
-indices can be force merged in parallel. 
+you might need to increase the size of the force merge threadpool so
+indices can be force merged in parallel.
 To do this, configure the `thread_pool.force_merge.size` <<cluster-get-settings,cluster setting>>.
 
-IMPORTANT: This can have cascading performance impacts. 
+IMPORTANT: This can have cascading performance impacts.
 Monitor cluster performance and increment the size of the thread pool slowly to reduce the backlog.
 
 Force merging will be performed by the nodes within the current phase of the index. A forcemerge in
@@ -44,7 +43,7 @@ without impacting ingestion in the `hot` tier.
 ==== Options
 
 `max_num_segments`::
-(Required, integer) 
+(Required, integer)
 Number of segments to merge to. To fully merge the index, set to `1`.
 
 `index_codec`::

docs/reference/ilm/actions/ilm-searchable-snapshot.asciidoc

@@ -45,7 +45,7 @@ indices. These costs are typically lower but, in some environments, may be
 higher. See <<searchable-snapshots-costs>> for more details.
 
 By default, this snapshot is deleted by the <<ilm-delete, delete action>> in the delete phase.
-To keep the snapshot, set `delete_searchable_snapshot` to `false` in the delete action. This 
+To keep the snapshot, set `delete_searchable_snapshot` to `false` in the delete action. This
 snapshot retention runs off the index lifecycle management (ILM) policies and is not effected by snapshot lifecycle management (SLM) policies.
 
 [[ilm-searchable-snapshot-options]]
@@ -64,8 +64,7 @@ If the managed index was already force merged using the
 the `searchable snapshot` action force merge step will be a no-op.
 
 [NOTE]
-The `forcemerge` action is best effort. It might happen that some of
-the shards are relocating, in which case they will not be merged.
+Shards that are relocating during a `forcemerge` will not be merged.
 The `searchable_snapshot` action will continue executing even if not all shards
 are force merged.
 

docs/reference/indices/data-stream-stats.asciidoc

@@ -160,9 +160,8 @@ the {wikipedia}/Unix_time[Unix epoch].
 +
 [NOTE]
 =====
-This timestamp is provided as a best effort. The data stream may contain
-`@timestamp` values higher than this if one or more of the following conditions
-are met:
+The data stream may contain `@timestamp` values higher than this if one or more
+of the following conditions are met:
 
 * The stream contains <<indices-open-close,closed>> backing indices.
 * Backing indices with a <<data-streams-generation,lower generation>> contain

docs/reference/indices/diskusage.asciidoc

@@ -185,6 +185,6 @@ files are ignored and some parts of data files might not be scanned by the API.
 <3> The stored size of the `_id` field
 
 <4> The stored size of the `_source` field. As stored fields are stored
-together in a compressed format, the estimated sizes of stored fields are
-best efforts and can be inaccurate. The stored size of the `_id` field
+together in a compressed format, the sizes of stored fields are
+estimates and can be inaccurate. The stored size of the `_id` field
 is likely underestimated while the `_source` field is overestimated.

docs/reference/mapping/fields/synthetic-source.asciidoc

@@ -4,7 +4,7 @@
 IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
 (indices that have `index.mode` set to `time_series`). For other indices
 synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will apply best effort to fix
+be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 

docs/reference/mapping/types/aggregate-metric-double.asciidoc

@@ -255,7 +255,7 @@ The search returns the following hit. The value of the `default_metric` field,
 IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
 (indices that have `index.mode` set to `time_series`). For other indices
 synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will apply best effort to fix
+be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 

docs/reference/mapping/types/boolean.asciidoc

@@ -229,7 +229,7 @@ The following parameters are accepted by `boolean` fields:
 IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
 (indices that have `index.mode` set to `time_series`). For other indices
 synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will apply best effort to fix
+be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 

docs/reference/mapping/types/date.asciidoc

@@ -236,7 +236,7 @@ Which will reply with a date like:
 IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
 (indices that have `index.mode` set to `time_series`). For other indices
 synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will apply best effort to fix
+be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 

docs/reference/mapping/types/date_nanos.asciidoc

@@ -146,7 +146,7 @@ field. This limitation also affects <<transforms,{transforms}>>.
 IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
 (indices that have `index.mode` set to `time_series`). For other indices
 synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will apply best effort to fix
+be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 

docs/reference/mapping/types/dense-vector.asciidoc

@@ -228,7 +228,7 @@ neighbors for each new node. Defaults to `100`.
 IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
 (indices that have `index.mode` set to `time_series`). For other indices
 synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will apply best effort to fix
+be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 

docs/reference/mapping/types/flattened.asciidoc

@@ -320,7 +320,7 @@ The following mapping parameters are accepted:
 IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
 (indices that have `index.mode` set to `time_series`). For other indices
 synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will apply best effort to fix
+be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 

docs/reference/mapping/types/geo-point.asciidoc

@@ -215,7 +215,7 @@ def lon      = doc['location'].lon;
 IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
 (indices that have `index.mode` set to `time_series`). For other indices
 synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will apply best effort to fix
+be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.
 

docs/reference/mapping/types/histogram.asciidoc

@@ -74,7 +74,7 @@ index data in that manner (e.g. centroids for T-Digest or intervals for HDRHisto
 IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
 (indices that have `index.mode` set to `time_series`). For other indices
 synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will apply best effort to fix
+be changed or removed in a future release. Elastic will work to fix
 any issues, but features in technical preview are not subject to the support SLA
 of official GA features.