Merge branch '7.x' into ua_batch_reindex_docs

dev_docs/key_concepts/persistable_state.mdx

@@ -0,0 +1,83 @@
+---
+id: kibDevDocsPersistableStateIntro
+slug: /kibana-dev-docs/persistable-state-intro
+title: Persistable State
+summary: Persitable state is a key concept to understand when building a Kibana plugin.
+date: 2021-02-02
+tags: ['kibana','dev', 'contributor', 'api docs']
+---
+
+ “Persistable state” is developer-defined state that supports being persisted by a plugin other than the one defining it. Persistable State needs to be serializable and the owner can/should provide utilities to migrate it, extract and inject any <DocLink id="kibDevDocsSavedObjectsIntro" section="references" text="references to Saved Objects"/> it may contain, as well as telemetry collection utilities.
+
+## Exposing state that can be persisted
+
+Any plugin that exposes state that another plugin might persist should implement <DocLink id="kibKibanaUtilsPluginApi " section="def-common.PersistableStateService" text="`PersistableStateService`"/> interface on their `setup` contract. This will allow plugins persisting the state to easily access migrations and other utilities.
+
+Example: Data plugin allows you to generate filters. Those filters can be persisted by applications in their saved 
+objects or in the URL. In order to allow apps to migrate the filters in case the structure changes in the future, the Data plugin implements `PersistableStateService` on <DocLink id="kibDataQueryPluginApi " section="def-public.FilterManager" text="`data.query.filterManager`"/>.
+
+note: There is currently no obvious way for a plugin to know which state is safe to persist. The developer must manually look for a matching `PersistableStateService`, or ad-hoc provided migration utilities (as is the case with Rule Type Parameters).
+In the future, we hope to make it easier for producers of state to understand when they need to write a migration with changes, and also make it easier for consumers of such state, to understand whether it is safe to persist.
+
+## Exposing state that can be persisted but is not owned by plugin exposing it (registry)
+
+Any plugin that owns collection of items (registry) whose state/configuration can be persisted should implement `PersistableStateService`
+interface on their `setup` contract and each item in the collection should implement <DocLink id="kibKibanaUtilsPluginApi" section="def-common.PersistableStateDefinition" text="`PersistableStateDefinition`"/> interface.
+
+Example: Embeddable plugin owns the registry of embeddable factories to which other plugins can register new embeddable factories. Dashboard plugin
+stores a bunch of embeddable panels input in its saved object and URL. Embeddable plugin setup contract implements `PersistableStateService`
+interface and each `EmbeddableFactory`  needs to implement `PersistableStateDefinition` interface.
+
+Embeddable plugin exposes this interfaces:
+```
+// EmbeddableInput implements Serializable
+
+export interface EmbeddableRegistryDefinition extends PersistableStateDefinition<EmbeddableInput> {
+  id: string;
+  ...
+}
+
+export interface EmbeddableSetup extends PersistableStateService<EmbeddableInput>;
+```
+
+Note: if your plugin doesn't expose the state (it is the only one storing state), the plugin doesn't need to implement the `PersistableStateService` interface.
+If the state your plugin is storing can be provided by other plugins (your plugin owns a registry) items in that registry still need to implement `PersistableStateDefinition` interface.
+
+## Storing persistable state as part of saved object
+
+Any plugin that stores any persistable state as part of their saved object should make sure that its saved object migration 
+and reference extraction and injection methods correctly use the matching `PersistableStateService` implementation for the state they are storing.
+
+Take a look at [example saved object](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/server/searchable_list_saved_object.ts#L32) which stores an embeddable state. Note how the `migrations`, `extractReferences` and `injectReferences` are defined.
+
+## Storing persistable state as part of URL
+
+When storing persistable state as part of URL you must make sure your URL is versioned. When loading the state `migrateToLatest` method
+of `PersistableStateService` should be called, which will migrate the state from its original version to latest.
+
+note: Currently there is no recommended way on how to store version in url and its up to every application to decide on how to implement that.
+
+## Available state operations
+
+### Extraction/Injection of References
+
+In order to support import and export, and space-sharing capabilities, Saved Objects need to explicitly list any references they contain to other Saved Objects.
+To support persisting your state in saved objects owned by another plugin, the <DocLink id="kibKibanaUtilsPluginApi" section="def-common.PersistableState.extract" text="`extract`"/> and <DocLink id="kibKibanaUtilsPluginApi" section="def-common.PersistableState.inject" text="`inject`"/> methods of Persistable State interface should be implemented.
+
+<DocLink id="kibDevTutorialSavedObject" section="references" text="Learn how to define Saved Object references"/>
+
+[See example embeddable providing extract/inject functions](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/public/migrations/migrations_embeddable_factory.ts)
+
+### Migrations and Backward compatibility
+
+As your plugin evolves, you may need to change your state in a breaking way. If that happens, you should write a migration to upgrade the state that existed prior to the change.
+
+<DocLink id="kibDevTutorialSavedObject" section="migrations" text="How to write a migration"/>.
+
+[See an example saved object storing embeddable state implementing saved object migration function](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/server/searchable_list_saved_object.ts)
+
+[See example embeddable providing migration functions](https://github.com/elastic/kibana/blob/master/examples/embeddable_examples/public/migrations/migrations_embeddable_factory.ts)
+
+## Telemetry
+
+You might want to collect statistics about how your state is used. If that is the case you should implement the telemetry method of Persistable State interface. 

docs/dev-tools/console/console.asciidoc

@@ -134,6 +134,7 @@ shortcuts, click *Help*.
 [[console-settings]]
 === Disable Console
 
+deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
 If you don’t want to use *Console*, you can disable it by setting `console.enabled`
 to `false` in your `kibana.yml` configuration file. Changing this setting
 causes the server to regenerate assets on the next startup,

docs/settings/alert-action-settings.asciidoc

@@ -32,6 +32,7 @@ Be sure to back up the encryption key value somewhere safe, as your alerting rul
 ==== Action settings
 
 `xpack.actions.enabled`::
+deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
 Feature toggle that enables Actions in {kib}.
 If `false`, all features dependent on Actions are disabled, including the *Observability* and *Security* apps.  Default: `true`.
 

docs/settings/apm-settings.asciidoc

@@ -46,7 +46,8 @@ Changing these settings may disable features of the APM App.
 [cols="2*<"]
 |===
 | `xpack.apm.enabled`
-  | Set to `false` to disable the APM app. Defaults to `true`.
+  | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+  Set to `false` to disable the APM app. Defaults to `true`.
 
 | `xpack.apm.maxServiceEnvironments`
   | Maximum number of unique service environments recognized by the UI. Defaults to `100`.

docs/settings/dev-settings.asciidoc

@@ -13,6 +13,7 @@ They are enabled by default.
 ==== Grok Debugger settings
 
 `xpack.grokdebugger.enabled` {ess-icon}::
+deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
 Set to `true` to enable the <<xpack-grokdebugger,Grok Debugger>>. Defaults to `true`.
 
 
@@ -21,11 +22,13 @@ Set to `true` to enable the <<xpack-grokdebugger,Grok Debugger>>. Defaults to `t
 ==== {searchprofiler} settings
 
 `xpack.searchprofiler.enabled`::
+deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
 Set to `true` to enable the <<xpack-profiler,{searchprofiler}>>. Defaults to `true`.
 
 [float]
 [[painless_lab-settings]]
 ==== Painless Lab settings
 
 `xpack.painless_lab.enabled`::
+deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
 When set to `true`, enables the <<painlesslab, Painless Lab>>. Defaults to `true`.

docs/settings/fleet-settings.asciidoc

@@ -21,7 +21,8 @@ See the {fleet-guide}/index.html[{fleet}] docs for more information.
 [cols="2*<"]
 |===
 | `xpack.fleet.enabled` {ess-icon}
-  | Set to `true` (default) to enable {fleet}. 
+  | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+  Set to `true` (default) to enable {fleet}. 
 | `xpack.fleet.agents.enabled` {ess-icon}
   | Set to `true` (default) to enable {fleet}. 
 |===

docs/settings/general-infra-logs-ui-settings.asciidoc

@@ -1,7 +1,8 @@
 [cols="2*<"]
 |===
 | `xpack.infra.enabled`
-  | Set to `false` to disable the Logs and Metrics app plugin {kib}. Defaults to `true`.
+  | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+  Set to `false` to disable the Logs and Metrics app plugin {kib}. Defaults to `true`.
 
 | `xpack.infra.sources.default.logAlias`
   | Index pattern for matching indices that contain log data. Defaults to `filebeat-*,kibana_sample_data_logs*`. To match multiple wildcard patterns, use a comma to separate the names, with no space after the comma. For example, `logstash-app1-*,default-logs-*`.

docs/settings/graph-settings.asciidoc

@@ -8,4 +8,5 @@
 You do not need to configure any settings to use the {graph-features}.
 
 `xpack.graph.enabled` {ess-icon}::
+deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
 Set to `false` to disable the {graph-features}.

docs/settings/ml-settings.asciidoc

@@ -14,7 +14,8 @@ enabled by default.
 [cols="2*<"]
 |===
 | `xpack.ml.enabled` {ess-icon}
-  | Set to `true` (default) to enable {kib} {ml-features}. +
+  | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+  Set to `true` (default) to enable {kib} {ml-features}. +
   +
   If set to `false` in `kibana.yml`, the {ml} icon is hidden in this {kib}
   instance. If `xpack.ml.enabled` is set to `true` in `elasticsearch.yml`, however,

docs/settings/monitoring-settings.asciidoc

@@ -32,7 +32,8 @@ For more information, see
 [cols="2*<"]
 |===
 | `monitoring.enabled`
-  | Set to `true` (default) to enable the {monitor-features} in {kib}. Unlike the
+  | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+  Set to `true` (default) to enable the {monitor-features} in {kib}. Unlike the
   <<monitoring-ui-enabled, `monitoring.ui.enabled`>> setting, when this setting is `false`, the
   monitoring back-end does not run and {kib} stats are not sent to the monitoring
   cluster.

docs/settings/security-settings.asciidoc

@@ -15,7 +15,8 @@ You do not need to configure any additional settings to use the
 [cols="2*<"]
 |===
 | `xpack.security.enabled`
-  | By default, {kib} automatically detects whether to enable the
+  | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+  By default, {kib} automatically detects whether to enable the
   {security-features} based on the license and whether {es} {security-features}
   are enabled. +
   +

docs/settings/spaces-settings.asciidoc

@@ -15,8 +15,9 @@ roles when Security is enabled.
 [cols="2*<"]
 |===
 | `xpack.spaces.enabled`
-  | Set to `true` (default) to enable Spaces in {kib}.
-    This setting is deprecated. Starting in 8.0, it will not be possible to disable this plugin.
+  | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+  Set to `true` (default) to enable Spaces in {kib}.
+  This setting is deprecated. Starting in 8.0, it will not be possible to disable this plugin.
 
 | `xpack.spaces.maxSpaces`
   | The maximum amount of Spaces that can be used with this instance of {kib}. Some operations

docs/settings/url-drilldown-settings.asciidoc

@@ -9,7 +9,8 @@ Configure the URL drilldown settings in your `kibana.yml` configuration file.
 [cols="2*<"]
 |===
 | [[url-drilldown-enabled]] `url_drilldown.enabled`
- | When `true`, enables URL drilldowns on your {kib} instance.
+ | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+ When `true`, enables URL drilldowns on your {kib} instance.
 
 | [[external-URL-policy]] `externalUrl.policy`
  | Configures the external URL policies. URL drilldowns respect the global *External URL* service, which you can use to deny or allow external URLs.

docs/setup/docker.asciidoc

@@ -125,7 +125,7 @@ Some example translations are shown here:
 **Environment Variable**:: **Kibana Setting**
 `SERVER_NAME`:: `server.name`
 `SERVER_BASEPATH`:: `server.basePath`
-`MONITORING_ENABLED`:: `monitoring.enabled`
+`ELASTICSEARCH_HOSTS`:: `elasticsearch.hosts`
 
 In general, any setting listed in <<settings>> can be configured with this technique.
 

docs/setup/settings.asciidoc

@@ -21,7 +21,8 @@ configuration using `${MY_ENV_VAR}` syntax.
 |===
 
 | `console.enabled:`
- | Toggling this causes the server to regenerate assets on the next startup,
+ | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+Toggling this causes the server to regenerate assets on the next startup,
 which may cause a delay before pages start being served.
 Set to `false` to disable Console. *Default: `true`*
 
@@ -794,12 +795,13 @@ sources and images. When false, Vega can only get data from {es}. *Default: `fal
  | Enables you to view the underlying documents in a data series from a dashboard panel. *Default: `false`*
 
 | `xpack.license_management.enabled`
- | Set this value to false to
-disable the License Management UI. *Default: `true`*
+ | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+Set this value to false to disable the License Management UI.
+*Default: `true`*
 
 | `xpack.rollup.enabled:`
- | Set this value to false to disable the
-Rollup UI. *Default: true*
+ | deprecated:[7.16.0,"In 8.0 and later, this setting will no longer be supported."]
+Set this value to false to disable the Rollup UI. *Default: true*
 
 | `i18n.locale` {ess-icon}
  | Set this value to change the {kib} interface language.

docs/user/plugins.asciidoc

@@ -149,6 +149,8 @@ NOTE: Removing a plugin will result in an "optimize" run which will delay the ne
 [[disable-plugin]]
 == Disable plugins
 
+deprecated:[7.16.0,"In 8.0 and later, this setting will only be supported for a subset of plugins that have opted in to the behavior."]
+
 Use the following command to disable a plugin:
 
 [source,shell]
@@ -158,7 +160,7 @@ Use the following command to disable a plugin:
 
 NOTE: Disabling or enabling a plugin will result in an "optimize" run which will delay the start of {kib}.
 
-<1> You can find a plugin's plugin ID as the value of the `name` property in the plugin's `package.json` file.
+<1> You can find a plugin's plugin ID as the value of the `name` property in the plugin's `kibana.json` file.
 
 [float]
 [[configure-plugin-manager]]

examples/embeddable_examples/kibana.json

@@ -9,7 +9,7 @@
     "githubTeam": "kibana-app-services"
   },
   "description": "Example app that shows how to register custom embeddables",
-  "requiredPlugins": ["embeddable", "uiActions", "savedObjects", "dashboard"],
+  "requiredPlugins": ["embeddable", "uiActions", "savedObjects", "dashboard", "kibanaUtils"],
   "optionalPlugins": [],
   "extraPublicDirs": ["public/todo", "public/hello_world", "public/todo/todo_ref_embeddable"],
   "requiredBundles": ["kibanaReact"]

examples/embeddable_examples/public/migrations/migrations_embeddable_factory.ts

@@ -7,6 +7,7 @@
  */
 
 import { i18n } from '@kbn/i18n';
+import { EmbeddableStateWithType } from '../../../../src/plugins/embeddable/common';
 import {
   IContainer,
   EmbeddableInput,
@@ -35,6 +36,16 @@ export class SimpleEmbeddableFactoryDefinition
     '7.3.0': migration730,
   };
 
+  public extract(state: EmbeddableStateWithType) {
+    // this embeddable does not store references to other saved objects
+    return { state, references: [] };
+  }
+
+  public inject(state: EmbeddableStateWithType) {
+    // this embeddable does not store references to other saved objects
+    return state;
+  }
+
   /**
    * In our simple example, we let everyone have permissions to edit this. Most
    * embeddables should check the UI Capabilities service to be sure of

examples/embeddable_examples/server/merge_migration_function_maps.ts

@@ -0,0 +1,25 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0 and the Server Side Public License, v 1; you may not use this file except
+ * in compliance with, at your election, the Elastic License 2.0 or the Server
+ * Side Public License, v 1.
+ */
+
+import { mergeWith } from 'lodash';
+import type { SerializableRecord } from '@kbn/utility-types';
+import { MigrateFunctionsObject, MigrateFunction } from '../../../src/plugins/kibana_utils/common';
+
+export const mergeMigrationFunctionMaps = (
+  obj1: MigrateFunctionsObject,
+  obj2: MigrateFunctionsObject
+) => {
+  const customizer = (objValue: MigrateFunction, srcValue: MigrateFunction) => {
+    if (!srcValue || !objValue) {
+      return srcValue || objValue;
+    }
+    return (state: SerializableRecord) => objValue(srcValue(state));
+  };
+
+  return mergeWith({ ...obj1 }, obj2, customizer);
+};

examples/embeddable_examples/server/searchable_list_saved_object.ts

@@ -9,9 +9,12 @@
 import { mapValues } from 'lodash';
 import { SavedObjectsType, SavedObjectUnsanitizedDoc } from 'kibana/server';
 import { EmbeddableSetup } from '../../../src/plugins/embeddable/server';
+// NOTE: this should rather be imported from 'plugins/kibana_utils/server' but examples at the moment don't
+// allow static imports from plugins so this code was duplicated
+import { mergeMigrationFunctionMaps } from './merge_migration_function_maps';
 
 export const searchableListSavedObject = (embeddable: EmbeddableSetup) => {
-  return {
+  const searchableListSO: SavedObjectsType = {
     name: 'searchableList',
     hidden: false,
     namespaceType: 'single',
@@ -30,14 +33,22 @@ export const searchableListSavedObject = (embeddable: EmbeddableSetup) => {
       },
     },
     migrations: () => {
-      // we assume all the migration will be done by embeddables service and that saved object holds no extra state besides that of searchable list embeddable input\
-      // if saved object would hold additional information we would need to merge the response from embeddables.getAllMigrations with our custom migrations.
-      return mapValues(embeddable.getAllMigrations(), (migrate) => {
+      // there are no migrations defined for the saved object at the moment, possibly they would be added in the future
+      const searchableListSavedObjectMigrations = {};
+
+      // we don't know if embeddables have any migrations defined so we need to fetch them and map the received functions so we pass
+      // them the correct input and that we correctly map the response
+      const embeddableMigrations = mapValues(embeddable.getAllMigrations(), (migrate) => {
         return (state: SavedObjectUnsanitizedDoc) => ({
           ...state,
           attributes: migrate(state.attributes),
         });
       });
+
+      // we merge our and embeddable migrations and return
+      return mergeMigrationFunctionMaps(searchableListSavedObjectMigrations, embeddableMigrations);
     },
-  } as SavedObjectsType;
+  };
+
+  return searchableListSO;
 };

package.json

@@ -666,7 +666,7 @@
     "callsites": "^3.1.0",
     "chai": "3.5.0",
     "chance": "1.0.18",
-    "chromedriver": "^92.0.1",
+    "chromedriver": "^93.0.1",
     "clean-webpack-plugin": "^3.0.0",
     "cmd-shim": "^2.1.0",
     "compression-webpack-plugin": "^4.0.0",