Convert repository plugins to modules #81870

pugnascotia · 2021-12-17T15:09:25Z

Convert the repository-azure, repository-gcs and repository-s3 plugins into modules, so that they are always included in the Elasticsearch distribution. Also change plugin installation, removal and syncing so that attempting to add or remove these plugins still succeeds but is now a no-op.

…s-to-modules

elasticmachine · 2021-12-17T15:09:28Z

Pinging @elastic/es-distributed (Team:Distributed)

elasticmachine · 2021-12-17T15:09:28Z

Pinging @elastic/es-delivery (Team:Delivery)

elasticmachine · 2021-12-17T15:09:29Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

elasticsearchmachine · 2021-12-17T15:09:52Z

Hi @pugnascotia, I've created a changelog YAML for you.

mark-vieira

Couple of minor comments but otherwise LGTM. Looks like we have some minor packaging test fallout but I don't think there are any fundamental issues with the implementation.

mark-vieira · 2021-12-17T20:10:30Z

...ribution/tools/plugin-cli/src/main/java/org/elasticsearch/plugins/cli/SyncPluginsAction.java

@@ -115,7 +116,14 @@ public void execute() throws Exception {
    PluginChanges getPluginChanges(PluginsConfig pluginsConfig, Optional<PluginsConfig> cachedPluginsConfig) throws PluginSyncException {
        final List<PluginInfo> existingPlugins = getExistingPlugins();

-        final List<PluginDescriptor> pluginsThatShouldExist = pluginsConfig.getPlugins();
+        // For plugins that have migrated to modules, in order to help transition it's OK to still specify these plugins


If that's the case should we even bother filtering here? Wouldn't it be better to just delegate on and get the warning message which would inform the user to remove these from their plugin config? As it is now we'd be silencing that message, no?

The problem with delegating is that in this situation, the config file might say "repository-s3" should be installed, but actually we need to remove that plugin because it's now a module. So we need to change the "this plugin should be present" into a "this plugin should be removed". As a result, we don't reach the logic in InstallPluginAction that logs the warning about modularized plugins. However I realised that this implementation wasn't printing that warning at all when using a config file, so I've corrected that.

mark-vieira · 2021-12-17T20:11:31Z

...bution/tools/plugin-cli/src/main/java/org/elasticsearch/plugins/cli/InstallPluginAction.java

+                if (PLUGINS_CONVERTED_TO_MODULES.contains(pluginId)) {
+                    // This deliberately does not throw an exception in order to avoid failing automation that relies on installing this
+                    // plugin during deployment.
+                    terminal.errorPrintln(


I wonder if there are additionally any concerns with sending this to stderrr? Would stdout similarly break stuff depending how folks are consuming the output?

The thing is, the output from the CLI tools is completely unstructured. The only contract is the exit code of the tool, which is zero when installing or removing a modularized plugin. We can't reasonably be expected never to change the output of the CLI when it has no documented output format.

mark-vieira · 2021-12-17T20:20:27Z

docs/plugins/repository.asciidoc

-<<repository-s3,S3 Repository>>::
-
-The S3 repository plugin adds support for using S3 as a repository.
+NOTE: The S3, GCS and Azure repository plugins are now included in the {es}


Since they're no longer "plugins" I'd reword this to say

NOTE: Support for S3, GCS and Azure repository is now bundled in {es} by default

I'd also rename the section "Core repository plugins" to "Official repository plugins".

@elastic/es-docs thoughts?

modules/repository-s3/licenses/httpclient-NOTICE.txt

modules/build.gradle

debadair · 2021-12-21T23:33:36Z

@elasticmachine retest this please

debadair · 2021-12-21T23:57:37Z

The last broken links from the Kibana link service are fixed by elastic/kibana#121834. Once this PR is ready to go except for the broken links, it can be merged with the failing doc check and the Kibana PR can be retested & merged.

pugnascotia · 2021-12-22T13:32:19Z

@elasticmachine run elasticsearch-ci/cloud-deploy

pugnascotia · 2021-12-22T14:40:30Z

This PR requires changes from Cloud before we can merge.

…s-to-modules

…nto repository-plugins-to-modules

Closes elastic#81652. Convert the `repository-azure`, `repository-gcs` and `repository-s3` plugins into modules, so that they are always included in the Elasticsearch distribution. Also change plugin installation, removal and syncing so that attempting to add or remove these plugins still succeeds but is now a no-op.

The repository plugins for S3, Azure, and GCS are being moved into Elasticsearch as modules. This PR updates the Kibana links to point to the Elasticsearch guide rather than the Plugin guide. Related to elastic/elasticsearch#81870

Backport of #81870. Closes #81652. Convert the `repository-azure`, `repository-gcs` and `repository-s3` plugins into modules, so that they are always included in the Elasticsearch distribution. Also change plugin installation, removal and syncing so that attempting to add or remove these plugins still succeeds but is now a no-op.

The repository plugins for S3, Azure, and GCS are being moved into Elasticsearch as modules. This PR updates the Kibana links to point to the Elasticsearch guide rather than the Plugin guide. Related to elastic/elasticsearch#81870 Co-authored-by: debadair <debadair@elastic.co>

With elastic/elasticsearch#81870, we migrated docs for the Azure, GCS, and S3 snapshot repositories to the ES guide from the ES plugin docs. However, this change only applies to Elastic 8.0+. These attributes let the Cloud docs link to the "plugin" version of the docs when referencing 7.x and 6.x Elastic deployments without breaking links. Relates to https://github.com/elastic/cloud/pull/95621 and #2312

With elastic/elasticsearch#81870, the Azure, GCS, and AWS snapshot repository types have built-in support in Elasticsearch and no longer require plugins in 8.0+. This PR updates step one of the **Register Repository** wizard to: - Include Azure, GCS, and AWS as default repository types - Tweak UI copy and links referring to repository plugins. Co-authored-by: Yulia Čech <6585477+yuliacech@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

#123768) With elastic/elasticsearch#81870, the Azure, GCS, and AWS snapshot repository types have built-in support in Elasticsearch and no longer require plugins in 8.0+. This PR updates step one of the **Register Repository** wizard to: - Include Azure, GCS, and AWS as default repository types - Tweak UI copy and links referring to repository plugins. Co-authored-by: Yulia Čech <6585477+yuliacech@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 1e57604) # Conflicts: # x-pack/plugins/snapshot_restore/server/routes/api/repositories.test.ts

With #81870, the Azure, GCS, and S3 repository types have separate, dedicated pages in the Elasticsearch guide. For consistency, this PR creates separate pages for the shared file system, read-only URL, and source-only repository types. Related changes: - Adds redirects to the plugins docs - Fixes a few breaking changes that refer to the Azure, GCS, and S3 repositories as plugins. Co-authored-by: Adam Locke <adam.locke@elastic.co>

With #81870, the Azure, GCS, and S3 repository types have separate, dedicated pages in the Elasticsearch guide. For consistency, this PR creates separate pages for the shared file system, read-only URL, and source-only repository types. Related changes: - Adds redirects to the plugins docs - Fixes a few breaking changes that refer to the Azure, GCS, and S3 repositories as plugins. Co-authored-by: Adam Locke <adam.locke@elastic.co> (cherry picked from commit cb6265f)

pugnascotia added 4 commits December 14, 2021 15:46

Move repository s3, gcs and azure plugins to modules

87cf3d8

Update plugin-cli code for modularised plugins

aef9477

Add test coverage for migrated plugins

98ae5dc

Merge remote-tracking branch 'upstream/master' into repository-plugin…

b1e25cd

…s-to-modules

pugnascotia added >enhancement :Delivery/Packaging RPM and deb packaging, tar and zip archives, shell and batch scripts :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs :Core/Infra/Plugins Plugin API and infrastructure v8.0.0 v8.1.0 labels Dec 17, 2021

pugnascotia requested a review from mark-vieira December 17, 2021 15:09

elasticmachine added Team:Delivery Meta label for Delivery team Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. Team:Core/Infra Meta label for core/infra team labels Dec 17, 2021

pugnascotia and others added 7 commits December 17, 2021 15:09

Update docs/changelog/81870.yaml

03505e8

Update 81870.yaml

ba6cd2c

Fix test class name

f0d0ed2

Fix test to refer to stderr, not stdout

b95da9b

Fix test failures

1cf74ba

Attempt to update docs

eeec446

Fixy McFix-face

33955ee

mark-vieira approved these changes Dec 17, 2021

View reviewed changes

pugnascotia added 3 commits December 20, 2021 14:23

Permit modules to contain log4j2.properties config files

bd738a0

Fixes when syncing

c62997f

Docs tweak

0b287bb

pugnascotia commented Dec 20, 2021

View reviewed changes

modules/build.gradle Outdated Show resolved Hide resolved

pugnascotia added the cloud-deploy Publish cloud docker image for Cloud-First-Testing label Dec 22, 2021

pugnascotia added 3 commits January 10, 2022 10:25

Merge remote-tracking branch 'upstream/master' into repository-plugin…

c0c9631

…s-to-modules

Merge remote-tracking branch 'upstream/master' into repository-plugin…

fa11fd7

…s-to-modules

Merge remote-tracking branch 'origin/repository-plugins-to-modules' i…

1127e30

…nto repository-plugins-to-modules

pugnascotia merged commit d2dbef5 into elastic:master Jan 10, 2022

pugnascotia mentioned this pull request Jan 10, 2022

Convert repository plugins to modules (#81870) #82353

Merged

This was referenced Jan 19, 2022

Azure, GCS, and AWS snapshot repos no longer require plugins elastic/kibana#123387

Merged

[DOCS] Move snapshot repo types to separate pages #82826

Merged

[DOCS] Add plugins-7x and plugins-6x attributes elastic/docs#2358

Merged

jrodewig mentioned this pull request Jan 27, 2022

[DOCS] ES Guide and ESS docs contain redundant snapshot docs #83223

Closed

pgomulka mentioned this pull request Feb 8, 2022

Azure repository is missing log4j-slf4j-impl dependency #83652

Closed

This was referenced Feb 11, 2022

[Snapshot & Restore] Default repository types missing on Cloud elastic/kibana#125424

Closed

[Snapshot & Restore] Fix repository types on Cloud elastic/kibana#125656

Merged

b-deam mentioned this pull request Nov 18, 2022

Clean up handling of repository-s3/gcs/azure for 8.0+ elastic/rally#1622

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert repository plugins to modules #81870

Convert repository plugins to modules #81870

pugnascotia commented Dec 17, 2021

elasticmachine commented Dec 17, 2021

elasticmachine commented Dec 17, 2021

elasticmachine commented Dec 17, 2021

elasticsearchmachine commented Dec 17, 2021

mark-vieira left a comment

mark-vieira Dec 17, 2021

pugnascotia Dec 20, 2021

mark-vieira Dec 17, 2021

pugnascotia Dec 20, 2021

mark-vieira Dec 17, 2021

debadair commented Dec 21, 2021

debadair commented Dec 21, 2021

pugnascotia commented Dec 22, 2021

pugnascotia commented Dec 22, 2021

Convert repository plugins to modules #81870

Convert repository plugins to modules #81870

Conversation

pugnascotia commented Dec 17, 2021

elasticmachine commented Dec 17, 2021

elasticmachine commented Dec 17, 2021

elasticmachine commented Dec 17, 2021

elasticsearchmachine commented Dec 17, 2021

mark-vieira left a comment

Choose a reason for hiding this comment

mark-vieira Dec 17, 2021

Choose a reason for hiding this comment

pugnascotia Dec 20, 2021

Choose a reason for hiding this comment

mark-vieira Dec 17, 2021

Choose a reason for hiding this comment

pugnascotia Dec 20, 2021

Choose a reason for hiding this comment

mark-vieira Dec 17, 2021

Choose a reason for hiding this comment

debadair commented Dec 21, 2021

debadair commented Dec 21, 2021

pugnascotia commented Dec 22, 2021

pugnascotia commented Dec 22, 2021