Reload secure settings for plugins

[albertzaharovits]

This is the merge PR of the reload-secure-store-action feature branch, progress of which has been tracked in #29135. This is not intended to be reviewed as a whole, small incremental changes have been reviewed independently. The PR is opened to have CI run (several times).

CC @elastic/es-security
Closes #29135

[elasticmachine]

Pinging @elastic/es-core-infra

[albertzaharovits]

Both the gradle build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/11892 and the packaging tests https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request+packaging-tests-sample/3495 have failed because this PR introduced stricter behavior of not allowing the installation of the azure repository plugin if its client settings are not already defined. This strictness has been reverted. After all, this new functionality allows one to defer specifying or changing the secure settings (e27f097)

[albertzaharovits]

test this please

[albertzaharovits]

The previous gradle build failure (https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/11893/console) was unrelated to the changes herein. I've taken care of it, see #30942 (comment)

[albertzaharovits]

Previous gradle build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/11894 and packaging tests https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request+packaging-tests-sample/3497 have completed successfully.
Diligence++
test this please

[albertzaharovits]

Previous gradle build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/11895 and packaging tests https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request+packaging-tests-sample/3498 have completed successfully.
Diligence++
test this please

[albertzaharovits]

Diligence paid off.
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/11896/console is a true failure due to changes herein. The reproduction line is:

REPRODUCE WITH: ./gradlew :server:integTest -Dtests.seed=9045C467BBF675E4 -Dtests.class=org.elasticsearch.action.admin.ReloadSecureSettingsIT -Dtests.method="testMissingKeystoreFile" -Dtests.security.manager=true -Dtests.locale=es-PR -Dtests.timezone=America/Yellowknife
12:01:52 FAILURE 0.16s J0 | ReloadSecureSettingsIT.testMissingKeystoreFile <<< FAILURES!
12:01:52    > Throwable #1: java.lang.AssertionError: 
12:01:52    > Expected: <3>
12:01:52    >      but: was <2>
12:01:52    > 	at __randomizedtesting.SeedInfo.seed([9045C467BBF675E4:A532E37150607DE1]:0)
12:01:52    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
12:01:52    > 	at org.elasticsearch.action.admin.ReloadSecureSettingsIT$1.onResponse(ReloadSecureSettingsIT.java:73)
12:01:52    > 	at org.elasticsearch.action.admin.ReloadSecureSettingsIT$1.onResponse(ReloadSecureSettingsIT.java:67)

It reproduces locally 80% of the time. I have pushed 56f741e to mitigate it.
The problem is explained in #31261 (comment) , although I was not right there.

In short, the problem is that the SecureString holding the password, which is used to decrypt the keystore, is cleared before it is broadcasted. This happens because the original nodes request, NodesReloadSecureSettingsRequest (holding the password) is passed by reference to each NodeRequest. The password is cleared for each node request, but if the request is not serialized and deserialized, i.e. it is submitted locally

elasticsearch/server/src/main/java/org/elasticsearch/transport/TransportService.java

Line 658 in 3d5f113

private void sendLocalRequest(long requestId, final String action, final TransportRequest request, TransportRequestOptions options) {

, then clearing the password on the NodeRequest will clear the password on the original NodeSRequest.

[albertzaharovits]

Previous gradle build https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request/11899 and packaging tests https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request+packaging-tests-sample/3502 have completed successfully.
Diligence++
test this please