Add support for secondary azure storage account #13779

dadoonet · 2015-09-24T20:19:55Z

In #13228, @craigwi proposed a PR to support multiple storage accounts and secondary endpoints (which are readonly).

As per discussion on the thread I started to revisit the PR.

This PR rebased @craigwi work on master and modify the way you define secondary storage account.

In elasticsearch.yml:

cloud:
    azure:
        storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2

When creating a repository, you can choose which azure account you want to use for it:

# This one will use the default account (my_account1)
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
  "type": "azure"
}'

# This one will use the second account (it will make it readonly)
curl -XPUT localhost:9200/_snapshot/my_backup2?pretty -d '{
  "type": "azure",
  "settings": { 
    "account" : "my_account2",
    "location_mode": "secondary_only"
  }
}'

@craigwi I ran some manual tests and it seems to work as expected but I'd love if you could confirm that.
Do you think you can build my branch and run a test on your side?

I don't think it's totally ready for a review though (I want at least to squash/reword some of commits).

craigwi · 2015-09-25T17:12:44Z

Thanks @dadoonet. I'll review and get back to you shortly.

dadoonet · 2015-10-12T10:38:03Z

@craigwi FYI I added a new commit and rebased on master.

dadoonet · 2015-10-27T14:32:52Z

@imotov Do you think you could review this PR?

craigwi · 2015-10-27T16:18:42Z

Thanks @dadoonet; as mentioned in a separate note, I have reviewed this and we are good to go.

imotov · 2015-11-10T14:28:36Z

...itory-azure/src/main/java/org/elasticsearch/cloud/azure/storage/AzureStorageServiceImpl.java

        try {
-            if (account != null) {
-                logger.trace("creating new Azure storage client using account [{}], key [{}], blob [{}]", account, key, blob);
+            String blob = "https://" + azureStorageSettings.getAccount() + ".blob.core.windows.net/";


Since the account settings are supplied by user, I would feels better if we used URI to build this string. This way we will have at least some basic validation of the things that go into this URL.

Well looking at the code again it sounds like we don't really use that blob string here. We just use it for logger.trace just below...

So I think I should remove that :D

imotov · 2015-11-10T14:37:46Z

Left one minor comment. The snapshot/restore side looks good to me.

Follow up for elastic#13228. This commit adds support for a secondary storage account: ```yml cloud: azure: storage: my_account1: account: your_azure_storage_account1 key: your_azure_storage_key1 default: true my_account2: account: your_azure_storage_account2 key: your_azure_storage_key2 ``` When creating a repository, you can choose which azure account you want to use for it: ```sh curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{ "type": "azure" }' curl -XPUT localhost:9200/_snapshot/my_backup2?pretty -d '{ "type": "azure", "settings": { "account" : "my_account2", "location_mode": "secondary_only" } }' ``` `location_mode` supports `primary_only` or `secondary_only`. Defaults to `primary_only`. Note that if you set it to `secondary_only`, it will force `read_only` to true.

dadoonet · 2015-11-18T16:03:47Z

Merged also in 2.x branch with 04b51d4

Bug introduced in elastic#13779: we don't filter anymore credentials because we were filtering `cloud.azure.storage.account` and `cloud.azure.storage.key` but now credentials are like `cloud.azure.storage.XXX.account` and `cloud.azure.storage.XXX.key` where `XXX` can be a storage setting id. Closes elastic#14843.

Bug introduced in #13779: we don't filter anymore credentials because we were filtering `cloud.azure.storage.account` and `cloud.azure.storage.key` but now credentials are like `cloud.azure.storage.XXX.account` and `cloud.azure.storage.XXX.key` where `XXX` can be a storage setting id. Closes #14843. (cherry picked from commit 149279f)

Using a single azure account is now rejected. This commit fixes this issue and adds a test for it. This regression was introduced with elastic#13779. Hopefully no elasticsearch version has been released since then. Needs to be merged in 2.2, 2.x and master branches.

Using a single azure account is now rejected. This commit fixes this issue and adds a test for it. This regression was introduced with #13779. Hopefully no elasticsearch version has been released since then. Backport of #15982 in 2.x branch. (cherry picked from commit ed45ad6)

Using a single azure account is now rejected. This commit fixes this issue and adds a test for it. This regression was introduced with #13779. Hopefully no elasticsearch version has been released since then. Backport of #15982 in 2.2 branch. (cherry picked from commit ed45ad6) (cherry picked from commit 2fdcd35)

One test we forgot in elastic#14843 and elastic#13779 is the default client selection. Most of the time, users won't define explicitly which client they want to use because they are providing only one connection to Azure storage: ```yml cloud: azure: storage: my_account: account: your_azure_storage_account key: your_azure_storage_key ``` Then using the default client like this: ```sh # This one will use the default account (my_account1) curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{ "type": "azure" }' ``` This commit adds tests to check that the right client is still selected when no client is explicitly set when creating the snapshot.

One test we forgot in #14843 and #13779 is the default client selection. Most of the time, users won't define explicitly which client they want to use because they are providing only one connection to Azure storage: ```yml cloud: azure: storage: my_account: account: your_azure_storage_account key: your_azure_storage_key ``` Then using the default client like this: ```sh # This one will use the default account (my_account1) curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{ "type": "azure" }' ``` This commit adds tests to check that the right client is still selected when no client is explicitly set when creating the snapshot. Backport of #16088 in 2.2 branch

One test we forgot in #14843 and #13779 is the default client selection. Most of the time, users won't define explicitly which client they want to use because they are providing only one connection to Azure storage: ```yml cloud: azure: storage: my_account: account: your_azure_storage_account key: your_azure_storage_key ``` Then using the default client like this: ```sh # This one will use the default account (my_account1) curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{ "type": "azure" }' ``` This commit adds tests to check that the right client is still selected when no client is explicitly set when creating the snapshot. Backport of #16088 in 2.x branch

This regression has been introduced in 2.2.0 by elastic#13779 where we removed support for `cloud.azure.storage.account` and replaced by `cloud.azure.storage.my_account.account`. But Azure plugin tries to detect when it starts if all needed settings to use an `azure` repository have been set. If not the case, the plugin does not expose `azure` as an available repository. Sadly, this check has been badly updated in 2.2.0 so it can never find the expected settings to start correctly. This gives the following effect: ```yaml cloud: azure: storage: my_account: account: your_azure_storage_account key: your_azure_storage_key ``` When you try to execute the following API you get Unknown [repository] type [azure]: ```sh [msimos@msi-gs60 elasticsearch-2.2.0]$ curl -XPUT http://localhost:9200/_snapshot/mybackup?pretty -d '{ "type": "azure" }' ``` ```js { "error" : { "root_cause" : [ { "type" : "repository_exception", "reason" : "[mybackup] failed to create repository" } ], "type" : "repository_exception", "reason" : "[mybackup] failed to create repository", "caused_by" : { "type" : "illegal_argument_exception", "reason" : "Unknown [repository] type [azure]" } }, "status" : 500 } ``` ```sh [msimos@msi-gs60 elasticsearch-2.2.0]$ curl -XGE http://localhost:9200/_cat/plugins?v name component version type url node01 cloud-azure 2.2.0 j node01 license 2.2.0 j node01 shield 2.2.0 j ``` In the elasticsearch log file you see this error: ``` [2016-02-19 10:54:47,357][DEBUG][cloud.azure ] [node01] starting azure services [2016-02-19 10:54:47,357][DEBUG][cloud.azure ] [node01] azure repository is not set using [repositories.azure.account] and [cloud.azure.storage.key] properties ``` Closes elastic#16734

This regression has been introduced in 2.2.0 by #13779 where we removed support for `cloud.azure.storage.account` and replaced by `cloud.azure.storage.my_account.account`. But Azure plugin tries to detect when it starts if all needed settings to use an `azure` repository have been set. If not the case, the plugin does not expose `azure` as an available repository. Sadly, this check has been badly updated in 2.2.0 so it can never find the expected settings to start correctly. This gives the following effect: ```yaml cloud: azure: storage: my_account: account: your_azure_storage_account key: your_azure_storage_key ``` When you try to execute the following API you get Unknown [repository] type [azure]: ```sh [msimos@msi-gs60 elasticsearch-2.2.0]$ curl -XPUT http://localhost:9200/_snapshot/mybackup?pretty -d '{ "type": "azure" }' ``` ```js { "error" : { "root_cause" : [ { "type" : "repository_exception", "reason" : "[mybackup] failed to create repository" } ], "type" : "repository_exception", "reason" : "[mybackup] failed to create repository", "caused_by" : { "type" : "illegal_argument_exception", "reason" : "Unknown [repository] type [azure]" } }, "status" : 500 } ``` ```sh [msimos@msi-gs60 elasticsearch-2.2.0]$ curl -XGE http://localhost:9200/_cat/plugins?v name component version type url node01 cloud-azure 2.2.0 j node01 license 2.2.0 j node01 shield 2.2.0 j ``` In the elasticsearch log file you see this error: ``` [2016-02-19 10:54:47,357][DEBUG][cloud.azure ] [node01] starting azure services [2016-02-19 10:54:47,357][DEBUG][cloud.azure ] [node01] azure repository is not set using [repositories.azure.account] and [cloud.azure.storage.key] properties ``` Closes #16734 (cherry picked from commit 7ffd6aa)

dadoonet added >enhancement :Plugin Cloud Azure labels Sep 24, 2015

dadoonet self-assigned this Sep 24, 2015

dadoonet force-pushed the pr/azure-multi-account branch from 69a1cf7 to edb779c Compare October 12, 2015 10:35

dadoonet force-pushed the pr/azure-multi-account branch from edb779c to 0057dee Compare October 27, 2015 10:36

dadoonet added review v2.1.0 v5.0.0-alpha1 v2.2.0 labels Oct 27, 2015

dadoonet assigned msimos and imotov and unassigned dadoonet and msimos Oct 27, 2015

dadoonet mentioned this pull request Oct 27, 2015

For Azure, support multiple storage accounts and secondary endpoints (which are readonly) #13228

Closed

imotov reviewed Nov 10, 2015
View reviewed changes

dadoonet removed the v2.1.0 label Nov 18, 2015

dadoonet force-pushed the pr/azure-multi-account branch from 4a9546b to 79a4d9c Compare November 18, 2015 15:35

dadoonet merged commit 79a4d9c into elastic:master Nov 18, 2015

dadoonet deleted the pr/azure-multi-account branch November 18, 2015 16:02

dadoonet mentioned this pull request Nov 18, 2015

Filter cloud azure credentials #14843

Closed

dadoonet mentioned this pull request Nov 19, 2015

Filter cloud azure credentials #14863

Merged

This was referenced Dec 18, 2015

[cloud-azure] Allow multiple repositories with different settings #12759

Closed

[snapshot] support multiple credentials in elasticsearch.yml #12935

Closed

dadoonet mentioned this pull request Jan 8, 2016

Remove -Xlint:-deprecation from all but core #15848

Merged

dadoonet mentioned this pull request Jan 14, 2016

Fix Azure repository with only one primary account #15982

Merged

dadoonet mentioned this pull request Jan 19, 2016

Add more tests for Azure Repository client selection #16088

Merged

dadoonet mentioned this pull request Feb 21, 2016

Fix Unknown [repository] type [azure] error with 2.2.0 #16747

Merged

dadoonet mentioned this pull request May 25, 2016

Read-Only RA-GRS Repository elastic/elasticsearch-cloud-azure#90

Closed

clintongormley added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Plugin Cloud Azure labels Feb 14, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for secondary azure storage account #13779

Add support for secondary azure storage account #13779

dadoonet commented Sep 24, 2015

craigwi commented Sep 25, 2015

dadoonet commented Oct 12, 2015

dadoonet commented Oct 27, 2015

craigwi commented Oct 27, 2015

imotov Nov 10, 2015

dadoonet Nov 18, 2015

imotov commented Nov 10, 2015

dadoonet commented Nov 18, 2015

Add support for secondary azure storage account #13779

Add support for secondary azure storage account #13779

Conversation

dadoonet commented Sep 24, 2015

craigwi commented Sep 25, 2015

dadoonet commented Oct 12, 2015

dadoonet commented Oct 27, 2015

craigwi commented Oct 27, 2015

imotov Nov 10, 2015

Choose a reason for hiding this comment

dadoonet Nov 18, 2015

Choose a reason for hiding this comment

imotov commented Nov 10, 2015

dadoonet commented Nov 18, 2015