S3/Azure IT should use ESBlobStoreRepositoryIntegTestCase #16472
Assignees
Labels
:Distributed Coordination/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
>test
Issues or PRs that are addressing/adding tests
Comments
dadoonet
added
>test
dadoonet
added a commit
to dadoonet/elasticsearch
that referenced
this issue
May 18, 2016
Probably when we updated Azure SDK, we introduced a regression.
Actually, we are not able to remove files anymore.
For example, if you register a new azure repository, the snapshot service tries to create a temp file and then remove it.
Removing does not work and you can see in logs:
```
[2016-05-18 11:03:24,914][WARN ][org.elasticsearch.cloud.azure.blobstore] [azure] can not remove [tests-ilmRPJ8URU-sh18yj38O6g/] in container {elasticsearch-snapshots}: The specified blob does not exist.
```
This fix deals with that. It now list all the files in a flatten mode, remove in the full URL the server and the container name.
As an example, when you are removing a blob which full name is `https://dpi24329.blob.core.windows.net/elasticsearch-snapshots/bar/test` you need to actually call Azure SDK with `bar/test` as the path, `elasticsearch-snapshots` is the container.
To run the test, you need to pass some parameters: `-Dtests.thirdparty=true -Dtests.config=/path/to/elasticsearch.yml`
Where `elasticsearch.yml` contains something like:
```
cloud.azure.storage.default.account: account
cloud.azure.storage.default.key: key
```
Related to elastic#16472
Closes elastic#18436.
dadoonet
added a commit
to dadoonet/elasticsearch
that referenced
this issue
May 25, 2016
Probably when we updated Azure SDK, we introduced a regression.
Actually, we are not able to remove files anymore.
For example, if you register a new azure repository, the snapshot service tries to create a temp file and then remove it.
Removing does not work and you can see in logs:
```
[2016-05-18 11:03:24,914][WARN ][org.elasticsearch.cloud.azure.blobstore] [azure] can not remove [tests-ilmRPJ8URU-sh18yj38O6g/] in container {elasticsearch-snapshots}: The specified blob does not exist.
```
This fix deals with that. It now list all the files in a flatten mode, remove in the full URL the server and the container name.
As an example, when you are removing a blob which full name is `https://dpi24329.blob.core.windows.net/elasticsearch-snapshots/bar/test` you need to actually call Azure SDK with `bar/test` as the path, `elasticsearch-snapshots` is the container.
Related to elastic#16472.
Related to elastic#18436.
Backport of elastic#18451 in 2.x branch
To test it, I ran some manual tests:
On my laptop, create a file `/path/to/azure/config/elasticsearch.yml`:
```yml
cloud.azure.storage.default.account: ACCOUNT
cloud.azure.storage.default.key: KEY
```
Run `AzureRepositoryF#main()` with `-Des.cluster.routing.allocation.disk.threshold_enabled=false -Des.path.home=/path/to/azure/` options.
Then run:
```sh
curl -XDELETE localhost:9200/foo?pretty
curl -XDELETE localhost:9200/_snapshot/my_backup1?pretty
curl -XPUT localhost:9200/foo/bar/1?pretty -d '{
"foo": "bar"
}'
curl -XPOST localhost:9200/foo/_refresh?pretty
curl -XGET localhost:9200/foo/_count?pretty
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
"type": "azure"
}'
curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1?pretty&wait_for_completion=true"
curl -XDELETE localhost:9200/foo?pretty
curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1/_restore?pretty&wait_for_completion=true"
curl -XGET localhost:9200/foo/_count?pretty
```
Then check files we have on azure platform using the console.
Then run:
```
curl -XDELETE localhost:9200/_snapshot/my_backup1/snap1?pretty
```
Then check files we have on azure platform using the console and verify that everything has been cleaned.
dadoonet
added a commit
to dadoonet/elasticsearch-cloud-azure
that referenced
this issue
May 25, 2016
Probably when we updated Azure SDK, we introduced a regression.
Actually, we are not able to remove files anymore.
For example, if you register a new azure repository, the snapshot service tries to create a temp file and then remove it.
Removing does not work and you can see in logs:
```
[2016-05-18 11:03:24,914][WARN ][org.elasticsearch.cloud.azure.blobstore] [azure] can not remove [tests-ilmRPJ8URU-sh18yj38O6g/] in container {elasticsearch-snapshots}: The specified blob does not exist.
```
This fix deals with that. It now list all the files in a flatten mode, remove in the full URL the server and the container name.
As an example, when you are removing a blob which full name is `https://dpi24329.blob.core.windows.net/elasticsearch-snapshots/bar/test` you need to actually call Azure SDK with `bar/test` as the path, `elasticsearch-snapshots` is the container.
Related to elastic/elasticsearch#16472.
Related to elastic/elasticsearch#18436.
Backport of elastic/elasticsearch#18451 for 1.7 series
clintongormley
added
:Distributed Coordination/Snapshot/Restore
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this issue
Apr 5, 2018
The test framework contains a base class for testing blob store repository implementations, but the S3 plugin does not use it. This commit adds the S3BlobStoreRepositoryTests class that extends the base testing class for S3. It also cleans up the S3BlobStoreTests and S3BlobStoreContainerTests so that they are now based on pure mock S3 clients. It also removes some usage of socket servers that emulate socket connections in unit tests. It was added to trigger security exceptions, but this won't be needed anymore once elastic#29296 will be merged. closes elastic#16472
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
As we now have #14050, we can try use it for Azure and S3 Integration tests as well.
See for example: https://github.com/elastic/elasticsearch/blob/master/plugins/repository-s3/src/test/java/org/elasticsearch/repositories/s3/AbstractS3SnapshotRestoreTest.java
The text was updated successfully, but these errors were encountered: