-
Notifications
You must be signed in to change notification settings - Fork 24.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Delete shard store files before restoring a snapshot (#27476)
Pull request #20220 added a change where the store files that have the same name but are different from the ones in the snapshot are deleted first before the snapshot is restored. This logic was based on the `Store.RecoveryDiff.different` set of files which works by computing a diff between an existing store and a snapshot. This works well when the files on the filesystem form valid shard store, ie there's a `segments` file and store files are not corrupted. Otherwise, the existing store's snapshot metadata cannot be read (using Store#snapshotStoreMetadata()) and an exception is thrown (CorruptIndexException, IndexFormatTooOldException etc) which is later caught as the begining of the restore process (see RestoreContext#restore()) and is translated into an empty store metadata (Store.MetadataSnapshot.EMPTY). This will make the deletion of different files introduced in #20220 useless as the set of files will always be empty even when store files exist on the filesystem. And if some files are present within the store directory, then restoring a snapshot with files with same names will fail with a FileAlreadyExistException. This is part of the #26865 issue. There are various cases were some files could exist in the store directory before a snapshot is restored. One that Igor identified is a restore attempt that failed on a node and only first files were restored, then the shard is allocated again to the same node and the restore starts again (but fails because of existing files). Another one is when some files of a closed index are corrupted / deleted and the index is restored. This commit adds a test that uses the infrastructure provided by IndexShardTestCase in order to test that restoring a shard succeed even when files with same names exist on filesystem. Related to #26865
- Loading branch information
Showing
5 changed files
with
209 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
142 changes: 142 additions & 0 deletions
142
...c/test/java/org/elasticsearch/repositories/blobstore/BlobStoreRepositoryRestoreTests.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| /* | ||
| * Licensed to Elasticsearch under one or more contributor | ||
| * license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright | ||
| * ownership. Elasticsearch licenses this file to you under | ||
| * the Apache License, Version 2.0 (the "License"); you may | ||
| * not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, | ||
| * software distributed under the License is distributed on an | ||
| * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| * KIND, either express or implied. See the License for the | ||
| * specific language governing permissions and limitations | ||
| * under the License. | ||
| */ | ||
|
|
||
| package org.elasticsearch.repositories.blobstore; | ||
|
|
||
| import org.apache.lucene.store.Directory; | ||
| import org.apache.lucene.util.IOUtils; | ||
| import org.apache.lucene.util.TestUtil; | ||
| import org.elasticsearch.cluster.metadata.RepositoryMetaData; | ||
| import org.elasticsearch.cluster.routing.ShardRouting; | ||
| import org.elasticsearch.cluster.routing.ShardRoutingHelper; | ||
| import org.elasticsearch.common.UUIDs; | ||
| import org.elasticsearch.common.settings.Settings; | ||
| import org.elasticsearch.env.Environment; | ||
| import org.elasticsearch.index.shard.IndexShard; | ||
| import org.elasticsearch.index.shard.IndexShardState; | ||
| import org.elasticsearch.index.shard.IndexShardTestCase; | ||
| import org.elasticsearch.index.shard.ShardId; | ||
| import org.elasticsearch.index.store.Store; | ||
| import org.elasticsearch.index.store.StoreFileMetaData; | ||
| import org.elasticsearch.repositories.IndexId; | ||
| import org.elasticsearch.repositories.Repository; | ||
| import org.elasticsearch.repositories.fs.FsRepository; | ||
| import org.elasticsearch.snapshots.Snapshot; | ||
| import org.elasticsearch.snapshots.SnapshotId; | ||
|
|
||
| import java.io.IOException; | ||
| import java.nio.file.Files; | ||
| import java.nio.file.Path; | ||
| import java.util.Arrays; | ||
| import java.util.List; | ||
|
|
||
| import static org.elasticsearch.cluster.routing.RecoverySource.StoreRecoverySource.EXISTING_STORE_INSTANCE; | ||
|
|
||
| /** | ||
| * This class tests the behavior of {@link BlobStoreRepository} when it | ||
| * restores a shard from a snapshot but some files with same names already | ||
| * exist on disc. | ||
| */ | ||
| public class BlobStoreRepositoryRestoreTests extends IndexShardTestCase { | ||
|
|
||
| /** | ||
| * Restoring a snapshot that contains multiple files must succeed even when | ||
| * some files already exist in the shard's store. | ||
| */ | ||
| public void testRestoreSnapshotWithExistingFiles() throws IOException { | ||
| final IndexId indexId = new IndexId(randomAlphaOfLength(10), UUIDs.randomBase64UUID()); | ||
| final ShardId shardId = new ShardId(indexId.getName(), indexId.getId(), 0); | ||
|
|
||
| IndexShard shard = newShard(shardId, true); | ||
| try { | ||
| // index documents in the shards | ||
| final int numDocs = scaledRandomIntBetween(1, 500); | ||
| recoverShardFromStore(shard); | ||
| for (int i = 0; i < numDocs; i++) { | ||
| indexDoc(shard, "doc", Integer.toString(i)); | ||
| if (rarely()) { | ||
| flushShard(shard, false); | ||
| } | ||
| } | ||
| assertDocCount(shard, numDocs); | ||
|
|
||
| // snapshot the shard | ||
| final Repository repository = createRepository(); | ||
| final Snapshot snapshot = new Snapshot(repository.getMetadata().name(), new SnapshotId(randomAlphaOfLength(10), "_uuid")); | ||
| snapshotShard(shard, snapshot, repository); | ||
|
|
||
| // capture current store files | ||
| final Store.MetadataSnapshot storeFiles = shard.snapshotStoreMetadata(); | ||
| assertFalse(storeFiles.asMap().isEmpty()); | ||
|
|
||
| // close the shard | ||
| closeShards(shard); | ||
|
|
||
| // delete some random files in the store | ||
| List<String> deletedFiles = randomSubsetOf(randomIntBetween(1, storeFiles.size() - 1), storeFiles.asMap().keySet()); | ||
| for (String deletedFile : deletedFiles) { | ||
| Files.delete(shard.shardPath().resolveIndex().resolve(deletedFile)); | ||
| } | ||
|
|
||
| // build a new shard using the same store directory as the closed shard | ||
| ShardRouting shardRouting = ShardRoutingHelper.initWithSameId(shard.routingEntry(), EXISTING_STORE_INSTANCE); | ||
| shard = newShard(shardRouting, shard.shardPath(), shard.indexSettings().getIndexMetaData(), null, null, () -> {}); | ||
|
|
||
| // restore the shard | ||
| recoverShardFromSnapshot(shard, snapshot, repository); | ||
|
|
||
| // check that the shard is not corrupted | ||
| TestUtil.checkIndex(shard.store().directory()); | ||
|
|
||
| // check that all files have been restored | ||
| final Directory directory = shard.store().directory(); | ||
| final List<String> directoryFiles = Arrays.asList(directory.listAll()); | ||
|
|
||
| for (StoreFileMetaData storeFile : storeFiles) { | ||
| String fileName = storeFile.name(); | ||
| assertTrue("File [" + fileName + "] does not exist in store directory", directoryFiles.contains(fileName)); | ||
| assertEquals(storeFile.length(), shard.store().directory().fileLength(fileName)); | ||
| } | ||
| } finally { | ||
| if (shard != null && shard.state() != IndexShardState.CLOSED) { | ||
| try { | ||
| shard.close("test", false); | ||
| } finally { | ||
| IOUtils.close(shard.store()); | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| /** Create a {@link Repository} with a random name **/ | ||
| private Repository createRepository() throws IOException { | ||
| Settings settings = Settings.builder().put("location", randomAlphaOfLength(10)).build(); | ||
| RepositoryMetaData repositoryMetaData = new RepositoryMetaData(randomAlphaOfLength(10), FsRepository.TYPE, settings); | ||
| return new FsRepository(repositoryMetaData, createEnvironment(), xContentRegistry()); | ||
| } | ||
|
|
||
| /** Create a {@link Environment} with random path.home and path.repo **/ | ||
| private Environment createEnvironment() { | ||
| Path home = createTempDir(); | ||
| return new Environment(Settings.builder() | ||
| .put(Environment.PATH_HOME_SETTING.getKey(), home.toAbsolutePath()) | ||
| .put(Environment.PATH_REPO_SETTING.getKey(), home.resolve("repo").toAbsolutePath()) | ||
| .build()); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters