add RemoveCorruptedShardDataCommand #32281

vladimirdolzhenko · 2018-07-23T13:28:16Z

add elasticsearch-shard command-line tool

We should have one tool for dealing with corruptions, both in the translog and in the lucene index
The tool will refuse to run if there are no existing corruption markers (i.e., it will only work on known corrupted shards)
- adds elasticsearch-shard remove-corrupted-segments tool instead of dropped in drop index.shard.check_on_startup: fix #32279 index.shard.check_on_startup: fix setting
- actually translog does not create corrupted file marker - it has to be address in following PR
The tool will first run a dry run and show an analysis of what it's going to do to the user, get confirmation and then perform required operations.
- -dry-run option could be there to provide an overview
The tool should fail when check index fails to drop corrupted segments in Lucene. In the future we can offer users to only recover the translog, if needed. We don't feel we need the complexity right now.
We should document the implication of the tool to join relationships as it may be unexpected to users.
The tool should generate a new history uuid to prevent ops based recoveries and CCR.
The tool should generate a new allocation id and tell the user what command they need to run in order for the cluster to use this shard (allocate stale primary).

elasticmachine · 2018-07-23T13:28:18Z

Pinging @elastic/es-distributed

jpountz · 2018-07-23T13:55:15Z

Such a tool would be very helpful when no sane shards are not available anymore as it is often better to lose a single segment than an entire shard! I left some comment on #31389 about the API.

Relates elastic#31389

jpountz · 2018-07-31T08:50:14Z

server/src/main/java/org/elasticsearch/index/shard/RemoveCorruptedSegmentsCommand.java

+        confirm("Continue and clean up Lucene index files ?", terminal);
+        // TODO: waiting for Lucene fixes or IndexWriter (to be able to create a new segment while other files there)
+        // or wait while Lucene CheckIndex makes files non corrupted
+        Lucene.cleanLuceneIndex(dir);


I just had a chat with @vladimirdolzhenko and dropping the whole index is probably the safest option at this point when CheckIndex.Status.missing is true

I added a link to LUCENE-6762 to the comment

…elasticsearch-shard

vladimirdolzhenko · 2018-08-20T12:46:16Z

@lcawl could you please have a look into documentation part change ?

lcawl · 2018-08-20T21:20:53Z

I've added a "Command-line tools" section in the Index Modules page, rather than including that link under the settings. @debadair will also provide some feedback.

vladimirdolzhenko · 2018-09-11T14:47:10Z

@DaveCTurner I've addressed your comments, could you please have another look ?

DaveCTurner

Minor stuff now, and I asked for another reviewer.

DaveCTurner · 2018-09-13T14:19:24Z

docs/reference/commands/shard-tool.asciidoc

+  ]
+}
+
+You must accept the possibility of data loss changing parameter `accept_data_loss` to `true`.


nits, sorry:

You must accept the possibility of data loss by changing the parameter accept_data_loss to true.

DaveCTurner · 2018-09-13T14:31:22Z

server/src/main/java/org/elasticsearch/index/shard/RemoveCorruptedShardDataCommand.java

+        for (int possibleLockId = fromNodeId; possibleLockId < toNodeId; possibleLockId++) {
+            final NodeEnvironment.NodeLock nodeLock;
+            try {
+                nodeLock = new NodeEnvironment.NodeLock(possibleLockId, logger, environment, p -> {});


Could this just be try (Releasable nodeLock = new NodeEnvironment.NodeLock(possibleLockId, logger, environment, p -> {})) {... with the catch at the bottom of this method?

Also, if the directory doesn't exist, I think that FSDirectory.open() creates it, which is not what we want here, so the consumer passed to NodeLock's constructor needs to check for this and bail out sooner. See the protected FSDirectory(Path path, LockFactory lockFactory) constructor for instance. I think this warrants a test that we don't accidentally create a bunch of directories if we can't find the index we were looking for.

DaveCTurner · 2018-09-13T15:37:30Z

server/src/main/java/org/elasticsearch/env/NodeEnvironment.java

+        private final Lock[] locks;
+        private final NodePath[] nodePaths;
+
+        public NodeLock(final int nodeId, final Logger logger,


This looks good to me but I would like another pair of eyes, as it's quite important not to break this. @ywelsch WDYT?

ywelsch

The NodeLock change looks ok to me. I would also like to revive the discussion that was initiated here about removing the max-local-storage "feature".

server/src/main/java/org/elasticsearch/env/NodeEnvironment.java

server/src/main/java/org/elasticsearch/index/shard/RemoveCorruptedShardDataCommand.java

DaveCTurner

LGTM, great work @vladimirdolzhenko

The docs are much improved (and much shorter) compared with the version that was reviewed

vladimirdolzhenko · 2018-09-19T11:40:31Z

thank you @DaveCTurner for the valuable comments and review 💯

* master: (46 commits) Fixing assertions in integration test (elastic#33833) [CCR] Rename idle_shard_retry_delay to poll_timout in auto follow patterns (elastic#33821) HLRC: Delete ML calendar (elastic#33775) Move DocsStats into Engine (elastic#33835) [Docs] Clarify accessing Date methods in painless (elastic#33560) add elasticsearch-shard tool (elastic#32281) Cut over to unwrap segment reader (elastic#33843) SQL: Fix issue with options for QUERY() and MATCH(). (elastic#33828) Emphasize that filesystem-level backups don't work (elastic#33102) Use the global doc id to generate a random score (elastic#33599) Add minimal sanity checks to custom/scripted similarities. (elastic#33564) Profiler: Don’t profile NEXTDOC for ConstantScoreQuery. (elastic#33196) [CCR] Change FollowIndexAction.Request class to be more user friendly (elastic#33810) SQL: day and month name functions tests locale providers enforcement (elastic#33653) TESTS: Set SO_LINGER = 0 for MockNioTransport (elastic#32560) Test: Relax jarhell gradle test (elastic#33787) [CCR] Fail with a descriptive error if leader index does not exist (elastic#33797) Add ES version 6.4.2 (elastic#33831) MINOR: Remove Some Dead Code in Scripting (elastic#33800) Ensure realtime `_get` and `_termvectors` don't run on the network thread (elastic#33814) ...

Relates elastic#31389 (cherry picked from commit a3e8b83)

#32281 adds elasticsearch-shard to provide bwc version of elasticsearch-translog for 6.x; have to remove elasticsearch-translog for 7.0 Relates to #31389

vladimirdolzhenko added >enhancement WIP :Distributed/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. v7.0.0 labels Jul 23, 2018

drop index.shard.check_on_startup: fix

843f977

Relates elastic#31389

vladimirdolzhenko force-pushed the fix/31389_1 branch from 5828119 to 843f977 Compare July 27, 2018 15:21

vladimirdolzhenko force-pushed the fix/31389_2 branch 4 times, most recently from 183b3da to 4116084 Compare July 30, 2018 12:38

jpountz reviewed Jul 31, 2018

View reviewed changes

vladimirdolzhenko force-pushed the fix/31389_2 branch from 4116084 to 2f116d8 Compare July 31, 2018 09:26

Merge remote-tracking branch 'remotes/origin/master' into fix/31389_1

4f01609

vladimirdolzhenko force-pushed the fix/31389_2 branch 7 times, most recently from 9b6aece to 8e0891f Compare August 20, 2018 12:19

add RemoveCorruptedSegmentsCommand; merge elasticsearch-translog and …

a8f1488

…elasticsearch-shard

vladimirdolzhenko force-pushed the fix/31389_2 branch from 8e0891f to a8f1488 Compare August 20, 2018 12:26

vladimirdolzhenko removed the WIP label Aug 20, 2018

vladimirdolzhenko requested review from DaveCTurner and lcawl August 20, 2018 12:36

fix test with ClusterAllocationExplanation

5f6b084

Vladimir Dolzhenko added 3 commits September 10, 2018 14:27

dropped unrelated checkIndexOnStartup = fix setting

674d1ba

nodeEnv code style clean up

53b404a

do not expose node lock; code style adjustment; text comment adjustment

24ffdd1

vladimirdolzhenko requested a review from DaveCTurner September 10, 2018 15:25

Merge remote-tracking branch 'remotes/origin/master' into fix/31389_2

2a3f58d

DaveCTurner reviewed Sep 13, 2018

View reviewed changes

Vladimir Dolzhenko added 5 commits September 13, 2018 20:15

tiny doc amendment

e1eb32f

NodeEnvironment.NodeLock can skip node path if it is required

ee1f6a2

Merge remote-tracking branch 'remotes/origin/master' into fix/31389_2

1df4685

after merge fix

844adaf

after merge fix

8210f3b

ywelsch reviewed Sep 18, 2018

View reviewed changes

server/src/main/java/org/elasticsearch/env/NodeEnvironment.java Outdated Show resolved Hide resolved

server/src/main/java/org/elasticsearch/index/shard/RemoveCorruptedShardDataCommand.java Outdated Show resolved Hide resolved

Vladimir Dolzhenko added 2 commits September 18, 2018 12:34

inline nodeLock

dab5125

add javadoc comment for pathFunction

54c4030

vladimirdolzhenko requested a review from DaveCTurner September 18, 2018 10:48

DaveCTurner approved these changes Sep 19, 2018

View reviewed changes

vladimirdolzhenko merged commit a3e8b83 into elastic:master Sep 19, 2018

vladimirdolzhenko mentioned this pull request Sep 19, 2018

add elasticsearch-shard tool to 6.x #33848

Merged

vladimirdolzhenko added a commit to vladimirdolzhenko/elasticsearch that referenced this pull request Sep 19, 2018

add elasticsearch-shard tool (elastic#32281)

f2570f4

Relates elastic#31389 (cherry picked from commit a3e8b83)

vladimirdolzhenko deleted the fix/31389_2 branch September 22, 2018 10:39

vladimirdolzhenko added a commit that referenced this pull request Oct 1, 2018

drop elasticsearch-translog for 7.0 (#33373)

2e2ae19

#32281 adds elasticsearch-shard to provide bwc version of elasticsearch-translog for 6.x; have to remove elasticsearch-translog for 7.0 Relates to #31389

kcm pushed a commit that referenced this pull request Oct 30, 2018

drop elasticsearch-translog for 7.0 (#33373)

aeba58a

#32281 adds elasticsearch-shard to provide bwc version of elasticsearch-translog for 6.x; have to remove elasticsearch-translog for 7.0 Relates to #31389

DaveCTurner mentioned this pull request Jan 7, 2019

Tool to remove corrupted parts of corrupt shards #31389

Closed

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add RemoveCorruptedShardDataCommand #32281

add RemoveCorruptedShardDataCommand #32281

vladimirdolzhenko commented Jul 23, 2018 •

edited

Loading

elasticmachine commented Jul 23, 2018

jpountz commented Jul 23, 2018

jpountz Jul 31, 2018

vladimirdolzhenko Jul 31, 2018

vladimirdolzhenko commented Aug 20, 2018

lcawl commented Aug 20, 2018

vladimirdolzhenko commented Sep 11, 2018

DaveCTurner left a comment

DaveCTurner Sep 13, 2018

DaveCTurner Sep 13, 2018

vladimirdolzhenko Sep 13, 2018

DaveCTurner Sep 13, 2018

ywelsch left a comment

DaveCTurner left a comment

vladimirdolzhenko commented Sep 19, 2018

add RemoveCorruptedShardDataCommand #32281

add RemoveCorruptedShardDataCommand #32281

Conversation

vladimirdolzhenko commented Jul 23, 2018 • edited Loading

elasticmachine commented Jul 23, 2018

jpountz commented Jul 23, 2018

jpountz Jul 31, 2018

Choose a reason for hiding this comment

vladimirdolzhenko Jul 31, 2018

Choose a reason for hiding this comment

vladimirdolzhenko commented Aug 20, 2018

lcawl commented Aug 20, 2018

vladimirdolzhenko commented Sep 11, 2018

DaveCTurner left a comment

Choose a reason for hiding this comment

DaveCTurner Sep 13, 2018

Choose a reason for hiding this comment

DaveCTurner Sep 13, 2018

Choose a reason for hiding this comment

vladimirdolzhenko Sep 13, 2018

Choose a reason for hiding this comment

DaveCTurner Sep 13, 2018

Choose a reason for hiding this comment

ywelsch left a comment

Choose a reason for hiding this comment

DaveCTurner left a comment

Choose a reason for hiding this comment

vladimirdolzhenko commented Sep 19, 2018

vladimirdolzhenko commented Jul 23, 2018 •

edited

Loading