Dangling indices documentation #58751

pugnascotia · 2020-06-30T14:00:40Z

Part of #48366. This PR adds documentation for the dangling indices API added in #58176.

Further docs fixes. Update gateways docs to cover the new `gateway.gateway.auto_import_dangling_indices` setting, and rework text since the default behaviour is now not to import automatically.

…docs

elasticmachine · 2020-06-30T14:00:42Z

Pinging @elastic/es-docs (>docs)

elasticmachine · 2020-06-30T14:00:44Z

Pinging @elastic/es-distributed (:Distributed/Distributed)

DaveCTurner · 2020-07-01T07:06:17Z

The test failures look related, holding off on a proper review for now.

…docs

pugnascotia · 2020-07-01T10:15:34Z

@elasticmachine run elasticsearch-ci/docs

pugnascotia · 2020-07-01T12:10:29Z

@DaveCTurner CI is now green.

DaveCTurner · 2020-07-01T13:57:31Z

We discussed this in today's distrib team sync and agreed to default auto-importing to false in 7.x, although would rather not do so in the same PR as the one that adds the API docs. @pugnascotia would you split this into two PRs?

A common (unsafe) source of dangling indices is using elasticsearch-node detach-cluster; we should clarify in the docs for that tool, and its output, that users will need to manually import any dangling indices after using it.

pugnascotia · 2020-07-01T19:06:47Z

Yes, on reflection I should have raised a separate PR - curse my laziness 😅

pugnascotia · 2020-07-02T10:40:07Z

I split the server changes into #58898.

…docs

DaveCTurner

Looks good, I left a few nits.

docs/reference/indices/dangling-index-delete.asciidoc

docs/reference/indices/dangling-index-import.asciidoc

docs/reference/indices/dangling-indices-list.asciidoc

docs/reference/modules/cluster/misc.asciidoc

Co-authored-by: David Turner <david.turner@elastic.co>

DaveCTurner

LGTM; I believe @lockewritesdocs is also taking a look too.

lockewritesdocs

Overall, LGTM. I left some editorial comments that you can choose to incorporate.

One suggestion: consider adding an Examples section to the Import dangling index API page. This addition provides users with a more comprehensive example and shows the response structure.

docs/reference/indices/dangling-indices-list.asciidoc

docs/reference/indices/dangling-index-import.asciidoc

docs/reference/modules/cluster/misc.asciidoc

docs/reference/indices/dangling-index-delete.asciidoc

docs/reference/indices/dangling-index-import.asciidoc

docs/reference/modules/gateway.asciidoc

ywelsch

I've left two small comments, places where I think we should add a few more clarifications. Thank you

ywelsch · 2020-07-03T06:37:40Z

docs/reference/modules/gateway.asciidoc

+When a node joins the cluster, if it finds any shards stored in its local data
+directory that do not already exist in the cluster, it will consider those
+shards to be "dangling". You can automatically import dangling indices
+into the cluster using the following setting, however this is not safe, and


I'm a bit confused by the "this is not safe". Without further qualification, this would leave a reader puzzled as well.

ywelsch · 2020-07-03T06:42:58Z

docs/reference/modules/gateway.asciidoc

+    to `false`.
+
+WARNING: The auto-import functionality was intended as a best effort to help users
+who lose all master nodes. For example, if a new master node were to be


I'm wondering if we should add that that scenario (losing all master nodes) should be avoided at all cost, as it puts the cluster's metadata and data at risk. Also worth pointing out above (when talking about "this is not safe") that importing dangling indices does not provide any guarantees as to whether the data that's imported truly represents the latest state of the data as it was when the index was still part of the cluster.

@ywelsch I've reworked the text to and address your points.

Co-authored-by: Adam Locke <adam.locke@elastic.co>

pugnascotia · 2020-07-03T10:22:14Z

docs/reference/modules/gateway.asciidoc

+its use is strongly discouraged in favour of the
+<<dangling-indices-api,dedicated API>.
+
+WARNING: Losing all master nodes is a situation that should be avoided at


@lockewritesdocs Is it overkill to have two WARNING: sections? I felt that they were making separate points, and all the content here ought to be scary.

ywelsch

LGTM. Can you re-request a review next time please (click button next to my name in Github UI)? The PR will then appear in my list of PRs to review. Makes it easier to stay on top of things.

Part of #48366. Add documentation for the dangling indices API added in #58176. Co-authored-by: David Turner <david.turner@elastic.co> Co-authored-by: Adam Locke <adam.locke@elastic.co>

pugnascotia · 2020-07-09T13:03:36Z

Backported to 7.9 in 5debd09

pugnascotia added 4 commits June 24, 2020 13:10

Initial dangling indices docs

6fc5002

Fixes and polishing

2dce37c

Disable auto_import_dangling_indices by default

02ee01d

Further docs fixes. Update gateways docs to cover the new `gateway.gateway.auto_import_dangling_indices` setting, and rework text since the default behaviour is now not to import automatically.

Merge remote-tracking branch 'upstream/master' into dangling-indices-…

27f4bbc

…docs

pugnascotia added >enhancement >docs General docs changes >breaking :Distributed Indexing/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. labels Jun 30, 2020

pugnascotia requested a review from DaveCTurner June 30, 2020 14:00

elasticmachine added the Team:Docs Meta label for docs team label Jun 30, 2020

elasticmachine added the Team:Distributed Meta label for distributed team (obsolete) label Jun 30, 2020

pugnascotia added 2 commits July 1, 2020 10:31

Fixes

e8470f4

Merge remote-tracking branch 'upstream/master' into dangling-indices-…

4b1ab53

…docs

DaveCTurner added the team-discuss label Jul 1, 2020

pugnascotia mentioned this pull request Jul 2, 2020

Default gateway.auto_import_dangling_indices to false #58898

Merged

pugnascotia changed the title ~~Dangling indices documentation and default auto imports to false~~ Dangling indices documentation Jul 2, 2020

pugnascotia removed >breaking >enhancement labels Jul 2, 2020

pugnascotia marked this pull request as draft July 2, 2020 10:38

pugnascotia added 2 commits July 2, 2020 11:40

Revert server changes

df0943f

Merge remote-tracking branch 'upstream/master' into dangling-indices-…

aefca75

…docs

pugnascotia marked this pull request as ready for review July 2, 2020 10:42

DaveCTurner reviewed Jul 2, 2020

View reviewed changes

Apply suggestions from code review

a625812

Co-authored-by: David Turner <david.turner@elastic.co>

DaveCTurner approved these changes Jul 2, 2020

View reviewed changes

lockewritesdocs approved these changes Jul 2, 2020

View reviewed changes

ywelsch reviewed Jul 3, 2020

View reviewed changes

pugnascotia and others added 6 commits July 3, 2020 09:53

Apply suggestions from code review

8135376

Co-authored-by: Adam Locke <adam.locke@elastic.co>

Reference snippet instead of duplicating content

5188297

Strengthen warnings around dangling index imports

5bd84b5

Add an example of importing an dangling index

873d865

Add missing params to delete page

6c931a8

Typo

f0b53eb

pugnascotia commented Jul 3, 2020

View reviewed changes

ywelsch approved these changes Jul 3, 2020

View reviewed changes

DaveCTurner removed the team-discuss label Jul 3, 2020

pugnascotia merged commit 3f51fed into elastic:master Jul 3, 2020

pugnascotia deleted the dangling-indices-docs branch July 3, 2020 15:28

pugnascotia added v7.9.0 v8.0.0 labels Jul 9, 2020

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dangling indices documentation #58751

Dangling indices documentation #58751

pugnascotia commented Jun 30, 2020 •

edited

Loading

elasticmachine commented Jun 30, 2020

elasticmachine commented Jun 30, 2020

DaveCTurner commented Jul 1, 2020

pugnascotia commented Jul 1, 2020

pugnascotia commented Jul 1, 2020

DaveCTurner commented Jul 1, 2020

pugnascotia commented Jul 1, 2020

pugnascotia commented Jul 2, 2020

DaveCTurner left a comment

DaveCTurner left a comment

lockewritesdocs left a comment •

edited

Loading

ywelsch left a comment

ywelsch Jul 3, 2020

ywelsch Jul 3, 2020

pugnascotia Jul 3, 2020

pugnascotia Jul 3, 2020

ywelsch left a comment

pugnascotia commented Jul 9, 2020

Dangling indices documentation #58751

Dangling indices documentation #58751

Conversation

pugnascotia commented Jun 30, 2020 • edited Loading

elasticmachine commented Jun 30, 2020

elasticmachine commented Jun 30, 2020

DaveCTurner commented Jul 1, 2020

pugnascotia commented Jul 1, 2020

pugnascotia commented Jul 1, 2020

DaveCTurner commented Jul 1, 2020

pugnascotia commented Jul 1, 2020

pugnascotia commented Jul 2, 2020

DaveCTurner left a comment

Choose a reason for hiding this comment

DaveCTurner left a comment

Choose a reason for hiding this comment

lockewritesdocs left a comment • edited Loading

Choose a reason for hiding this comment

ywelsch left a comment

Choose a reason for hiding this comment

ywelsch Jul 3, 2020

Choose a reason for hiding this comment

ywelsch Jul 3, 2020

Choose a reason for hiding this comment

pugnascotia Jul 3, 2020

Choose a reason for hiding this comment

pugnascotia Jul 3, 2020

Choose a reason for hiding this comment

ywelsch left a comment

Choose a reason for hiding this comment

pugnascotia commented Jul 9, 2020

pugnascotia commented Jun 30, 2020 •

edited

Loading

lockewritesdocs left a comment •

edited

Loading