Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trim unreferenced translog when the safe commit advanced #32967

Merged
merged 4 commits into from
Aug 20, 2018
Merged

Trim unreferenced translog when the safe commit advanced #32967

merged 4 commits into from
Aug 20, 2018

Conversation

dnhatn
Copy link
Member

@dnhatn dnhatn commented Aug 17, 2018

Since #28140 when the global checkpoint is advanced, we try to move the
safe commit forward, and clean up old index commits if possible. However,
we forget to trim unreferenced translog.

This change makes sure that we prune both old translog and index commits
when the safe commit advanced.

Relates #28140
Closes #32089

/cc @bleskes

@dnhatn
Trim unreferenced translog the safe commit advanced
a8594ac 
Since elastic#28140  when the global checkpoint is advanced, we try to move the
safe commit forward, and clean old index commits if possible. However,
we forget to trim unreferenced translog.

This change makes sure that we prune both translog and index commits
when the safe commit advanced.

Relates elastic#28140
Closes elastic#32089
@dnhatn dnhatn added >bug v7.0.0 :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. v6.5.0 v6.3.3 v6.4.1 labels Aug 17, 2018
@dnhatn dnhatn requested review from s1monw and ywelsch August 17, 2018 21:19
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@dnhatn dnhatn mentioned this pull request Aug 17, 2018
Closed
ywelsch
ywelsch approved these changes Aug 20, 2018
View reviewed changes
Copy link
Contributor

@ywelsch ywelsch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I checked for other callers of indexWriter.deleteUnusedFiles(), and there is one more, namely releaseIndexCommit. Should we also do the trimming there?

s1monw
s1monw approved these changes Aug 20, 2018
View reviewed changes
Copy link
Contributor

@s1monw s1monw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I checked for other callers of indexWriter.deleteUnusedFiles(), and there is one more, namely releaseIndexCommit. Should we also do the trimming there?

+1

LGTM 2

@dnhatn
Copy link
Member Author

dnhatn commented Aug 20, 2018

Thanks @ywelsch and @s1monw for reviewing (and @ywelsch for an extra look).

@dnhatn dnhatn merged commit 40f1bb5 into elastic:master Aug 20, 2018
@dnhatn dnhatn deleted the trim-translog-gcp branch August 20, 2018 19:13
dnhatn added a commit that referenced this pull request Aug 20, 2018
@dnhatn
Trim translog when safe commit advanced (#32967)
712862c 
Since #28140 when the global checkpoint is advanced, we try to move the
safe commit forward, and clean up old index commits if possible. However,
we forget to trim unreferenced translog.

This change makes sure that we prune both old translog and index commits
when the safe commit advanced.

Relates #28140
Closes #32089
dnhatn added a commit that referenced this pull request Aug 20, 2018
@dnhatn
Trim translog when safe commit advanced (#32967)
bbdcd5a 
Since #28140 when the global checkpoint is advanced, we try to move the
safe commit forward, and clean up old index commits if possible. However,
we forget to trim unreferenced translog.

This change makes sure that we prune both old translog and index commits
when the safe commit advanced.

Relates #28140
Closes #32089
jasontedor added a commit that referenced this pull request Aug 20, 2018
@jasontedor
Merge branch 'master' into ccr
853eb1c 
* master:
  Generalize remote license checker (#32971)
  Trim translog when safe commit advanced (#32967)
  Fix an inaccuracy in the dynamic templates documentation. (#32890)
  Logging: Use settings when building daemon threads (#32751)
  All Translog inner closes should happen after tragedy exception is set (#32674)
  HLREST: AwaitsFix ML Test
  Pass DiscoveryNode to initiateChannel (#32958)
  Add mzn and dz to unsupported locales (#32957)
  Use settings from the context in BootstrapChecks (#32908)
  Update docs for node specifications (#30468)
  HLRC: Forbid all Elasticsearch logging infra (#32784)
  Only configure publishing if it's applied externally (#32351)
  Fixes libs:dissect when in eclipse
  Protect ScriptedMetricIT test cases against failures on 0-doc shards (#32959) (#32968)
  [Kerberos] Add documentation for Kerberos realm (#32662)
  Watcher: Properly find next valid date in cron expressions (#32734)
  Fix some small issues in the getting started docs (#30346)
  Set forbidden APIs target compatibility to compiler java version   (#32935)
  Move connection listener to ConnectionManager (#32956)
dnhatn added a commit that referenced this pull request Aug 20, 2018
@dnhatn
Trim translog when safe commit advanced (#32967)
fc5bdbc 
Since #28140 when the global checkpoint is advanced, we try to move the
safe commit forward, and clean up old index commits if possible. However,
we forget to trim unreferenced translog.

This change makes sure that we prune both old translog and index commits
when the safe commit advanced.

Relates #28140
Closes #32089
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Aug 21, 2018
@jasontedor
Merge remote-tracking branch 'elastic/master' into allowed-index-stor…
be8ce3b 
…e-types

* elastic/master: (89 commits)
  Fix assertion in AbstractSimpleTransportTestCase (elastic#32991)
  [DOC] Splits role mapping APIs into separate pages (elastic#32797)
  HLRC: ML Close Job (elastic#32943)
  Generalize remote license checker (elastic#32971)
  Trim translog when safe commit advanced (elastic#32967)
  Fix an inaccuracy in the dynamic templates documentation. (elastic#32890)
  Logging: Use settings when building daemon threads (elastic#32751)
  All Translog inner closes should happen after tragedy exception is set (elastic#32674)
  HLREST: AwaitsFix ML Test
  Pass DiscoveryNode to initiateChannel (elastic#32958)
  Add mzn and dz to unsupported locales (elastic#32957)
  Use settings from the context in BootstrapChecks (elastic#32908)
  Update docs for node specifications (elastic#30468)
  HLRC: Forbid all Elasticsearch logging infra (elastic#32784)
  Only configure publishing if it's applied externally (elastic#32351)
  Fixes libs:dissect when in eclipse
  Protect ScriptedMetricIT test cases against failures on 0-doc shards (elastic#32959) (elastic#32968)
  [Kerberos] Add documentation for Kerberos realm (elastic#32662)
  Watcher: Properly find next valid date in cron expressions (elastic#32734)
  Fix some small issues in the getting started docs (elastic#30346)
  ...
jasontedor pushed a commit that referenced this pull request Aug 21, 2018
@dnhatn @jasontedor
Trim translog when safe commit advanced (#32967)
55cff96 
Since #28140 when the global checkpoint is advanced, we try to move the
safe commit forward, and clean up old index commits if possible. However,
we forget to trim unreferenced translog.

This change makes sure that we prune both old translog and index commits
when the safe commit advanced.

Relates #28140
Closes #32089
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request Aug 24, 2018
@jasontedor
Merge branch '6.x' into ccr-6.x
d894679 
* commit '9088d811ce9cff922e6ef1befbeb0f1e0c27016a': (23 commits)
  Generalize remote license checker (elastic#32971)
  Trim translog when safe commit advanced (elastic#32967)
  Fix an inaccuracy in the dynamic templates documentation. (elastic#32890)
  HLREST: AwaitsFix ML Test
  Make Geo Context Mapping Parsing More Strict (elastic#32862)
  Add mzn and dz to unsupported locales (elastic#32957)
  Use settings from the context in BootstrapChecks (elastic#32908)
  Update docs for node specifications (elastic#30468)
  HLRC: Forbid all Elasticsearch logging infra (elastic#32784)
  Fix use of deprecated apis
  Only configure publishing if it's applied externally (elastic#32351)
  Protect ScriptedMetricIT test cases against failures on 0-doc shards (elastic#32959) (elastic#32968)
  Scripted metric aggregations: add deprecation warning and system (elastic#32944)
  Watcher: Properly find next valid date in cron expressions (elastic#32734)
  Fix some small issues in the getting started docs (elastic#30346)
  Set forbidden APIs target compatibility to compiler java version   (elastic#32935)
  [TEST] Add "ne" as an unsupported SimpleKdc locale (elastic#32700)
  MINOR: Remove `IndexTemplateFilter` (elastic#32841) (elastic#32974)
  INGEST: Create Index Before Pipeline Execute (elastic#32786) (elastic#32975)
  NETWORKING: Make RemoteClusterConn. Lazy Resolve DNS (elastic#32764) (elastic#32976)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ywelsch ywelsch ywelsch approved these changes

@s1monw s1monw s1monw approved these changes

Assignees
No one assigned
Labels
>bug :Distributed Indexing/Engine Anything around managing Lucene and the Translog in an open shard. v6.3.3 v6.4.1 v6.5.0 v7.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@dnhatn @elasticmachine @s1monw @ywelsch @colings86