Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use simpler write-once semantics for HDFS repository #30439

Merged
merged 1 commit into from
May 11, 2018
Merged

Use simpler write-once semantics for HDFS repository #30439

merged 1 commit into from
May 11, 2018

Conversation

ywelsch
Copy link
Contributor

@ywelsch ywelsch commented May 7, 2018

There's no need for an extra blobExists() call when writing a blob to the HDFS service. The writeBlob implementation for the HDFS repository already uses the CreateFlag.CREATE option on the file creation, which ensures that the blob that's uploaded does not already exist. This saves one network roundtrip.

Relates to #19749

@ywelsch ywelsch added >enhancement :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v7.0.0 v6.4.0 labels May 7, 2018
@ywelsch ywelsch requested review from tlrx and jbaiera May 7, 2018 20:45
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

jbaiera
jbaiera approved these changes May 9, 2018
View reviewed changes
Copy link
Member

@jbaiera jbaiera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ywelsch ywelsch merged commit fc870fd into elastic:master May 11, 2018
@ywelsch ywelsch changed the title Use proper write-once semantics for HDFS repository Use simpler write-once semantics for HDFS repository May 11, 2018
ywelsch added a commit that referenced this pull request May 11, 2018
@ywelsch
Use simpler write-once semantics for HDFS repository (#30439)
1219798 
There's no need for an extra `blobExists()` call when writing a blob to the HDFS service. The writeBlob implementation for the HDFS repository already uses the `CreateFlag.CREATE` option on the file creation, which ensures that the blob that's uploaded does not already exist. This saves one network roundtrip.
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request May 12, 2018
@jasontedor
Merge branch 'master' into no-copy-settings-for-you
790b0fd 
* master: (41 commits)
  Bump Gradle heap to 2 GB (elastic#30535)
  SQL: Use request flavored methods in tests (elastic#30345)
  Suppress hdfsFixture if there are spaces in the path (elastic#30302)
  Delete temporary blobs before creating index file (elastic#30528)
  Watcher: Remove TriggerEngine.getJobCount() (elastic#30395)
  [ML] Fix wire BWC for JobUpdate (elastic#30512)
  Use simpler write-once semantics for FS repository (elastic#30435)
  Derive max composite buffers from max content len
  Use simpler write-once semantics for HDFS repository (elastic#30439)
  SQL: Improve correctness of SYS COLUMNS & TYPES (elastic#30418)
  Mute two tests in FlushIT with @AwaitsFix.
  Fix incorrect template name in test case
  Build: Remove legacy bwc files from xpack (elastic#30485)
  Mute UnicastZenPingTests#testSimplePings with @AwaitsFix.
  Security: cleanup code in file stores (elastic#30348)
  Security: fix TokenMetaData equals and hashcode (elastic#30347)
  Mute two tests from SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.
  Mute SharedClusterSnapshotRestoreIT#testSnapshotSucceedsAfterSnapshotFailure with @AwaitsFix.
  SQL: Improve compatibility with MS query (elastic#30516)
  SQL: Fix parsing of dates with milliseconds (elastic#30419)
  ...
dnhatn added a commit that referenced this pull request May 14, 2018
@dnhatn
Merge branch 'master' into ccr
b971a81 
* master:
  Default to one shard (#30539)
  Unmute IndexUpgradeIT tests
  Forbid expensive query parts in ranking evaluation (#30151)
  Docs: Update HighLevelRestClient migration docs (#30544)
  Clients: Switch to new performRequest (#30543)
  [TEST] Fix typo in MovAvgIT test
  Add missing dependencies on testClasses (#30527)
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Document woes between auto-expand-replicas and allocation filtering (#30531)
  Moved tokenizers to analysis common module (#30538)
  Adjust copy settings versions
  Mute ShrinkIndexIT suite
  SQL: SYS TABLES ordered according to *DBC specs (#30530)
  Deprecate not copy settings and explicitly disallow (#30404)
  [ML] Improve state persistence log message
  Build: Add mavenPlugin cluster configuration method (#30541)
  Re-enable FlushIT tests
  Bump Gradle heap to 2 GB (#30535)
  SQL: Use request flavored methods in tests (#30345)
  Suppress hdfsFixture if there are spaces in the path (#30302)
  Delete temporary blobs before creating index file (#30528)
  Watcher: Remove TriggerEngine.getJobCount() (#30395)
  [ML] Fix wire BWC for JobUpdate (#30512)
  Use simpler write-once semantics for FS repository (#30435)
  Derive max composite buffers from max content len
  Use simpler write-once semantics for HDFS repository (#30439)
  SQL: Improve correctness of SYS COLUMNS & TYPES (#30418)
  Mute two tests in FlushIT with @AwaitsFix.
  Fix incorrect template name in test case
  Build: Remove legacy bwc files from xpack (#30485)
  Mute UnicastZenPingTests#testSimplePings with @AwaitsFix.
  Security: cleanup code in file stores (#30348)
  Security: fix TokenMetaData equals and hashcode (#30347)
  Mute two tests from SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.
  Mute SharedClusterSnapshotRestoreIT#testSnapshotSucceedsAfterSnapshotFailure with @AwaitsFix.
  SQL: Improve compatibility with MS query (#30516)
  SQL: Fix parsing of dates with milliseconds (#30419)
dnhatn added a commit that referenced this pull request May 14, 2018
@dnhatn
Merge branch '6.x' into ccr-6.x
1325c25 
* 6.x:
  Unmute IndexUpgradeIT tests
  Forbid expensive query parts in ranking evaluation (#30151)
  Docs: Update HighLevelRestClient migration docs (#30544)
  Clients: Switch to new performRequest (#30543)
  [TEST] Fix typo in MovAvgIT test
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Moved tokenizers to analysis common module (#30538)
  Document woes between auto-expand-replicas and allocation filtering (#30531)
  [ML] Hide internal Job update options from the REST API (#30537)
  Deprecate not copy settings and explicitly disallow (#30404)
  Mute ShrinkIndexIT suite
  SQL: SYS TABLES ordered according to *DBC specs (#30530)
  [ML] Improve state persistence log message
  Build: Add mavenPlugin cluster configuration method (#30541)
  Re-enable FlushIT tests
  Bump Gradle heap to 2 GB (#30535)
  Bump Gradle heap to 1792m (#30484)
  SQL: Use request flavored methods in tests (#30345)
  Suppress hdfsFixture if there are spaces in the path (#30302)
  Delete temporary blobs before creating index file (#30528)
  Watcher: Remove TriggerEngine.getJobCount() (#30395)
  Use simpler write-once semantics for FS repository (#30435)
  Use simpler write-once semantics for HDFS repository (#30439)
  SQL: Improve correctness of SYS COLUMNS & TYPES (#30418)
  Mute two tests in FlushIT with @AwaitsFix.
  Fix incorrect template name in test case
  Build: Remove legacy bwc files from xpack (#30485)
  Security: Simplify security index listeners (#30466)
  Mute SharedClusterSnapshotRestoreIT#testSnapshotSucceedsAfterSnapshotFailure with @AwaitsFix.
  Add proper longitude validation in geo_polygon_query (#30497)
  Mute UnicastZenPingTests#testSimplePings with @AwaitsFix.
  Security: cleanup code in file stores (#30348)
  Security: fix TokenMetaData equals and hashcode (#30347)
  Mute two tests from SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.
  Fix incorrect merged entry in changelog
  SQL: Improve compatibility with MS query (#30516)
  SQL: Fix parsing of dates with milliseconds (#30419)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jbaiera jbaiera jbaiera approved these changes

@tlrx tlrx Awaiting requested review from tlrx

Assignees
No one assigned
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >enhancement v6.4.0 v7.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ywelsch @elasticmachine @jbaiera @colings86