Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segment Replication - Fix NoSuchFileException errors caused when computing metadata snapshot on primary shards. #4366

Merged
merged 12 commits into from
Sep 6, 2022
Merged

Segment Replication - Fix NoSuchFileException errors caused when computing metadata snapshot on primary shards. #4366

merged 12 commits into from
Sep 6, 2022

Commits on Sep 3, 2022

  1. Segment Replication - Fix NoSuchFileException errors caused when comp… 

    …uting metadata snapshot on primary shards.

This change fixes the errors that occur when computing metadata snapshots on primary shards from the latest in-memory SegmentInfos.  The error occurs when a segments_N file that is referenced by the in-memory infos is deleted as part of a concurrent commit.  The segments themselves are incref'd by IndexWriter.incRefDeleter but the commit file (Segments_N) is not.  This change resolves this by ignoring the segments_N file when computing metadata for CopyState and only sending incref'd segment files to replicas.

Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 3, 2022
    Configuration menu
    Copy the full SHA
    a1e60d3 View commit details
    Browse the repository at this point in the history

  2. Fix spotless. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 3, 2022
    Configuration menu
    Copy the full SHA
    f424567 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2022

  1. Update StoreTests.testCleanupAndPreserveLatestCommitPoint to assert a… 

    …dditional segments are deleted.

Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 4, 2022
    Configuration menu
    Copy the full SHA
    f1ae2dd View commit details
    Browse the repository at this point in the history

  2. Rename snapshot to metadataMap in CheckpointInfoResponse. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 4, 2022
    Configuration menu
    Copy the full SHA
    6e25396 View commit details
    Browse the repository at this point in the history

  3. Refactor segmentReplicationDiff method to compute off two maps instea… 

    …d of MetadataSnapshots.

Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 4, 2022
    Configuration menu
    Copy the full SHA
    82901e9 View commit details
    Browse the repository at this point in the history

  4. Fix spotless. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 4, 2022
    Configuration menu
    Copy the full SHA
    7f6b9f8 View commit details
    Browse the repository at this point in the history

  5. Revert catchall in SegmentReplicationSourceService. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 4, 2022
    Configuration menu
    Copy the full SHA
    e7f9de5 View commit details
    Browse the repository at this point in the history

  6. Revert log lvl change. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 4, 2022
    Configuration menu
    Copy the full SHA
    e60f8d4 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2022

  1. Fix SegmentReplicationTargetTests 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 5, 2022
    Configuration menu
    Copy the full SHA
    a0ab4d7 View commit details
    Browse the repository at this point in the history

  2. Cleanup unused logger. 

    Signed-off-by: Marc Handalian <handalm@amazon.com>
    @mch2
    mch2 committed Sep 5, 2022
    Configuration menu
    Copy the full SHA
    d07446e View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2022

  1. Merge branch 'main' into fileNotFound

    @dreamer-89
    dreamer-89 authored Sep 6, 2022
    Configuration menu
    Copy the full SHA
    b3b9ac9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into fileNotFound

    @dreamer-89
    dreamer-89 authored Sep 6, 2022
    Configuration menu
    Copy the full SHA
    780266f View commit details
    Browse the repository at this point in the history