Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SegmentV2] Change the default storage format to SegmentV2 #4387

Merged
merged 3 commits into from
Aug 24, 2020

Conversation

morningman
Copy link
Contributor

@morningman morningman commented Aug 18, 2020

Proposed changes

Since the Segment V2 has been released for a long time, we should make it as default storage format for newly created table.

This CL mainly changes:

  1. For all newly created tables, their default storage format is Segment V2.
  2. For all already exist tablets, their storage format remain unchanged.
  3. Fix bugs described in Fix [Bug][SparkLoad] Spark load will create rowset with incorrect rowset type #4384 and Fix [Bug] tablet type in tablet meta is wrong #4385

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Checklist

  • I have create an issue on (Fix [SegmentV2] Change the default storage format to SegmentV2  #4386), and have described the bug/feature there in detail
  • Compiling and unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • If this change need a document change, I have updated the document
  • Any dependent changes have been merged

Further comments

We should provide a more friendly way to check the conversion progress of Segment V2

@morningman morningman added kind/fix Categorizes issue or PR as related to a bug. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API area/storage Issues or PRs related to storage engine labels Aug 18, 2020
@morningman morningman self-assigned this Aug 18, 2020
Copy link
Member

@yangzhg yangzhg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not change the default value of config default_rowset_type in common/config.h to BETA

chaoyli
chaoyli previously approved these changes Aug 20, 2020
@morningman
Copy link
Contributor Author

Why not change the default value of config default_rowset_type in common/config.h to BETA

Modifying this config will cause BE to automatically convert all existing data.
In fact, this is a very bad design because it is completely uncontrollable. . .
But since we have released this feature in version 0.12, even if we make changes, we cannot change the status quo.

I will add some explain in document.

@yangzhg
Copy link
Member

yangzhg commented Aug 20, 2020

Why not change the default value of config default_rowset_type in common/config.h to BETA

Modifying this config will cause BE to automatically convert all existing data.
In fact, this is a very bad design because it is completely uncontrollable. . .
But since we have released this feature in version 0.12, even if we make changes, we cannot change the status quo.

I will add some explain in document.

We will change it to BETA in 0.13?
the real default is BETA but the value in config says default is ALPHA this it too werid

@morningman
Copy link
Contributor Author

Why not change the default value of config default_rowset_type in common/config.h to BETA

Modifying this config will cause BE to automatically convert all existing data.
In fact, this is a very bad design because it is completely uncontrollable. . .
But since we have released this feature in version 0.12, even if we make changes, we cannot change the status quo.
I will add some explain in document.

We will change it to BETA in 0.13?
the real default is BETA but the value in config says default is ALPHA this it too werid

yes, but I can't find a good way to solve it. I think we can just leave it and explain it in document.

Copy link
Contributor

@EmmyMiao87 EmmyMiao87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove the config of storage format in BE?

@morningman
Copy link
Contributor Author

Should we remove the config of storage format in BE?

Can not. Because some user already use it.

Copy link
Contributor

@EmmyMiao87 EmmyMiao87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@EmmyMiao87 EmmyMiao87 added approved Indicates a PR has been approved by one committer. branch-0.13 PR which need to merge to branch 0.13 labels Aug 24, 2020
@morningman morningman merged commit 976820b into apache:master Aug 24, 2020
@EmmyMiao87 EmmyMiao87 mentioned this pull request Sep 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. area/storage Issues or PRs related to storage engine branch-0.13 PR which need to merge to branch 0.13 kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/fix Categorizes issue or PR as related to a bug.
Projects
None yet
4 participants