Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pd-simulator: update import-data case #2741

Merged
merged 11 commits into from
Aug 11, 2020
Merged

pd-simulator: update import-data case #2741

Changes from 6 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
17f6b0a
Fix mv peer log parse error
ZenoTan Aug 6, 2020
f71f6fc
Merge branch 'master' into sim
ZenoTan Aug 6, 2020
18f0f41
Delete unused var
ZenoTan Aug 6, 2020
47d1699
Rename
ZenoTan Aug 6, 2020
ef22f6a
Update case
ZenoTan Aug 11, 2020
10bb4d0
Merge branch 'master' into case
ZenoTan Aug 11, 2020
3f48876
Address comment
ZenoTan Aug 11, 2020
730ad0f
Update import_data.go
ZenoTan Aug 11, 2020
ff4ad2e
Add stop condition
ZenoTan Aug 11, 2020
45496fc
Merge branch 'master' into case
ZenoTan Aug 11, 2020
3b41894
Merge branch 'master' into case
nolouch Aug 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 15 additions & 16 deletions tools/pd-simulator/simulator/cases/import_data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ func newImportData() *Case {
})
}

storeIDs := rand.Perm(3)
for i := 0; i < 40; i++ {
for i := 0; i < 10000; i++ {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can it be configured?

Copy link
Contributor Author

@ZenoTan ZenoTan Aug 11, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to configurable.

storeIDs := rand.Perm(10)
peers := []*metapb.Peer{
{Id: IDAllocator.nextID(), StoreId: uint64(storeIDs[0] + 1)},
{Id: IDAllocator.nextID(), StoreId: uint64(storeIDs[1] + 1)},
Expand All @@ -63,23 +63,15 @@ func newImportData() *Case {
e := &WriteFlowOnSpotDescriptor{}
table2 := string(codec.EncodeBytes(codec.GenerateTableKey(2)))
table3 := string(codec.EncodeBytes(codec.GenerateTableKey(3)))
table5 := string(codec.EncodeBytes(codec.GenerateTableKey(5)))
e.Step = func(tick int64) map[string]int64 {
if tick < 100 {
return map[string]int64{
table3: 4 * MB,
table5: 32 * MB,
}
}
return map[string]int64{
table2: 2 * MB,
table3: 4 * MB,
table5: 16 * MB,
table2: 32 * MB,
}
}
simCase.Events = []EventDescriptor{e}

// Checker description
checkCount := uint64(0)
simCase.Checker = func(regions *core.RegionsInfo, stats []info.StoreStats) bool {
leaderDist := make(map[uint64]int)
peerDist := make(map[uint64]int)
Expand Down Expand Up @@ -120,21 +112,28 @@ func newImportData() *Case {
regionTotal := regions.GetRegionCount()
totalLeaderLog := fmt.Sprintf("%d leader:", regionTotal)
totalPeerLog := fmt.Sprintf("%d peer:", regionTotal*3)
isEnd := true
isEnd := checkCount > 1000
var regionProps []float64
for storeID := uint64(1); storeID <= 10; storeID++ {
regions.GetStoreRegionCount(storeID)
totalLeaderLog = fmt.Sprintf("%s [store %d]:%.2f%%", totalLeaderLog, storeID, float64(regions.GetStoreLeaderCount(storeID))/float64(regionTotal)*100)
regionProp := float64(regions.GetStoreRegionCount(storeID)) / float64(regionTotal*3) * 100
if regionProp > 13.8 {
isEnd = false
}
regionProps = append(regionProps, regionProp)
totalPeerLog = fmt.Sprintf("%s [store %d]:%.2f%%", totalPeerLog, storeID, regionProp)
}
simutil.Logger.Info("import data information",
zap.String("table-leader", tableLeaderLog),
zap.String("table-peer", tablePeerLog),
zap.String("total-leader", totalLeaderLog),
zap.String("total-peer", totalPeerLog))
checkCount++
dev := 0.0
for _, p := range regionProps {
dev += (p - 10) * (p - 10) / 100
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will it still be 10 after region num is configured?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It means if balanced, store each get 10 % of data, no matter what the total is.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

}
if dev > 0.005 {
simutil.Logger.Warn("Not balanced, consider higher store limit", zap.Float64("dev score", dev))
}
return isEnd
}
return &simCase
Expand Down