Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wal: use page buffered writer for writing records #6310

Merged
merged 2 commits into from
Aug 31, 2016

Conversation

heyitsanthony
Copy link
Contributor

Fixes #6271

/cc @gyuho

@gyuho
Copy link
Contributor

gyuho commented Aug 30, 2016

Ok I will try this branch in failpoints cluster.

@@ -36,7 +36,7 @@ type encoder struct {

func newEncoder(w io.Writer, prevCrc uint32) *encoder {
return &encoder{
bw: bufio.NewWriter(w),
bw: ioutil.NewPageWriter(w, 4096),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

define 4096 somewhere?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also doc that this is buffering same as bufio https://github.com/golang/go/blob/master/src/bufio/bufio.go#L18?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gyuho it's only a coincidence the wal page size is the bufio default buffer size; the full buffer size for a pagewriter is 128k. I'll add a comment about the choice of the value.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Thanks!

@xiang90
Copy link
Contributor

xiang90 commented Aug 30, 2016

LGTM. Have you tried to run the benchmark in wal pkg to see if there is any performance gain?

@heyitsanthony
Copy link
Contributor Author

@xiang90 benchmark found a bug in the page writer 👍

Master:

BenchmarkWrite100EntryWithoutBatch-4             200      10144426 ns/op       0.01 MB/s
BenchmarkWrite100EntryBatch10-4                 2000        943462 ns/op       0.11 MB/s
BenchmarkWrite100EntryBatch100-4               10000        116260 ns/op       0.93 MB/s
BenchmarkWrite100EntryBatch500-4               50000         28449 ns/op       3.80 MB/s
BenchmarkWrite100EntryBatch1000-4             100000         15095 ns/op       7.15 MB/s
BenchmarkWrite1000EntryWithoutBatch-4            100      10645949 ns/op       0.09 MB/s
BenchmarkWrite1000EntryBatch10-4                2000       1016150 ns/op       0.99 MB/s
BenchmarkWrite1000EntryBatch100-4              10000        144891 ns/op       6.96 MB/s
BenchmarkWrite1000EntryBatch500-4              50000         42404 ns/op      23.79 MB/s
BenchmarkWrite1000EntryBatch1000-4             50000         28502 ns/op      35.40 MB/s

This patch:

BenchmarkWrite100EntryWithoutBatch-4             100      10629690 ns/op       0.01 MB/s
BenchmarkWrite100EntryBatch10-4                 2000        963754 ns/op       0.11 MB/s
BenchmarkWrite100EntryBatch100-4               10000        107374 ns/op       1.01 MB/s
BenchmarkWrite100EntryBatch500-4              100000         23026 ns/op       4.69 MB/s
BenchmarkWrite100EntryBatch1000-4             200000         11675 ns/op       9.25 MB/s
BenchmarkWrite1000EntryWithoutBatch-4            100      10485759 ns/op       0.10 MB/s
BenchmarkWrite1000EntryBatch10-4                2000        997684 ns/op       1.01 MB/s
BenchmarkWrite1000EntryBatch100-4              10000        121113 ns/op       8.33 MB/s
BenchmarkWrite1000EntryBatch500-4              50000         25324 ns/op      39.84 MB/s
BenchmarkWrite1000EntryBatch1000-4            100000         13901 ns/op      72.58 MB/s

Anthony Romano added 2 commits August 30, 2016 15:49
A buffered writer that only writes full pages or when explicitly flushed.
Forces torn writes to only happen on sector boundaries.

Fixes etcd-io#6271
@gyuho
Copy link
Contributor

gyuho commented Aug 30, 2016

I can confirm that this fixes #6271 since it used to fail with less than 3 rounds. Now it's over 10 rounds. Thanks!

@xiang90
Copy link
Contributor

xiang90 commented Aug 31, 2016

lgtm

@heyitsanthony heyitsanthony merged commit e29c79c into etcd-io:master Aug 31, 2016
@gyuho gyuho mentioned this pull request Aug 31, 2016
@heyitsanthony heyitsanthony deleted the wal-page-write branch September 1, 2016 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants