-
Notifications
You must be signed in to change notification settings - Fork 9.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
wal: check out of range slice in "ReadAll", entry limits in "decodeRecord" #11793
Conversation
wal/decoder.go
Outdated
@@ -79,6 +80,9 @@ func (d *decoder) decodeRecord(rec *walpb.Record) error { | |||
} | |||
|
|||
recBytes, padBytes := decodeFrameSize(l) | |||
if recBytes >= math.MaxInt64-padBytes { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think we have a max wal size, also a max key-value pair size. i feel these can be set as a better limit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think we have a max wal size, also a max key-value pair size. i feel these can be set as a better limit.
Do we mean define one for max wal entry size? We have request size limit, but I don't see anywhere we define limit?
Lines 777 to 786 in 1166b1f
func (w *WAL) saveEntry(e *raftpb.Entry) error { | |
// TODO: add MustMarshalTo to reduce one allocation. | |
b := pbutil.MustMarshal(e) | |
rec := &walpb.Record{Type: entryType, Data: b} | |
if err := w.encoder.encode(rec); err != nil { | |
return err | |
} | |
w.enti = e.Index | |
return nil | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we have a request limit. also we know that the throughput wont be unlimited. we probably will only be able to batch at most 1000 requests into one WAL entry at most, usually far less... So I guess a 2GB wal entry limit is large enough...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added a check for entry upper limit. Here I assume no project ever allows more than 10 MB raft message entry.
@@ -412,7 +415,12 @@ func (w *WAL) ReadAll() (metadata []byte, state raftpb.HardState, ents []raftpb. | |||
case entryType: | |||
e := mustUnmarshalEntry(rec.Data) | |||
if e.Index > w.start.Index { | |||
ents = append(ents[:e.Index-w.start.Index-1], e) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we simply check 0 <= e.Index-w.start.Index - 1 < len(ents)
?
wal: add slice bound checks in decoder CHANGELOG-3.5: add wal slice bound check CHANGELOG-3.5: add "decodeRecord" Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
Codecov Report
@@ Coverage Diff @@
## master #11793 +/- ##
==========================================
- Coverage 66.69% 66.00% -0.69%
==========================================
Files 403 403
Lines 36976 36981 +5
==========================================
- Hits 24660 24411 -249
- Misses 10822 11068 +246
- Partials 1494 1502 +8
Continue to review full report at Codecov.
|
lgtm |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm Thanks @gyuho
…-upstream-release-3.4 Automated cherry pick of #11793
…-upstream-release-3.3 Automated cherry pick of #11793
Signed-off-by: Gyuho Lee leegyuho@amazon.com
Please read https://github.com/etcd-io/etcd/blob/master/CONTRIBUTING.md#contribution-flow.