Use too many heap objects to get the log stream's data size. #210
Assignees
Comments
ijsong
added a commit
to ijsong/varlog
that referenced
this issue
Nov 2, 2022
…torage The previous method to get the data size of storage belonging to a log stream was costly. Concretely it used too many heap objects to call `filepath.Walk` API. To avoid this problem, this patch changes the implementation of `internal/storage.(*Storage).DiskUsage` to use the built-in method of pebble DB - `DiskSpaceUsage`. Resolves kakao#210
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When clients ask for the metadata, a log stream replica gets its data size through
internal/storage.(*Storage).DiskUsage. Internally, the DiskUsage method walks each item in a given directory and sums those file sizes. However, it causes the use of too many heap objects like the below:Rather than directly computing a log stream's data size, we can use functions of underlying storage - pebble. See https://pkg.go.dev/github.com/cockroachdb/pebble#Metrics.DiskSpaceUsage. The DiskSpaceUsage method returns a value already calculated with just a straightforward sum.
The text was updated successfully, but these errors were encountered: