Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improves the size estimation when cutting new rows #3038

Merged
merged 7 commits into from
Oct 20, 2023
Merged

Improves the size estimation when cutting new rows #3038

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
13da350
Improves the size estimation when cutting new rows
joe-elliott Oct 19, 2023
564c15d
changelog
joe-elliott Oct 19, 2023
9530964
lint
joe-elliott Oct 19, 2023
52c89b1
changelog fix
joe-elliott Oct 20, 2023
dd8db57
adjust magic factor to be slightly more aggressive
joe-elliott Oct 20, 2023
94a9680
wrong vresion
joe-elliott Oct 20, 2023
febad03
Merge branch 'main' into adjust-sz-calculations
joe-elliott Oct 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
* [CHANGE] Restructure Azure backends into versioned backends. Introduce `use_v2_sdk` config option for switching. [#2952](https://github.com/grafana/tempo/issues/2952) (@zalegrala)
v1: [azure-storage-blob-go](https://github.com/Azure/azure-storage-blob-go) original (now deprecated) SDK
v2: [azure-sdk-for-go](https://github.com/Azure/azure-sdk-for-go)
* [CHANGE] Adjust trace size estimation to better honor row group size settings. [#3038](https://github.com/grafana/tempo/pull/3038) (@joe-elliott)
* [CHANGE] Update alpine image version to 3.18. [#3046](https://github.com/grafana/tempo/pull/) (@joe-elliott)
* [ENHANCEMENT] Add block indexes to vParquet2 and vParquet3 to improve trace by ID lookup [#2697](https://github.com/grafana/tempo/pull/2697) (@mdisibio)
* [ENHANCEMENT] Assert ingestion rate limits as early as possible [#2640](https://github.com/grafana/tempo/pull/2703) (@mghildiy)
Expand Down
21 changes: 20 additions & 1 deletion tempodb/encoding/vparquet3/compactor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,7 +315,26 @@ func estimateProtoSizeFromParquetRow(row parquet.Row) (size int) {
// estimateMarshalledSizeFromParquetRow estimates the byte size as marshalled into parquet.
// this is a very rough estimate and is generally 66%-100% of actual size.
func estimateMarshalledSizeFromParquetRow(row parquet.Row) (size int) {
return len(row)
for _, v := range row {
sz := 1

switch v.Kind() {
case parquet.ByteArray:
sz = len(v.ByteArray())

case parquet.FixedLenByteArray:
sz = len(v.ByteArray())
}

if sz > 1 {
// for larger values, estimate 1 byte per 20 bytes. this is a very rough estimate
// but works well enough for our purposes. TODO: improve this calculation
sz = max(sz/20, 1) // nolint: typecheck
}

size += sz
}
return
}

// countSpans counts the number of spans in the given trace in deconstructed
Expand Down
Loading