Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: update “Reduce storage” #9693

Merged
merged 2 commits into from
Nov 30, 2022
Merged

docs: update “Reduce storage” #9693

merged 2 commits into from
Nov 30, 2022

Conversation

bmorelli25
Copy link
Member

Summary

This PR updates the reduce storage documentation:

  • Adds span compression
  • Updates sample rate content
  • Links to storage explorer docs in Kibana

@bmorelli25 bmorelli25 added Team:Docs Label for the Observability docs team backport-8.5 Automated backport with mergify backport-8.6 Automated backport with mergify labels Nov 29, 2022
@bmorelli25 bmorelli25 requested a review from a team November 29, 2022 23:58
@bmorelli25 bmorelli25 self-assigned this Nov 29, 2022
@apmmachine
Copy link
Contributor

apmmachine commented Nov 30, 2022 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-11-30T20:30:19.877+0000

  • Duration: 25 min 28 sec

Test stats 🧪

Test Results
Failed 0
Passed 153
Skipped 0
Total 153

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate and publish the docker images.

  • /test windows : Build & tests on Windows.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@apmmachine
Copy link
Contributor

apmmachine commented Nov 30, 2022 •
edited
Loading

📚 Go benchmark report

Diff with the main branch

name                                                                                              old time/op    new time/op    delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
FetchAndAdd/FetchFromCache-12                                                                       46.2ns ± 0%    41.2ns ± 0%  -10.88%  (p=0.008 n=5+5)
FetchAndAdd/FetchAndAddToCache-12                                                                    106ns ± 2%      96ns ± 3%   -9.90%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
ContextResetContentEncoding/empty-12                                                                 132ns ± 1%     119ns ± 0%   -9.92%  (p=0.008 n=5+5)
ContextResetContentEncoding/uncompressed-12                                                          158ns ± 1%     142ns ± 1%  -10.22%  (p=0.008 n=5+5)
ContextResetContentEncoding/deflate-12                                                              3.89µs ± 4%    4.01µs ± 1%   +3.02%  (p=0.032 n=5+5)
ContextResetContentEncoding/deflate_sniff-12                                                        3.88µs ± 2%    3.10µs ±58%  -20.11%  (p=0.032 n=5+5)
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/errors_rum.ndjson-12                                                               20.7µs ±11%    34.5µs ±22%  +66.38%  (p=0.008 n=5+5)
BackendProcessor/transactions_spans_rum.ndjson-12                                                   19.2µs ±30%    23.8µs ±14%  +24.22%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/spans.ndjson-12                         44.5µs ±20%    33.0µs ±12%  -25.81%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/invalid-json-event.ndjson-12            1.45µs ± 2%    1.47µs ± 2%   +1.57%  (p=0.048 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/ratelimit.ndjson-12                     6.32µs ± 3%    6.73µs ± 1%   +6.61%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/span-links.ndjson-12                    1.58µs ± 3%    1.64µs ± 1%   +3.63%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/spans.ndjson-12                         11.1µs ± 2%    11.6µs ± 1%   +4.12%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12      5.70µs ± 1%    5.79µs ± 2%   +1.62%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions.ndjson-12                  11.3µs ± 1%    11.5µs ± 1%   +1.74%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans.ndjson-12            11.3µs ± 1%    11.6µs ± 2%   +2.63%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum.ndjson-12        2.07µs ± 1%    2.11µs ± 0%   +2.13%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/unknown-span-type.ndjson-12             7.39µs ± 1%    7.58µs ± 1%   +2.58%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_rum.ndjson-12                  1.98µs ± 1%    2.06µs ± 2%   +3.87%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_transaction_id.ndjson-12       5.52µs ± 2%    5.66µs ± 2%   +2.65%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/heavy.ndjson-12                        507µs ± 2%     530µs ± 7%   +4.48%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event-type.ndjson-12           815ns ± 2%     840ns ± 1%   +3.09%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event.ndjson-12               3.37µs ± 1%    3.47µs ± 1%   +3.14%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-event.ndjson-12          1.11µs ± 2%    1.15µs ± 2%   +3.09%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12       1.89µs ± 0%    1.95µs ± 1%   +3.21%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12           475ns ± 1%     500ns ± 1%   +5.33%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata.ndjson-12             480ns ± 1%     505ns ± 1%   +5.14%  (p=0.016 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal.ndjson-12                     2.01µs ± 2%    2.05µs ± 2%   +2.15%  (p=0.024 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/optional-timestamps.ndjson-12         1.49µs ± 1%    1.53µs ± 2%   +2.96%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/otel-bridge.ndjson-12                 3.27µs ± 2%    3.35µs ± 2%   +2.55%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/span-links.ndjson-12                  1.21µs ± 1%    1.24µs ± 1%   +2.32%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/spans.ndjson-12                       9.60µs ± 1%    9.81µs ± 1%   +2.12%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions.ndjson-12                9.53µs ± 1%    9.74µs ± 2%   +2.21%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum.ndjson-12      1.62µs ± 2%    1.67µs ± 2%   +3.16%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum_2.ndjson-12    1.55µs ± 2%    1.61µs ± 2%   +3.43%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/unknown-span-type.ndjson-12           6.24µs ± 2%    6.46µs ± 4%   +3.55%  (p=0.008 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
Publisher-12                                                                                         1.00s ± 0%     1.00s ± 0%   +0.07%  (p=0.016 n=5+5)
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
WriteTransaction/json_codec-12                                                                      11.9µs ±40%     4.5µs ±11%  -62.16%  (p=0.008 n=5+5)
WriteTransaction/json_codec_big_tx-12                                                               11.3µs ±39%     5.4µs ± 4%  -52.38%  (p=0.008 n=5+5)
ReadEvents/json_codec/0_events-12                                                                    347ns ± 6%     310ns ± 6%  -10.74%  (p=0.008 n=5+5)
ReadEvents/json_codec_big_tx/0_events-12                                                             342ns ± 4%     313ns ± 9%   -8.60%  (p=0.032 n=5+5)
ReadEvents/nop_codec/0_events-12                                                                     338ns ± 5%     314ns ± 6%   -7.15%  (p=0.016 n=5+5)
ReadEvents/nop_codec_big_tx/0_events-12                                                              336ns ± 5%     302ns ± 6%  -10.27%  (p=0.008 n=5+5)
IsTraceSampled/sampled-12                                                                           75.6ns ± 4%    68.6ns ± 4%   -9.32%  (p=0.008 n=5+5)
IsTraceSampled/unsampled-12                                                                         77.7ns ± 1%    70.1ns ± 1%   -9.85%  (p=0.016 n=5+4)
IsTraceSampled/unknown-12                                                                            425ns ± 2%     369ns ± 2%  -13.13%  (p=0.008 n=5+5)

name                                                                                              old alloc/op   new alloc/op   delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/errors_rum.ndjson-12                                                               7.91kB ± 1%    8.08kB ± 1%   +2.07%  (p=0.008 n=5+5)
BackendProcessor/invalid-event.ndjson-12                                                            7.76kB ± 1%    7.90kB ± 2%   +1.76%  (p=0.016 n=5+5)
BackendProcessor/transactions-huge_traces.ndjson-12                                                 14.9kB ± 1%    15.1kB ± 1%   +1.28%  (p=0.008 n=5+5)
BackendProcessor/transactions_spans_rum.ndjson-12                                                   5.76kB ± 2%    5.97kB ± 2%   +3.62%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel4/invalid-metadata-2.ndjson-12            3.19kB ± 0%    3.23kB ± 2%   +1.17%  (p=0.016 n=5+5)
ReadBatch/errors.ndjson-12                                                                          20.8kB ± 0%    20.8kB ± 0%   +0.09%  (p=0.024 n=5+5)
ReadBatch/events.ndjson-12                                                                          38.6kB ± 0%    38.7kB ± 0%   +0.23%  (p=0.008 n=5+5)
ReadBatch/heavy.ndjson-12                                                                            831kB ± 0%     830kB ± 0%   -0.14%  (p=0.032 n=5+5)
ReadBatch/invalid-json-event.ndjson-12                                                              3.23kB ± 0%    3.23kB ± 0%   -0.03%  (p=0.016 n=5+4)
ReadBatch/logs.ndjson-12                                                                            18.4kB ± 0%    18.5kB ± 0%   +0.03%  (p=0.048 n=5+5)
ReadBatch/metadata.ndjson-12                                                                        4.00kB ± 0%    4.00kB ± 0%   +0.02%  (p=0.016 n=4+5)
ReadBatch/metricsets.ndjson-12                                                                      15.4kB ± 0%    15.5kB ± 0%   +0.94%  (p=0.008 n=5+5)
ReadBatch/unknown-span-type.ndjson-12                                                               16.8kB ± 0%    16.8kB ± 0%   +0.07%  (p=0.048 n=5+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
ReadEvents/json_codec/1000_events-12                                                                3.46MB ± 0%    3.45MB ± 0%   -0.48%  (p=0.048 n=5+4)
ReadEvents/nop_codec_big_tx/100_events-12                                                            250kB ± 0%     250kB ± 0%   +0.05%  (p=0.032 n=5+5)
ReadEvents/nop_codec_big_tx/1000_events-12                                                          2.19MB ± 0%    2.18MB ± 0%   -0.35%  (p=0.016 n=5+5)

name                                                                                              old allocs/op  new allocs/op  delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/beater/request goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/events.ndjson-12                                                                      716 ± 0%       717 ± 0%   +0.14%  (p=0.016 n=4+5)
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64

name                                                                                              old speed      new speed      delta
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
BackendProcessor/errors_rum.ndjson-12                                                             92.2MB/s ±10%  56.0MB/s ±20%  -39.25%  (p=0.008 n=5+5)
BackendProcessor/transactions_spans_rum.ndjson-12                                                 62.3MB/s ±38%  48.8MB/s ±15%  -21.64%  (p=0.032 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel2/spans.ndjson-12                        185MB/s ±18%   246MB/s ±11%  +32.82%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/ratelimit.ndjson-12                    667MB/s ± 3%   626MB/s ± 1%   -6.22%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/span-links.ndjson-12                   432MB/s ± 3%   416MB/s ± 1%   -3.52%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/spans.ndjson-12                        724MB/s ± 2%   695MB/s ± 1%   -3.98%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions-huge_traces.ndjson-12     556MB/s ± 1%   547MB/s ± 2%   -1.59%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions.ndjson-12                 498MB/s ± 1%   490MB/s ± 1%   -1.70%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans.ndjson-12           517MB/s ± 1%   503MB/s ± 2%   -2.56%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/transactions_spans_rum.ndjson-12       559MB/s ± 1%   547MB/s ± 0%   -2.10%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel8/unknown-span-type.ndjson-12            448MB/s ± 1%   436MB/s ± 1%   -2.51%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_rum.ndjson-12                 956MB/s ± 1%   920MB/s ± 2%   -3.71%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/errors_transaction_id.ndjson-12      693MB/s ± 2%   675MB/s ± 2%   -2.59%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/heavy.ndjson-12                      787MB/s ± 2%   754MB/s ± 7%   -4.18%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event-type.ndjson-12         480MB/s ± 2%   465MB/s ± 1%   -3.01%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-event.ndjson-12              228MB/s ± 1%   221MB/s ± 1%   -3.04%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-event.ndjson-12         527MB/s ± 2%   511MB/s ± 2%   -2.99%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-json-metadata.ndjson-12      237MB/s ± 0%   230MB/s ± 1%   -3.13%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata-2.ndjson-12         918MB/s ± 1%   872MB/s ± 1%   -5.06%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/invalid-metadata.ndjson-12           929MB/s ± 1%   884MB/s ± 1%   -4.89%  (p=0.016 n=4+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/minimal.ndjson-12                    513MB/s ± 2%   502MB/s ± 2%   -2.11%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/optional-timestamps.ndjson-12        691MB/s ± 1%   671MB/s ± 2%   -2.88%  (p=0.016 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/otel-bridge.ndjson-12                576MB/s ± 2%   562MB/s ± 2%   -2.49%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/span-links.ndjson-12                 562MB/s ± 1%   549MB/s ± 1%   -2.26%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/spans.ndjson-12                      840MB/s ± 1%   822MB/s ± 1%   -2.08%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions.ndjson-12               592MB/s ± 1%   580MB/s ± 2%   -2.16%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum.ndjson-12     713MB/s ± 2%   691MB/s ± 2%   -3.07%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/transactions_spans_rum_2.ndjson-12   720MB/s ± 2%   696MB/s ± 2%   -3.31%  (p=0.008 n=5+5)
BackendProcessorParallel/BenchmarkBackendProcessorParallel200/unknown-span-type.ndjson-12          530MB/s ± 2%   512MB/s ± 4%   -3.40%  (p=0.008 n=5+5)

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

simitt
simitt approved these changes Nov 30, 2022
View reviewed changes
Copy link
Contributor

@simitt simitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@bmorelli25 bmorelli25 enabled auto-merge (squash) November 30, 2022 20:30
@bmorelli25 bmorelli25 merged commit 8cace45 into elastic:main Nov 30, 2022
@mergify mergify bot mentioned this pull request Nov 30, 2022
Merged
mergify bot pushed a commit that referenced this pull request Nov 30, 2022
@bmorelli25 @mergify
docs: update “Reduce storage” (#9693)
c4b69cf 
(cherry picked from commit 8cace45)
@mergify mergify bot mentioned this pull request Nov 30, 2022
Merged
mergify bot pushed a commit that referenced this pull request Nov 30, 2022
@bmorelli25 @mergify
docs: update “Reduce storage” (#9693)
7a05431 
(cherry picked from commit 8cace45)
@bmorelli25 bmorelli25 deleted the update-reduce-storage-docs branch November 30, 2022 21:01
bmorelli25 added a commit that referenced this pull request Nov 30, 2022
@mergify @bmorelli25
docs: update “Reduce storage” (#9693) (#9708)
455a9ba 
(cherry picked from commit 8cace45)

Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co>
@bmorelli25 bmorelli25 mentioned this pull request Dec 1, 2022
Closed
bmorelli25 added a commit that referenced this pull request Dec 6, 2022
@mergify @bmorelli25
docs: update “Reduce storage” (#9693) (#9707)
429c670 
(cherry picked from commit 8cace45)

Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co>
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@simitt simitt simitt approved these changes

Assignees

@bmorelli25 bmorelli25

Labels
backport-8.5 Automated backport with mergify backport-8.6 Automated backport with mergify Team:Docs Label for the Observability docs team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Update APM "Reduce storage page"
3 participants
@bmorelli25 @apmmachine @simitt