-
Notifications
You must be signed in to change notification settings - Fork 309
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[DI] Drop snapshot if JSON payload is too large #4818
Conversation
This stack of pull requests is managed by Graphite. Learn more about stacking. |
Overall package sizeSelf size: 7.65 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | @datadog/native-appsec | 8.2.1 | 19.18 MB | 19.19 MB | | @datadog/native-iast-taint-tracking | 3.2.0 | 13.9 MB | 13.91 MB | | @datadog/pprof | 5.4.1 | 9.76 MB | 10.13 MB | | protobufjs | 7.2.5 | 2.77 MB | 5.16 MB | | @datadog/native-iast-rewriter | 2.5.0 | 2.51 MB | 2.65 MB | | @opentelemetry/core | 1.14.0 | 872.87 kB | 1.47 MB | | @datadog/native-metrics | 2.0.0 | 898.77 kB | 1.3 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.11.2 | 112.74 kB | 826.22 kB | | msgpack-lite | 0.1.26 | 201.16 kB | 281.59 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.0 | 109.9 kB | 109.9 kB | | semver | 7.6.3 | 95.82 kB | 95.82 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.1 | 51.46 kB | 51.46 kB | | int64-buffer | 0.1.10 | 49.18 kB | 49.18 kB | | shell-quote | 1.8.1 | 44.96 kB | 44.96 kB | | istanbul-lib-coverage | 3.2.0 | 29.34 kB | 29.34 kB | | rfdc | 1.3.1 | 25.21 kB | 25.21 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | dc-polyfill | 0.1.4 | 23.1 kB | 23.1 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | path-to-regexp | 0.1.10 | 6.38 kB | 6.38 kB | | module-details-from-path | 1.0.3 | 4.47 kB | 4.47 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
BenchmarksBenchmark execution time: 2024-10-29 13:41:37 Comparing candidate commit 3ef1ead in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 260 metrics, 6 unstable metrics. |
f0041a7
to
87e143e
Compare
|
||
if (Buffer.byteLength(json) > MAX_PAYLOAD_SIZE) { | ||
// TODO: This is a very crude way to handle large payloads. Proper pruning will be implemented later (DEBUG-2624) | ||
delete payload['debugger.snapshot'] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
kinda harsh :D
remove 'debugger.snapshot' would cause probe-tracker to mark the probe as error: as the probe would trigger emitting start but there are not events marked with the probe-id and would default to error state.
I would suggest remove 'debugger.snapshot.captures' or replace it with notCaptureReasons: 'snapshot was too large' - at least give the user some feedback on why the don't see the snapshot.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, good point. My intent was to only rmeove the captures
part, but the JSON schema threw me for a loop since we're collecting a snapshot, the property is called snapshot
and it's called "snapshot pruning" 🙄 I'll push a fix 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed 👍
The log track has a 1MB limit of the JSON payload. The client is not notified if the payload is too large, but it is simply never indexed. This is a very crude approach. In the future a more sophsticated algorithm will be implemented that reduces the size of the snapshot instead of removing it completely.
87e143e
to
1d824e5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
very cool idea to make the tests robust to line changes using // BREAKPOINT
The log track has a 1MB limit of the JSON payload. The client is not notified if the payload is too large, but it is simply never indexed. This is a very crude approach. In the future a more sophsticated algorithm will be implemented that reduces the size of the snapshot instead of removing it completely.
The log track has a 1MB limit of the JSON payload. The client is not notified if the payload is too large, but it is simply never indexed. This is a very crude approach. In the future a more sophsticated algorithm will be implemented that reduces the size of the snapshot instead of removing it completely.
The log track has a 1MB limit of the JSON payload. The client is not notified if the payload is too large, but it is simply never indexed. This is a very crude approach. In the future a more sophsticated algorithm will be implemented that reduces the size of the snapshot instead of removing it completely.
The log track has a 1MB limit of the JSON payload. The client is not notified if the payload is too large, but it is simply never indexed. This is a very crude approach. In the future a more sophsticated algorithm will be implemented that reduces the size of the snapshot instead of removing it completely.
The log track has a 1MB limit of the JSON payload. The client is not notified if the payload is too large, but it is simply never indexed. This is a very crude approach. In the future a more sophsticated algorithm will be implemented that reduces the size of the snapshot instead of removing it completely.
The log track has a 1MB limit of the JSON payload. The client is not notified if the payload is too large, but it is simply never indexed.
This is a very crude approach. In the future a more sophsticated algorithm will be implemented that reduces the size of the snapshot instead of removing it completely.