Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: early validate all messages before encoding #454

Merged
merged 8 commits into from
Sep 18, 2024

Conversation

muktihari
Copy link
Owner

@muktihari muktihari commented Sep 18, 2024

Validating messages early give several advantages:

  1. If invalid message occurs, we can return early without doing unnecessary works such as marshaling, calculate CRC or even do a syscall to write to the file.
  2. If normal writer is given (a writer without offset writer capability), the encoding speed is improved by geomean 16% since we don't need to validate messages twices. Benchmark:
goos: linuxl goarch: amd64l pkg: benchfitl cpu: Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
                           │   old.txtnew.txt               │
                           │   sec/opsec/op     vs baseEncode/muktihari/fit_raw-4   93.57m ± 8%   74.63m ± 3%  -20.24% (p=0.000 n=10)
Encode/muktihari/fit-4       157.2m ± 7%   136.7m ± 2%  -13.01% (p=0.000 n=10)
geomean                      121.3m        101.0m       -16.71%
  1. We can release all objects generated by the message validator early, so it can be garbage-collected as soon as we are done validating. The objects are only be generated when it encounters developer data.

@muktihari muktihari added the performance Changes related to performance improvement label Sep 18, 2024
@muktihari muktihari added this to the v0.23.2 milestone Sep 18, 2024
@muktihari muktihari self-assigned this Sep 18, 2024
@muktihari muktihari merged commit c0b1138 into master Sep 18, 2024
5 checks passed
@muktihari muktihari deleted the perf/validate-messages-first branch September 18, 2024 12:12
@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.94%. Comparing base (e3d3e16) to head (27b5306).
Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##            master     #454      +/-   ##
===========================================
- Coverage   100.00%   99.94%   -0.06%     
===========================================
  Files           42       42              
  Lines         3659     3687      +28     
===========================================
+ Hits          3659     3685      +26     
- Misses           0        2       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Changes related to performance improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants