Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up JSON output #1114

Merged
merged 4 commits into from
Sep 2, 2019
Merged

Speed up JSON output #1114

merged 4 commits into from
Sep 2, 2019

Commits on Aug 14, 2019

  1. perf(jsonOutput): speed up json output

    The commit includes three main changes:
    1. Don't write on Collect instead (as all other outputs) append to a
    buffer and write the buffer in a separate goroutine every 100
    milliseconds
    2. Use json.Encoder for speed up
    3. If the output file ends in `.gz` gzip the output which in my testing
    decreases the file size by 30x. I couldn't measure any performance
    degradation from the fact we need to compress if anything there is some
    performance gain because of the smaller io writes
    
    I was testing with simple script doing batch requests for a local zero
    length file with 40VUS, that makes around 8k RPS using around 600-700mb
    of memory. When running before those I was getting 5.8-6k RPS, after
    this I am getting 6.5-6.9k RPS with around 3GB of memory usage.
    
    At this point my cpu and memory profiling shows that the problem is the
    json encoding upwards of 10 seconds and as such the next possible
    optimizations are in using some kind of faster json encoding.
    mstoykov committed Aug 14, 2019
    Configuration menu
    Copy the full SHA
    7ebd06b View commit details
    Browse the repository at this point in the history
  2. fix golangci warning

    mstoykov committed Aug 14, 2019
    Configuration menu
    Copy the full SHA
    7a3d0d8 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2019

  1. Configuration menu
    Copy the full SHA
    f26c24f View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    67cede7 View commit details
    Browse the repository at this point in the history