Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve performance for large files #22

Merged
merged 1 commit into from
Mar 9, 2017
Merged

improve performance for large files #22

merged 1 commit into from
Mar 9, 2017

Conversation

targos
Copy link
Contributor

@targos targos commented Mar 8, 2017

Using buf = Buffer.concat([buf, data]); on the data event copies the
data in a new larger buffer everytime. Instead, store the intermediate
small buffers in an array and concat only at the end.

Test with a 117MB file:

  • before: 83.868 seconds
  • after: 2.230 seconds

@coveralls
Copy link

coveralls commented Mar 8, 2017

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling 156bba8 on targos:improve-perf into ac309ee on richardgirges:master.

Using `buf = Buffer.concat([buf, data]);` on the data event copies the
data in a new larger buffer everytime. Instead, store the intermediate
small buffers in an array and concat only at the end.
@coveralls
Copy link

coveralls commented Mar 8, 2017

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

4 similar comments
@coveralls
Copy link

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

@coveralls
Copy link

coveralls commented Mar 8, 2017

Coverage Status

Coverage increased (+0.2%) to 86.765% when pulling e23f337 on targos:improve-perf into ac309ee on richardgirges:master.

@richardgirges
Copy link
Owner

Beautiful. Amazing increase in performance! Thank you very much :)

@richardgirges richardgirges merged commit 90267a3 into richardgirges:master Mar 9, 2017
@targos targos deleted the improve-perf branch March 9, 2017 19:16
@richardgirges richardgirges mentioned this pull request Mar 9, 2017
@mikeatlas
Copy link

Wow. This is a huge difference. Thanks @targos !

This was referenced Nov 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants