You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Performance on large files is atrocious.
You can look at code here https://github.com/Cpecific/twwh2_ancillary/blob/master/node_modules/tokenizr/lib/tokenizr.js
Mainly added Tokenizr._pendingPos. Switched from concatting and shifting arrays to simply iterating over them.
In my case it was an improvement from unknown amount of hours/days to barely a second. Probably issue crops up because of using transactions, but not sure, I wrote this code couple months back.
I would like to spend some time to improve it further, but sadly barely have time for such a side project.
Not sure if it is even the right way to do it, as I didn't try to understand all uses of library that much.
Probably have some bugs.
The text was updated successfully, but these errors were encountered:
Hmmm.... that's interesting. Sure, array operations might cost more than a simple integer operation, but I guess it mainly has to do in your case because of an excessive amount of backtracking with transactions and in this case the array operation costs accumulate dramatically. I'll check this in more detail when I find time again for the Tokenizr project.
Performance on large files is atrocious.
You can look at code here https://github.com/Cpecific/twwh2_ancillary/blob/master/node_modules/tokenizr/lib/tokenizr.js
Mainly added
Tokenizr._pendingPos
. Switched from concatting and shifting arrays to simply iterating over them.In my case it was an improvement from unknown amount of hours/days to barely a second. Probably issue crops up because of using transactions, but not sure, I wrote this code couple months back.
I would like to spend some time to improve it further, but sadly barely have time for such a side project.
Not sure if it is even the right way to do it, as I didn't try to understand all uses of library that much.
Probably have some bugs.
The text was updated successfully, but these errors were encountered: