refactor(utils): replace with lodash

[bd82]

WIP

[NaridaL]

I think import from "lodash" with tree shaking also works (might need lodash-es).

If you want to go this way you should add https://eslint.org/docs/rules/no-restricted-imports with "lodash".

[bd82]

I think import from "lodash" with tree shaking also works, (might need lodash-es).

Without lodash-es it bundled the whole lodash package.
I will try lodash-es

[bd82]

Adding import {isEmpty} from "lodash-es" increased the bundle size from 166kb to 280kb.
I suspect it may be because we are first transpiling to ES5.1 using TSC and only than minifying.

[NaridaL]

The problem is more likely the tsconfig module option. That should be set to ES and then webpack/esbuild should handle the conversion to cjs for the bundle.

[bd82]

The problem is more likely the tsconfig module option. That should be set to ES and then webpack/esbuild should handle the conversion to cjs for the bundle.

Good point. 👍
Although I will probably finish the utils replacement without changes to tsconfig as it consumes too much time anyhow
and leave other changes as a separate possible future upgarde.

[bd82]

Looks like this will cause a 15-20% increase in the bundles size, which I consider reasonable considering
the improvements in maintainability of the project.

Some of this increase will be nullified with the upcoming deprecation of features such as syntactic content assist.

• Evaluate Deprecating Syntactic Content Assist Support #1165

Might be worth looking into more ways to decrease the bundle size, although that is not the highest priority...

[bd82]

Need to:

• check if there is an eslint rule to ensure lodash functions are imported with import foo from "lodash/foo" style.
  • Edit: yes there is: eslint-plugin-lodash
• check if the change in the lookahead data structures from sparse array to object dictionary causes any meaningful performance regression (_.has assumes the array is packed and uses the length to return true/false)

Edit:

Object dictionary seems to have a slight performance advantage.
I guess it depends how "holey" the array becomes and under the hood details of V8

[NaridaL]

Can we just move these into the chevrotain package?

[bd82]

• There may be future utils that will be common/shared, so that gave me an incentive to keep the utils bundle.
• I believe the remaining three utils are all used in both the Parser and the Lexer, and I would like at some point in the future to extract the lexer to a separate package.

[bd82]

this is a JS only option, for scripts / configuration which enables using ... syntax

[bd82]

There is no proper benchmark for this, but it seems introducing lodash may have decreased the parser/lexer init time by ~10%
Possibly due to deferring certain utils to the native implementations where possible.

[bd82]

has in lodash on array assumes the array is non-holey and seems to use the length property.
Changing to an object dictionary seems to have very slightly improved performance as well.